# Recommendations for Different Tasks Based on the Uniform Multimodal Joint Representation
## 3 Multimodal Joint Representation of Pins
### Multimodal Fusion  
Different modalities can provide both consistent and complementary information, while their distinct statistical properties make it difficult to combine them into a joint representation that maintains their specific characteristics using a shallow architecture. A multimodal DBM [32] can effectively model a joint distribution over modalities, which adds a shared hidden layer on top of DBMs to combine them.  
As illustrated in Figure 5, a multimodal DBM is an undirected graphical model with fully bipartite connections between adjacent layers. Each pathway of it is a DBM, which is structured by stacking two restricted Bolzmann machines (RBMs) in a hierarchical manner. All layers, except the two bottom layers, use standard binary units. An RBM with hidden units \(H=\left(h_{j}\right)\in\left\{0,1\right\}^{F}\) and visible units \(V=\left(v_{i}\right)\in\left\{0,1\right\}^{D}\) defines the energy function as follows:  
\[\mathrm{E}\left(V,H;\theta\right)=-\sum_{i=1}^{D}\sum_{j=1}^{F}v_{i}w_{ij}h_{j} -\sum_{i=1}^{D}a_{i}v_{i}-\sum_{j=1}^{F}b_{j}h_{j}, \tag{5}\]  
where \(\theta=\left\{\left(w_{ij}\right)\in\mathbb{R}^{D\times F},\left(a_{i}\right) \in\mathbb{R}^{D},\left(b_{j}\right)\in\mathbb{R}^{F}\right\}\) are model parameters comprising the symmetric interaction term \(w_{ij}\) between the hidden unit and the visible unit, the visible unit bias term \(a_{i}\), and the hidden unit bias term \(b_{j}\). RBMs can be considered autoencoders, and one of their applications is dimensionality reduction by reducing \(F\). Both bottom layers of our model change to Gaussian-Bernoulli RBMs which use Gaussian distribution to model real-valued inputs. The energy function of a Gaussian-Bernoulli RBM with visible variables \(V=\left(v_{i}\right)\in\mathbb{R}^{D}\) and hidden variables \(H=\left(h_{j}\right)\in\left\{0,1\right\}^{F}\) is defined as  
\[\mathrm{E}\left(V,H;\theta\right)=\sum_{i=1}^{D}\frac{\left(v_{i}-a_{i}\right) ^{2}}{2\sigma_{i}{}^{2}}-\sum_{i=1}^{D}\sum_{j=1}^{F}\frac{v_{i}}{\sigma_{i}} w_{ij}h_{j}-\sum_{j=1}^{F}b_{j}h_{j}, \tag{6}\]  
where \(\sigma_{i}\) denotes the standard deviation of the \(i\)-th visible unit and \(\theta=\left\{\left(w_{ij}\right)\in\mathbb{R}^{D\times F},\left(a_{i}\right) \in\mathbb{R}^{D},\left(b_{j}\right)\in\mathbb{R}^{F},\left(\sigma_{i}\right) \in\mathbb{R}^{D}\right\}\). During the unsupervised pretraining process of the multimodal DBM, modalities can be thought of as labels for each other. Each of the multimodal DBM layers has a small contribution to eliminating modality-specific correlations. Therefore, in contrast to the modality-full input layers, the top layer can learn representations that are relatively modality free. The joint representation of the image and text inputs can be represented as follows:  
\[\mathrm{P}\left(V_{I},V_{T};\theta\right)=\sum_{H_{12},H_{12},H_{13}}\mathrm{ P}\left(H_{12},H_{12},H_{13}\right)\left(\sum_{H_{11}}\mathrm{P}\left(V_{I},H_{1 1},H_{12}\right)\right)\left(\sum_{H_{T1}}\mathrm{P}\left(V_{T},H_{T1},H_{T2} \right)\right), \tag{7}\]  
where \(\theta\) denotes all model parameters. The reader may refer to the original paper for more details of multimodal DBMs.  
Figure 5: Architecture of a multimodal deep Boltzmann machine (DBM) for fusing image and text representations.  
An advantage of multimodal DBMs is that they can deal with the absence of some modalities. After training our multimodal DBM, even though some pins may have no descriptions, the activation probabilities of \(H_{3}\), which are used as our final multimodal joint representation of pins, could be inferred from different conditional distributions with the standard Gibbs sampler. In addition, the multimodal DBMs are used to generate the missing text representation in a similar manner. Moreover, multimodal DBMs can be trained supervised by connecting additional label layers on top of them.