# Recommendations for Different Tasks Based on the Uniform Multimodal Joint Representation
## 2 Related Work  
With the rise of CCSNs, several studies have been performed, of which search engine, user modeling, and recommender systems are the most relevant. Most prior work only studied monomodal data. [PERSON] et al. [16] recommended boards re-ranked with image representations based on boards with the text representations model. [PERSON] et al. [22] recommended pins with two unimodal representations separately. [PERSON] et al. [11] predicted categories of pins with two kinds of unimodal representations and fused the two modality results using decision fusion. All the models are late fusion models that do not concern multimodal joint representations. Multimodal joint representation includes unimodal representation models and multimodal fusion schemes. For image representation, CNNs have achieved remarkable performance in the field of computer vision. Creating a large labelled dataset is the key to train CNNs. [PERSON] et al. [11] and [PERSON] et al. [12] directly used a pin's category as its label. However, this label may not be absolutely correct since the same image may have different categories selected by different users. [PERSON] et al. [10] trained a multitask CNN with ontological concepts, but the ontology was constructed in the fashion domain and was difficult to extend to all other domains. [PERSON] et al. [21] extracted more detailed labels on Pinterest by taking top text search queries, but the quality and consumption of this annotation highly depends on the search engine. Inspired by the fact that the predefined categories on CCSNs are not independent objects but related notions, labels formed by statistics category distributions are used and a CNN is fine-tuned as a multilabel regressor. With regard to text representation, one-hot representations [13; 16] and distributed representations, such as the word2 vec tool [11], have been used. From the practical point of view, the word2 vec tool [26], which can capture syntactic and semantic relationships between words in the corpus, is more scalable. In addition, mean vectors [27] of the word2 vec tool can obtain usable text representation without further learning. Several multimodal fusion studies are being performed on classification and retrieval. Except for directly concatenating modalities, most existing schemes are designed based on models such as CNNs [28] and recurrent neural networks [8]. These models mainly learn the consistency between multiple modalities and cannot deal with missing input modalities well. On the generative side, latent Dirichlet allocations (LDAs) [29], restricted Bolzmann machines (RBMs) [30], deep autoencoders (DAEs) [31], and deep Boltzmann machines (DBMs) [32] have been proven to be feasible methods for learning the consistency and complementarity between modalities and can easily deal with some missing modalities. However, limited studies have focused on fusing features obtained from these deep learning models. [PERSON] et al. [33] used a DAE for fusing the textual features extracted by training the Word2 Vec tool [26] and visual features generated by the 6-th layer of AlexNet [34]. However, there are no existing studies that have used information from all modalities from CCSNs for recommendation tasks. In this paper, we trained a multimodal DBM to handle a situation in which the data from CCSNs are unlabeled and some modality inputs are missing, and we used features obtained by deep learning as the input to make our multimodal representation more accurate and compact. Compared to pin and board category recommendation, few studies have been performed on board and user recommendations. [PERSON] et al. and [PERSON] et al. [13; 23] model boards and users, respectively, with text data and some collaborative filtering methods [15; 20; 25] to recommend users with user behaviors, but they do not take images, which are the essential content on CCSNs, into account. [PERSON] et al. [17] represent boards by sparse coding the descriptors of images, but similarly to [PERSON] et al.