# Recommendations for Different Tasks Based on the Uniform Multimodal Joint Representation
## 5 Experiments and Results
### Datasets and Implementation Details  
We crawled data used in experiments from Huaban, a typical Chinese CCSN. Huaban provides certain applications similar to those in Pinterest, while the main differences between the two networks are as follows: There are \"like\" pins or board operations on Huaban but not on Pinterest; Huaban records both users and the paths in a re-pin tree, while Pinterest only records all the users and the initially created user.  
We first crawled the pins of 5957 users without images, and then sampled 88 users according to board categories and pin counts. Some extremely active and cold-start users had been confirmed among them to make our dataset diverse and to take the influence of pin counts into account. We then crawled all images of the sampled users and all their \"like\" pins. In addition, we crawled the top 1000 pins recommended by the system of each category to fine-tune AlexNet and their re-pin paths for automatic annotation. The dataset for recommendation included 151,631 pins, which were categorized into 33 categories from 1694 boards, and the number of unique images for both fine-tuning and recommendation was 167,747. All pins were used as supplement elements for obtaining distributions of the recommended pin categories. The average re-pin path length was 47.57.  
After a little manual label balancing, labelled images were split into 80% for training and validating and the remaining 20% for testing. Because the input dimension of AlexNet should be constant, every image was firstly rescaled so that the shorter side was 256 pixels, and then the central 256 \(\times\) 256 patch of the processed image was cropped out. The loss layer of our AlexNet was replaced. As a comparison, the most frequent category was used as the label to fine-tune a multiclass AlexNet. The dimensions of the FC8 layers of both Alexnets were changed to 33. Image representations were generated from the FC7 layer of the multilabel Alexnet.  
We trained our Word2 Vec model on Wikipedia dumps ([[https://dumps.wikimedia.org/](https://dumps.wikimedia.org/)]([https://dumps.wikimedia.org/](https://dumps.wikimedia.org/))) and Sougou Lab dataset ([[http://www.sogou.com/labs/resource/list_news.php](http://www.sogou.com/labs/resource/list_news.php)]([http://www.sogou.com/labs/resource/list_news.php](http://www.sogou.com/labs/resource/list_news.php))) with the CBOW (Continuous Bag of Words) model and negative sampling. In addition, the vector dimension was 300. The words with a frequency lower than five were ignored. Word preprocessing, such as removing punctuation, traditional and simplified Chinese conversion, word tokenization, machine translation, and removing stop words, was applied on pin descriptions.  
All image and text representations were exploited for the multimodal DBM training. The dimensions of \(H_{\text{T1}}\), \(H_{\text{T2}}\) and \(H_{\text{V1}}\) were the same as their corresponding visible inputs, and dimensions of \(H_{\text{V2}}\) and \(H_{\text{3}}\) were set to 2048 to compress the vectors, as the FV would increase the dimension. Each layer was pretrained using a contrastive divergence strategy to accelerate the training of the DBM. Then, missing text representations were extracted using Gibbs sampler and the multimodal joint representation of pins was inferred.  
\(K\) in Equation (10) was set to 1 such that the dimension of the FV of a board was twice that of the pin vector. \(\alpha\) in Equation (15) was set to 0.5.  
To evaluate the effectiveness of the proposed model, we compared it with the following multimodal deep architectures: the Multimodal Autoencoder (MAE), which was proposed in [31] and connects two deep autoencoders of multimodalities by a shared hidden layer; and ICMAE, which imposes Independent Component Analysis (ICA) constraints in the MAE architecture to de-correlate the relationships among the variables. All the baseline methods had the same number of layers, and we used the same features as inputs to ensure that the comparisons were fair.