# Recommendations for Different Tasks Based on the Uniform Multimodal Joint Representation
## 4 Implementation of Recommendations for Different Tasks
### Board and User Recommendation  
As the pins are assembled, the theme of a board emerges. Users can easily collect pins with well-organized boards. For this reason, board recommendation is another important function for content discovery on CCSNs. In this section, we discuss how to model boards and users using the acquired multimodal joint representations of pins. There is an analogy between user contents on CCSNs and articles, as user contents consist of boards which consist of pins, while articles are composed of paragraphs or sentences that are composed of words. One clear difference between them is that the order of pins or boards may not be that important. Therefore, the loss of order information is not an issue when modeling. Inspired by this, we consider that applying pooling methods to transform a different number of pins into a constant dimension vector, as we mentioned in Section 3.2, is reasonable for board and user modeling. Among pooling methods, the Fisher vector (FV) was chosen as our solution for board and user modeling. The FV [37] was designed for encoding patch descriptors of an image into a high-dimensional vector. Since boards and users are image collections, a pin can be treated as a descriptor of them. A common method to encode a set of descriptors is to assign them into a visual dictionary, which is composed of prototypical elements such as cluster centers, while the FV approximates the distribution of descriptors with a GMM, whose Gaussian distributions can be treated as a universal probabilistic visual dictionary. As for representation of pins \(V_{X_{i}}=\left(v_{ij}\right)\in\mathbb{R}^{I}\), the GMM is defined as  
\[\text{GMM}\left(V_{X}\right)=\sum\limits_{k=1}^{K}\omega_{k}\text{norm}_{k} \left(V_{X}\right), \tag{10}\]  
where \(\text{norm}_{k}\) denotes the \(k\)-th multivariate normal distribution, \(\omega_{k}\) is the weight of the \(k\)-th mixture component and is subject to the following constraints: \(\forall_{k}:\omega_{k}\geq 0\) and \(\sum\limits_{k=1}^{K}\omega_{k}=1\); and \(K\) is the number of mixture components. The parameters of the GMM also include \(\left(\mu_{kj}\right)\in\mathbb{R}^{J}\) and \(\Sigma_{k}\), which are the mean vector and covariance matrix of the \(k\)-th mixture component, respectively. The FV first computes the partial derivatives with respect to the parameters of the logarithm of the GMM, and then it normalizes them with the Fisher information matrix. The simplified normalized partial derivatives of a board \(B\) are given by \[g_{\omega_{k}} =\frac{1}{N_{B}\sqrt{\omega_{k}}}\sum_{i=1}^{N_{B}}(\gamma_{ik}- \omega_{k}), \tag{11}\] \[g_{\mu_{k}} =\left(\frac{1}{N_{B}\sqrt{\omega_{k}}}\sum_{i=1}^{N_{B}}\gamma_{ ik}\left(\frac{v_{ij}-\mu_{kj}}{\sigma_{kj}}\right)\right)\in\mathbb{R}^{I},\] (12) \[g_{\sigma_{k}} =\left(\frac{1}{N_{B}\sqrt{2\omega_{k}}}\sum_{i=1}^{N_{B}}\gamma_ {ik}\left[\frac{\left(v_{ij}-\mu_{kj}\right)^{2}}{\sigma_{kj}^{2}}-1\right] \right)\in\mathbb{R}^{I}, \tag{13}\]  
where \(\sigma_{kj}\) denotes the standard deviation of the \(j\)-th dimension of the \(k\)-th mixture component, and \(\gamma_{ik}\) is the soft assignment of \(V_{P_{\text{i}}}\) to the \(k\)-th mixture component, which is written as  
\[\gamma_{ik}=\frac{\omega_{k}\text{norm}_{k}\left(V_{X_{i}}\right)}{\sum\limits \limits_{k=1}^{K}\omega_{k}\text{norm}_{k}\left(V_{X_{i}}\right)}=\frac{ \omega_{k}\text{norm}_{k}\left(V_{X_{i}}\right)}{\text{GMM}\left(V_{X_{i}} \right)}, \tag{14}\]  
and is also known as the posterior probability or responsibility. All partial derivatives are concatenated to compose the FV. Since one of \(\omega_{k}\) is redundant because of the constraints, the dimension of the FV is \((2J+1)\)\(K-1\). Power normalization and \(L2\)-normalization [38] are applied to improve the quality of the FV as follows:  
\[g_{i} \leftarrow\text{sgn}\left(g_{i}\right)|g_{i}|^{\rho}, \tag{15}\] \[g_{i} \leftarrow\frac{g_{i}}{\sqrt{\frac{(2J+1)^{K-1}}{i}{g_{i}}^{2}}}, \tag{16}\]  
where \(\rho\in[0,1]\) is the normalization parameter.