# Cross-Modal Manifold Propagation for Image Recommendation
## 3 Cross-Modal Manifold Propagation
### Interest Manifold Propagation  
Users' historical records intrinsically reflect users' preferences for multimedia information. This inspired us to investigate users' personalized preferences with their historical records. However, considering the volume of multimedia content, the amount of user-image records is incomplete to uncover users' entire interests, not to mention for learning interest representations for users. To leverage precious historical records, the proposed CMP evaluates common interests among users and constructs a manifold of user interest distribution. User interest manifold takes a user interest graph \(G_{u}=(U,W^{u})\) with \(W^{u}=[W_{ij}^{u}]\) representing the common interest degree of users \(U=[u_{1},\cdots,u_{C}]\) as Equation (1), \(i,j=1,2,...,C\). \[W_{ij}^{u}=\frac{\big{|}S_{i}\cap S_{j}\big{|}}{\sqrt{\big{|}S_{i}\big{|} \big{|}S_{j}\big{|}}} \tag{1}\]  
where \(S_{i}/S_{j}\) represents the set of images with which user \(u_{i}/u_{j}\) interacts, \(|S_{i}|/|S_{i}|\) is the size of \(S_{i}/S_{j}\), and \(\big{|}S_{i}\cap S_{j}\big{|}\) measures the number of images that users \(u_{i}\) and \(u_{j}\) share in common, \(\cap\) is the intersection operation, \(i,j=1,2,...,C\). Equation (1) evaluates the proportion of images shared by users \(u_{i}\) and \(u_{j}\) to represent their common interests. The more common the images the two users share are, the higher the common interest degree \(W_{ij}^{u}\) exists between them is. On the interest manifold, as shown in the blue module in Figure 2, characteristics of user image records could be spread along with the distribution of user interest. The interest-based recommendation module uncovers interest-aware cross-modal correlation \(F_{u}\) in view of user interests. Suppose that \(V=[v_{1},\cdots,v_{n}]\) represents an image set, and \(Y=[y_{ij}]_{n\times C}\) are corresponding interactions from user historical records. \(y_{ij}=1\) records existing interactions between image \(v_{i}\) and user \(u_{j}\); otherwise, \(y_{ij}=0\), for \(i=1,2,\cdots,n\), \(j=1,2,\cdots,C\). CMP employs the following constrained interest smoothness operation to perform interest manifold propagation and infer personalized interest degree \(F_{u}\) to images \(V\) as  
\[F_{u}^{*}=\operatorname*{arg\,min}_{F_{u}}(1-\alpha)\|F_{u}-Y\|_{F}^{2}+\alpha tr (<F_{u},L^{u}F_{u}>) \tag{2}\]  
where \(F_{u}=[f_{ij}^{u}]_{n\times C}\) represents estimated correlation \(f_{ij}^{u}\) of user \(u_{j}\) to image \(v_{i}\) with superscript \(u\) denoting the user interest manifold used in propagating users' records, visual image index \(i=1,2,\cdots,n\), user index \(j=1,2,\cdots,C\). \(\|*\|_{F}^{2}\) calculates the _Frobenius_ norm, \(\alpha\) is a trade-off parameter to constrain between the global interest smoothness and the distinguishing interest of users. User graph Laplacian \(L^{u}=I-D^{-1/2}W^{u}D^{-1/2}\) is derived from the common interest degree matrix \(W^{u}=[w_{ij}^{u}]_{n\times n}\), identity matrix \(I\) and user degree matrix \(D\) diagonalized by column sum of \(W^{u}\), and \(tr(*)\) denotes the trace operator. Higher \(f_{ij}^{u}\) implies more possibility that user \(u_{j}\) would prefer image \(v_{i}\). Furthermore, an analytical solution of interest manifold propagation was derived as Equation (3), representing estimated interest-aware user-image correlation. \[F_{u}^{*}=((1-\alpha)I+\alpha L^{u})^{-1}Y \tag{3}\]  
The user interest manifold propagation module is referred to as UMP in blue of the collaborative learning framework in Figure 2. The user interest-based recommendation module endows users visible to and is aware of others' interests on visual images, i.e., to know the trend of general visual preferences. Taking into consideration the incompleteness of interest manifold in UMP module, user-image correlation in Equation (3) reflects only users' interest tendency, and not a detailed interest degree.