# Cross-Modal Manifold Propagation for Image Recommendation
## 3 Cross-Modal Manifold Propagation
### Cross-Modal Collaborative Ranking  
As described in Section 3.1, the interest-based recommendation module selects interest-aware image candidates in the view of user interest where local details tend to be coarse, ignoring attributes of user interested content. The incompleteness of user-image correspondence in UMP worsens this. In contrast, VMP greatly focuses on local semantic distribution. It was designed to prefer images that are as similar as possible to the user's records. This may limit the interest scope of recommendation in a semantic space. As shown in the framework of Figure 2, the proposed CMP collaboratively fuses the cross-modal estimation of UMP and VMP modules to incorporate their advantages in both global interest and local visual semantic views. The final dense user-image correlation is estimated by jointly optimizing UMP and VMP as Equation (8).  
\[\begin{cases}F_{u}^{*}=\operatorname*{arg\,min}_{F_{u}}(1-\alpha)\|F_{u}-Y\| _{F}^{2}+\alpha tr(<F_{u},L^{a}F_{u}>)\\ F_{v}^{*}=\bigcup\limits_{a\in A}\operatorname*{arg\,min}_{F_{v}}(1-\alpha)\| F_{v}-Y\|_{F}^{2}+\alpha tr(<F_{v},L^{a}F_{v}>)\end{cases} \tag{8}\]where their first terms focus on respecting historical records \(Y\), and the second terms propagate records on user interest manifold and visual semantic manifold for user-image correlation estimation.  
**Fusion discussion**: with user-image correlations \(F_{u}\) and \(F_{v}\) of both user interest and visual semantic views, CMP introduces fusion for estimating user-image correlation as follows:  
\[F=F_{u}\odot F_{v} \tag{9}\]  
where \(F_{u}\) and \(F_{v}\) are estimated user-image correlations propagated along user interest manifold by UMP and visual semantic manifold by VMP, respectively. \(\odot\) could be any commonly used fusion operators, such as min, max, and ave pooling. By considering the merits of UMP and VMP modules, CMP designs an interest-aware semantic fusion to derive the final user-image correlation for recommendation. On the one hand, an interest-aware image candidate set was selected by ranking \(F_{u}\) to match each user's interests. On the other hand, the detailed semantic user-image correlations by VMP were provided to rank the candidates to conduct the multi-view fusion of user interest and visual semantics. Lastly, CMP recommends images with high correlation \(F\) as customized visual contents for a specific user. In CMP, user interest manifold propagation plays a role in selecting user-interest image candidates, while visual manifold propagation estimates detailed user-image semantic scores for the candidates. This endows the recommended images with characters of both user and visual spaces while following users' individual preferences. Algorithm 2 summarizes the proposed cross-modal collaborative manifold propagation for image recommendation.  
```
0:\(V=[v_{1},\cdots,v_{n}]\)--visual image database; \(U=[u_{1},\cdots,u_{C}]\)--set of users; \(Y=[y_{ji}]_{n\times C}\)--historical records between \(V\) and \(U\).
0: Image list for personalized recommendation.
1: Construct interest manifold of users and visual manifold of images with graphs \(G_{u}\) and \(G_{v}\) by measuring common interests between users and semantic visual correlation of images, respectively.
2: Propagate users' historical image records along user interest manifold for estimating interest-aware user-image correlations \(F_{u}\) by Equation (3).
3: Generate an interest-aware image candidate set for each user on the basis of UMP.
4: Decompose semantic visual manifold into semantic compact submanifolds by measuring modularity gain as Equation (5).
5: Propagate users' historical image records along modularized semantic visual submanifolds by Equation (7) for semantic-aware user-image correlations \(F_{v}\).
6: Fuse \(F_{u}\) and \(F_{v}\) with Equation (9), and rank the candidate images of each user according to their semantic-aware user-image correlations for personalized recommendation.
```  
**Algorithm 2** Cross-modal collaborative manifold propagation.