# Cross-Modal Manifold Propagation for Image Recommendation
## 3 Cross-Modal Manifold Propagation
### Visual Manifold Propagation  
Informative visual content in social multimedia environments brings a massive amount of visual information at low costs. It inspired us to take informative visual semantics of images as supplementary over sparse user-image records. The green module in Figure 2 shows the constructed visual manifold propagation (VMP) to estimate semantic detailed cross-modal correlation. VMP transfers characteristics of users' visual records along with visual semantic distribution and produces a semantic-aware user-image correlation estimation in view of visual semantics. Visual distribution-oriented propagation greatly benefits from images' dense semantic visual correlation. Distribution provides a more informative detailed correlation than the sparse user-image correlation in UMP. Since deep models, such as AlexNet [39], have achieved great success in discriminative feature learning, high-level semantic visual features could be robustly extracted instead of mining explicit semantics by classification or clustering. On analyzing visual semantics, CMP immediately adopts AlexNet [39] to capture visual semantic features to construct visual manifold distribution. The visual recommendation module in Figure 2 builds a visual manifold by a visual adjacency graph \(G_{v}=(V,W^{v})\) on image set \(V\), where \(W^{v}=[W^{v}_{ij}]\) contains pairwise visual similarity between images \(v_{i}\) and \(v_{j}\) with cosine measurement on their semantic features. Unlike data incompleteness in the interest-based recommendation module, visual correlations require massive volumes and tend to be more complicated. This leads the visual manifold to a wide range of visual semantics. In the case of complex distribution, [PERSON] et al. [5] found that manifold propagation would result in biased structure learning with high possibility. A predefined learning strategy is expected to simplify and guide the learning procedure without bias from complex data correlation. Therefore, the visual module was designed to learn user-image correlation in a decomposed manner. Since the nature of semantic-similar contents is distributed to be relatively compact, the proposed CMP employs semantic-compact distribution of the visual manifold to break down the global manifold into several submanifolds. The operation is called manifold modularization. It immediately relieves the semantic complexity of the visual manifold propagation and alleviates the computational burden on the global visual manifold. The modularity \(Q\) of visual manifold is measured by Equation (4). \[ \begin{split} Q&=\frac{1}{2m}\sum\limits_{i,j=1,2, \ldots,n}\bigg{[}W^{v}_{ij}-\frac{k_{i}k_{j}}{2m}\bigg{]}\delta\big{(}o_{i},o_{ j}\big{)}\\ &=\frac{1}{2m}\sum\limits_{a\in A}\bigg{[}a_{in}-\frac{a_{tot}^{2 }}{2m}\bigg{]}\end{split}  \tag{4}\]  
where \(W^{v}_{ij}\) indicates semantic visual correlation between images \(v_{i}\) and \(v_{j}\) in \(W^{v}\), \(m=\frac{1}{2}\sum\limits_{i,j}W^{v}_{ij}\) provides the normalized weights in \(W^{v}\), \(k_{i}=\sum\limits_{j}W^{v}_{ij}\) is the corresponding degree of \(v_{i}\) in graph \(G_{v}\), and \(o_{i}\) indexes the corresponding sub-manifold of image \(v_{i}\). \(\delta(o_{i},o_{j})\) indicates whether \(v_{i}\) and \(v_{j}\) belong to the same sub-manifold with \(\delta\big{(}o_{i},o_{j}\big{)}=1\) if \(o_{i}=o_{j}\), otherwise \(\delta\big{(}o_{i},o_{j}\big{)}=0\). \(A\) is the set of sub-manifolds decomposed by modularity, \(a_{in}\) is the sum of the weights in a specific sub-manifold \(a\), and \(a_{tot}\) is the sum of weights of the edges incident to the corresponding sub-manifold \(a\), for \(a\in A\), \(i,j=1,2,...,n\). In the beginning of manifold modularization, images in \(V=[v_{1},\cdots,v_{n}]\) are individually treated as a single submanifold to initialize modularized manifold. The change in modularity takes place when allotting an image \(v_{i}\) into its neighboring submanifolds. Extracting an image \(v_{i}\) or entering it into a submanifold would both result in a gain to global modularity.