# Cross-Modal Manifold Propagation for Image Recommendation
## 1 Introduction  
The recent ubiquitous impact of social platforms and related techniques has highly accelerated the creation and spread of multimedia information [1]. The social multimedia environment provides rich interactions for users to generate and enrich content, and even collaborate. As reported on November 2021 ([[https://www.visualcapitalist.com/from-amazon-to-zoom-what-happens-in-an-internet-minute-in-2021/](https://www.visualcapitalist.com/from-amazon-to-zoom-what-happens-in-an-internet-minute-in-2021/)]([https://www.visualcapitalist.com/from-amazon-to-zoom-what-happens-in-an-internet-minute-in-2021/](https://www.visualcapitalist.com/from-amazon-to-zoom-what-happens-in-an-internet-minute-in-2021/)), accessed on 15 January 2022), during an Internet minute, the Google engine processes 5.7 million searches, Facebook receives 44.0 million views, TikTok serves 167.0 million videos, and 12.0 million emails are sent. Such a variety of social platforms usher in the evolution of people communicating with each other via information sharing, and following the activities and comments of others. This inevitably produces a wide range of overloaded multimedia content, which lays tremendous obstacles to quality of service, especially response speed. To this end, personalized recommendations are promising for guaranteeing effective and efficient access to users' desired multimedia content. However, users' diverse preferences can not be easily uncovered due to extremely sparse user-image correspondence in current social environments [2]. Figure 1 takes content from Huaban.com to roughly illustrate the relationship between user space and visual content space with intention and semantic gaps. Since both users and visual content play critical roles in the modern social environment, it is crucial to understand and bridge user intention and multimedia semantic gaps for personalized recommendations. Personalized image recommendation is intrinsically modeled as the cross-modal matching problem between users and images [1]. Recent works [3; 4] successfully modeled user interests with graph-based learning due to the inherently heterogeneous pairwise interactions. Their advance inspired further investigating the heterogeneous graph of interactions by manifold propagation [5] to learn users' interests, since a manifold actively works on learning the global graph structure. Contextual structures are involved in compensating collaborative signals to mine cross-modal correlation on personalized recommendations [2; 6; 7]. We thus propose cross-modal manifold propagation (CMP) to match pairwise user images with correlation estimation. Figure 2 illustrates the framework of CMP to perform cross-modal learning collaboratively on both user interest and visual distributions. Figure 2 shows that CMP takes user interest distribution (blue) to spread characters of users' image records over users' interest manifold, which induces a set of interest-aware image candidates for each user. On the other hand, the visual recommendation module (green) is desired to spread users' image records along the visual semantic manifold of images and produce detailed visual estimation as dense user-image correlation. An interest-aware semantic fusion strategy is further introduced to incorporate detailed visual-aware user-image correlation with interest-aware image candidates to rank personalized candidates and recommend high-score images to the user. The main contributions of this work are as follows. * We designed user interest-oriented semantic image ranking for cross-modal collaborative recommendation by investigating the distributions of both user interest and visual semantics. * The proposed CMP reveals the trend of interests for users and estimates interest-aware user-image correlation by spreading users' image records on users' interest manifold. * CMP leverages visual manifold modularization to help reduce the computational burden on visual manifold propagation, and promote estimating semantic visual-aware user-image scores for recommendation.