# A Multimodal Recommender System Using Deep Learning Techniques Combining Review Texts and Images
###### Abstract  
Online reviews that consist of texts and images are an essential source of information for alleviating data sparsity in recommender system studies. Although texts and images provide different types of information, they can provide complementary or substitutive advantages. However, most studies are limited in introducing the complementary effect between texts and images in the recommender systems. Specifically, they have overlooked the informational value of images and proposed recommender systems solely based on textual representations. To address this research gap, this study proposes a novel recommender model that captures the dependence between texts and images. This study uses the RoBERTa and VGG-16 models to extract textual and visual information from online reviews and applies a co-attention mechanism to capture the complementarity between the two modalities. Extensive experiments were conducted using Amazon datasets, confirming the superiority of the proposed model. Our findings suggest that the complementarity of texts and images is crucial for enhancing recommendation accuracy and performance. recommender system; multimodal review; deep learning; co-attention mechanism; complementarity +
Footnote †: journal: _Sciences_  
0000-0001-5000-0000-0000-0000-0000-0000-0000-0000-0000-00000-00000-00000-00000-00000-00000-00000-00000-00000-000000-00000-00000-00000-0000  
for extracting textual representations for recommendations. [PERSON], [PERSON] and [PERSON] [8] used Bidirectional Encoder Representations from Transformers (BERT) and Robustly optimized BERT approach (RoBERTa) models as the encoder to extract textual information from review texts. The experimental results showed such methods can significantly enhance recommendation performance, as they incorporate more diverse information from the review texts. Although mitigating the issue of data sparsity and improving recommendation performance, these review-based recommender system studies have a limitation in that they fail to capture user preferences for visual representations [9]. Nowadays, most online reviews are composed of multimodal contents including both images and texts [10]. Images with visual information can convey detailed aspects of items and users' preferences in an intuitive manner that review texts may not be able to deliver [11]. From the sentence \"I love this design\" in the review text shown in Figure 1, it is evident that the user appreciates a specific aspect of the phone case design. However, relying solely on the text, it is difficult to determine which specific part of the design the user appreciates. In other words, images can directly deliver user preferences that are not expressed in the text [9]. Therefore, texts as textual information and images as visual information in multimodal reviews provide different types of information that can be complementary or substitutive [11]. Meanwhile, some studies confirm that fusing these two types of information enhances the effectiveness of the recommendations [10]. Therefore, it is necessary to propose a recommender system that can capture complementary and substitutive effects between texts and images. For the fusion strategy, most studies used multiplication or concatenation operations, which do not fully reflect the interactions between different modalities [12]. To achieve the proposed idea, this study applies the advanced fusion method based on attention mechanism to capture the complementarity between texts and images. Specifically, this study leverages the advantages of co-attention to model the complementarity between review texts and images by focusing on aligned features. Therefore, this study proposes a novel recommender system model called CAMRec (Co-Attention based Multimodal Recommender system), which effectively integrates the complementary effects of texts and images into the recommendation tasks. First, we use the RoBERTa model to extract textual information, which demonstrates excellent performance on various Natural Language Processing (NLP) tasks.