# A Multimodal Recommender System Using Deep Learning Techniques Combining Review Texts and Images
## 3 Methodology
### CAMRec Architecture
#### 3.2.2 Feature Extraction Network  
The Feature Extraction Network aims to extract the textual and visual features from texts and images and fuse them to represent users' preferences. \(R_{u,j}=\{r_{1},r_{2},\cdots r_{l}\}\) represents a review text left by user \(u\) on item \(i\), where \(l\) represents the length of a review text. Meanwhile, \(M_{u,i}=\{m_{1},m_{2},\cdots m_{n}\}\) represents the image accompanied by the review text, where \(n\) represents the number of images. To extract the textual feature of review texts, we use the pre-trained RoBERTa model, which has demonstrated its excellent performance in the NLP field. The RoBERTa model can be seen as an extension of BERT, which was trained using a dynamic masking method, larger datasets, and additional training optimizations. The pre-trained RoBERTa base model consists of 12 transformer encoders, each with 12 attention heads, and each output token is represented as a 768-dimensional vector. Since the [CLS] token of RoBERTa contains detailed meanings behind the sentence [8], we use the 768-dimensional vector of the [CLS] token as the textual feature. Therefore, the textual feature \(O^{Text}\) can be defined as follows:  
\[O^{Text}=\text{RoBERTa}(R_{u,i}), \tag{5}\]  
To extract the visual feature of review images, we use the pre-trained VGG-16 model. VGG-16 is a deep CNN architecture widely used in image recognition and classification tasks. The VGG-16 model consists of 13 convolutional layers followed by three fully connected layers. Each convolutional layer employs \(3\times 3\) filters to extract features from the input images. Before inputting images to the VGG-16 model, each image should be resized to 224 \(\times\) 224 pixels, converted to the RGB color space, and normalized by subtracting the mean of each color channel [26]. Since the model is used to perform image recognition, we decided to use the 4096-dimensional vector of the last convolutional layer as the visual feature. Considering the number of images varies with the reviews, we decided to extract the vectors of each image and average the value to represent the visual features for the review effectively. Therefore, the visual feature \(O^{Image}\) can be defined as follows:  
\[O^{Image}=\frac{1}{n}\sum_{i=1}^{n}\text{VGG}-16(M_{u,i}), \tag{6}\]  
where \(n\) represents the number of images. For both review text and image features, they are fed into the MLP to enhance nonlinearity and ensure the same dimension for each feature. Therefore, the final review text and image feature vectors can be represented as \(O^{Text}_{M}\) and \(O^{Image}_{M}\). The goal of this study is to complementarily fuse features based on the relative importance of each modality in the recommendation process. Therefore, this study leverages the advantages of the co-attention mechanism to model the complementarity between extracted final review text and image feature vectors. For a clear comparison, we recap the scaled dot product attention mechanism as follows:  
\[\text{Attention}(Q,\ K,\ V)=\text{Softmax}(\frac{QK^{T}}{\sqrt{d_{k}}})\ V, \tag{7}\]  
where \(Q\), \(K\), and \(V\) indicate query, key, and value, respectively. \(\sqrt{d_{k}}\) indicates the square root of the dimension of the key vector \(d_{k}\). The attention mechanism is typically implemented using a multi-head structure, where multiple attention heads operate in parallel to capture different aspects of the information. The formula of the multi-head attention mechanism can be defined as:  
\[\text{MultiHead}(Q,K,V)=(\text{head}_{1}\oplus\text{head}_{2} \oplus\ldots\oplus\text{head}_{\text{h}})W^{O}, \tag{8}\] \[\text{where head}_{\text{i}}=\text{Attention}(QW^{O}_{i},KW^{K}_{ i},PW^{V}_{i}),\]where \(W_{i}^{Q}\in\mathbb{R}^{d_{k}\times d_{p}}\), \(W_{i}^{K}\in\mathbb{R}^{d_{k}\times d_{p}}\), and \(W_{i}^{V}\in\mathbb{R}^{d_{u}\times d_{p}}\) indicate the weight matrices for the queries \(Q\), keys \(K\), and values \(V\) of the \(i\)-th head, respectively.