## 3 The Proposed Method
### Feature Extraction  
People often express their opinions on selected topics by posting related texts and images they are interested in. These behaviors might show their implicit user preferences. Thus, to better understand what people like, in this paper, we assume that user preferences can be represented by the characteristics of texts and images in their posts and comments.  
Previous studies used the title or tags as text features for classification. However, user preferences cannot be effectively learned since tags might be ambiguous and diverse in their meanings. Since text contents usually contain more semantic information such as emotions or stances than tags, text contents are used instead of tags. In this paper, we focus on deep learning methods such as the TextCNN model [41] for text feature extraction. First, text documents are represented by a word embedding model such as Word2 Vec [42; 43]. This is a distributional representation of words among different contexts in fix-sized vectors. Next, we use TextCNN [41] to extract text features. This is a slight variant of the CNN architecture by [PERSON] et al. [44]. The TextCNN architecture is shown in Figure 2.  
Figure 1: The system architecture of our proposed method.  
Figure 2: Convolutional neural networks (CNNs) for text classification [41].  
As shown in Figure 2, there are four layers in the TextCNN model: word embedding layer, convolutional layer, max pooling layer, and fully connected layer. First, given a sentence of length \(n\), we denote it by its word embeddings as follows:  
\[x_{1:n}=x_{1}\varnothing x_{2}\varnothing\dots\varnothing x_{n}, \tag{1}\]  
where \(x_{i}\) is the embedding of word \(i\) with dimension \(k\), and \(\varnothing\) is the concatenation operator. Second, in the convolutional layer, the convolution operation involves a filter \(w\in R^{hk}\), which is applied to a window of \(h\) words to produce a feature \(c_{i}\) for the \(i\)-th window as follows:  
\[c_{i}=f(w*x_{i:i+h-1}+b), \tag{2}\]  
where \(b\in R\) is a bias term and \(f\) is a nonlinear function, such as the hyperbolic tangent. For the given sentence \(x_{1:n}\), this filter produces a feature map \(c\) as follows:  
\[c=[c_{1},c_{2},\dots,c_{n-h+1}], \tag{3}\]  
where \(c\in R^{n-h+1}\). Then, in the max pooling layer, we take the maximum value \(c^{\prime}=max(c_{1}\), \(c_{2}\),..., \(c_{n-h+1})\) as the feature map corresponding to this particular filter. Lastly, the fully connected layer outputs the final classification result or category. In this paper, we also extract outputs from the layer before the last layer to compare the effects of feature fusion.  
To extract image features, we utilize a CNN model called VGG16 [45] to convolve the image pixels represented in the Red-Green-Blue (RGB) color model. This architecture contains several differences from the previous convolutional networks. It is a multilayer convolutional network that is a thorough evaluation of networks of increasing depth using an architecture with very small (3 \(\times\) 3) convolution filters, which shows a significant improvement by increasing the depth to 16 layers (13 convolutional layers and three fully connected layers). Layers of configurations are designed using the same principles, inspired by [PERSON] et al. [46] and [PERSON] et al. [47]. Compared with other models such as [48], VGG16 is a CNN architecture with fewer parameters but comparable accuracy. For this reason, it is used as the Image CNN model to extract the feature map from images.