# Visual Enhancement Capsule Network for Aspect-Based Multimodal Sentiment Analysis
## 3 Methodology
### Visual Enhancement Capsule Network Based on Multimodal Fusion
#### 3.2.3 Multimodal Fusion Based on Interactive Learning  
The interactive learning of the memory network is mainly to capture the auxiliary information between the text and the image for improving the performance of multimodal sentiment analysis. To explore the complementarity information of text and images, we propose a multimodal multi-round fusion module based on interactive learning using the capsule features from MemCapsNet and AgCapsNet for aspect-based sentiment classification. A new visual enhanced feature fusion network is designed for interactive learning of the different modalities. The network iteratively queries the text and image features by using the next multi-hop storage system to explore the relationship between text and images. The upper part of Figure 1 illustrates the architecture of the module, including three parts of processes: an aspect-guided image learning based on a Text2 Vision attention mechanism, an image-guided text learning based on a Vision2 Text attention mechanism, and a capsule convolution. **Aspect-guided image learning. ** In order to learn the image features related to the affective entities, an aspect-guided Text2 Vision attention mechanism is designed for allowing the aspect features to help the model find the key feature mapping of an image, that is, using the aspect capsules to conduct the gating selection on the image capsules of different types and at different locations. Let \(p_{j}^{l}\in\mathbb{R}^{B\times(4\times 4)}\) be the pose matrix vector of the \(j\)th image \(I_{j}\) at the \(l\)th position and \(l\in[1,d\times d]\). \(v^{a}\) is an aspect mapping vector from the hidden vector of aspect phrases in MemCapsNet. As shown in Figure 1, an aspect mapping is obtained by performing a temporal maximum pooling on the aspect phrase indices of the text source capsule sequence \(P^{pri}\):  
\[v^{a}=pooling(p_{x}^{pri}|x\in[n_{1}+1,n]) \tag{16}\]  
where \(p_{x}^{pri}\) is the pose vector of \(3*d\) capsules of the \(x\)th aspect word. In order to gain the aspect-guided attention, we first compute the projects of the pose vector \(p_{j}^{l}\) and the aspect vector \(v^{a}\) onto the Text2 Vision attention space as \(k_{l}\) and \(q^{a}\), respectively:\[k_{l} = tanh(w_{k}p_{j}^{l}+b_{k}) \tag{17}\] \[q^{a} = tanh(w_{q}v^{a}+b_{q}) \tag{18}\]  
where \(w_{k}\in\mathbb{R}^{d_{att}\times B\times(4\times 4)}\) and \(w_{q}\in\mathbb{R}^{d_{att}\times B}\) are the coefficient matrixes of projecting, \(b_{k}\), \(b_{q}\in\mathbb{R}^{d_{att}}\) are their corresponding biases, and \(d_{att}\) is the size of the attention space. Then, the aspect-guided correlation score \(g_{l}\) and the attention score \(\beta_{l}\) are computed as follows:  
\[g_{l}=V\cdot(k_{l}\odot q^{a}+k_{l}) \tag{19}\]  
\[\beta_{l}=\frac{exp(g_{l})}{\sum_{l}exp(g_{l})} \tag{20}\]  
where \(V\in\mathbb{R}^{d_{att}}\) is a randomly initialized contextual vector that can be trained, and we can obtain the attention matrix of images \(\beta_{j}=\{\beta_{jl}|1\leq l\leq d*d\}(1\leq j\leq m)\). Lastly, the most relevant image with the given affective entity is found as below:  
\[J=\operatorname*{arg\,max}_{l}(\|\beta_{l}\|_{2}) \tag{21}\]  
where \(\|\cdot\|_{2}\) is the 2-norm function of the matrix. We can complete the gating selection of the image capsule using attention matrix \(\beta_{l}\) to weight the capsule features of \(l_{J}\), and obtain the capsule feature map \(p_{J}\) related to the affective entity. Subsequently, we conduct the multilayer attention computation on \(p_{J}\) to get the final attention matrix \(\beta_{J}\) and the weighted capsule feature mapping \(\beta_{j}\odot p_{J}\), which will be inputted into the add and norm layer for the further sentiment computation. At the same time, with \(\beta_{j}\) and \(p_{J}\) as the inputs for \(\sum\) operation, we can obtain the global feature matrix:  
\[v^{img}=\frac{\sum_{k}\beta_{jk}\odot p_{jk}}{\sum_{k}\beta_{jk}} \tag{22}\]  
Then, \(v^{img}\) can be inputted into the Vision2 Text attention layer for image-guided text feature learning.