# Visual Enhancement Capsule Network for Aspect-Based Multimodal Sentiment Analysis
## 3 Methodology
### Training and Predicting  
During training, MemCapsNet and AgCapsNet are separately used for encoding text and images. In the other words, the capsule representations for the text and images are obtained by two models, respectively. Multimodal fusion based on interactive learning is applied to select and enhance the aspect-based sentimental information of capsule features using the mutual information between text and images. Finally, the probabilities of aspect-based sentiment labels are outputted from the capsule convolution layer. Our training objective is to maximize the interval between the active value \(act_{t}\) of the objective label and \(act_{i}\) of the other labels, and the loss function is defined as  
\[Loss=\sum_{i\
eq t}Loss_{i} \tag{33}\]  
\[Loss_{i}=(\max(0,m-(act_{t}-act_{i})))^{2} \tag{34}\]  
where \(m\) is an interval coefficient. \(m\) is set as 0.2 at the beginning, and linearly increased to 0.9 with the training. We start training with a smaller interval value to avoid the problem of dead capsules due to early excessive punishment. Since the two modalities of text and image correspond to two losses \(Loss^{txt}\) and \(Loss^{img}\), respectively, in order to make the text capsules with more aspect-oriented information play a leading role in the training, we use the weighted interval loss as the joint objective function of the network, i.e.,  
\[Loss=Loss^{txt}+w^{img}Loss^{img} \tag{35}\]  
where weighted coefficient \(w^{img}\in[0,0.5]\). \(Loss^{txt}\) and \(Loss^{img}\) are two modal losses obtained from Equations (33) and (34), respectively.  
Algorithm 1 is the operational flow of VECapsNet.  
**Input:** Multimodal triplet test dataset: \(\{<T,I,A_{t}>\}\)  
**Output:** The sentiment label set \(L_{t}\) at aspect \(a_{t}\)  
1. Extract the context word vector \(e^{t}\), the aspect word vector \(e^{a}\) and the image feature mapping set \(M\)  
2. Obtain the text hidden vectors \(l^{c}\&hl^{a}\) from Bi-LSTMs, and concatenate them as sequence \(H\)  
3. Get the pose matrix \(p^{pri}\) and the active matrix \(a^{pri}\) of \(H\) from source capsule layer  
4. Obtain the attention \(p^{att}\) from adaptive mask attention layer  
5. Obtain the aspect mapping \(v^{t}\) using Equation (16).  
6. Get the pose matrix \(p^{pri}_{m}\) and the active matrix \(a^{pri}_{m}\) of each image \(m\) in \(M\), and obtain its attention matrix \(\beta_{m}\) using \(p^{pri}_{m}\) and \(v^{a}\) from Text2 vision module according to Equations (16)-(20)  
7. Compute the most relevant image index \(J\) from all \(\beta_{m}\) using Equations (21)  
8. Compute the weighted image pose matrix \(p^{img}_{J}\) using \(\beta_{J}\) and \(p^{pri}_{J}\)  
9. Repeat  
10. Compute the Text2 vision attention \(\beta_{J}\) using \(p^{img}_{J}\) and \(v^{a}\)  
11. Obtain the global image vector \(v^{img}\) using Equation (22)  
12. Compute the weighted matrix \(p^{img}_{J}\) using \(\beta_{J}\) and \(p^{img}_{J}\)  
13. Compute the Vision2 text attention \(p^{txt}\) using \(v^{img}\) and \(p^{att}\) according to Equations (23)-(26)  
14. Obtain new \(p^{img}_{J}\) and \(a^{img}_{J}\) by image convolution layer  
15. Obtain new \(p^{txt}\) and \(a^{txt}\) by text convolution layer  
16. Update Loss using Equation (35)  
17. Until the accuracy of the validation dataset no longer increases over ten epochs or reaches default \(N\) times  
18. Obtain the class capsules \(p^{img\_class}\) and \(p^{txt\_class}\), and the class active \(a^{img\_class}\) and \(a^{txt\_class}\) from last fully connected routing layer  
19. Retrieve the predicted label set \(L_{t}\) from the aggregated distribution of \(a^{img\_class}\) and \(a^{txt\_class}\).  
**Algorithm 1** Multimodal aspect-based sentiment analysis based on a VECapsNet.