# Visual Enhancement Capsule Network for Aspect-Based Multimodal Sentiment Analysis
## 3 Methodology
### Visual Enhancement Capsule Network Based on Multimodal Fusion
#### 3.2.1 MemCapsNet
\[i_{t}=\sigma(W_{i}[h_{t-1},e_{t}]+b_{i}) \tag{1}\]  
\[o_{t}=\sigma(W_{o}[h_{t-1},e_{t}]+b_{o}) \tag{2}\]  
\[f_{t}=\sigma(W_{f}[h_{t-1},e_{t}]+b_{f}) \tag{3}\]  
\[\widetilde{c_{t}}=tanh(W_{c}[h_{t-1},e_{t}]+b_{c}) \tag{4}\]  
\[c_{t}=f_{t}\odot c_{t-1}+i_{t}\odot\widetilde{c_{t}} \tag{5}\]  
\[h_{t}=o_{t}\odot tanh(c_{t}) \tag{6}\]  
where \(\sigma\) is the sigmoid function, \(\theta_{bi}=\{W_{i},W_{o},W_{f},W_{c},b_{i},b_{o},b_{f},b_{c}\}\) is the set of training parameters of the Bi-LSTM; \(i\), \(o\), and \(f\) are the input gate, output gate, and forget gate, respectively; \(h\) and \(c\) are the hidden vector and the cell vector, respectively; and \(\odot\) expresses the dot product operation of two vectors. A Bi-LSTM model runs a forward and backward LSTM on a sequence starting from the left and right ends, respectively. The hidden states of \(v_{t}\) generated by these two LSTMs are concatenated to represent the \(t\)th word vector and its context. Hence, the vector representation of the \(t\)th word can be expressed as follows:  
\[\overrightarrow{h_{t}}=\overrightarrow{LSTM}(h_{t-1},v_{t},\theta), \overleftarrow{h_{t}}=\overleftarrow{LSTM}(h_{t-1},v_{t},\theta) \tag{7}\]  
\[h_{t}=\overrightarrow{h_{t}}\|\overleftarrow{h_{t}},t\in\{1,2,...,n\} \tag{8}\]  
where \(\overrightarrow{h_{t}},\overleftarrow{h_{t}}\in\mathbb{R}^{k},h_{t}\in \mathbb{R}^{2k}\), \(k\) is the number of hidden units in LSTM, and \(\|\) represents the concatenation of two vectors. In the model, two different Bi-LSTMs are used on the document and the aspect phrase sequence, respectively, to prevent them from mixing their features through the hidden states. Thus, the information of the document and the aspect phrases can be sufficiently distinguished in the upper layers of the network, and it is more conducive to extracting semantic information correlated to affective entities from the document. So from the encoding layer, we can obtain the hidden states (\(h_{t}^{e},i\in\{1,2,...,n_{1}\}\)) of the document \(E^{c}=\{e_{1}^{e},e_{2}^{e},\ldots,e_{n_{1}}^{e}\}\) and the hidden states (\(h_{j}^{a},j\in\{1,2,...,n_{2}\}\)) of the aspect phrases \(E^{a}=\{e_{1}^{a},e_{2}^{a},\ldots,e_{n_{2}}^{a}\}\), respectively generated by two Bi-LSTMs according to the above process, and then concatenate their text representation vectors as \(H=\{h_{1}^{e},h_{2}^{e},\ldots,h_{n_{1}}^{e},h_{1}^{a},h_{2}^{a},\ldots,h_{n_{2 }}^{a}\}\). **N-gram source capsule layer. ** In this layer, we extract N-gram source capsule features of the hidden vector \(H\) as the input for the capsule network by one-dimensional convolution operation using different sizes of convolution kernels, where each source capsule contains a \(4\times 4\) pose matrix and an activation value. Set the set of one-dimensional convolution kernel sizes \(k=\{k_{i}|k_{i}\in\mathbf{N}\}\), where \(\mathbf{N}\) is a natural number set. In the model, we use \(k=\{3,5,7\}\) to extract the text local features of 3-gram, 5-gram, and 7-gram, respectively. Suppose \(H_{si}(1\leq s\leq t\leq n)\) express the column vectors in \(H\) from the \(s\)th to \(t\)th column, the one-dimensional convolution operation on the \(j\)th vector is  
\[c_{j}^{k_{i}}=ReLu(W^{k_{i}}\cdot H_{-\frac{k_{i}-1}{2}+j}\frac{k_{i}-1}{2+j}+ b^{k_{i}}) \tag{9}\]  
where \(W^{k_{i}}\in\mathbb{R}^{k\times n}(n=n_{1}+n_{2})\) is the weight coefficient of the convolution kernel with a size of \(k_{i}\), \(b^{k_{i}}\in\mathbb{R}\) is its corresponding bias, and Relu is a rectified linear unit. The \(k_{i}\)-gram output feature map of \(H\) is \(c^{k_{i}}=[c_{0}^{k_{i}},c_{1}^{k_{i}},\ldots,c_{n_{1}}^{k_{i}}]\). The implicit vector needs to be pad on its two sides before the one-dimensional convolution operation so that the length of the generated vector \(c^{k_{i}}\) is equal to that of the implicit vector \(H\). The pose matrix sequence \(P^{pri}\) and the active value vector \(a^{pri}\) of source capsules can be computed after multiple repeated one-dimensional convolution operations through Equations (10) and (11).