# Integration of Multi-Branch GCNs Enhancing Aspect Sentiment Triplet Extraction
## 3 Framework of MBGCN
### Embedding via Structure-Biased BERT  
As aforementioned, BERT has an impressive performance in modeling contextual representation in various NLP tasks [46; 47; 48; 49]. Therefore, in our proposed Multi-branches Graph Convolutional Network (MBGCN),we also utilize it to generate the semantic features by the version of the bert-uncased-base. To be precise, before feeding the review **X** into the MBGCN, the input is always formulated in three formats: segment embedding \(X_{s}\), position embedding \(X_{p}\), and tokens embedding \(X_{t}\). Then, these three aspects of embedding are summarized as the input to the selective feature generator, which is shown in Equation (1),  
\[\textbf{E}=X_{s}+X_{p}+X_{t}, \tag{1}\]  
where \(\textbf{E}=[E_{1},E_{2},E_{3},\ldots,E_{n}]\) is the input of self-attention (Equation (2)). In detail, BERT is a PLM with the structure of a stacked transformer, which has 12 transformer layers in total. Furthermore, in each transformer layer, the feature representations are trans \begin{table}
\begin{tabular}{c c c} \hline \hline
**Items** & **Relation** & **Definition** \\ \hline
1 & B-A & beginning of aspect term. \\
2 & I-A & inside of aspect term. \\
3 & A & aspect term. \\
4 & B-O & beginning of opinion term. \\
5 & I-O & inside of opinion term. \\
6 & O & opinion term. \\
7 & POS & sentiment polarity is positive. \\
8 & NEU & sentiment polarity is neutral. \\
9 & NEG & sentiment polarity is negative. \\
10 & N & belong to no aforementioned relations. \\ \hline \hline \end{tabular}
\end{table} 
Table 1: The definitions of our defined relations. Figure 2: The overall framework of the proposed Multi-branched Graph Convolutional Network (MBGCN). [\(E_{1}\), \(E_{2}\), \(E_{3}\),..., \(E_{n}\)] is the input vector **E** of self-attention (Equations (1) and (2)). formed by multi-head self-attention with a residual structure (Figure 3a). Furthermore, this transformation can be formulated as follows:  
\[h^{0}=LN(\mathbf{E}), \tag{2}\]  
\[\hat{h}^{l}=LN(h^{l-1}+MHSA(h^{l-1})), \tag{3}\]  
\[h^{l}=LN(\hat{h}^{l}+FFN(\hat{h}^{l})), \tag{4}\]  
where \(l\in[1,12]\) is the \(l\)-th layer transformer, and \(h^{0}\) means the input embedding of BERT, which is built from \(\mathbf{E}\) with a liner function. The outputs of 12-layer transformers are denoted as \([h^{1},h^{2},\ldots,h^{12}]\). \(FFN\) includes two linear functions with a \(ReLU\) activation function between. \(MHSA\) is the core of the transformer, which has a stacked structure with 12 heads of self-attention. Thus, we can formulate the architecture of attention as follows:  
\[\hat{h}^{l}_{Mj}=softmax(e^{l}_{j})(h^{l-1}\mathbf{W}_{V}), \tag{5}\]  
\[e^{l}_{j}=\frac{h^{l-1}\mathbf{W}_{Q}(h^{l-1}\mathbf{W}_{K})^{\top}}{\sqrt{d} }), \tag{6}\]  
\[\hat{h}^{l}_{M}=\sum_{j=1}^{N}\hat{h}^{l}_{Mj}, \tag{7}\]  
where parameters \(\mathbf{W}_{Q}\), \(\mathbf{W}_{K}\), and \(\mathbf{W}_{V}\) are the learnable weights for query \(\mathbf{Q}\), key \(\mathbf{K}\), and value \(\mathbf{V}\), and \(d\) is the head dimensionality. \(\hat{h}^{l}_{Mj}\) is the single attention, and \(\hat{h}^{l}_{M}\) denotes the sum of \(N\) heads attention (MHSA). Inspired by the structure-biased BERT utilized in [24; 50], we also introduce it into our MBGCN for generating more informative feature maps. In the optimized approach, self-attention is re-constructed by inserting the relative distance or the dependency between words. Furthermore, the effectiveness of this modification has been obviously proven by  
Figure 3: The mechanism of structure bias utilized in BERT. Furthermore, \(\mathbf{Q}\), \(\mathbf{K}\), and \(\mathbf{V}\) denote the query vector, key vector, and value vector, respectively, which are standardized inputs for the transformer module. \(\mathbf{R}\) indicates the relation distance embedding (Equation (8)). the NLP task [51]. Thus, we describe this change in our model as Equation (8), which can be implemented in Equation (6) directly.