# AgriMFLN: Mixing Features LSTM Networks for Sentiment Analysis of Agricultural Product Reviews
## 3 Materials and Methods
### Research Methods
#### 3.3.2 Transformer  
Transformer is a machine learning model based on the attention mechanism, which was proposed in 2017. The Transformer model discards the sequential structure of traditional RNN and uses a mechanism based on self-attention, which has better parallel computing capability.  
The encoder and decoder, respectively, depicted in the left and right halves of Figure 2, are implemented by the Transformer employing stacked self-attention and pointwise, fully connected layers. In this experiment, we only use the encoder module of the Transformer model.  
First, the input text sequence is embedded to learn the feature of the text, similarly to other sequence transduction models. Since the original Transformer model is unable to capture sequential information, sine and cosine functions are used to add positional information, which is shown in Formulas (7) and (8). This method can not only obtain the  
Figure 2: Structure of the Transformer.  
absolute position information of words but also the relative position information, which can help the text to better obtain the order of words.  
\[PE_{(pos,2i+1)}=\cos\Bigl{(}pos/10,000^{2i/d_{model}}\Bigr{)} \tag{7}\]  
\[PE_{(pos,2i+1)}=\cos\Bigl{(}pos/10,000^{2i/d_{model}}\Bigr{)} \tag{8}\]  
The term _pos_ refers to the position of the current character in the sentence and \(i\) refers to the dimension.  
The core of the Transformer model is the attention mechanism, which can be described as mapping a Query and a set of Key-Value pairs to an output, where the Query, Keys, Values, and output are all vectors. The output is calculated as a weighted sum of the values, with each Value's weight determined by the Query's compatibility function with its corresponding Key, according to Formulas (9)-(12).  
\[Q=XW_{Q} \tag{9}\]  
\[K=XW_{K} \tag{10}\]  
\[V=XW_{V} \tag{11}\]  
\[X_{attention}=selfAttention(Q,K,V) \tag{12}\]  
Scaled dot-product attention is implemented to calculate the self-attention score, according to Formula (13).  
\[Attention(Q,K,V)=softmax(\frac{QK^{T}}{\sqrt{d_{k}}})V \tag{13}\]  
Then, the multi-head attention mechanism connects multiple self-attention mechanisms to enable the model to learn related information in different subspaces, according to Formula (14). Through different \(Q\), \(K\), and \(V\) matrices, the model's ability to focus on different positions is increased by the multiple attention mechanism, which also provides the self-attention layer with several \"representation subspaces\", improving the ability to learn different meanings of words. Compressing multiple \(Q\), \(K\), and \(V\) matrices into a weight matrix can obtain the output of the multi-head attention mechanism, that is, the input of the feedforward network.  
\[MultiHead(Q,K,V)=Concat(head_{1},\dots,head_{h})W^{O} \tag{14}\]  
where \(head_{i}=Attention(QW_{i}^{Q},KW_{i}^{K},VW_{i}^{V})\).  
The encoder and decoder block also contain a fully connected feed-forward network, according to Formula (15).  
\[FFN(x)=\max(0,xW_{1}+b_{1})W_{2}+b_{2} \tag{15}\]  
In Transformer, data passing through the attention layer and _FFN_ layer will be processed by an _Add & Norm_. The _Add & Norm_ process refers to a step where a residual connection is added to the output of a sub-layer, followed by layer normalization. This process helps in preserving information from the original input while mitigating the vanishing gradient problem and improving training efficiency.