# An Effective ELECTRA-Based Pipeline for Sentiment Analysis of Tourist Attraction Reviews
## 3 Methodology
### ELECTRA System Architecture  
While BERT and its variants produce excellent results on downstream NLP tasks, they require a large amount of computation to be effective. This is because such masked language models (MLM) mask a few words randomly during the training process and predict very limited words. Furthermore, because of the arbitrary choice of the masked token, it would be challenging to learn as many meaningful tokens as possible, such as emotional words and opinion words, for the sentiment analysis task. As an alternative,ELECTRA proposes a more efficient pre-training task that can compensate well for these shortcomings, so we introduce it as a sentiment classification model for attraction review to achieve a more satisfying classification performance. The ELECTRA system architecture is shown in Figure 2.  
To go into more detail, just like Generative Adversarial Networks (GAN), the ELECTRA architecture consists of two networks: a generator and a discriminator. Both parts use transformer-based encoding networks to obtain the vector representation of the input word sequence. In the Chinese dataset, as Chinese characters are different from English, which is formed by alphabet-like symbols, the word tokenizer uses the traditional Chinese Word Segmentation (CWS) tool to split the text into several words instead of small fragments. In this way, whole word masking in Chinese could be adopted to mask the word instead of individual Chinese characters. In the generator, the goal is to train a masked language model. Its structure is similar to BERT, i.e., given an initial input sequence = {\(\mathbb{H}\) (I), \(\mathcal{T}\) (not), \(\mathcal{R}\) (like), \(\mathbb{E}\) (eat), \(\mathbb{H}\) (apple)}, the Chinese words \(\mathcal{R}\) (like) and \(\mathbb{H}\) (apple) in the masked sequence are first replaced by [MASK] according to a certain percentage to obtain the generator's input. The process can be formulated as:  
\[x^{masked}=REPLACE(x,index,[MASK]), \tag{1}\]  
where \(index=[id_{1},\dots,id_{k}]\) is the index sequence for selected positions. Then, a vector representation is obtained through a generative network, typically a small MLM. Followed by a softmax layer, a sample word \(\hat{x}_{i}\) is predicted for the location of the mask in the generator's input:  
\[\hat{x}_{i}\sim p_{G}(x_{i}\Big{|}x^{masked}) \tag{2}\]  
The objective function of training is to maximize the likelihood of the masked words. The prediction result replaces the original masked word, e.g., the Chinese words\(\mathcal{R}\) (like) and \(\mathcal{R}\) (apple) are replaced by \(\mathcal{R}\) (like) and \(\mathcal{R}\) (pear), respectively:  
\[x^{corrpt}=REPLACE(x,index,\hat{x}), \tag{3}\]  
where \(index\) is the same as above. In the discriminator, a new pre-training task known as \"Replaced Token Detection (RTD)\" is applied. More specifically, a discriminative model is trained to predict whether the word at each position of the discriminator's input sequence has been replaced by the generator, with \"Original\" or \"Replaced\" as the classification result. Here, only the word \(\mathcal{R}\) (apple) can be found to have changed. The generator's loss function \(L_{MLM}(x,\theta_{G})\) can be formulated as:  
\[x^{corrpt}=REPLACE(x,index,\hat{x}), \tag{4}\]  
Figure 2: ELECTRA architecture overview. and the discriminator's loss function \(L_{Disc}(x,\theta_{D})\) can be formulated as:  
\[L_{Disc}(x,\theta_{D})=E\left(\sum_{t=1}^{n}-1(x_{t}^{corrupt}=x_{t})\log D(x^{ corrupt},t)-1(x_{t}^{corrupt}\
eq x_{t})\log(1-D(x^{corrupt},t))\right) \tag{5}\]  
The final loss function is the weighted sum of two loss functions:  
\[\min_{\theta_{C},\theta_{D}}\sum_{x\in X}L_{MLM}(x,\theta_{C})+\lambda L_{Disc} (x,\theta_{D}), \tag{6}\]  
where \(\lambda\) is a weighting factor and X is a large raw text corpus. After pre-training, we use the discriminator to fine-tune the sentiment analysis task.