## 2 Literacy Review
### Bidirectional Encoder Representations from Transformers (BERT)  
BERT (Bidirectional Encoder Representations from Transformers), the deep learning model with Nature Language Processing proposed by Google in 2018, proved the importance of bidirectional pre-training in language models and largely reduced the training time for model use. BERT solves the one-way training of current language training models, e.g., ELMo and OpenAI GPT. BERT is also applied to different tasks, such as downstream tasks of sentence pairs in paraphrasing, hypothesis-premise pairs in entailment, SQuAD (Question-Passage Pairs in Question Answering), and a degenerate text-\(\mathcal{O}\) pair in text classification or sequence tagging [34]. BERT stresses pre-training and fine-tuning; during pre-training, different pre-training tasks would precede model training of unlabeled data [34]. The training data of BERT are BooksCorpus (800 M words) and English Wikipedia (2500 M words). First, the input representation of text is preceded by Wordpiece Embedding to mark the first label of each sequence as [CLS], and the special symbol [SEP] is then used for separating two sentences and transforming them into token embedding, segment embedding, and position embedding for the successive pre-training task. Two unsupervised tasks, Masked LM (MLM) and Next Sentence Prediction (NSP), are preceded by pre-training in Google. In comparison with Word2 Vec and ELMo, BERT, with its deep bidirectional model, strengthens the function. In order to train the deep bidirection, BERT inputs Tokens with random cover for the model, predicting such covered Tokens; such a process is termed Masked LM [34]. Moreover, BERT precedes Next Sentence Prediction through the corpus, allowing the model to comprehend sentence relationships in order to judge two sentences [35]. BERT could extract features through Transformer, which is composed of an encoder and decoder. The encoder enters the decoder through a multi-head attention mechanism and a fully connected feed-forward network. With the multi-head attention mechanism, fully connected feed-forward network, and pure fully connected layer (Linear), the Softmax function is eventually transformed into a probability value for the output. BERT, with MLM, NSP, and the encoder mechanism of Transformer, first transforms the token embedding of input text into three matrices of Q (Query), K (Key), and V (Value), precedes the dot product and computation of matrices through the multi-head attention mechanism, and finally, inputs the token embedding. Each Q, K, and V is a set of Heads, and the number of Heads could be set in the multi-head attention mechanism so that the model could simultaneously consider the vector of each word during training, and precede training in parallel, to better understand the tightness between words. After completing the pre-training process, the model that could be applied to multiple Nature Language Processing tasks is first trained using the Encoder of Transformer, a large number of texts, and two pre-training tasks, employing the idea of transfer learning and then fine-tuning several downstream tasks. The self-attention mechanism in Transformer allows BERT to precede different downstream tasks, either single text or paired texts [34]. BERT, through the pre-training model, allows the model to better understand vocabulary and keywords and can more accurately correspond sentences to integrate the semantic information of the entire article. In terms of the application, BERT is currently applied to text classification [36], text abstracting [37], sequence labeling [38], Q&A [39], task searching [40], and dialogue robots [41]. The application of BERT is broadened through the pre-training process of Transformer. The latest applications of BERT include text sentiment analysis [42], the protection of private health information [43], recommender systems [44], and other fields. [PERSON] et al.