# Enhancement of Question Answering System Accuracy via Transfer Learning and BERT
## 3 Related Technologies
### Bert  
Using the pre-trained word vector for representing the words is better than using the randomly initialized vector, but the vector representation of the pre-trained word vector is the same in various contexts, which is a context-independent word vector. For example, \"Who founded Apple?\" and \"How much does an apple cost?\" The \"apple\" in the two sentences represents different meanings but uses the same vector, so the pre-trained word vector cannot solve the case of polysemy. Therefore, pre-trained language models such as ELMo (Embeddings from Language Models) [37], GPT (Generative pre-training) [38], and BERT [39], build text sequences to obtain context-relevant Representations. The structures of these three models are shown in Figure 4. Elmo uses double-layer LSTM connected by residuals and encodes text sequences with forward and reverse directions. Its pre-training objective function is given as follows:  
\[\sum_{n=1}^{N}\log P(x_{n}\mid x_{1},\dots,x_{n-1})+\log P(x_{n}\mid x_{n+1}, \dots,x_{N}) \tag{7}\]  
The objective is to maximize the sum of probabilities that the context predicts the current word. After pre-training, word vectors encoded by ELMo can be input into the downstream model and applied to specific tasks. It can be seen from the objective function, however, that ELMo is a one-way language model essentially. In 2018, Google proposed BERT, which can better model bidirectional language models.  
As shown in BERT's model structure diagram, BERT mainly comprises the input, coding, and mask layers. Among them, the input layer splits the input text sequence into three vector representations: token vector, position vector, and clause vector, which are added together and input into the coding layer. The coding layer comprises encoders of  
Figure 4: Illustration of pre-trained language models (Redrawn from [37; 38; 39]).  
Figure 3: Illustration of a LSTM cell (Redrawn from [36]).  
a bi-directional connected Transformer [40]. The basic BERT model has 12 layers, which is the core structure of BERT. The transformer is a model structure suggested by Google in 2017. Figure 5 presents the structure of its encoder part, and it consists of a multi-attentional network and a feedforward network. The structure of task layer depends primarily on the task target. The SoftMax layer handles classification tasks, and the CRF layer follows sequence annotation. Since the input and coding layers are task-independent, data fine-tuning BERT is often used to achieve good results in downstream tasks.  
BERT has two pre-training tasks: Mask Language Model (MLM) and Next Sentence Continuity Prediction (NSP). MLM replaces certain words in the text with [MASK] tags with certain rules and then predicts these words in the output layer. NSP is to predict whether two sentences are connected. It is often used in sentence pair similarity tasks. BERT has significantly improved 11 NLP tasks, which have become the focus of academic attention. After that, some BERT improvements have emerged continuously, such as XLNet [41], ALBERT [42], ELECTRA [43], ERNIE [44], etc. The work in this article is also on the basis of the BERT pre-training language model and performed well on the question and answering task.