# Enhancement of Question Answering System Accuracy via Transfer Learning and BERT
## 4 Method
### Model For Entity Linking
#### 4.2.1 Named Entity Recognition
The formula is as follows:  
\[\text{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V})=\text{softmax}\Bigg{(} \frac{\mathbf{Q}\mathbf{K}^{T}}{\sqrt{\mathbf{d}}}\Bigg{)}\mathbf{V} \tag{10}\]  
where \(Q\in R^{N\times 2 dh}\), \(K\in R^{N\times 2 dh}\)and \(V\in R^{N\times 2 dh}\) are query matrix, keys matrix and value matrix, respectively. \(d\) is the dimension of BiLSTM hidden layers. More information about self-attention mechanisms are detailed in [40]. The next layer is the CRF [47]. The CRF layer can add some constraints to the final predicted labels to ensure the reasonableness of the predicted labels. Because the labels are different, we create a unique CRF layer for each task separately. For each sentence of task m, the final representation is to concatenate the BiLSTM layer with the self-attention representation.The final decoding process uses the Viterbi algorithm. As shown in the Figure 7, the middle layer enters our task recognizer. We propose a task recognizer to determine from which task the sentences come. The following formula can signify the task recognizer:  
\[\mathbf{s}^{m}=\text{ Maxpooling}\left(\mathbf{S}^{\prime}m\right) \tag{11}\]  
\[D\big{(}\mathbf{s}^{\prime m};\theta_{d}\big{)}=\text{softmax}\big{(}\mathbf{ W}_{d}\mathbf{s}^{\prime}+\mathbf{b}_{d}\big{)} \tag{12}\]  
where \(\theta_{d}\) denotes the parameters of task recognizer. \(W_{d}\in R^{K\times 2 dh}\) and \(b_{d}\in R^{K}\) are trainable parameters. \(m\) denotes the number of tasks. The middle layer is the shared features of CWS and NER tasks, to prevent task-specific features from being incorporated into the shared features. Inspired by adversarial networks, we introduce adversarial loss to train the shared BiLSTM, resulting in a task recognizer that cannot reliably identify the task from which the sentence comes. The following equation calculates the adversarial loss:  
\[L_{Adv}=\min_{\theta_{s}}\Bigg{(}\max_{\theta_{d}}\sum_{m=1}^{m}\sum_{i=1}^{ T_{m}}\log D\Big{(}E_{s}\Big{(}\mathbf{x}_{m}^{(i)}\Big{)}\Big{)}\Bigg{)} \tag{13}\]  
where \(\theta_{s}\) refers to the shared BiLSTM's trainable parameters. \(E_{s}\) is the shared feature extractor. \(T_{m}\) is the number of training examples of task \(m\) and \(x_{m}^{(i)}\) is the \(i\)-th example. The shared BiLSTM develops a representation to deceive the task recognizer as part of a minimax optimization, while the task recognizer tries its best to accurately recognize the type of work. To tackle the minimal-extreme optimization problem, we add a gradient inversion layer below the Softmax layer. We reduce the task recognizer's error during the training stage, and invert the gradient through the gradient inversion layer, which favors the shared feature extractor for studying the word boundary information shared by the task. The shared feature extractor and the task recognizer arrive at a point when the recognizer is unable to discriminate tasks based on the representations acquired from the shared feature extractor after training phrases. In training, the following is the final loss function of our model:  
\[L=L_{NER}\cdot I(\mathbf{x})+L_{\text{CWS}}\cdot(1-I(\mathbf{x}))+\lambda L_{Adv} \tag{14}\]  
where \(\lambda\) denotes a hyper-parameter. Equation (21) can be used to calculate \(L_{NER}\) and \(L_{\text{CWS}}\). \(I(x)\) is a switching function determining which task the input is from. The formula is as follows:  
\[I(\mathbf{x})=\left\{ \begin{array}{ll}1,&\text{if}\quad\mathbf{x}\in \mathcal{D}_{NER}\\ 0,&\text{if}\quad\mathbf{x}\in\mathcal{D}_{\text{CWS}}\end{array} \right. \tag{15}\]  
where \(D_{NER}\) is Chinese \(NER\) training corpora and \(D_{\text{CWS}}\) is \(CWS\) training corpora. In each iteration of the training phase, we sequentially select a task from \(\{NER,CWS\}\) and obtain different training samples to update the parameters. We optimize the final loss function by utilizing the Adam algorithm.