# Chinese Named Entity Recognition Model Based on Multi-Task Learning
## 3 Chinese Named Entity Recognition Model Based on Multi-Task Learning
### Loss Function Design  
In addition to the multi-task learning design that represents the knowledge of all tasks in the same parameter space, it is equally important to consider how to design the interaction pattern between subtasks. As such, the loss function design takes into account the feature interactions between subtasks, and the auxiliary information of the boundary labeling task is incorporated into the entity type labeling learning. Given the significant number of non-entity words in the sequence (for example, in the CoNLL-2003 dataset, 'O' labels account for 83% of the total number of labels), and the fact that the type labeling task does not require feature learning of non-entity words, the network can greatly reduce computational costs by eliminating non-entity words in the boundary labeling task after learning that they are not entities. This allows the type labeling task to focus on feature learning of only a small number of entity words. The loss function optimization is performed according to the above idea. The loss function is designed as follows:  
\[[y_{1},y_{2},\cdot\cdot\cdot,y_{n}]=Softmax(Linear(h_{1},h_{2},\cdot\cdot\cdot,h_ {n})) \tag{5}\]  
\[M=\left( \begin{bmatrix}y_{1}\\ y_{2}\\ \cdots\\ y_{n}\end{bmatrix} != \begin{bmatrix}\text{'}O\text{'}\\ \text{'}O\text{'}\\ \ldots\\ \text{'}O\text{'}\end{bmatrix} \right) \tag{6}\]  
\[L_{task1}=\sum_{j=1}^{n}-\text{log}(p_{j}\big{(}y_{j})^{T}\big{)} \tag{7}\]  
\[L_{task2}=\sum_{j=1}^{n}-\text{log}(p_{j}\big{(}y_{j})^{T}\big{)}*M_{j} \tag{8}\]  
In Equation (5), \(y_{n}\) denotes the boundary annotation result obtained using _Softmax_ as the decoding method, where '\(O\)' represents the non-entity annotation type. Equation (6) represents the mask matrix M generated based on the decoding result of boundary annotation. In Equations (7) and (8), \(L_{task1}\) and \(L_{task2}\) represent the cross-entropy loss functions of entity boundary annotation and type annotation, respectively. The predicted probability and label of the network for each word are represented by \(p_{j}\) and \(y_{j}\), respectively, and \(n\) is the sentence length. It is observed that the mask matrix \(M\) eliminates the loss values of task 2, thereby preventing redundant learning of the entity-type labeling task on non-entity words. Regarding the calculation of the overall loss value of the network, the simplest way is to directly add up the loss values of two tasks. However, the unreasonableness of this calculation is obvious. Since different tasks have different learning difficulties, the direct summation is likely to lead to the learning of multiple tasks being dominated by one task, which will affect the final learning effect. In this paper, we propose a method to express the loss value of the total task as a weighted sum of the loss values of the two subtasks. Specifically, we set fixed weight parameters \(\alpha\) and \(\beta\) for the loss value of each task, as shown in Equation (9). This approach enables manual adjustment of the importance ratio of the two tasks. \[L=\alpha L_{task1}+\beta L_{task2} \tag{9}\]  
However, this fixed hyperparameter will accompany the whole training cycle, which may bring disadvantages to multi-task learning. Inspired by previous work, this paper also incorporates dynamic weighted average [23] and uncertainty weighting [15]. Two dynamic weighting approaches are added to the experimental comparison as a way to explore the best solution to the multi-task learning balance problem. The main idea of the dynamic weighted average (DWA) approach is to expect each task to converge at a similar learning rate, and when the loss function converges faster, it indicates that the task is easy to learn and the corresponding weight will be reduced, and vice versa the weight will be increased.