# Mixup Based Cross-Consistency Training for Named Entity Recognition
## 3 Method
### Stage 3: Pseudo-Labeling and Joint Training
#### 3.4.2 Joint Training  
In this process, strongly labeled data and weakly labeled data are used together for training. For the strongly labeled data, we calculate the loss function as shown in Equation (2). We calculated losses \(\mathcal{L}_{u}\) and \(\mathcal{L}_{w}\), which contribute to the training of language model \(F\) and auxiliary classifiers \(G_{a}\), from the weakly labeled data. Firstly, we sample the data from \(D^{\prime}\), which is twice the batch size. Then, the labels of the samples were sampled by pseudo-labels for one half and weak labels for the other half, which are represented as \((\mathbf{x}^{\prime},\mathbf{y}^{p})\) and \((\mathbf{x}^{\prime\prime},\mathbf{y}^{w})\), respectively. Then, \(F\) receives sampled sentences and generates hidden representations. The hidden representations for sentence \(\mathbf{x}\), \(\mathbf{x}^{\prime}\), and \(\mathbf{x}^{\prime\prime}\) from the \(l\)-th layer of \(F\), which has \(L\) layers, are defined as follows:  
\[ \begin{split}\mathbf{h}_{l}=& F_{l}(\mathbf{h}_{l-1}; \theta),\qquad l\in[1,k]\\ \mathbf{h}_{l}^{\prime}=& F_{l}(\mathbf{h}_{l-1}^{ \prime};\theta),\qquad l\in[1,k]\\ \end{split}  \tag{3}\]where \(F_{l}\) refers to a \(l\)-th layer of \(F\) having \(\theta\) as parameters. \(\mathbf{h}_{0}\) is the word embedding for the tokens of sentence \(\mathbf{x}\), and \(\mathbf{h}_{l}=\{h_{1},h_{2},\,\ldots,h_{n}\}\) is the \(l\)-th hidden representation of the sentence \(\mathbf{x}\). Then, the intermediate representations continue forward passing to the upper layers in three different ways. The first is \(\mathbf{h}_{L}\) to calculate the loss on strongly labeled data. \(\mathbf{h}_{L}\) is the result of passing through all remaining \(F\) layers without mixing up even after the \(k\)-th layer. \(\mathbf{h}_{L}\) is defined as follows. \[\tilde{\mathbf{h}}_{l}= F_{l}(\tilde{\mathbf{h}}_{l-1};\theta),\qquad l\in[k+1,L] \tag{4}\] \[\mathbf{h}_{L}= F_{L}(\tilde{\mathbf{h}}_{L-1};\theta)\]  
The second is \(\mathbf{h}_{L}^{mix}\), which is linearly interpolated with \(\mathbf{h}_{k}^{\prime}\) and \(\mathbf{h}_{k}^{\prime\prime}\) via the mixup. \(\mathbf{h}_{k}^{\prime}\) and \(\mathbf{h}_{k}^{\prime\prime}\) refer to \(k\)-th representations of the language model from Equation (3). and then continues forward to the remaining layers of \(F\):  
\[\tilde{\mathbf{h}}_{k}^{mix}= \lambda\mathbf{h}_{k}^{\prime}+(1-\lambda)\mathbf{h}_{k}^{\prime \prime} \tag{5}\] \[\tilde{\mathbf{h}}_{l}^{mix}= F_{l}(\tilde{\mathbf{h}}_{l-1}^{mix};\theta),\qquad l\in[k+1,L]\] \[\mathbf{h}_{L}^{mix}= F_{L}(\tilde{\mathbf{h}}_{L-1}^{mix};\theta)\]  
where \(\lambda\) is the same as Equation (1). In this study, \(\lambda\) follows \(\text{Beta}(0.75,0.75)\). The last one is \(\mathbf{h}_{L}^{\prime}\) from \(\mathbf{h}_{l}^{\prime}\), which is a result of passing through the remaining layers of \(F\) without mixup. \[\tilde{\mathbf{h}}_{l}^{\prime}= F_{l}(\tilde{\mathbf{h}}_{l-1}^{\prime};\theta),\qquad l\in[k+1,L] \tag{6}\] \[\mathbf{h}_{L}^{\prime}= F_{L}(\tilde{\mathbf{h}}_{L-1}^{\prime};\theta)\]  
In training, models \(F\) and \(G\) are supervised by strongly labeled data in Stage 2. The supervised loss for the strongly labeled data is defined as \(\mathcal{L}_{s}\). \[\mathbf{p}=G(\mathbf{h}_{L})=G\cdot F(\mathbf{x}) \tag{7}\] \[\mathcal{L}_{s}=Focal(\mathbf{y},\mathbf{p})\]  
For weakly labeled data, \(F\) and \(G_{a}\) are supervised. We use the same loss function as mentioned above. Note that we use mixed labels that are linearly interpolated with pseudo-labels and the weak label. The loss, \(\mathcal{L}_{w}\), for the auxiliary classifier is defined as follows:  
\[\mathbf{p}^{mix(k)}=G_{a}^{(k)}(\mathbf{h}_{L}^{mix(k)}) \tag{8}\] \[\mathbf{h}_{L}= F_{L}(\tilde{\mathbf{h}}_{L-1};\theta)\]0  
where upper subscript \((k)\) denotes the language model that mixes up examples in the \(k\)-th layer. \(\tilde{\mathbf{y}}=\lambda\mathbf{y}^{p}+(1-\lambda)\mathbf{y}^{w}\) refers to the mixed label.