# Mixup Based Cross-Consistency Training for Named Entity Recognition
## 4 Experiments
### Experimental Setup  
In the experiments conducted in this study, the weight of BERT varies depending on the dataset. Biomedical NER used the weights of BioBERT [39] pretrained on biomedical corpus, while tech term NER used \"bert-base-cased\" weights. When using only strongly labeled data, we trained the models for one epoch with a batch size of 128, while we used a batch size of 32 for one epoch for the models using both strongly labeled data and weakly labeled data (16 for strong and 16 for weak). For a fair comparison, we evaluate Dev and Test set using batch size 16 for all experiments. We used a maximum sentence length of 256 tokens in all experiments. The learning rate is fixed on \(5\times 10^{-5}\) in the pretraining stage. In contrast, for iterative Stage 3, we conducted pseudo-labeling after performing joint training 5 times (\(\phi=5\)). And during Stage 3, we used learning rate schedule with the warm-up in 10% of steps, peaked at \(5\times 10^{-5}\), and cool-down in 90% of steps. The Adam optimizer was used for optimization. Following the studies in [18,40], we mixed hidden representations at the 7-th, 9-th, and 12-th layers, which have the most representation power of the language model, i.e., \(K=\{7,9,12\}\).  
We compared the proposed framework with the following options to evaluate the proposed framework:  
* BERT baseline: A supervised learning baseline. We construct the representative baseline model for NER. The BERT baseline is structured with the multi-layer pretrained model BERT and some fully connected layers. That is, we stacked the main classifier on top of the language model.
* Mixup baseline: A baseline to show the effect of mixup on the NER task. The mixup baseline has the same structure as the BERT baseline. However, it is trained with examples generated by mixup as well as strongly labeled data.
* Mixup-CCT: The method proposed in [1]. CCT means cross-consistency training. The architecture is the same as the one proposed in this paper. There is no pseudo-labeling; instead, it gradually increases the affection of the loss \(\mathcal{L}_{w}\) and \(\mathcal{L}_{u}\).
* Proposed: It performs consistency regularization from mixed examples generated from the pseudo-label and weak label, while mixup-CCT is mixed by strongly labeled and weakly labeled data.  
For every method except mixup-CCT, we used the focal loss for supervision on strongly labeled data. The results of mixup-CCT were derived from [1].  
We used span-level Precision, Recall, and F1-score as evaluation metrics, which are the most significant metrics for classification tasks. First of all, we obtained results with respect to True Positive (TP), True Negative (TN), False Positive (FP), and False Negative (FN) for each span in sentences. Then, using their notions, we calculated Precision, Recall, and F1-score as follows.  
\[\begin{split}\text{Precision}&=\frac{\text{TP}}{ \text{TP}+\text{FP}}\\ \text{Recall}&=\frac{\text{TP}}{\text{TP}+\text{FN} }\\ \text{F1-score}&=2\cdot\frac{\text{Precision}\cdot \text{Recall}}{\text{Precision}+\text{Recall}}\end{split} \tag{11}\]  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**Dataset**} & \multicolumn{4}{c}{**The Number of Examples**} \\  & **Train** & **Dev** & **Test** & **Weak** \\ \hline BC5 CDR-chem & 4560 & 4581 & 4797 & 107,827 1 \\ BC5 CDR-disease & 4560 & 4581 & 4797 & 67,182 1 \\ NCBI-disease & 5424 & 923 & 940 & 67,182 1 \\ LaptopReview & 2436 & 609 & 800 & 28,063 2 \\ \hline \hline \end{tabular} 1
[FOOTNOTE:1]Footnote 1: PubMed corpus + the combination of MeSH and CTD dictionary. 2  
\end{table}
Table 1: The statistical results on experimental datasets.