SlowGuess/NAACL_2022_OCR huggingface dataset:
Linzhi Wu $^{1*}$ , Pengjun Xie $^{*}$ , Jie Zhou $^{2*}$ , Meishan Zhang $^{3\dagger}$ , Chunping Ma, Guangwei Xu, Min Zhang $^{3}$
$^{1}$ School of New Media and Communication, Tianjin University
$^{2}$ School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University
$^{3}$ Institute of Computing and Intelligence, Harbin Institute of Technology (Shenzhen)
{tjuwlz2020, machunpingjj}@163.com, sanny02@sjtu.edu.cn
{xpjandy, mason.zms, ahxgwOnePiece}@gmail.com, zhangmin2021@hit.edu.cn
Self-augmentation has received increasing research interest recently to improve named entity recognition (NER) performance in low-resource scenarios. Token substitution and mixup are two feasible heterogeneous self-augmentation techniques for NER that can achieve effective performance with certain specialized efforts. Noticeably, self-augmentation may introduce potentially noisy augmented data. Prior research has mainly resorted to heuristic rule-based constraints to reduce the noise for specific self-augmentation methods individually. In this paper, we revisit these two typical self-augmentation methods for NER, and propose a unified meta-reweighting strategy for them to achieve a natural integration. Our method is easily extensible, imposing little effort on a specific self-augmentation method. Experiments on different Chinese and English NER benchmarks show that our token substitution and mixup method, as well as their integration, can achieve effective performance improvement. Based on the meta-reweighting mechanism, we can enhance the advantages of the self-augmentation techniques without much extra effort.
Named entity recognition (NER), which aims to extract predefined named entities from a piece of unstructured text, is a fundamental task in the natural language processing (NLP) community, and has been studied extensively for several decades (Hammerton, 2003; Huang et al., 2015; Chiu and Nichols, 2016; Ma and Hovy, 2016). Recently, supervised sequence labeling neural models have been exploited most popularly for NER, leading to state-of-the-art (SOTA) performance (Zhang and Yang, 2018; Li et al., 2020a; Ma et al., 2020).
Although great progress has been made, developing an effective NER model usually requires a
Figure 1: The main idea of our work, where the two heterogeneous self-augmentation methods (i.e., token substitution and mixup) are integrated by a unified meta reweighting framework.
large-scale and high-quality labeled training corpus, which is often difficult to be obtained in real-world scenarios due to the expensive and time-consuming annotations by human experts. Moreover, it would be extremely serious because the target language, target domain, and the desired entity type could all be infinitely varied. As a result, the low-resource setting with only a small amount of annotated corpus available is far more common in practice, even though it may result in significant performance degradation due to the overfitting problem.
Self-augmentation is a prospective solution to this problem, which has received widespread attention (Zhang et al., 2018; Wei and Zou, 2019; Dai and Adel, 2020; Chen et al., 2020; Karimi et al., 2021). The major motivation is to generate a pseudo training example set deduced from the original gold-labeled training data automatically. For NER, a token-level task, the feasible self-augmentation techniques include token substitution (Dai and Adel, 2020; Zeng et al., 2020) and mixup (Zhang et al., 2020a; Chen et al., 2020), which are deformed at the ground-level inputs and the high-level hidden representations, respectively.
Nonetheless, there are still some limitations currently for the above token substitution and mixup methods. For one thing, both of them require some specialized efforts to improve their effectiveness due to the potential noise introduced by the self-augmentation, which may restrict the valid seman
Human Language Technologies, pages 4049 - 4060
tic representation of the augmented data. For instance, token substitution is typically limited to the named entities in the training corpus (Wu et al., 2019), and the mixup tends to be imposed on the example pairs with small semantic distance gaps (Chen et al., 2020). For another thing, though the two techniques seem to be orthogonal and probably complementary to each other, it remains a potential challenge to effectively and naturally integrate them.
In this work, we revisit the token substitution and mixup methods for NER, and investigate the two heterogeneous techniques under a unified meta reweighting framework (as illustrated in Figure 1). First, we try to relax the previous constraints to a broader scope for these methods, allowing for more diverse and larger-scale pseudo training examples. However, this would inevitably produce some low-quality augmented examples (i.e., noisy pseudo data) in terms of linguistic correctness, which may negatively affect the model performance. To this end, we present a meta reweighting strategy for controlling the quality of the augmented examples and leading to noise-robust training. Also, we can naturally integrate the two methods by using the example reweighting mechanism, without any specialization in a specific self-augmentation method.
Finally, we carry out experiments on several Chinese and English NER benchmark datasets to evaluate our proposed methods. We mainly focus on the low-resource settings, which can be simulated by using only part of the standard training set when the scale is large. Experimental results show that both our token substitution and mixup method coupled with the meta-reweighting can effectively improve the performance of our baseline model, and the combination can bring consistent improvement. Positive gains become more significant as the scale of the training data decreases, indicating that our self-augmentation methods can handle the low-resource NER well. In addition, our methods can still work even with a large amount of training data. The code is available at https://github.com/LindgeW/MetaAug4NER.
In this section, we firstly describe our baseline model. Then, we present our self-augmentation methods to enhance the baseline model in the low-resource settings. Finally, we elaborate on our meta reweighting strategy, which aims to alleviate the
negative impact of the noisy augmented examples caused by the self-augmentation while also elegantly combining these augmentation methods.
NER task is typically formulated as a sequence labeling problem, which transforms entities/nonentities into token-level boundary label sequence by using the BIO or BIOES schema (Huang et al., 2015; Lample et al., 2016). In this work, we adopt BERT-BiLSTM-CRF as our basic model architecture which consists of four components: (1) input representation, (2) BiLSTM encoding, (3) CRF decoding, and (4) training objective.
Input Representation Given an input sequence  $X = (x_{1},\dots ,x_{n})$  of length  $n$ , we first convert it into sequential hidden vectors using the pre-trained BERT (Devlin et al., 2019):
$$
\boldsymbol {e} _ {1}, \dots , \boldsymbol {e} _ {n} = \operatorname {B E R T} (X), \tag {1}
$$
where each token is mapped to a contextualized representation correspondingly.
Encoding We use a bidirectional LSTM layer to further extract the contextual representations, where the process can be formalized as:
$$
\boldsymbol {h} _ {1}, \dots , \boldsymbol {h} _ {n} = \operatorname {B i L S T M} \left(\boldsymbol {e} _ {1}, \dots , \boldsymbol {e} _ {n}\right), \tag {2}
$$
where  $\pmb{h}_i$  is the hidden state output of the  $i$ -th token in the sequence  $(i \in [1, n])$ .
Decoding First, a linear transformation layer is used to calculate the initial label scores. Then, a label transition matrix  $\pmb{T}$  is used to model the label dependency. Let  $Y = (y_{1},\dots ,y_{n})$  be a label sequence, the score  $s(Y|X)$  can be computed by:
$$
\boldsymbol {o} _ {i} = \boldsymbol {W h} _ {i} + \boldsymbol {b},
$$
$$
s (Y | X) = \sum_ {i = 1} ^ {n} \left(\boldsymbol {T} _ {\mathcal {Y} _ {i - 1}, \mathcal {Y} _ {i}} + \boldsymbol {o} _ {i} [ \mathcal {Y} _ {i} ]\right), \tag {3}
$$
where  $W, b$  and  $\pmb{T}$  are the model parameters. Finally, we employ the Viterbi algorithm (Viterbi, 1967) to find the best label sequence  $Y$ .
Training Objective We exploit the sentence-level cross-entropy objective for training. Given a gold-labeled training example  $(X,Y)$ , we have the conditional probability  $p(Y|X)$  based on the
Figure 2: An overview of the self-augmentation framework for NER.
scoring function defined in Equation 3, and then apply a cross-entropy function to calculate the single example loss:
$$
p (Y | X) = \frac {\exp (s (Y | X))}{\sum_ {\widetilde {Y}} \exp (s (\widetilde {Y} | X))}, \tag {4}
$$
$$
\mathcal {L} (X, Y) = - \log p (Y | X).
$$
where  $\widetilde{Y}$  denotes the candidate label sequences.
2.2 Self-Augmentation
Self-augmentation methods can reduce the demand for abundant manually-annotated examples, which can be implemented at the input level and representation level. Token substitution and mixup are two popular methods for NER that correspond to these two distinct levels. Here, we try to extend these two self-augmentation methods.
Token Substitution Token substitution aims to generate pseudo examples based on the original gold-labeled training data by replacing the tokens of input sentence with their synonym alternatives (Wu et al., 2019; Dai and Adel, 2020). For NER, Wu et al. (2019) adopted this method to obtain performance gains on Chinese datasets where the substituted objects are limited to named entities. Dai and Adel (2020) empirically demonstrated the superiority of synonym replacement among various augmentation schemes where the synonyms are retrieved from the off-the-shelf WordNet thesaurus.
Our token substitution is performed by building a synonym dictionary, which covers the named entity synonyms as well as numerous normal word synonyms. Following Wu et al. (2019), we treat all entities of the same type from the training set
as synonyms, which are added to the entity dictionary. We name it as entity mention substitution (EMS). Meanwhile, we extend the substitution to non-entity tokens (i.e., the corresponding label is 'O'), which is named as normal word substitution (NWS). Since unlabeled data in a specific domain is easily accessible, we adopt the word2vec-based algorithm (Mikolov et al., 2013; Pennington et al., 2014) to mine tokens with similar semantics on Wikidata via distributed word representation (Yamada et al., 2020), and build a normal word synonym dictionary from the  $k$ -nearest token set based on cosine similarity distance. Note that this scheme does not require access to thesaurus for a specific domain in order to obtain synonyms.
Figure 2 presents an example of token substitution, where EMS and NWS are both involved. Specifically, for a given gold-labeled training example  $(X,Y)$ , we replace the entity token of  $X$  with a sampled entity from the entity dictionary which has the same entity type, and meanwhile replace the non-entity token of  $X$  with a sampled synonym. Then, we can obtain a pseudo example  $(\bar{X},\bar{Y})$ . Especially, we balance the EMS and NWS strategies based on a ratio  $\gamma$  by adjusting the percentage of EMS operations, aiming for a good trade-off between entity diversity and context diversity. And, we refer to this method as TS in the rest of this paper for short.
Mixup for CRF Unlike token substitution performed at the ground input, the mixup technique (Zhang et al., 2018) generates virtual examples at the feature representation level in the NLP field (Guo et al., 2019). The main idea is to perform linear interpolations on both the input and ground-truth output between randomly sampled example
pairs from the given training set. Chen et al. (2020) presented the first work based on the token classification framework for the NER task, and their mixup strategy is constrained to the examples pairs where the input sentences are semantically similar by using specific heuristic rules. Different from their method, we extend the mixup technique to the CRF decoding.
Formally, give an example pair  $(X_{1},Y_{1})$  and  $(X_{2},Y_{2})$  randomly sampled from the gold-labeled training set, we firstly obtain their vector representations through Equation 1, resulting in  $e_{1,1}\dots e_{1,n_1}$ , and  $e_{2,1}\dots e_{2,n_2}$ , respectively. Then we apply the linear interpolation to obtain a new virtual training example  $(\bar{X},\bar{Y})$ . Here we assume a regularization of pair-wise linear interpolation over the input representations and the output scores, where the following attributes should be satisfied:
$$
\bar {X}: \left\{ \begin{array}{l} \operatorname {B E R T} (\bar {X}) = \bar {e} _ {1} \dots \bar {e} _ {n} \\ \bar {e} _ {i} = \lambda e _ {1, i} + (1 - \lambda) e _ {2, i}, i \in [ 1, n ] \end{array} \right. \tag {5}
$$
$$
\bar {Y}: s (\bar {Y} | \bar {X}) = \lambda s (Y _ {1} | \bar {X}) + (1 - \lambda) s (Y _ {2} | \bar {X}),
$$
where  $n = \max (n_1,n_2)^1$  and  $\lambda$  is sampled from a  $\operatorname {Beta}(\alpha ,\alpha)$  distribution  $(\lambda \in [0,1]$  and  $\alpha >0$  According to this formulation, the loss function can be reformulated as:
$$
\begin{array}{l} \mathcal {L} (\bar {X}, \bar {Y}) = - \log \frac {\exp (s (\bar {Y} | \bar {X}))}{\sum_ {\widetilde {Y}} \exp (s (\widetilde {Y} | \bar {X}))} \tag {6} \\ = \lambda \mathcal {L} (\bar {X}, Y _ {1}) + (1 - \lambda) \mathcal {L} (\bar {X}, Y _ {2}). \\ \end{array}
$$
which aligns with the training objective of Equation 4. In this way, our mixup method can fit well with the structural decoding.
2.3 Meta Reweighting
Although the self-augmentation techniques can efficiently generate numerous pseudo training examples, how to control the quality of augmented examples is a potential challenge that cannot be overlooked. In particular, unlike sentence-level classification tasks, entity recognition is highly sensitive to the semantics of the context. While positive augmented examples can help our model advance, some low-quality augmented examples that are inevitably introduced during self-augmentation may hurt the final model performance.
In this paper, we leverage a meta reweighting mechanism to dynamically and adaptively assign
Algorithm 1 The training procedure of the meta reweighting strategy
Input: Initial model parameters  $\Theta^{(0)}$ , clean training dataset  $\mathcal{D}$ , augmented training dataset  $\hat{\mathcal{D}}$ , batch size  $m, n$ , training steps  $T$
Output: Updated model parameters  $\Theta^{(T)}$
1: for  $t = 1$  to  $T$  do
2: Initialize the trainable parameter  $\epsilon$ .
3:  $\{x_{c},y_{c}\} \gets$  SampleMiniBatch(D,m).
4:  $\{x_{a},y_{a}\} \gets$  SampleMiniBatch(D,n).
5:  $\mathcal{L}_a\gets \sum_{i = 1}^n\epsilon_i\mathcal{L}(f(x_{a,i};\Theta^{(t)}),y_{a,i}).$
6:  $\nabla \Theta^{(t)}\gets \operatorname {Grad}(\mathcal{L}_a,\Theta^{(t)})$
7:  $\hat{\Theta}^{(t)}\gets \Theta^{(t)} - \beta \nabla \Theta^{(t)}$
8:  $\mathcal{L}_c\gets \frac{1}{m}\sum_{i = 1}^m\mathcal{L}(f(x_{c_i};\hat{\Theta}^{(t)}),y_{c_i}).$
9:  $\nabla \epsilon \gets \operatorname{Grad}(\mathcal{L}_c, \epsilon)$ .
10:  $\hat{w} \gets \operatorname{Sigmoid}(-\nabla \epsilon)$ .
11:  $w\gets \frac{\hat{w}}{\sum_{j}\hat{w}_{j} + \delta}.$
12:  $\hat{\mathcal{L}}_a\gets \sum_{i = 1}^n w_i\mathcal{L}(f(x_{a,i};\Theta^{(t)}),y_{a,i}).$
13:  $\nabla \Theta^{(t)}\gets \operatorname {Grad}(\hat{\mathcal{L}}_a,\Theta^{(t)})$
14:  $\Theta^{(t + 1)}\gets$  OptimizerStep  $(\Theta^{(t)},\nabla \Theta^{(t)})$
15: end for
the example-wise weights to each mini-batch of training data, motivated by Ren et al. (2018). The key idea is that a small and clean meta-data set is applied to guide the training of model parameters, and the loss produced by the mini-batch of meta-data is exploited to reweight the augmented examples in each batch online. Intuitively, if the data distribution and gradient-descent direction of the augmented example are similar to those of the sample in the meta-data set, our model could better fit this positive augmented sample and increase its weight, and vice versa. In other words, the clean and valid augmented examples are more likely to be fully trained.
More specifically, suppose that we have a set of  $N$  augmented training examples  $\hat{\mathcal{D}} = \{(X_i,Y_i)\}_{i=1}^N$ , our final optimizing objective can be formulated as a weighted loss as follows:
$$
\Theta^ {*} (w) = \underset {\Theta} {\arg \min } \sum_ {i = 1} ^ {N} w _ {i} \mathcal {L} \left(f \left(X _ {i}; \Theta\right), Y _ {i}\right), \tag {7}
$$
where  $w_{i}\geq 0$  is the learnable weight for the loss of  $i$ -th training example.  $f(\cdot ;\Theta)$  represents the forward process of our model (with parameter  $\Theta$ ). The optimal parameter  $w$  is further determined by minimizing the following loss computed on the meta example set  $\mathcal{D} = \{(X_i^m,Y_i^m)\}_{i = 1}^M$  ( $M\ll$
<sup>1</sup>Special zero-vector pads are used to align two sequences with different lengths.
$N)$
$$
w ^ {*} = \arg \min  _ {w} \frac {1}{M} \sum_ {i = 1} ^ {M} \mathcal {L} \left(f \left(X _ {i} ^ {m}; \Theta^ {*} (w)\right), Y _ {i} ^ {m}\right), \tag {8}
$$
Accordingly, we need to calculate the optimal  $\Theta^{*}$  and  $w^{*}$  in Equation 7 and 8 based on two nested loops of optimization iteratively. For simplicity and efficiency, we take a single gradient-descent step for each training iteration to update them via an online-approximation manner. At every training step  $t$ , we sample a mini-batch augmented examples  $\{(X_{i},Y_{i})\}_{i = 1}^{n}$  initialized with the learnable weights  $\epsilon$ . After a single optimization step, we have:
$$
\hat {\Theta} ^ {(t + 1)} (\epsilon) = \Theta^ {(t)} - \beta \nabla_ {\Theta} \sum_ {i = 1} ^ {n} \epsilon_ {i} \mathcal {L} (f (X _ {i}; \Theta), Y _ {i}), \tag {9}
$$
where  $\beta$  is the inner-loop step size. Based on the updated parameters, we then calculate the loss of the sampled mini-batch meta examples  $\{(X_{j}^{meta},Y_{j}^{meta})\}_{j = 1}^{m}$
$$
\mathcal {L} ^ {\text {m e t a}} (\hat {\Theta}) = \frac {1}{m} \sum_ {j = 1} ^ {m} \mathcal {L} \left(f \left(X _ {j} ^ {\text {m e t a}}; \hat {\Theta} ^ {(t + 1)}\right), Y _ {j} ^ {\text {m e t a}}\right), \tag {10}
$$
To generalize the parameters  $\hat{\Theta}$  well to the metadata set, we take the gradients of  $\epsilon$  w.r.t the meta loss to produce example weights and normalize it along mini-batch:
$$
\hat {w} _ {i} = \sigma \left(- \nabla_ {\epsilon_ {i}} \mathcal {L} ^ {m e t a} (\hat {\Theta}) \right| _ {\epsilon_ {i} = 0}),
$$
$$
w _ {i} = \frac {\hat {w} _ {i}}{\sum_ {j} \hat {w} _ {j} + \delta}.
$$
where  $\sigma (\cdot)$  is the sigmoid function and  $\delta$  is a small value to avoid division by zero. Finally, we optimize the model parameters over augmented examples with the calculated weights.
Algorithm 1 illustrates the detailed training procedure of the meta reweighting strategy. It is noteworthy that the augmented training examples contain the original clean training examples, which serve as the unbiased meta-data. Since the algorithm execution just requires a clear definition of the training objective for the input examples, it is also well adaptable for the virtual augmented examples generated by our mixup method.
Datasets To validate our methods, we conduct experiments on Chinese benchmarks: OntoNotes 4.0 (Weischedel et al., 2011) and Weibo NER (Peng and Dredze, 2015), as well as English benchmarks: CoNLL 2003 (Sang and Meulder, 2003) and OntoNotes  $5.0^{2}$  (Pradhan et al., 2013). The Chinese datasets are split into training, development and test sections following Zhang and Yang (2018) while we take the same data split as Benikova et al. (2014) and Pradhan et al. (2012) on the English datasets. We follow Lample et al. (2016) to use the BIOES tagging scheme for all datasets. The detailed statistics can be found in Table 4.
Dataset	Type	Train	Dev	Test
OntoNotes 4	#sent	15.7k	4.3k	4.3k
#char	491.9k	200.5k	208.1k
#entity	12.6k	6.6k	7.3k
Weibo	#sent	1.4k	0.27k	0.27k
#char	73.8k	14.5k	14.8k
#entity	1.9k	0.4k	0.4k
CoNLL03	#sent	15.0k	3.5k	3.7k
#token	203.6k	51.4k	46.4k
#entity	23.5k	5.9k	5.6k
OntoNotes 5	#sent	59.9k	8.5k	8.3k
#token	1088.5k	147.7k	152.7k
#entity	81.8k	11.1k	11.3k
Table 4: Statistics of datasets. #sent and #entity stand for the number of sentences and entity words, respectively.
Implementation Details We use one-layer BiLSTM and the hidden size is set to 768. The dropout ratio is set to 0.5 for the input and output of BiLSTM. Regarding BERT, we adopt BERT-base model $^3$  (BERT-base-cased for the English NER) and fine-tune the inside parameters together with all other module parameters. We use the AdamW(Loshchilov and Hutter, 2019) optimizer to update the trainable parameters with  $\beta_{1} = 0.9$  and  $\beta_{2} = 0.99$ . For the BERT parameters, the learning rate is set to  $2e - 5$ . For other module parameters excluding BERT, a learning rate of  $1e - 3$  and weight decay of  $1e - 4$  are used. Gradient clipping is used to avoid gradient explosion by a maximum value of 5.0. All the models are trained on NVIDIA
$^{2}$ https://catalog.ldc.upenn.edu/LDC2013T19
3https://github.com/huggingface/ transformers
Models	ON 4	ON 5	CoNLL03
5%	10%	30%	5%	10%	30%	5%	10%	30%
Baseline	75.07	76.14	80.88	81.22	83.51	86.27	85.12	87.11	89.24
+ TS w/o MR	74.58	75.94	79.83	82.12	83.82	86.23	85.93	87.66	89.14
+ TS w/ MR	76.08	76.85	81.23	82.58	83.92	86.50	86.25	88.00	89.55
+ Mixup w/o MR	75.21	76.03	80.00	82.63	83.77	86.04	86.18	87.75	89.48
+ Mixup w/ MR	76.15	76.75	80.97	82.83	84.12	86.60	86.33	88.03	89.75
+ Both w/o MR	76.33	76.91	81.40	82.85	84.33	86.88	86.51	88.10	89.96
+ Both (Final)	76.82	77.13	81.66	82.98	84.52	87.09	86.76	88.25	90.12
Dai and Adel (2020)	75.05	76.75	81.24	82.47	83.90	86.55	86.22	87.86	89.91
Chen et al. (2020)	-	-	-	-	-	-	84.85	87.85	89.87
Chen et al. (2020) (Semi)	-	-	-	-	-	-	86.33	88.78	90.25
Table 1: Results on OntoNotes and CoNLL03 using  $5\%$ ,  $10\%$ , and  $30\%$  of the training data. Semi: additional 10,000 unlabeled training examples are used.
Models	ON 4	Weibo
Baseline	81.73	69.10
+ TS w/o MR	81.38	68.69
+ TS w/ MR	81.85	69.61
+ Mixup w/o MR	81.68	69.96
+ Mixup w/ MR	82.15	70.53
+ Both w/ MR	82.33	71.15
+ Both (Final)	82.48	71.42
Meng et al. (2019)†	81.63	67.60
Hu and Wei (2020)	80.20	64.00
Mengge et al. (2020)	80.60	69.23
Li et al. (2020a)	81.82	68.55
Nie et al. (2020a)†	81.18	69.78
Nie et al. (2020b)	-	69.80
Li et al. (2020b)†	82.11	-
Ma et al. (2020)	82.81	70.50
Xuan et al. (2020)†	82.04	71.25
Liu et al. (2021)	82.08	70.75
Tesla V100 (32G) GPUs. The higher library $^4$  is utilized for the implementation of second-order optimization involved in Algorithm 1.
For the NWS, we use the word vectors trained on Wikipedia data based on the GloVe model (Pennington et al., 2014) and build the synonym set for any given non-entity word based on the top-5 cosine similarity, where stop-words are excluded.
Table 2: Performance comparisons using the full training data on OntoNotes 4 (ON 4) and Weibo. Previous SOTA results are also offered for comparisons.  $\dagger$  denotes external knowledge is used.
Models	CoNLL03	ON 5
Baseline	91.23	88.22
+ TS w/o MR	90.98	87.55
+ TS w/ MR	91.64	88.84
+ Mixup w/o MR	91.04	87.46
+ Mixup w/ MR	91.42	88.98
+ Both w/ MR	91.88	89.24
+ Both (Final)	92.15	89.43
Chen et al. (2020)	91.83	-
Clark et al. (2018)‡	92.60	88.80
Fisher and Vlachos (2019)	-	89.20
Li et al. (2020b)†	93.04	91.11
Yu et al. (2020)†	93.50	91.30
Xu et al. (2021)†	-	90.85
Table 3: Performance comparisons using the full training data on CoNLL03 and OntoNotes 5 (ON 5). Previous SOTA results are also offered for comparisons.  $\ddagger$  means the multi-task learning with more unlabeled data.  $\dagger$  denotes external knowledge is used.