SlowGuess/ACL_2020_OCR huggingface dataset:
Xiaoya Li\*, Xiaofei Sun\*, Yuxian Meng\*, Junjun Liang\*, Fei Wu\* and Jiwei Li\*\*
{xiaoya_li, xiaofei_sun, yuxian_meng, jiwei_li} @shannonai.com, wufei@cs.zju.edu.cn
Many NLP tasks such as tagging and machine reading comprehension (MRC) are faced with the severe data imbalance issue: negative examples significantly outnumber positive ones, and the huge number of easy-negative examples overwhelms training. The most commonly used cross entropy criteria is actually accuracy-oriented, which creates a discrepancy between training and test. At training time, each training instance contributes equally to the objective function, while at test time F1 score concerns more about positive examples.
In this paper, we propose to use dice loss in replacement of the standard cross-entropy objective for data-imbalanced NLP tasks. Dice loss is based on the Sørensen-Dice coefficient (Sørensen, 1948) or Tversky index (Tversky, 1977), which attaches similar importance to false positives and false negatives, and is more immune to the data-imbalance issue. To further alleviate the dominating influence from easy-negative examples in training, we propose to associate training examples with dynamically adjusted weights to deemphasize easy-negative examples. Experimental results show that this strategy narrows down the gap between the F1 score in evaluation and the dice loss in training.
With the proposed training objective, we observe significant performance boosts over a wide range of data imbalanced NLP tasks. Notably, we are able to achieve SOTA results on CTB5, CTB6 and UD1.4 for the part of speech tagging task, and competitive or even better results on CoNLL03, OntoNotes5.0, MSRA and OntoNotes4.0 for the named entity recognition task along with the machine reading comprehension and paraphrase identification tasks. The code can be found at https://github.com/ShannonAI/dice_loss_for_NLP.
Task	# neg	# pos	ratio
CoNLL03 NER	170K	34K	4.98
OntoNotes5.0 NER	1.96M	239K	8.18
SQuAD 1.1 (Rajpurkar et al., 2016)	10.3M	175K	55.9
SQuAD 2.0 (Rajpurkar et al., 2018)	15.4M	188K	82.0
QUOREF (Dasigi et al., 2019)	6.52M	38.6K	169
Table 1: Number of positive and negative examples and their ratios for different data-imbalanced NLP tasks.
Data imbalance is a common issue in a variety of NLP tasks such as tagging and machine reading comprehension. Table 1 gives concrete examples: for the Named Entity Recognition (NER) task (Sang and De Meulder, 2003; Nadeau and Sekine, 2007), most tokens are backgrounds with tagging class  $O$ . Specifically, the number of tokens with tagging class  $O$  is 5 times as many as those with entity labels for the CoNLL03 dataset and 8 times for the OntoNotes5.0 dataset; Data-imbalanced issue is more severe for MRC tasks (Rajpurkar et al., 2016; Nguyen et al., 2016; Rajpurkar et al., 2018; Kocisky et al., 2018; Dasigi et al., 2019) with the value of negative-positive ratio being 50-200, which is due to the reason that the task of MRC is usually formalized as predicting the starting and ending indexes conditioned on the query and the context, and given a chunk of text of an arbitrary length, only two tokens are positive (or of interest) with all the rest being background.
Data imbalance results in the following two issues: (1) the training-test discrepancy: Without balancing the labels, the learning process tends to converge to a point that strongly biases towards class with the majority label. This actually creates a discrepancy between training and test: at training time, each training instance contributes equally to the objective function, whereas at test time, F1 gives equal weight to positive and negative examples; (2) the overwhelming effect of easy-negative examples. As pointed out by Meng et al. (2019), a significantly large number of negative examples also
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 465-476 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
means that the number of easy-negative example is large. The huge number of easy examples tends to overwhelm the training, making the model not sufficiently learn to distinguish between positive examples and hard-negative examples. The cross-entropy objective (CE for short) or maximum likelihood (MLE) objective, which is widely adopted as the training objective for data-imbalanced NLP tasks (Lample et al., 2016; Wu et al., 2019; Devlin et al., 2018; Yu et al., 2018a; McCann et al., 2018; Ma and Hovy, 2016; Chen et al., 2017), handles neither of the issues.
To handle the first issue, we propose to replace CE or MLE with losses based on the Sørensen-Dice coefficient (Sørensen, 1948) or Tversky index (Tversky, 1977). The Sørensen-Dice coefficient, dice loss for short, is the harmonic mean of precision and recall. It attaches equal importance to false positives (FPs) and false negatives (FNs) and is thus more immune to data-imbalanced datasets. Tversky index extends dice loss by using a weight that trades precision and recall, which can be thought as the approximation of the  $F_{\beta}$  score, and thus comes with more flexibility. Therefore, we use dice loss or Tversky index to replace CE loss to address the first issue.
Only using dice loss or Tversky index is not enough since they are unable to address the dominating influence of easy-negative examples. This is intrinsically because dice loss is actually a soft version of the F1 score. Taking the binary classification task as an example, at test time, an example will be classified as negative as long as its probability is smaller than 0.5, but training will push the value to 0 as much as possible. This gap isn't a big issue for balanced datasets, but is extremely detrimental if a big proportion of training examples are easy-negative ones: easy-negative examples can easily dominate training since their probabilities can be pushed to 0 fairly easily. Meanwhile, the model can hardly distinguish between hard-negative examples and positive ones. Inspired by the idea of focal loss (Lin et al., 2017) in computer vision, we propose a dynamic weight adjusting strategy, which associates each training example with a weight in proportion to  $(1 - p)$ , and this weight dynamically changes as training proceeds. This strategy helps deemphasize confident examples during training as their probability  $p$  approaches 1, making the model attentive to hard-negative examples, and thus alleviates the dominating effect of easy-negative exam-
plies. Combining both strategies, we observe significant performance boosts on a wide range of data imbalanced NLP tasks.
The rest of this paper is organized as follows: related work is presented in Section 2. We describe different proposed losses in Section 3. Experimental results are presented in Section 4. We perform ablation studies in Section 5, followed by a brief conclusion in Section 6.
2.1 Data Resampling
The idea of weighting training examples has a long history. Importance sampling (Kahn and Marshall, 1953) assigns weights to different samples and changes the data distribution. Boosting algorithms such as AdaBoost (Kanduri et al., 2018) select harder examples to train subsequent classifiers. Similarly, hard example mining (Malisiewicz et al., 2011) downsamples the majority class and exploits the most difficult examples. Oversampling (Chen et al., 2010; Chawla et al., 2002) is used to balance the data distribution. Another line of data resampling is to dynamically control the weights of examples as training proceeds. For example, focal loss (Lin et al., 2017) used a soft weighting scheme that emphasizes harder examples during training. In self-paced learning (Kumar et al., 2010), example weights are obtained through optimizing the weighted training loss which encourages learning easier examples first. At each training step, self-paced learning algorithm optimizes model parameters and example weights jointly. Other works (Chang et al., 2017; Katharopoulos and Fleuret, 2018) adjusted the weights of different training examples based on training loss. Besides, recent work (Jiang et al., 2017; Fan et al., 2018) proposed to learn a separate network to predict sample weights.
2.2 Data Imbalance Issue in Computer Vision
The background-object label imbalance issue is severe and thus well studied in the field of object detection (Li et al., 2015; Girshick, 2015; He et al., 2015; Girshick et al., 2013; Ren et al., 2015). The idea of hard negative mining (HNM) (Girshick et al., 2013) has gained much attention recently. Pang et al. (2019) proposed a novel method called IoU-balanced sampling and Chen et al. (2019) designed a ranking model to replace the conventional classification task with an average-precision loss
to alleviate the class imbalance issue. The efforts made on object detection have greatly inspired us to solve the data imbalance issue in NLP.
Sudre et al. (2017) addressed the severe class imbalance issue for the image segmentation task. They proposed to use the class re-balancing property of the Generalized Dice Loss as the training objective for unbalanced tasks. Shen et al. (2018) investigated the influence of Dice-based loss for multi-class organ segmentation using a dataset of abdominal CT volumes. Kodym et al. (2018) proposed to use the batch soft Dice loss function to train the CNN network for the task of segmentation of organs at risk (OAR) of medical images. Shamir et al. (2019) extended the definition of the classical Dice coefficient to facilitate the direct comparison of a ground truth binary image with a probabilistic map. In this paper, we introduce dice loss into NLP tasks as the training objective and propose a dynamic weight adjusting strategy to address the dominating influence of easy-negative examples.
3 Losses
For illustration purposes, we use the binary classification task to demonstrate how different losses work. The mechanism can be easily extended to multi-class classification. Let  $X$  denote a set of training instances and each instance  $x_{i} \in X$  is associated with a golden binary label  $y_{i} = [y_{i0}, y_{i1}]$  denoting the ground-truth class  $x_{i}$  belongs to, and  $p_{i} = [p_{i0}, p_{i1}]$  is the predicted probabilities of the two classes respectively, where  $y_{i0}, y_{i1} \in \{0, 1\}, p_{i0}, p_{i1} \in [0, 1]$  and  $p_{i1} + p_{i0} = 1$ .
3.2 Cross Entropy Loss
The vanilla cross entropy (CE) loss is given by:
$$
\mathrm {C E} = - \frac {1}{N} \sum_ {i} \sum_ {j \in \{0, 1 \}} y _ {i j} \log p _ {i j} \tag {1}
$$
As can be seen from Eq.1, each  $x_{i}$  contributes equally to the final objective. Two strategies are normally used to address the case where we wish that not all  $x_{i}$  are treated equally: associating different classes with different weighting factor  $\alpha$  or resampling the datasets. For the former, Eq.1 is adjusted as follows:
$$
\text {W e i g h t e d} \quad \mathrm {C E} = - \frac {1}{N} \sum_ {i} \alpha_ {i} \sum_ {j \in \{0, 1 \}} y _ {i j} \log p _ {i j} \tag {2}
$$
where  $\alpha_{i}\in [0,1]$  may be set by the inverse class frequency or treated as a hyperparameter to set by cross validation. In this work, we use  $\lg \left(\frac{n - n_t}{n_t} +K\right)$  to calculate the coefficient  $\alpha$  where  $n_t$  is the number of samples with class  $t$  and  $n$  is the total number of samples in the training set.  $K$  is a hyperparameter to tune. Intuitively, this equation assigns less weight to the majority class and more weight to the minority class. The data resampling strategy constructs a new dataset by sampling training examples from the original dataset based on human-designed criteria, e.g. extracting equal training samples from each class. Both strategies are equivalent to changing the data distribution during training and thus are of the same nature. Empirically, these two methods are not widely used due to the trickiness of selecting  $\alpha$  especially for multi-class classification tasks and that inappropriate selection can easily bias towards rare classes (Valverde et al., 2017).
3.3 Dice Coefficient and Tversky Index
Sorensen-Dice coefficient (Sorensen, 1948; Dice, 1945), dice coefficient (DSC) for short, is an F1-oriented statistic used to gauge the similarity of two sets. Given two sets  $A$  and  $B$ , the vanilla dice coefficient between them is given as follows:
$$
\operatorname {D S C} (A, B) = \frac {2 | A \cap B |}{| A | + | B |} \tag {3}
$$
In our case,  $A$  is the set that contains all positive examples predicted by a specific model, and  $B$  is the set of all golden positive examples in the dataset. When applied to boolean data with the definition of true positive (TP), false positive (FP), and false negative (FN), it can be then written as follows:
$$
\begin{array}{l} D S C = \frac {2 T P}{2 T P + F N + F P} = \frac {2 \frac {T P}{T P + F N} \frac {T P}{T P + F P}}{\frac {T P}{T P + F N} + \frac {T P}{T P + F P}} \\ = \frac {2 \operatorname {P r e} \times \operatorname {R e c}}{\operatorname {P r e} + \operatorname {R e c}} = F 1 \tag {4} \\ \end{array}
$$
For an individual example  $x_{i}$ , its corresponding dice coefficient is given as follows:
$$
\operatorname {D S C} \left(x _ {i}\right) = \frac {2 p _ {i 1} y _ {i 1}}{p _ {i 1} + y _ {i 1}} \tag {5}
$$
As can be seen, a negative example  $(y_{i1} = 0)$  does not contribute to the objective. For smoothing purposes, it is common to add a  $\gamma$  factor to both the nominator and the denominator, making the form to be as follows (we simply set  $\gamma = 1$  in the rest of
Loss	Formula (one sample xi)
CE	- ∑j∈{0,1} yij log pij
WCE	-αi ∑j∈{0,1} yij log pij
DL	1 - 2pi1y1i + γ / pi2i + y2i + γ
TL	1 - pi1y1i + γ / pi1y1i + α pi1y10 + β pi0y1i + γ
DSC	1 - 2(1-pi1)pi1·y1i + γ / (1-pi1)pi1 + yi + γ
FL	-αi ∑j∈{0,1} (1 - pij)γ log pij
Table 2: Different losses and their formulas. We add +1 to DL, TL and DSC so that they are positive.
this paper):
$$
\operatorname {D S C} \left(x _ {i}\right) = \frac {2 p _ {i 1} y _ {i 1} + \gamma}{p _ {i 1} + y _ {i 1} + \gamma} \tag {6}
$$
As can be seen, negative examples whose DSC is  $\frac{\gamma}{p_{i1} + \gamma}$ , also contribute to the training. Additionally, Milletari et al. (2016) proposed to change the denominator to the square form for faster convergence, which leads to the following dice loss (DL):
$$
\mathrm {D L} = \frac {1}{N} \sum_ {i} \left[ 1 - \frac {2 p _ {i 1} y _ {i 1} + \gamma}{p _ {i 1} ^ {2} + y _ {i 1} ^ {2} + \gamma} \right] \tag {7}
$$
Another version of DL is to directly compute set-level dice coefficient instead of the sum of individual dice coefficient, which is easier for optimization:
$$
\mathrm {D L} = 1 - \frac {2 \sum_ {i} p _ {i 1} y _ {i 1} + \gamma}{\sum_ {i} p _ {i 1} ^ {2} + \sum_ {i} y _ {i 1} ^ {2} + \gamma} \tag {8}
$$
Tversky index (TI), which can be thought as the approximation of the  $F_{\beta}$  score, extends dice coefficient to a more general case. Given two sets  $A$  and  $B$ , tversky index is computed as follows:
$$
\mathrm {T I} = \frac {\left| A \cap B \right|}{\left| A \cap B \right| + \alpha \left| A \backslash B \right| + \beta \left| B \backslash A \right|} \tag {9}
$$
Tversky index offers the flexibility in controlling the tradeoff between false-negatives and false-positives. It degenerates to DSC if  $\alpha = \beta = 0.5$ . The Tversky loss (TL) is thus given as follows:
$$
\mathrm {T L} = \frac {1}{N} \sum_ {i} \left[ 1 - \frac {p _ {i 1} y _ {i 1} + \gamma}{p _ {i 1} y _ {i 1} + \alpha p _ {i 1} y _ {i 0} + \beta p _ {i 0} y _ {i 1} + \gamma} \right] \tag {10}
$$
3.4 Self-adjusting Dice Loss
Consider a simple case where the dataset consists of only one example  $x_{i}$ , which is classified as positive as long as  $p_{i1}$  is larger than 0.5. The computation of  $F1$  score is actually as follows:
$$
\mathrm {F} 1 \left(x _ {i}\right) = 2 \frac {\mathbb {I} \left(p _ {i 1} > 0 . 5\right) y _ {i 1}}{\mathbb {I} \left(p _ {i 1} > 0 . 5\right) + y _ {i 1}} \tag {11}
$$
Figure 1: An illustration of derivatives of the four losses. The derivative of DSC approaches zero right after  $p$  exceeds 0.5, and for the other losses, the derivatives reach 0 only if the probability is exactly 1, which means they will push  $p$  to 1 as much as possible.
Comparing Eq.5 with Eq.11, we can see that Eq.5 is actually a soft form of  $F1$ , using a continuous  $p$  rather than the binary  $\mathbb{I}(p_{i1} > 0.5)$ . This gap isn't a big issue for balanced datasets, but is extremely detrimental if a big proportion of training examples are easy-negative ones: easy-negative examples can easily dominate training since their probabilities can be pushed to 0 fairly easily. Meanwhile, the model can hardly distinguish between hard-negative examples and positive ones, which has a huge negative effect on the final F1 performance.
To address this issue, we propose to multiply the soft probability  $p$  with a decaying factor  $(1 - p)$ , changing Eq.11 to the following adaptive variant of DSC:
$$
\operatorname {D S C} \left(x _ {i}\right) = \frac {2 \left(1 - p _ {i 1}\right) p _ {i 1} \cdot y _ {i 1} + \gamma}{\left(1 - p _ {i 1}\right) p _ {i 1} + y _ {i 1} + \gamma} \tag {12}
$$
One can think  $(1 - p_{i1})$  as a weight associated with each example, which changes as training proceeds. The intuition of changing  $p_{i1}$  to  $(1 - p_{i1})p_{i1}$  is to push down the weight of easy examples. For easy examples whose probability are approaching 0 or 1,  $(1 - p_{i1})p_{i1}$  makes the model attach significantly less focus to them.
A close look at Eq.12 reveals that it actually mimics the idea of focal loss (FL for short) (Lin et al., 2017) for object detection in vision. Focal loss was proposed for one-stage object detector to handle foreground-background tradeoff encountered during training. It down-weights the loss assigned to well-classified examples by adding a  $(1 - p)^{\gamma}$  factor, leading the final loss to be  $-(1 - p)^{\gamma} \log p$ .
CTB5	CTB6	UD1.4
Model	Prec.	Rec.	F1	Prec.	Rec.	F1	Prec.	Rec.	F1
Joint-POS(Sig)(Shao et al., 2017)	93.68	94.47	94.07	-	-	90.81	89.28	89.54	89.41
Joint-POS(Ens)(Shao et al., 2017)	93.95	94.81	94.38	-	-	-	89.67	89.86	89.75
Lattice-LSTM(Zhang and Yang, 2018)	94.77	95.51	95.14	92.00	90.86	91.43	90.47	89.70	90.09
BERT-Tagger(Devlin et al., 2018)	95.86	96.26	96.06	94.91	94.63	94.77	95.42	94.17	94.79
BERT+FL	96.11	97.42	96.76	95.80	95.08	95.44	96.33	95.85	96.81
(+0.70)	(+0.67)	(+2.02)
BERT+DL	96.77	98.87	97.81	94.08	96.12	95.09	96.10	97.79	96.94
(+1.75)	(+0.32)	(+2.15)
BERT+DSC	97.10	98.75	97.92	96.29	96.85	96.57	96.24	97.73	96.98
(+1.86)	(+1.80)	(+2.19)
Table 3: Experimental results for Chinese POS datasets including CTB5, CTB6 and UD1.4.
English WSJ
Model	Prec.	Rec.	F1
Meta BiLSTM(Bohnet et al., 2018)	-	-	98.23
BERT-Tagger (Devlin et al., 2018)	99.21	98.36	98.86
BERT-Tagger+FL	98.36	98.97	98.88 (+0.02)
BERT-Tagger+DL	99.34	98.22	98.91 (+0.05)
BERT-Tagger+DSC	99.41	98.93	99.38 (+0.52)
English Tweets
Model	Prec.	Rec.	F1
FastText+CNN+CRF(Godin, 2019)	-	-	91.78
BERT-Tagger (Devlin et al., 2018)	92.33	91.98	92.34
BERT-Tagger+FL	91.24	93.22	92.47 (+0.13)
BERT-Tagger+DL	91.44	92.88	92.52 (+0.18)
BERT-Tagger+DSC	92.87	93.54	92.58 (+0.24)
Table 4: Experimental results for English POS datasets.
In Table 2, we summarize all the aforementioned losses. Figure 1 gives an explanation from the perspective in derivative: The derivative of DSC approaches zero right after  $p$  exceeds 0.5, which suggests the model attends less to examples once they are correctly classified. But for the other losses, the derivatives reach 0 only if the probability is exactly 1, which means they will push  $p$  to 1 as much as possible.
We evaluated the proposed method on four NLP tasks, part-of-speech tagging, named entity recognition, machine reading comprehension and paraphrase identification. Hyperparameters are tuned on the corresponding development set of each dataset. More experiment details including datasets and hyperparameters are shown in supplementary material.
4.1 Part-of-Speech Tagging
Settings Part-of-speech tagging (POS) is the task of assigning a part-of-speech label (e.g., noun, verb, adjective) to each word in a given text. In this paper, we choose BERT (Devlin et al., 2018) as the backbone and conduct experiments on three widely used Chinese POS datasets including Chinese Treebank (Xue et al., 2005) 5.0/6.0 and UD1.4 and English datasets including Wall Street Journal (WSJ) and the dataset proposed by Ritter et al. (2011). We report the span-level micro-averaged precision, recall and F1 for evaluation.
Baselines We used the following baselines:
- Joint-POS: Shao et al. (2017) jointly learns Chinese word segmentation and POS.
- Lattice-LSTM: Zhang and Yang (2018) constructs a word-character lattice network.
- Bert-Tagger: Devlin et al. (2018) treats part-of-speech as a tagging task.
Results Table 3 presents the experimental results on Chinese datasets. As can be seen, the proposed DSC loss outperforms the best baseline results by a large margin, i.e., outperforming BERT-tags by  $+1.86$  in terms of F1 score on CTB5,  $+1.80$  on CTB6 and  $+2.19$  on UD1.4. As far as we know, we are achieving SOTA performances on the three datasets. Focal loss only obtains a little performance improvement on CTB5 and CTB6, and the dice loss obtains huge gain on CTB5 but not on CTB6, which indicates the three losses are not consistently robust in solving the data imbalance issue.
Table 4 presents the experimental results for English datasets.
English CoNLL 2003
Model	Prec.	Rec.	F1
ELMo(Peters et al., 2018)	-	-	92.22
CVT(Clark et al., 2018)	-	-	92.6
BERT-Tagger(Devlin et al., 2018)	-	-	92.8
BERT-MRC(Li et al., 2019)	92.33	94.61	93.04
BERT-MRC+FL	93.13	93.09	93.11 (+0.06)
BERT-MRC+DL	93.22	93.12	93.17 (+0.12)
BERT-MRC+DSC	93.41	93.25	93.33 (+0.29)
English OntoNotes 5.0
Model	Prec.	Rec.	F1
CVT (Clark et al., 2018)	-	-	88.8
BERT-Tagger (Devlin et al., 2018)	90.01	88.35	89.16
BERT-MRC(Li et al., 2019)	92.98	89.95	91.11
BERT-MRC+FL	90.13	92.34	91.22 (+0.11)
BERT-MRC+DL	91.70	92.06	91.88 (+0.77)
BERT-MRC+DSC	91.59	92.56	92.07 (+0.96)
Chinese MSRA
Model	Prec.	Rec.	F1
Lattice-LSTM (Zhang and Yang, 2018)	93.57	92.79	93.18
BERT-Tagger (Devlin et al., 2018)	94.97	94.62	94.80
Glyce-BERT (Wu et al., 2019)	95.57	95.51	95.54
BERT-MRC(Li et al., 2019)	96.18	95.12	95.75
BERT-MRC+FL	95.45	95.89	95.67 (-0.08)
BERT-MRC+DL	96.20	96.68	96.44 (+0.69)
BERT-MRC+DSC	96.67	96.77	96.72 (+0.97)
Chinese OntoNotes 4.0
Model	Prec.	Rec.	F1
Lattice-LSTM (Zhang and Yang, 2018)	76.35	71.56	73.88
BERT-Tagger (Devlin et al., 2018)	78.01	80.35	79.16
Glyce-BERT (Wu et al., 2019)	81.87	81.40	80.62
BERT-MRC(Li et al., 2019)	82.98	81.25	82.11
BERT-MRC+FL	83.63	82.97	83.30 (+1.19)
BERT-MRC+DL	83.97	84.05	84.01 (+1.90)
BERT-MRC+DSC	84.22	84.72	84.47 (+2.36)
Table 5: Experimental results for NER task.
4.2 Named Entity Recognition
Settings Named entity recognition (NER) is the task of detecting the span and semantic category of entities within a chunk of text. Our implementation uses the current state-of-the-art model proposed by Li et al. (2019) as the backbone, and changes the MLE loss to DSC loss. Datasets that we use include OntoNotes4.0 (Pradhan et al., 2011), MSRA (Levow, 2006), CoNLL2003 (Sang and Meulder, 2003) and OntoNotes5.0 (Pradhan et al., 2013). We report span-level micro-averaged precision, recall and F1.
Baselines We use the following baselines:
- ELMo: a tagging model with pretraining from Peters et al. (2018).
- Lattice-LSTM: Zhang and Yang (2018) constructs a word-character lattice, only used in Chinese datasets.
- CVT: Clark et al. (2018) uses Cross-View Training(CVT) to improve the representations of a Bi-LSTM encoder.
- Bert-Tagger: Devlin et al. (2018) treats NER as a tagging task.
- Glyce-BERT: Wu et al. (2019) combines Chinese glyph information with BERT pretraining.
- BERT-MRC: Li et al. (2019) formulates NER as a machine reading comprehension task and achieves SOTA results on Chinese and English NER benchmarks.
Results Table 5 shows experimental results on NER datasets. DSC outperforms BERT-MRC(Li et al., 2019) by  $+0.29$ ,  $+0.96$ ,  $+0.97$  and  $+2.36$  respectively on CoNLL2003, OntoNotes5.0, MSRA and OntoNotes4.0. As far as we are concerned, we are setting new SOTA performances on all of the four NER datasets.
4.3 Machine Reading Comprehension