SlowGuess/NAACL_2022_OCR huggingface dataset:
The shape of  $\mathbf{X}^{\top}\mathbf{A}_c(\mathbf{P} - \mathbf{Y})$  is  $m\times k$ , the same as  $\mathbf{W}$ .
B.2 Dice Loss Gradient
Dice loss computes per-class F-1 scores. Suppose the ground truth is the  $c$ -th class for a given example  $\mathbf{x}$ .
$$
\begin{array}{l} L _ {D L} (\mathbf {x}, \mathbf {y}) (18) \\ = \sum_ {j ^ {\prime} = 1} ^ {k} \left[ 1 - \mathbf {1} \{c = j ^ {\prime} \} \frac {\gamma + 2 p _ {c}}{\gamma + p _ {c} ^ {2} + 1} \right. \\ \left. + 1 - \mathbf {1} \{c \neq j ^ {\prime} \} \frac {\gamma}{\gamma + p _ {j ^ {\prime}} ^ {2}} \right] (19) \\ = 1 - \frac {\gamma + 2 p _ {c}}{\gamma + p _ {c} ^ {2} + 1} + \sum_ {j ^ {\prime} \neq c} \left[ 1 - \frac {\gamma}{\gamma + p _ {j ^ {\prime}} ^ {2}} \right] (20) \\ = k - \frac {\gamma + 2 p _ {c}}{\gamma + p _ {c} ^ {2} + 1} - \sum_ {j ^ {\prime} \neq c} \frac {\gamma}{\gamma + p _ {j ^ {\prime}} ^ {2}} (21) \\ \end{array}
$$
Take gradient with respect to  $w_{ji}$ , the weight of the  $j$ -th class, the  $i$ -th dimension.
$$
\begin{array}{l} \frac {\partial L _ {D L} (\mathbf {x} , \mathbf {y})}{\partial w _ {j i}} (22) \\ = - \frac {\partial}{\partial p _ {c}} \left[ \frac {\gamma + 2 p _ {c}}{\gamma + p _ {c} ^ {2} + 1} \right] \cdot \frac {\partial p _ {c}}{\partial w _ {j i}} \\ - \sum_ {j ^ {\prime} \neq c} \frac {\partial}{\partial p _ {j ^ {\prime}}} \left[ \frac {\gamma}{\gamma + p _ {j ^ {\prime}} ^ {2}} \right] \cdot \frac {\partial p _ {j ^ {\prime}}}{\partial w _ {j i}} (23) \\ = - \frac {2 (\gamma + p _ {c} ^ {2} + 1) - (\gamma + 2 p _ {c}) 2 p _ {c}}{(\gamma + p _ {c} ^ {2} + 1) ^ {2}} \cdot \frac {\partial p _ {c}}{\partial w _ {j i}} \\ - \sum_ {j ^ {\prime} \neq c} \frac {- \gamma \cdot 2 p _ {j ^ {\prime}}}{\left(\gamma + p _ {j ^ {\prime}} ^ {2}\right) ^ {2}} \cdot \frac {\partial p _ {j ^ {\prime}}}{\partial w _ {j i}} (24) \\ = - \frac {2 (\gamma + p _ {c} ^ {2} + 1) - (\gamma + 2 p _ {c}) 2 p _ {c}}{(\gamma + p _ {c} ^ {2} + 1) ^ {2}} \\ \cdot \left[ \mathbf {1} \{j = c \} - p _ {j} \right] p _ {c} x _ {i} \\ - \sum_ {j ^ {\prime} \neq c} \frac {- \gamma \cdot 2 p _ {j ^ {\prime}}}{\left(\gamma + p _ {j ^ {\prime}} ^ {2}\right) ^ {2}} \cdot \left[ \mathbf {1} \{j = j ^ {\prime} \} - p _ {j} \right] p _ {j ^ {\prime}} x _ {i} (25) \\ = - \frac {2 \left(1 - p _ {c}\right) \left(1 + \gamma + p _ {c}\right) p _ {c}}{\left(\gamma + p _ {c} ^ {2} + 1\right) ^ {2}} \cdot \left[ \mathbf {1} \{j = c \} - p _ {j} \right] x _ {i} (26) \\ \end{array}
$$
$$
\begin{array}{l} + \sum_ {j ^ {\prime} \neq c} \frac {\gamma \cdot 2 p _ {j ^ {\prime}} ^ {2}}{(\gamma + p _ {j ^ {\prime}} ^ {2}) ^ {2}} \cdot [ \mathbf {1} \{j = j ^ {\prime} \} - p _ {j} ] x _ {i} (27) \\ = a _ {c} \left[ p _ {j} - \mathbf {1} \{j = c \} \right] x _ {i} - \sum_ {j ^ {\prime} \neq c} b _ {j ^ {\prime}} \left[ p _ {j} - \mathbf {1} \{j = j ^ {\prime} \} \right] x _ {i} (28) \\ \end{array}
$$
$$
a _ {c} = \frac {2 \left(1 - p _ {c}\right) \left(1 + \gamma + p _ {c}\right) p _ {c}}{\left(\gamma + p _ {c} ^ {2} + 1\right) ^ {2}} \tag {29}
$$
$$
b _ {j ^ {\prime}} = \frac {\gamma \cdot 2 p _ {j ^ {\prime}} ^ {2}}{(\gamma + p _ {j ^ {\prime}} ^ {2}) ^ {2}} \tag {30}
$$
$a_{c}$  depends on the ground truth label of example  $\mathbf{x}$ .  $b_{j'}$  depends on the current predicted probabilities for  $\mathbf{x}$ .
Generalizing the scalar gradient in Equation (28) to matrix gradient, we have
$$
\begin{array}{l} \frac {\partial L _ {D L (\mathbf {x} , \mathbf {y})}}{\partial \mathbf {W}} (31) \\ = \mathbf {x} ^ {\top} a _ {c} (\mathbf {p} - \mathbf {y}) \\ - \mathbf {x} ^ {\top} \left[ \underbrace {\dots , \sum_ {j ^ {\prime} \neq c} b _ {j ^ {\prime}} \left[ p _ {j} - \mathbf {1} \{j = j ^ {\prime} \} \right] , \cdots} _ {j = 1, \dots , k} \right] (32) \\ = \mathbf {x} ^ {\top} a _ {c} (\mathbf {p} - \mathbf {y}) - \mathbf {x} ^ {\top} \mathbf {v} (33) \\ \end{array}
$$
$\mathbf{v}$  is a vector specific to the example  $\mathbf{x}$ .
$$
\frac {\partial L _ {D L} (\mathbf {X} , \mathbf {Y})}{\partial \mathbf {W}} = \mathbf {X} ^ {\top} \mathbf {A} _ {c} (\mathbf {P} - \mathbf {Y}) - \mathbf {X} ^ {\top} \mathbf {V} \tag {34}
$$
where  $\mathbf{V}$  has shape  $n\times k$ , and the  $l$ -th row in matrix  $\mathbf{V}$  is a  $k$  dimensional vector computed in the same manner as Equation (32) with respect to the  $l$ -th training example,  $1\leq l\leq n$ .
C Additional Performance Analysis
C.1 Micro-averaged Metrics
In the main paper we reported macro-averaged F1 scores for each dataset. To provide a more complete comparison of performance changes, here we report micro-averaged F1 scores in Tables 7-10. Micro-averaged metrics lump together all named entities without distinguishing their types, and therefore the majority types have more influence on these metrics than minority types. Overall, the trend is consistent with the macro-averaged metrics. Sentence-level resampling methods tend to deliver more stable gains and generally outperform baseline methods.
C.2 Per-Entity-Type Metrics
To further examine the impacts of our method on entity types, we also report per-entity-type precision, recall, and F1 scores for each dataset in Tables 11-14. We compare the performance of using the original corpus and a representative of our methods (sCR). Red up-arrows  $(\uparrow)$  means sCR obtains better precision/recall/F1 score compared to using the original corpus.
Here we observe that at the level of entity types, either precision and recall simultaneously improve or drop, or precision improves at a slight cost of recall. It is rare that recall improves at the cost of precision (only the GPE type in Table 13). This pattern indicates that the BERT-CRF model trained on the original corpus has many 'false negatives' (tagging entity tokens as "other"). In other words, the model is extremely reluctant to predict non-other entity types. Our sentence-level resampling methods encourage the model to correctly assign entity types to more tokens. Another trend is that improvements on small corpora (AnEM, WNUT, GMB subset) are more salient than on large corpus (CoNLL). Note that sentence resampling does not necessarily favor minority entity types as all entity types are very rare already, compared to the Other tokens (see the last column of Tables 11-14, "Token %").
Shallow model	Bi-LSTM	BERT
Softmax	CRF	Softmax	CRF	Softmax	CRF
Original corpus	33.96	56.74	55.94	58.81	67.02	66.69
Balanced undersampling	31.12	56.03	24.95	30.90	69.62	69.38
Data augmentation	32.70	56.80	53.12	56.19	68.09	71.63
Focal loss	28.89	-	53.12	-	68.24	-
Dice loss	35.62	-	8.62	-	64.50	-
sC	35.00	55.75	57.03	59.24	70.90	70.55
sCR	35.25	55.89	57.12	60.37	71.76	69.71
sCRD	34.04	55.65	55.57	57.02	70.22	69.57
nsCRD	35.28	56.87	54.08	58.52	71.08	69.82
Table 7: Micro-averaged F1-scores on AnEM. The three NER models using either softmax or CRF output layer are reported.  $\underline{\underline{\mathbf{\Pi}}}$  means that focal loss and Dice loss do not apply to CRF outputs. In each column, the highest F1-score is shown in **boldface** and the second highest is shown in **underline**.
Shallow model	Bi-LSTM	BERT
Softmax	CRF	Softmax	CRF	Softmax	CRF
Original corpus	21.74	5.28	27.89	29.32	61.31	60.70
Balanced undersampling	22.77	5.69	22.51	24.64	53.70	51.66
Data augmentation	17.33	5.47	28.73	14.37	60.00	61.86
Focal loss	21.62	-	30.23	-	62.18	-
Dice loss	24.01	-	29.53	-	57.14	-
sC	22.74	5.62	25.49	28.55	63.54	64.62
sCR	23.25	6.48	31.38	27.76	65.67	65.99
sCRD	23.09	6.01	27.74	32.69	66.02	64.68
nsCRD	23.17	5.68	28.83	34.29	62.00	63.41
Table 8: Micro-averaged F1-scores on the WNUT corpus. See the caption of Table 7 above for details.
Shallow model	Bi-LSTM	BERT
Softmax	CRF	Softmax	CRF	Softmax	CRF
Original corpus	50.60	66.67	75.92	76.87	80.43	81.68
Balanced undersampling	50.94	66.18	70.91	73.86	77.87	78.41
Data augmentation	49.09	66.13	74.75	76.17	81.48	81.86
Focal loss	49.84	-	72.44	-	81.61	-
Dice loss	57.70	-	70.46	-	80.02	-
sC	50.68	66.32	75.85	77.11	81.26	81.16
sCR	51.68	66.09	74.76	76.29	80.45	82.12
sCRD	51.12	66.58	72.43	74.21	80.71	81.09
nsCRD	50.76	66.87	73.38	74.62	81.69	81.54
Table 9: Micro-averaged F1-scores on the GMB subset corpus. See the caption of Table 7 above for details.
Shallow model	Bi-LSTM	BERT
Softmax	CRF	Softmax	CRF	Softmax	CRF
Original corpus	42.47	71.11	79.84	81.03	92.08	92.23
Balanced undersampling	43.22	70.43	71.46	76.75	90.97	90.78
Data augmentation	42.37	70.84	81.87	81.40	91.96	92.33
Focal loss	43.78	-	81.53	-	92.15	-
Dice loss	48.77	-	77.51	-	91.94	-
sC	43.42	70.86	81.10	78.70	92.53	92.48
sCR	42.77	70.57	82.26	78.18	91.88	92.59
sCRD	44.48	70.84	82.29	81.36	91.93	92.07
nsCRD	43.24	70.67	83.05	83.42	91.37	91.87
Table 10: Micro-averaged F1-scores on the CoNLL corpus. See the caption of Table 7 above for details.
Entity Type	Original corpus	sCR	Token %
P	R	F	P	R	F
Developing anatomical structure	86.96	90.91	88.89	87.50↑	95.45↑	91.30↑	0.03
Immaterial anatomical entity	24.14	23.33	23.73	40.00↑	40.00↑	40.00↑	0.06
Anatomical system	52.17	85.17	64.86	41.38↓	85.71↑	55.81↓	0.08
Organism subdivision	42.86	54.00	47.79	64.44↑	58.00↑	61.05↑	0.16
Cellular component	26.37	27.27	26.81	41.79↑	31.82↑	36.13↑	0.21
Tissue	35.62	52.34	42.46	46.97↑	52.54↑	49.60↑	0.27
Organism substance	57.58	77.87	66.21	74.62↑	80.83↑	77.60↑	0.30
Organ	78.95	75.47	77.17	78.47↓	71.07↓	74.59↓	0.35
Pathological formation	62.18	65.54	63.82	72.73↑	59.46↓	65.43↑	0.51
Multi-tissue structure	50.17	60.08	54.68	56.45↑	57.61↓	57.02↑	0.86
Cell	77.02	79.07	78.03	79.79↑	76.33↓	78.02↓	1.08
Macro-avg	54.00	62.89	57.68	62.19↑	64.44↑	62.41↑	-
Table 11: Per-entity-type precision (P), recall (R), and F1 scores (F) on AnEM corpus using BERT-CRF model.
Entity Type	Original corpus	sCR	Token %
P	R	F	P	R	F
Corporation	33.33	66.67	44.44	20.00↓	33.33↓	43↓	0.43
Creative work	0	0	0	100.00↑	20.00↑	33.33↑	0.55
Product	33.33	33.33	33.33	25.00↓	16.67↓	20.00↑	0.55
Group	30.77	12.90	18.18	47.37↑	29.03↑	36.00↑	0.66
Location	70.00	72.41	71.18	91.30↑	72.41	80.76↑	1.27
Person	71.11	74.42	72.72	76.19↑	74.42	75.29↑	1.59
Macro-avg	39.76	43.29	39.98	59.98↑	40.98↓	45.06↑	-
Table 12: Per-entity-type precision (P), recall (R), and F1 scores (F) on WNUT corpus using BERT-CRF model.
Entity Type	Original corpus	sCR	Token %
P	R	F	P	R	F
NAT	0	0	0	0	0	0	0.03
ART	12.00	10.34	11.11	0↓	0↓	0↓	0.07
EVE	26.67	36.36	30.77	35.71↑	45.45↑	40.00↑	0.11
GPE	54.38	52.49	53.41	54.20↓	52.99↑	53.59↑	1.58
TIM	76.69	88.73	88.27	77.22↑	89.71↑	83.00↑	2.40
ORG	74.39	77.06	76.72	77.43↑	75.76↓	76.59↓	3.31
PER	86.87	87.31	87.09	88.56↑	89.90↑	89.22↑	3.56
GEO	93.19	93.19	93.19	92.26↓	92.26↓	92.26↓	4.00
Macro-avg	53.27	55.68	54.32	53.17↓	55.86↑	54.33↑	-
Table 13: Per-entity-type precision (P), recall (R), and F1 scores (F) on GMB subset using BERT-CRF model.
Entity Type	Original corpus	sCR	Token %
P	R	F	P	R	F
MISC	77.26	81.76	79.44	82.61↑	82.49↑	82.54↑	2.26
LOC	92.17	92.50	92.33	91.76↓	91.49↓	91.62↓	4.07
ORG	86.16	88.83	87.47	85.92↓	88.59↓	87.23↓	4.92
PER	94.44	94.66	94.55	95.11↑	94.97↑	95.03↑	5.46
Macro-avg	87.51	89.44	88.45	88.85↑	89.38↓	89.11↑	-
Table 14: Per-entity-type F1 scores on CoNLL using BERT-CRF model.