SlowGuess/NAACL_2021_OCR huggingface dataset:
Jie Yang, Shuailong Liang, and Yue Zhang. 2018. Design challenges and misconceptions in neural sequence labeling. arXiv: Computation and Language.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, pages 5753-5763.
Zihuiwen Ye, Pengfei Liu, Jinlan Fu, and Graham Neubig. 2021. Towards more fine-grained and reliable nlp performance prediction. arXiv preprint arXiv:2102.05486.
Jianfei Yu, Jing Jiang, Li Yang, and Rui Xia. 2020. Improving multimodal named entity recognition via entity span detection with unified multimodal transformer. Association for Computational Linguistics.
A Aggregator Setting
Tab. 4 illustrates the window size  $k(k \neq 1)$  when the larger-context aggregator achieves the best performance. The window size  $k$  when seq achieves the best performance will be chosen to set the document-length of the graph aggregator.
B Quantifying and Understanding Dataset Bias
In this section, we will supplement some analyses related to Sec. 5.3.
B.1 Data-level Measure
Tab. 5 gives the data-level measure  $\zeta_{p}$  in seven (four) datasets with respect to eight (seven) attributes in NER (CWS) task. The data-level measure  $\zeta_{p}$  will be used to compute the correlation measure in Sec. 5.3.
Tab. 6 displays (using spider charts) measure  $\zeta_{p}$  of seven datasets with respect to diverse attributes, and correlation measure  $\rho$  in NER task. We have given a detail analysis on seq and cPre-seq on the main text, here, we will provide the suggestion for choosing the datasets for bow and graph aggregator.
(1) regarding bow aggregator, it negatively correlated with  $\zeta_{tCon}$  and  $\zeta_{eDen}$  with larger correlation values. Therefore, bow aggregator is more appropriate to deal with datasets WB, TC, BC, since these four datasets are with lower value of  $\zeta_p$  with respect to the attribute tCon (TC, WB) and eDen (BC, WB, TC). Additionally, bow aggregator obtained the highest positive correlation with  $\zeta_{tFre}$ ,
$\zeta_{eLen}$ , and  $\zeta_{sLen}$ . Besides,  $\zeta_{tFre}(\mathsf{MZ})$ ,  $\zeta_{eLen}(\mathsf{NW})$ , and  $\zeta_{sLen}(\mathsf{NW})$ , also achieved the highest value, suggesting that bow aggregator is suitable for MZ and NW.
(2) regarding graph aggregator, it negatively correlated with  $\zeta_{\mathrm{eCon}}$ ,  $\zeta_{\mathrm{eFre}}$ ,  $\zeta_{\mathrm{tFre}}$ , and  $\zeta_{\mathrm{eLen}}$ , with larger correlation values. Therefore, graph aggregator is more appropriate to deal with datasets WB, TC, NW, and CN03, since these four datasets are with lower value of  $\zeta_p$  with respect to the attribute eCon (TC, WB), eFre (NW, TC), tFre (CN03, WB), and eLen (CN03).
Tab. 7 illustrates the measures  $\zeta_{p}$  in four CWS datasets with respect to seven attributes (e.g., wCon) and correlation measure  $\rho$ . We can conduct similar analysis like NER for CWS. We highlight the suitable CWS datasets for each aggregator as follows:
- bow: NCC, and SXU.
- graph: PKU and CITYU.
- seq: SXU, and NCC.
- cPre-seq: CITYU, PKU, SXU.
Agg.	CITYU	NCC	SXU	PKU	CN03	BC	BN	MZ	WB	NW	TC	CN00	PTB
bow	5	6	2	2	2	5	4	2	10	2	5	2	4
graph	7	3	3	6	10	6	4	3	7	6	10	3	10
seq	7	3	3	6	10	6	4	3	7	6	10	3	10
cPre	7	3	3	6	10	7	4	3	7	6	10	2	10
Table 4: The window size  $k$  when the four larger-context aggregators achieve the final performance.
Task	Data	eCon	tCon	eFre	tFre	eLen	dOov	sLen	eDen
NER	CN03	0.485	0.514	0.109	0.017	1.436	0.067	13.4	0.232
BC	0.486	0.440	0.113	0.108	1.905	0.064	16.3	0.085
BN	0.627	0.552	0.147	0.089	1.623	0.004	19.5	0.148
MZ	0.496	0.487	0.129	0.119	1.832	0.015	22.9	0.124
WB	0.294	0.269	0.111	0.084	1.631	0.024	22.9	0.050
NW	0.567	0.512	0.083	0.108	2.015	0.014	26.1	0.179
TC	0.261	0.258	0.082	0.105	1.598	0.043	8.3	0.040
Task	Data	wCon	cCon	wFre	cFre	wLen	dOov	sLen
CWS	CITYU	0.763	0.285	1.834	0.489	1.634	0.010	62.400
NCC	0.743	0.274	3.856	1.016	1.546	0.021	64.400
SXU	0.781	0.293	3.832	1.005	1.590	0.020	69.700
PKU	0.777	0.292	1.765	0.466	1.615	0.019	59.200
Table 5: The data-level measure  ${\zeta }_{p}$  in seven (four) datasets with respect to eight (seven) attributes in NER (CWS) task. The value of wFre and cFre on CWS task needs to multiply by  ${10}^{-7}$  .
Attr.	eCon	tCon	eFre	tFre	eLen	dOov	sLen	eDen
ζp	BN MZ CW NB NC03 TC	BN MZ CW NC03 TC	BN MZ CW NC03 TC	BN MZ CW NC03 TC	BN MZ CW NC03 TC	BN MZ CW NC03 TC	BN MZ CW NC03 TC	BN MZ CW NC03 TC
bow	-0.179	-0.607	0.143	0.396	0.643	-0.036	0.468	-0.571
graph	-0.571	-0.143	-0.393	-0.919	-0.643	0.286	-0.288	-0.107
ρ seq	-0.643	-0.857	-0.429	0.162	0.143	0.071	0.180	-0.714
cPre	-0.714	-0.750	-0.571	-0.306	0.000	0.357	-0.180	-0.643
Table 6: Illustration of measures  $\zeta_p$  in seven datasets (CN03, TC, NW, WB, MZ, BN, BC) with respect to eight attributes (e.g., eCon) and correlation measure  $\rho$  in NER task. A higher absolute value  $\rho$  (e.g  $| - 0.714|$ ) represents the improvement of corresponding aggregator (e.g., seq) heavily correlate with corresponding attribute (e.g. eCon). The number with the highest absolute value of each column is colored by green. "cPre" represents "cPre-seq" and the value in grey denotes correlation value does not pass a significance test ( $p = 0.05$ ).
Attr.	wCon	cCon	wFre	cFre	wLen	dOov	sLen
ζp	NCC
SUX	CITYU
PKU	NCC
SUX	CITYU
PKU	NCC
SUX	CITYU
PKU	NCC
SUX	CITYU
PKU
ρ	bow	-0.657	-0.771	0.086	0.257	-0.600	0.319	-0.486
graph	-0.029	0.257	-0.543	-0.429	0.143	-0.319	-0.486
seq	0.086	-0.200	0.371	0.314	-0.257	0.464	0.257
cPre	0.580	0.493	-0.261	-0.203	-0.203	0.544	-0.232
Table 7: Illustration of measures  ${\zeta }_{p}$  in four datasets ( CITYU, NCC, SXU, PKU) with respect to seven attributes (e.g., wCon) and correlation measure  $\rho$  in CWS task. A higher absolute value  $\rho \left( {\mathrm{e}.\mathrm{g}\left| {-{0.657}}\right| }\right)$  represents the improvement of corresponding aggregator (e.g., bow) heavily correlate with corresponding attribute (e.g. wCon). The number with the highest absolute value of each column is colored by green. "cPre" represents "cPre-seq" and the value in grey denotes correlation value does not pass a significance test  $\left( {p = {0.05}}\right)$  .