SlowGuess/ACL_2020_OCR huggingface dataset:
Hansen K. Lars. and Salamon Peter. 1990. Neural network ensembles. IEEE Trans. Pattern Anal. Mach. Intell., pages 993-1001.
Peters Matthew, Neumann Mark, Iyyer Mohit, Gardner Matt, Clark Christopher, Lee Kenton, and Zettlemoyer Luke. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, (NAACL).
Johnson Melvin, Schuster Mike, Le V. Quoc, Krikun Maxim, Wu Yonghui, Chen Zhifeng, Thorat Nikhil, Viégas Fernanda, Wattenberg Martin, Corrado Greg, Hughes Macduff, and Dean Jeffrey. 2017. Google's multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics, 5:339-351.
David W. Opitz and Jude W. Shavlik. 1996. Actively searching for an effective neural network ensemble. Connect. Sci., 8:337-354.
Sennrich Rico, Haddow Barry, and Birch Alexandra. 2016. Controlling politeness in neural machine translation via side constraints. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, (NAACL), pages 35-40, San Diego, California.
Erik TK. Sang and Fien D. Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning, (NAACL), pages 142-147.
Erik TK. Sang and Buchholz Sabine. 2000. Introduction to the CoNLL-2000 shared task chunking. In Fourth Conference on Computational Natural Language Learning and the Second Learning Language in Logic Workshop.
Andrew M. Saxe, James L. McClelland, and Surya Ganguli. 2013. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15:1929-1958.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. CoRR, abs/1706.03762.
Lewis D. Davidand Yang Yiming, Rose G. Tony, and Li Fan. 2004. Rcv1: A new benchmark collection for text categorization research. J. Mach. Learn. Res., 5:361-397.
Jinhua Zhu, Yingce Xia, Lijun Wu, Di He, Tao Qin, Wengang Zhou, Houqiang Li, and Tieyan Liu. 2020. Incorporating bert into neural machine translation. In International Conference on Learning Representations, (ICLR).
A Hyper-parameters and Ensemble Strategy
Text Classification	Sequence Labeling
TFM:GLOVE	TFM:BERT	TFM:ELMO
Embedding dimension	200	768	256
Hidden dimension	200	768	256
Number of layers	6	6	6
Number of attention heads	8	8	8
Frozen vectors	GloVe 200	BERT-Large	ELMo 1024
-	-	0.5(Emb)
0.2 (Residual)	0.5 (Residual)	0.2 (Residual)
Dropout	0.1 (Attention)	-	0.1 (Attention)
-	-	0.1 (FF)
Label smoothing	0.1	0.1	-
Optimizer	Adam	Adam	Adam
Initial learning rate	0.0001	0.0001	0.0001
Batch size	64	128	32
Gradient Clipping	1.0	1.0	5.0
Aggregation Strategy	Averaging	Averaging	Voting
Sampling strategy	Normal & Bootstrapping	Normal & Bootstrapping	Normal
Table 5: Hyper-parameters and ensemble strategies for SINGLE, NORMALENS and SINGLEENS. For TFM:BERT, we followed the model architecture of Zhu et al. (2020). For TFM:ELMO on sequence labeling, we referenced the architecture of Matthew et al. (2018) with replacing the encoder with Transformer. It should be noted that for TFM:ELMO, we add Linear  $\rightarrow$  Relu  $\rightarrow$  LayerNorm between embedding and self-attention.
Text Classification	Sequence Labeling
TFM:GLOVE	TFM:BERT	TFM:ELMO
Embedding dimension	50	64	370
Hidden dimension	50	64	128
Frozen vectors	GloVe 50	BERT-Base	ELMo 256
Number of layers	3	3	4
Number of attention heads	10	8	8
Feed forward dimension	128	128	128
Aggregation Strategy	Averaging	Averaging	Voting
Sampling strategy	Normal	Normal	Normal
B Data Statistics
Table 6: Hyper-parameters and ensemble strategies for  $1 / \mathrm{K}$  ENS. The other values are same as Table 5. It should be noted that we ensemble  $K$  models of each sub model for final prediction.
Task	Dataset	Train	Valid	Test
Text Classification	IMDB	21,246	3,754	25,000
Rotten	8,636	960	1,066
RCV 1	14,007	1,557	49,838
Sequence Labeling	CoNLL-2003	14,987	3,466	3,684
CoNLL-2000	8,926	2,012	2,012
Table 7: Summary of the datasets. The values are the number of sentences contained in each dataset.