SlowGuess/NAACL_2021_OCR huggingface dataset:
Loic Vial, Benjamin Lecouteux, and Didier Schwab. 2019. Sense vocabulary compression through the semantic knowledge of WordNet for neural word sense disambiguation. In Proceedings of the 10th Global Wordnet Conference, pages 108-117, Wroclaw, Poland. Global Wordnet Association.
Warren Weaver. 1949. *Translation. In Machine Translation of Languages: Fourteen Essays* (written in 1949, published in 1955), pages 15-23, New York, NY. W. N. Locke and A. D. Booth, Eds. Technology Press of MIT, Cambridge, MA, and John Wiley & Sons.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, pages 5753-5763.
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: Enhanced language representation with informative entities. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguis
tics, pages 1441-1451, Florence, Italy. Association for Computational Linguistics.
A Transformer Architectures
In this Section we show the results attained when using RoBERTlarge (Liu et al., 2019) and XLNetlarge (Yang et al., 2019) as pretrained model for EsCHER. We note that 73 training examples out of 226,036 could not fit in 512 bpes, i.e., the maximum input length for both models, and we therefore discard them. In Table 6, we report the results on the English WSD evaluation framework of Raganato et al. (2017a) for EsCHER initialized with the aforementioned models along with its performance when using BARTlarge.
ESCHER Transformer	Model Parameters	SE07	ALL
RoBERTlarge	355M	76.0	80.5
XLNetlarge	340M	76.2	80.6
BARTlarge	406M	76.3	80.7
Table 6: EsCHER results on ALL when initialized with different transformers.
We use  $\mathrm{BART}_{\text{large}}$  as transformer architecture which consists of 12 encoder layers and 12 decoder layers with 1024 hidden size. We train the model with a constant learning rate of 0.00001, Rectified Adam as optimizer and a batch size of 700 tokens. We accumulate the gradient for 20 steps and clip it at 10. The model is trained for a maximum of 300,000 steps. We compute the F1 score on the validation dataset every 2000 steps and stop the training if the model does not improve for 15 consecutive tests (30,000 steps). The whole training is done with half precision and an amp-level of O1. It is worth noting that the training of EsCHER on SemCor (Miller et al., 1990) took less than 5 hours on a GeForce RTX 2080ti.