SlowGuess/NAACL_2022_OCR huggingface dataset:
Shlomo Bentin and Ram Frost. 1987. Processing lexical ambiguity and visual word recognition in a deep orthography. Memory & Cognition, 15(1):13-23.
Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of "bias" in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454-5476, Online. Association for Computational Linguistics.
Ya'akov Gal. 2002. An HMM approach to vowel restoration in Arabic and Hebrew. In Proceedings of the ACL-02 Workshop on Computational Approaches to Semitic Languages, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Yoav Goldberg and Michael Elhadad. 2010. Easy-first dependency parsing of modern Hebrew. In Proceedings of the NAACL HLT 2010 First Workshop on Statistical Parsing of Morphologically-Rich Languages, pages 103-107, Los Angeles, CA, USA. Association for Computational Linguistics.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Dror Kamir, Naama Soreq, and Yoni Neeman. 2002. A comprehensive NLP system for modern standard Arabic and modern Hebrew. In Proceedings of the ACL-02 Workshop on Computational Approaches to Semitic Languages, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Leonid Kontorovich. 2001. Problems in Semitic NLP: Hebrew vocalization using HMMs. In *Problems in Semitic NLP*, NIPS Workshop on Machine Learning Methods for Text and Images.
Amit Moryossef, Roee Aharoni, and Yoav Goldberg. 2019. Filling gender & number gaps in neural machine translation with black-box context injection. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 49-54, Florence, Italy. Association for Computational Linguistics.
Hamdy Mubarak, Ahmed Abdelali, Hassan Sajjad, Younes Samih, and Kareem Darwish. 2019. Highly effective Arabic diacritization using sequence to sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2390-2395, Minneapolis, Minnesota. Association for Computational Linguistics.
Jakub Náplava, Milan Straka, and Jana Straková. 2021. Diacritics restoration using bert with analysis on czech language. arXiv preprint arXiv:2105.11408.
Jakub Náplava, Milan Straka, Pavel Stražák, and Jan Hajic. 2018. Diacritics restoration using neural networks. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Barbara Plank, Anders Søgaard, and Yoav Goldberg. 2016. Multilingual part-of-speech tagging with bidirectional long short-term memory models and auxiliary loss. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 412-418, Berlin, Germany. Association for Computational Linguistics.
Danny Shacham and Shuly Wintner. 2007. Morphological disambiguation of Hebrew: A case study in classifier combination. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 439-447, Prague, Czech Republic. Association for Computational Linguistics.
Avi Shmidman, Shaltiel Shmidman, Moshe Koppel, and Yoav Goldberg. 2020. Nakdan: Professional Hebrew diacritizer. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 197-203, Online. Association for Computational Linguistics.
Leslie N Smith. 2017. Cyclic learning rates for training neural networks. In 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 464-472. IEEE.
Lukas Stankevicius, Mantas Lukosevicius, Jurgita KAPOciute-Dzikiene, Monika Briediene, and Tomas Krilavicius. 2022. Correcting diacritics and typos with ByT5 transformer model. arXiv e-prints.
Eran Tomer. 2012. Automatic Hebrew Text Vocalization. Ben-Gurion University of the Negev, Faculty of Natural Sciences, Department of Computer Science.
Reut Tsarfaty, Shoval Sadde, Stav Klein, and Amit Seker. 2019. What's wrong with Hebrew NLP? and how to make it right. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations, pages 259-264, Hong Kong, China. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Nasser Zalmout and Nizar Habash. 2017. Don't throw those morphological analyzers away just yet: Neural morphological disambiguation for Arabic. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 704-713,
Copenhagen, Denmark. Association for Computational Linguistics.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2979–2989, Copenhagen, Denmark. Association for Computational Linguistics.
System	Dec	Dicta – reported / reproduced	New test set (§2.3)	OOV
CHA	WOR	VOC	DEC	CHA	WOR	VOC	WOR	VOC
Baselines
MAJMOD	84.93	75.94	68.10	69.63	88.04	81.22	76.14	77.10	N/A	N/A
MAJALL	91.67	86.29	79.43	81.19	93.79	90.01	84.87	86.19	N/A	N/A
Lexicalized
SNOPI	87.81	78.96 / 79.92	66.41 / 66.57	70.35	91.29	85.84	76.45	78.91	40.83	42.39
MORFIX	94.91	90.32 / 91.29	80.90 / 82.24	86.48	96.84	94.92	90.38	92.39	63.91	69.20
DICTA	97.53	95.12 / 95.71	88.23 / 89.23	90.66	97.95	96.77	94.11	94.92	76.21	77.66
Unlexicalized
\( NAKDIMON_0 \)	95.78	92.59	79.00	83.01	94.59	91.70	84.94	87.54	47.05	50.96
NAKDIMON	97.91	96.37	89.75	91.64	57.46	62.06
Table 5: Document-level macro % accuracy on the test set from Shmidman et al. (2020) and on our new test set. We cannot report our full NAKDIMON's performance on the former, as we use the test set for parts of its training. MAJALL is reported as MAJORITY in the main text; MAJMOD only considers text in the MODERN portion of our training set.
A Full Script Reconciliation
We apply the following resolution tactics for added letters in undotted text: (a) We almost never remove or add letters to the original text (unless it is completely undiacritizable). (b) We keep dagesh in letters that follow a shuruk which replaces a kubuts, and similarly for yod (hirik male replacing hirik haser). (c) When we have double vav or double yod, the second letter is usually left undotted, except when it is impossible to have the correct vocalization this way.
Resolving ktiv haser discrepancies from Morfix outputs is done by adding missing vowel letters, or removing superfluous vowel letters, in such a way that would not count as an error if it is correct according to Academy regulations.
B Development Experiments
We tried to further improve NAKDIMON by initializing its parameters from a language model trained to predict masked characters in a large undotted Wikipedia corpus (440MB,  $30\%$  mask rate), but were only able to achieve an improvement of  $0.07\%$ . Attempted architectural modifications, including substituting a Transformer (Vaswani et al., 2017) for the LSTM; adding a CRF layer to the decoding process; and adding a residual connection between the character LSTM layers, yielded no substantial benefits in these experiments. Similarly, varying the number of LSTM layers between 2 and 5 (keeping the total number of parameters roughly constant, close to the 5,313,223 parameters of our final model) had little to no impact on the accuracy on the validation set.
Figure 1: WOR error rate on validation set as a function of training set size vs. Dicta, over five runs. Other metrics show similar trends.
Figure 1 shows the favorable effect of training NAKDIMON over an increasing amount of MODERN text.
C Dicta Test Set
We present results for the Dicta test set in Table 5. In order to provide fair comparison and to preempt overfitting on this test data, we ran this test in a preliminary setup on a variant of NAKDIMON which was not tuned or otherwise unfairly trained. This system,  $\mathrm{NAKDIMON_0}$ , differs from our final variant in three main aspects: it is not trained on the Dicta portion of our training corpus (§2.2), it is not trained on the AUTOMATIC corpus, and it employs a residual connection between the two character Bi-LSTM layers. Testing on the Dicta test set required some minimal evaluation adaptations resulting from encoding constraints (for example, we do not distinguish between kamatz katan and kamatz gadol). Thus, we copy the results reported in
Shmidman et al. (2020) as well as our replication.
We see that the untuned  $\mathrm{NAKDIMON_0}$  performs on par with the proprietary Morfix, which uses word-level dictionary data, consistent with our main results on our novel test set.
We tuned hyperparameters and architecture over a held-out validation set of 40 documents with 27,681 tokens, on which Dicta performs at  $91.56\%$  WOR accuracy.
In our chosen setup, we train NAKDIMON over PRE-MODERN for a single epoch, followed by two epochs over the AUTOMATIC corpus, and then by three epochs over the MODERN corpus. We optimize using Adam (Kingma and Ba, 2014). For the PRE-MODERN corpus we use a cyclical learning rate schedule (Smith, 2017), varying linearly from  $3 \cdot 10^{-3}$  through  $8 \cdot 10^{-3}$  and down to  $10^{-4}$ , which we found to be more useful than a constant learning rate. For each of AUTOMATIC and MODERN corpora we use epoch-wise decreasing learning rate:  $(3 \cdot 10^{-3}, 10^{-3})$  and  $(10^{-3}, 10^{-3}, 3 \cdot 10^{-4})$  respectively. We set maximum chunk size to 80 characters, and use batch size of 128. We set both character embedding and LSTM hidden dimensions to 400, and apply a dropout rate of 0.1.