SlowGuess/NAACL_2022_OCR huggingface dataset:
Petra Steiner. 2017. Merging the trees - building a morphological treebank for German from two resources. In Proceedings of the 16th International Workshop on Treebanks and Linguistic Theories, pages 146-160, Prague, Czech Republic.
Mariona Taule, M. Antonia Martí, and Marta Recasens. 2008. AnCora: Multilevel annotated corpora for Catalan and Spanish. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08), Marrakech, Morocco. European Language Resources Association (ELRA).
Clara Vania and Adam Lopez. 2017. From characters to words to in between: Do we capture morphology? In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2016-2027, Vancouver, Canada. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Hongzhi Xu, Jordan Kodner, Mitchell Marcus, and Charles Yang. 2020. Modeling morphological typology for unsupervised learning of language morphology. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6672-6681, Online. Association for Computational Linguistics.
Tim Zingler. 2018. Reduction without fusion: Grammaticalization and wordhood in turkish. *Folia Linguistica*, 52(2):415-447.
A Human Evaluation
A.1 Annotation Protocol
This study measures the translation quality of translations generated by a translation system. You are given a list of sentences where one column lists each word in the gold standard (correct) translation and the corresponding column the system-generated translations. The evaluation of the translations will rely on the two scores described below. The scores to use in the evaluation are:
Semantic score evaluates the meaning of the word used in the automatic translation (system output) and how it compares with the gold standard translation.
Please assign each word in the output one of the scores you find most appropriate:
1. There is no relationship between the two lemmas
2. The lemmas are different but the translation does not fit well in the context
3. The lemmas are different but it is still an acceptable translation (e.g. synonym)
4. It is the same lemma
Grammar score evaluates the grammatical form and how it compares with the gold standard translation.
1. The word is inflected in a different way and it is not necessarily correct
2. The word has different inflection but it is still grammatically correct
3. The words have the same inflection, and it is correct
Please annotate all words in the translations in the file shared with you. In your evaluation try assigning the two scores to each word independently. The inflection of the word measures the morphological feature and should also be evaluated independently from the analyzer output which is automated and may contain errors.
The file contains example annotations for your reference, please ask any questions related to unresolved annotation examples by contacting the project coordinators.
A.2 Annotators
For both Turkish and Spanish, the annotators were contacted directly due to their expertise in morphology (both of them are PhD students in Linguistics and Computational Linguistics, respectively), besides requiring that they are native speakers of the target languages. Also, they were paid more than the minimum wage per hour of annotation of their country of residence, and were told that the annotated data will be released upon acceptance of the study.
B Segment-level Analysis of Synthesis and Fusion
B.1 List of machine translation systems
- EnTr1: the same system used in §4.2
- EnTr2: Transformer-base model (Vaswani et al., 2017) with joint vocabulary size of 8k pieces (unigram language modelling from SentencePiece (Kudo and Richardson, 2018), and trained with a sample  $(10\%)$  of the corpus of EnTr1.
Predictor	Description
char.count	number of characters
word.count	number of words (no punct. or numbers)
morph.count	number of morphemes.
synthesis	ratio of morph.count / word.count
N+V(word.count	number of Nouns and Verbs
N+V.morph.count	number of morphemes of the Nouns and Verbs
N+V.synthesis	ratio of N+V.morph.count / word.count
swEnTr1.count	number of subwords processed by the EnTr1 model
swEnTr2.count	number of subwords processed by the EnTr2 model
swTrEn2.count	number of subwords processed by the TrEn2 model
syn.swEnTr1	ratio of swEnTr1.count / word.count (synthesis proxy)
syn.swEnTr2	ratio of swEnTr1.count / word.count (synthesis proxy)
syn.swTrEn2	ratio of swEnTr1.count / word.count (synthesis proxy)
Table 4: List of predictors for En-Tr and Tr-En. All variables are computed on the Turkish segment of the evaluation set.
Predictor	Description
char.count	number of characters
word.count	number of words (no punct. or numbers)
verb.count	number of verbs
fusion	sum of the degree of fusion of all the verbs in the segment
R.fusion verb	ratio of fusion / verb.count
R.fusion(word	ratio of fusion / word.count
swEsEn1.count	number of subwords processed by the EsEn1 model
swEsEn2.count	number of subwords processed by the EsEn2 model
R.fusion.swEsEn1	ratio of fusion / swEsEn1.count
R.fusion.swEsEn2	ratio of fusion / swEsEn2.count
swEsEn1.count	number of subwords processed by the EnEs1 model
swEsEn2.count	number of subwords processed by the EnEs2 model
R.fusion.swEnEs1	ratio of fusion / swEnEs1.count
R.fusion.swEnEs2	ratio of fusion / swEnEs2.count
Table 5: List of predictors for En-Es and Es-En. All variables are computed on the Spanish segment of the evaluation set.
- EnEs1: the same system used in §4.3
- EsEn1: similar configuration than EnEs1 but in the opposite direction
- EnEs2: same configuration as EnEs1 (model and vocabulary) but with smaller training data. It uses only newscommentary-v8 data, with around 300k sentences).
- EsEn2: similar configuration than EnEs2 but in the opposite direction.
B.2 List of predictors
Tables 4, 5 and 6 describes all the predictors used at the segment level analysis of English-Turkish, English-Spanish and English-German (both directions), respectively.
B.3 Results on English-German
Figure 8 shows the analogous results for English to German, where the synthesis-based variables presents a high impact w.r.t. the other predictors.
Figure 8: Overview of significant predictors for degree of synthesis across EN-DE models.
Predictor	Description
src.char.count	number of characters in the source side
ref.char.count	number of characters in the target side
src(word.count	number of words in the source side
ref(word.count	number of words in the target side
src.uniLM.count	number of subwords obtained by uniLM in the source
ref.uniLM.count	number of subwords obtained by uniLM in the target
src.SYN.uniLM	synthesis in source = src.uniLM.count / src(word.count
ref.SYN.uniLM	synthesis in target = ref.uniLM.count / ref(word.count
src.mfrsr.count	number of subwords obtained by Morfessor in the source
ref.mfrsr.count	number of subwords obtained by Morfessor in the target
src.SYN.mfrsr	synthesis in source = src.mfrsr.count / src(word.count
ref.SYN.mfrsr	synthesis in target = ref.mfrsr.count / ref(word.count
src.PtrNet.count	number of subwords obtained by PtrNet in the source
ref.PtrNet.count	number of subwords obtained by PtrNet in the target
src.SYN.PtrNet	synthesis in source = src.PtrNet.count / src(word.count
ref.SYN.PtrNet	synthesis in target = ref.PtrNet.count / ref(word.count
srcfeat.count	number of morph. features in the source (using spAcy)
src.R feat_token	ratio of src feat.count / src(word.count
ref feat.count	number of morph. features in the target (using spAcy)
ref.R feat_token	ratio of ref feat.count / ref(word.count
src-ref.mean/min	src.mean/min minus ref.mean/min
src-ref.R.mean/min	src.R.mean/min minus ref.R.mean/min
src-ref.R.mean/min	ref.R.mean/min minus src.R.mean/min
src-ref.R.mean/min	ref.R.mean/min minus src.R.mean/min
Table 6: List of predictors for En-De and De-En. Variables are computed either on source (src) or target (ref) side.