SlowGuess/NAACL_2022_OCR huggingface dataset:
a convolution kernel being able to capture nonlocal interactions in an EE (i.e., identical first and third words, and similar second and fourth words), whereas the linear nature of an LSTM encoder becomes restrictive in this task.
When half of the EEs in the form of  $AB_{1}AB_{2}$  are changed to  $AB_{2}AB_{1}$  and their tags are modified to B-fake and I-fake (swap clf. rows in the table), the model is still able to achieve high F1 scores that are only slightly lower than the unswapped counterpart, even though the B and I tags have split into two types. The fact that increasing the number of classes does not degrade the performance very much suggests that the model can learn to distinguish attested and unattested orderings very well. To quantify the model's ability to learn Hmong EE ordering, we calculate an in-context classification accuracy by examining how many correctly identi-
Word vectors are trained using Word2Vec (Mikolov et al., 2013). We find that SkipGram outperforms CBOw on this task, hence all results reported are SkipGram embeddings.  $\alpha$  is determined by grid search and we find that  $\alpha = 0.4$  works best.
fied EEs also have a correct prediction in whether the order has been swapped. We find that the incontext classification accuracy is  $99.1\%$  for LSTM and  $99.5\%$  for CNN, which are both exceptionally high. Note that this analysis excludes EEs that are not correct identified (both false positives and false negatives). Full confusion matrices are shown in Appendix C.
Interestingly, we find that adding phoneme level features to the input of either LSTM or CNN does not improve the performance in both the swapped and un swapped cases. This result is in contrast with other similar sequence tagging tasks (e.g., NER), where character level features are found to improve performance (Yang et al., 2018; Kuru et al., 2016). More importantly, this result presents a contrast to the robust phonological patterns found in the previous section, as it demonstrates that the model is able to tag elaborate expressions and classify their orders successfully without any reference to phonology. This suggests that the ordering  $(B_{1}, B_{2})$  can be predicted not only via phonology, but also via word-level features through the embeddings trained with the tagging model.
Visualization of Word Embeddings It is a rather perplexing result that a tagging model can learn the ordering of EEs via word embeddings only. Figure 1 shows the UMAP projection (McInnes et al., 2018) of two types of learned embeddings into 2D space. Embeddings from the tagging model show clear separation between words that tend to occur first in an EE (in the  $B_{1}$  position) and words that tend to occur second, where as embeddings trained separately on the SkipGram algorithm (Mikolov et al., 2013) show no separation. This suggests that the learned separation is unique to the tagging model. However, there is no way for the model to memorize EEs from the training set, since the test set contains non-overlapping EEs. How, then, would the tagging model learn what words tend to occur first and what words tend to occur second in an EE?
It appears that the model is able to learn the order of  $B_{1}$  and  $B_{2}$  from the occurrences of these component words in the training set. For an EE  $AB_{1}AB_{2}$  in the test set, although  $AB_{1}AB_{2}$  itself never appears in the training set,  $B_{1}$  and  $B_{2}$  do appear either as a coordinate compound  $B_{1}B_{2} / B_{2}B_{1}$ , or as parts
Figure 1: UMAP projection of word embeddings from neural sequence tagger (top) and separately trained SkipGram (bottom). Each circle is one Hmong word. The pie indicates proportion of times the word occurs first or second. Size of the dot indicates frequency in list of EEs.
of another EE  $XB_{1}XB_{2} / XB_{2}XB_{1}$  in the training set. As shown in Figure 2, appearances of them in the same order greatly outnumber those of the reversed order. As a result, the model may be able to learn which words tend to be  $B_{1}$  or  $B_{2}$  from these distributional properties of the EE words.
To further isolate this effect, we perform an experiment where the train/test splits are made so that even component words do no overlap between them (so that the box plot in Figure 2 would be completely empty). We confirm that the tagging performance drop considerably in this setting, with an average F1 score (59.78) unable to beat the strongest baseline. However, even in this setting, we do not find phoneme features to contribute to the tagging performance in a statistically signifi
<sup>8</sup>We also tried using character features or using only tones (the focal constituent for Hmong), but they were equally ineffective.
Occurrence of Test  $B_{1}B_{2}$  in training corpus
Figure 2: Occurrence of  $B_{1}$  and  $B_{2}$  words in test set EEs  $(AB_{1}AB_{2})$  in the training set as different forms  $(X \neq A)$ . Forms with the same ordering (blue and green) outnumber forms with the reversed ordering (orange and red)
cant way (F1=61.16, one-sided Wilcoxon signed-rank test  $p = 0.28$ ).
5.3 Learning Hmong EE Ordering with Classifiers over Word Vectors
Experiment To further investigate to what extent word embeddings determine the order of  $(B_{1}, B_{2})$  in Hmong EEs, we revisit the out-of-context classification experiment presented in section 5.1, this time adding word vector features. We experiment with both SkipGram embeddings and embeddings extracted from the CNN sequence tagging model (without swapping). Embeddings from the tagging model are expected to perform better on the classification task, since they are optimized to detect words contained in EEs in the attested order. On the other hand, embeddings separately trained via SkipGram are more "pure," as they only capture the distributional semantics of the words without additional information.
Results Table 6 shows the classification accuracies using word embedding features, as well as word embedding combined with one-hot phoneme features. We observe that embeddings trained with the tagger indeed perform better than those trained via SkipGram. What is surprising is that using embedding features from the tagger alone produces a classification accuracy comparable to using all phonemes (88%). Moreover, an even higher accuracy can be achieved by combining phoneme features with embeddings from the tagger. This suggests that EE ordering in Hmong can be pre
Data	All	Unique
focal constituent (tone)	89.1%	85.4%
all constituents	96.7%	88.3%
wv-sg	94.4%	71.1%
wv-tagsger	96.4%	88.3%
all constituents + wv-sg	96.6%	88.8%
all constituents + wv-tagsger	97.1%	93.8%
Table 6: Classification accuracies for Hmong using SVM with phoneme and word embedding features. First two rows are from Table 3.  $w\nu -sg$ : separately trained skipgram embeddings.  $w\nu -tagger$ : embeddings from the CNN sequence tagging model
dicted from two independent but mutually reinforcing routes, namely phonology and lexical distribution. Either method alone is a good predictor of the ordering, but combining the two achieves the best accuracy, because the two routes each offer additional information that are important in predicting the ordering of Hmong EEs.
Visualization of Feature Importance With a total 360 features from both phonemes and word vectors<sup>9</sup>, we can visualize which features the model find the most important in this classification task by examining the weights learned by the model<sup>10</sup>.
Figure 3 shows the proportion of feature types that have the highest importance when varying the number of features  $(k)$ . We see that when  $k$  is small, the model overwhelmingly uses phoneme features (especially tones) to perform classification. The test accuracy is impressively  $84\%$  with only 12 features – nearly  $40\%$  of which are tonal features. As  $k$  increases, word embedding features start to gain importance, and the test accuracy can be further improved when word embeddings are incorporated. By the time when  $k$  reaches 200, the proportion of each feature type becomes similar to the natural proportion before selection (dashed lines).
In this paper, we set out to explore the ways that the order of words in EEs and CCs in Hmong, Lahu, and Chinese can be learned by computational models. Motivated by earlier linguists' find
958 onsets, 14 rhymes, 8 tones, and 100 word vector dimensions for each of the two words  $B_{1}$ $B_{2}$ .
<sup>10</sup>We switch to an SVM with linear kernel to compare the importance for each feature directly
Figure 3: Proportion of feature types for the top k features ranked by importance weight.  $x$ -axis on the bottom shows the number of features to use on a log scale, while  $x$ -axis on the top shows the test accuracy for a linear SVM trained on the top  $k$  features. Dashed lines show the natural proportion of the tone, rhyme, onset and word vector features. Experiment was done with the Unique  $(B_1, B_2)$  words.
ings, we first use phonological features alone to discriminate between attested and unattested orders of words. We find that in the case of all three languages, the order of words can indeed be predicted phonologically, and the "phonetically unnatural" hierarchies do predict the ordering of EEs and CCs. Furthermore, a decision tree classifier is able to learn more-or-less the same hierarchies, suggesting that speakers of those languages could in principle learn the linear hierarchies through exposure to the language, and use these hierarchies to decide on the correct order of words in EEs and CCs. These findings provide positive evidence for hypotheses 1-3 from Section 3. We then explored the ways models can utilize context and distributional patterns of words to learn the orders in the sequence tagging experiments, and we were not able to find evidence for hypothesis 4. We were surprised to find that models can perform well using only word features, and that adding phonemes to the feature set does not help at all.
The seemingly contradictory results of our investigation point in an interesting direction. Information on which a model could rely to learn the ordering of these constructions is present redundantly in phonology (on the one hand) and in lexical and distributional patterns (on the other). When allowed to cooperate on a level playing field, embeddings and phonology-based features both con
tribute to the identification of well-formed EEs at a similar level. In other words, while it is possible that language users may use phonological hierarchies like those proposed in Mortensen (2006) to select appropriate orders for EEs and CCs, it is clearly not the case that they must (though they will perform a bit better if they do). These phonological hierarchies may have been more order-defining in the history of the languages, but as the sequence tagging experiments have suggested, they may also have become fossilized in the lexicon and in distributional patterns in the modern form. Many times, a  $(B_{1}, B_{2})$  pair appears abundantly in multiples EEs (as  $XB_{1}XB_{2}$ ), as a CC (as  $B_{1}B_{2}$ ), or in other—more complicated—discourse patterns in the same order, so that language users could learn whether a given word tends to appear in the  $B_{1}$  or  $B_{2}$  position. If a tagging model can learn a word representation that distinguishes between  $B_{1}$  and  $B_{2}$ , language users may do the same.
In a sense, these results should be pleasing to both the "structure" (Mortensen, Hale, Reiss) and the "substance" (Hayes, Flemming, Steriade) camps. They show, once again, that generalizations about sounds can be robust but phonetically arbitrary. However, they leave open the possibility that the relevant synchronic generalizations are not actually phonological.
We have shown two independent routes, namely phonology and lexical distribution, by which computational methods can predict the order of words in Hmong EEs. A language user could probably do the same, relying on both routes to some degree when they need to select the order of words in EEs. However, there is no way to know for sure without conducting a psycholinguistic experiment with native speakers, which would shed light on whether any of the modeling actually translates to human cognition. The Chinese and Lahu cases also raise interesting questions for future work: does the same two-route mechanism work for EEs and CCs in these languages as well? Answering this question will require additional data collection and annotation, but will shed significant light on this theoretically important issue.
Anonymous. 2016. Xiandai Hanyu Cidian. Commercial Press, Beijing.
Elizabeth M. Banker. 1964. Bahnar reduplication. Mon-Khmer Studies, 1:119-134.
Michael Becker, Nihan Ketrez, and Andrew Nevins. 2011. The surfeit of the stimulus: Analytic biases filter lexical statistics in turkish laryngeal alternations. Language, pages 84-125.
Sarah Bunin Benor and Roger Levy. 2006. The chicken or the egg? a probabilistic analysis of english binomials. Language, 82(2):233-278.
Joan Bresnan, Ash Asudeh, Ida Toivonen, and Stephen Wechsler. 2015. Lexical-functional syntax. John Wiley & Sons.
Noam Chomsky. 1965. Aspects of the Theory of Syntax, volume 11. MIT press.
Noam Chomsky. 1981. Lectures on Government and Binding: The Pisa Lectures. Studies in Generative Grammar. de Gruyter.
Noam Chomsky. 1995. The Minimalist Program. MIT Press.
Noam Chomsky and Morris Halle. 1968. The Sound Pattern of English. ERIC.
Qingxia Dai. 1986. Jingpo yu binglie jiegou fuheci de yuanyin hexie. Minzu Yuwen, 1986(5):23-29.
Patricia Donegan and David Stampe. 2009. Hypotheses of natural phonology. *Poznan Studies in Contemporary Linguistics*, 45(1):1-31.
Patricia J Donegan and David Stampe. 1979. The study of natural phonology. Current approaches to phonological theory, 126173.
David Filbeck. 1996. Couplets and duplication in Mal. Mon-Khmer Studies, 26:91-106.
Edward S Flemming. 2013. Auditory representations in phonology. Routledge.
Danny Fox and David Pesetsky. 2005. Cyclic linearization of syntactic structure. Theoretical Linguistics, 31(1-2):1-45.
Eric C Fudge. 1967. The nature of phonological primes. Journal of Linguistics, 3(1):1-36.
Mark Hale and Charles Reiss. 2000. "substance abuse" and "dysfunctionalism": current trends in phonology. Linguistic inquiry, 31(1):157-169.
Mark Hale and Charles Reiss. 2008. The phonological enterprise. Oxford University Press.
William J. Hanna. 2013. Elaborate expressions in Dai Lue. Linguistics of the Tibeto-Burman Area, 36(1):33-56.
Bruce Hayes. 2011. Introductory phonology, volume 32. John Wiley & Sons.
Bruce Hayes, Péter Siptár, Kie Zuraw, and Zsuzsa Londe. 2009. Natural and unnatural constraints in hungarian vowel harmony. Language, pages 822-863.
Bruce Hayes and James White. 2013. Phonological Naturalness and Phonotactic Learning. Linguistic Inquiry, 44(1):45-75.
Larry M Hyman. 1970. How concrete is phonology? Language, pages 58-76.
Roman Jakobson, C Gunnar Fant, and Morris Halle. 1951. Preliminaries to speech analysis: The distinctive features and their correlates. MIT press.
Brenda Johns and David Strecker. 1987. Lexical and phonological sources of Hmong elaborate expressions. Linguistics of the Tibet-O-Burman Area, 10(2):106-112.
Ronald Kaplan and Joan Bresnan. 1982. Lexical-functional grammar: A formal system for grammatical representation. In Joan Bresnan, editor, The Mental Representation of Grammatical Relations, chapter 4, pages 173-281. MIT Press.
Onur Kuru, Ozan Arkan Can, and Deniz Yuret. 2016. CharNER: Character-level named entity recognition. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 911-921, Osaka, Japan. The COLING 2016 Organizing Committee.
Nahyun Kwon and Keiko Masuda. 2019. On the ordering of elements in ideophonic echo-words versus prosaic dvandva compounds, with special reference to korean and japanese. Journal of East Asian Linguistics, 28(1):29-53.
Rong Li. 1952. Qieyun Yinxi. Chinese Academy of Sciences, Beijing.
James A. Matisoff. 1973. The Grammar of Lahu. Number 75 in University of California Publications in Linguistics. University of California Press, Berkeley.
James A. Matisoff. 1989. The Dictionary of Lahu. University of California Press, Berkeley and Los Angeles.
James A Matisoff. 2006. English-Lahu Lexicon. University of California Press, Berkeley and Los Angeles.
Leland McInnes, John Healy, Nathaniel Saul, and Lukas Großberger. 2018. Umap: Uniform manifold approximation and projection. Journal of Open Source Software, 3(29):861.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc.
Elliott Moreton and Joe Pater. 2012a. Structure and substance in artificial-phonology learning, part i: Structure. Language and Linguistics Compass, 6(11):686-701.
Elliott Moreton and Joe Pater. 2012b. Structure and substance in artificial-phonology learning, part ii: Substance. Language and Linguistics Compass, 6(11):702-718.
Emily Morgan and Roger Levy. 2016. Abstract knowledge versus direct experience in processing binomial expressions. Cognition, 157:384-402.
David R. Mortensen. 2006. Logical and Substantive Scales in Phonology. Ph.D. thesis, University of California, Berkeley.
David R. Mortensen, Patrick Littell, Akash Bharadwaj, Kartik Goyal, Chris Dyer, and Lori S. Levin. 2016. Panphon: A resource for mapping IPA segments to articulatory feature vectors. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3475-3484. ACL.
David R. Mortensen, Xinyu Zhang, Chenxuan Cui, and Katherine J. Zhang. 2022. A hmong corpus with elaborate expression annotations. In Language Resources and Evaluation Conference (LREC), Marseille, France.
Tina Obermüller. 2015. Co-Compounds vs. Coordinate Compounds: A formal, functional and semantic analysis. AV Akademikerverlag, Saarbrücken.
Yuanen Pan and Cuiyin Cao. 1972. Four-syllable coordinative constructions in the Miao languages of eastern Kweichow. In Herbert C. Purnell, editor, Miao and Yao Linguistic Studies, number 88 in Data Papers, pages 211-234. Cornell University Southeast Asia Program, Ithaca, New York.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830.
John Robert Ross. 1967. Constraints on variables in syntax. Ph.D. thesis, Massachusetts Institute of Technology.
Stephanie S Shih and Kie Zuraw. 2017. Phonological conditions on variable adjective and noun word order in tagalog. Language, 93(4):e317-e352.
Donca Steriade et al. 2001. Directional asymmetries in place assimilation: A perceptual account. The role of speech perception in phonology, pages 219-250.
Pa-Hsin Ting. 1975. Tonal relationship between the two consistent of the coordinate construction in the anaects, the meng-tze, and the book of odes.
Bulletin of the Institute of History and Philology, Academia Sinica, 47(1):17-52.
Thomas Wasow and Jennifer Arnold. 2003. Post-verbal constituent ordering in English, pages 119-154. De Gruyter Mouton.
Richard L. Watson. 1966. Reduplication in Pacoh. Master's thesis, The Hartford Seminary Foundation.
Julian Wheatley. 1982. *Burmese: A Grammatical Sketch*. Ph.D. thesis, University of California, Berkeley.
Jie Yang, Shuailong Liang, and Yue Zhang. 2018. Design challenges and misconceptions in neural sequence labeling. In Proceedings of the 27th International Conference on Computational Linguistics (COLING).
Jie Yang and Yue Zhang. 2018. Ncrf++: An opensource neural sequence labeling toolkit. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics.
A Trained Decision Trees
Figures 4, 5, 6 show what the trained decision tree looks like in the three languages. In each tree node, the top half of the box shows the current majority class, attested (ATT) or unattested (FAKE), as well as the number of votes. The bottom half of the box shows the variable to branch on. As noted in the main text, a linear ordering can be induced from the tree by following the branches. Take the Hmong tree (Figure 4) as an example. The first factor to split on is whether  $B_{1}$  has the  $j$  tone, and if the answer is yes, the majority of words are attested (255 attested vs 15 unattested/fake). This suggests that  $j$  has a strong tendency to occur in the  $B_{1}$  position, since it is the most distinguishing factor to split on. Hence  $j$  can be placed as the first tone on the scale. If  $B_{1}$  does not have the  $j$  tone, the next question to ask is whether  $B_{1}$  has the  $b$  tone. Since a yes answer again leads to majority attested words (273 attested vs 61 unattested/fake),  $b$  can be placed second on the scale. The next three questions to ask concern with the  $B_{2}$  word. Since a yes answer leads to attested words in all three cases, it suggests that  $\varnothing$ ,  $g$  and  $s$  have a tendency to appear in the  $B_{2}$  position, hence they can be placed on the end of the scale in that order. The next two factors concern with the  $j$  and  $b$  tones, which have already been placed on the scale, so we skip them. This process of following the left child (the no branch) and placing tones at either end of the scale is repeatedly applied, yielding the induced linear scales shown in Table 4.
Figure 4: Decision tree trained on Hmong elaborate expressions predicts the following order of tones:  $\mathrm{j} < \mathrm{b} < \mathrm{m} < \mathrm{v} < \mathrm{s} < \mathrm{g} < \varnothing$ .
Figure 5: Decision tree trained on Lahu elaborate expressions predicts the following order of rhymes: o < u
$$
\prec \dots \prec e \prec o \prec \varepsilon \prec a.
$$
Middle Chinese Tones
Figure 6: Decision tree trained on Middle Chinese coordinate compounds predicts the following order of tones: ping  $(\varnothing)\prec$  shang (X)  $\prec$  qu (H)  $\prec$  ru (p, t, k)
The Hmong corpus consists of 740k sentences with a positive rate of around  $3.1\%$  (i.e.  $96.9\%$  of sentences contain no EEs). The EEs are randomly split into disjoint train and val/test sets with approximate ratios of  $91\% / 4.5\% / 4.5\%$ . To reduce the possibility that certain splits are easier than others, three such splits are independently produced. The positive sentences are split into train and val/test sets according to the EE partitions, and the negative sentences are split with approximate ratios of  $91\% / 4.5\% / 4.5\%$ .
The sequence tagging model consists of a feature extractor followed by a fully connected layer to predict the tags:  $\{B,I,0\}$  in the unswapped case and  $\{B,B\text{-face},I,I\text{-face},0\}$  in the swapped classification experiments. Two feature extractors are used: 1) an LSTM with bidirectional encoding, and 2) a CNN, consisting of four layers of 1D convolution, ReLU, Dropout, and BatchNorm. When character or phoneme level features are used, the character embeddings go through a CharCNN before being concatenated with the word embedding. Details on model configuration is shown in Table 7. The LSTM model contains approximately 1.4M parameters and the CNN contains approximately 1.7M parameters. Our code is based on NCRF++ (Yang and Zhang, 2018).
Hyperparameter	Value
Word embed dim	100
Char embed dim	30
LSTM hidden dim	100
CNN hidden dim	200
CNN kernel size	3
CharCNN hidden dim	50
CharCNN kernel size	3
Dropout probability	0.5
Table 7: Model configuration hyperparameters.
An extensive architecture search was not performed, because the purpose of the experiments is not to achieve the best performing model.
$^{12}$  https://github.com/jiesutd/NCRFpp, under Apache 2.0 License which permits use for research purposes.
B.3 Training and Decoding