SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 3: POS induction performance on the 45-tag English Penn WSJ dataset. Numbers are the 5-run averages plus standard deviations. In the last row group, we include the numbers of baselines that have unspecified model selection procedures and no official code available (denoted by \*\*), or use a more carefully designed model selection method (denoted by \*)
representation over all the subwords and layers as the representation for each word. For all models, we train our model using Adam (Kingma and Ba, 2015) with an initial learning rate 0.001. The batch size is set to 80. The Gumbel softmax temperature is set to 2.0. The results on the Penn WSJ dataset are the mean of 5 runs, and the results on the universal treebank are the mean of 3 runs (see more details in Appendix C).
6 Results and Ablations
6.1 45-tag English Penn WSJ dataset.
The results are shown in Table 3. We reported two variants: the MPoSM model that does not use any external resource, and the MPoSM + emb model that uses the same pretrained word embeddings as He et al. (2018). Using pretrained embeddings does provide substantial improvements to our model. Overall, our model achieves competitive performance compared to SOTA models (Stratos, 2019; Gupta et al., 2022), reaching 78.6 oracle M-1. The oracle performance is 0.5 points higher
than the model in Stratos (2019) using a mutual information-based loss. Our fully unsupervised performance reaches 77.9 M-1, which is also similar to SOTA models (Stratos, 2019; Gupta et al., 2022), and is higher compared to previous models using the same pretrained embeddings (He et al., 2018) (75.6), models not using the Markov assumption (Abend et al., 2010) (75.1) or models using uni-directional long-term dependency (Tran et al., 2016) (75.0). Concurrent to our work, Gupta et al. (2022) achieve a higher M-1 of 79.5, but they use more additional resources, including mBERT representations and fastText (Joulin et al., 2017) morphological features.
6.2 12-tag Multilingual Results on Universal Treebank.
We also report results on all 10 languages on the universal treebank in Table 2 (the full table with standard deviations can be found in Table 6 of Appendix D). To make the settings practical to low-resource languages, we do not use any pretrained word embeddings on this dataset. Compared to the SOTA model (Stratos, 2019) that also does not use any external resources, our model achieves competitive performance, outperforming it on 4 out of 10 languages (es, fr, it, pt-br). Together with the English Penn WSJ dataset, we notice that MPoSM perform well on most of the languages, but may underperform the previous model on languages with weaker tag-level dependency (e.g., ko and id, statistics are in Table 1, more detailed analyses and discussions are in Appendix E and F) and on smaller datasets (e.g., ko and sv).
Concurrently, Gupta et al. (2022) showed substantial improvement on the universal treebank by leveraging knowledge in the pretrained mBERT representations. Inspired by their success, we also report the result using mBERT in MPoSM (as denoted by MPoSM + mBERT) in Table 2. Similarly,
en (Penn)	de (uni)	id (uni)	ko (uni)
MPoSM (full)	78.6 (±1.7)	71.8 (±2.5)	69.4 (±1.8)	55.2 (±1.3)
MPoSM (width=2)	77.3 (±0.3)	68.5 (±2.8)	70.0 (±1.0)	56.6 (±1.4)
Table 4: Oracle M-1 performance of different context types on the four different languages.
using mBERT substantially improves MPoSM's performance on the universal treebank. While on languages with weak tag-level dependency or smaller datasets, MPoSM + mBERT still does not perform most effectively (similar to the trend in MPoSM), MPoSM + mBERT achieves substantially higher results on most of the languages compared to MPoSM, and achieves results higher than Gupta et al. (2022) on French and Italian. On average, Gupta et al. (2022) still achieves higher results. This trend may imply that other factors (e.g. the clustering methods used in Gupta et al. (2022)) are important for their good performance. We have also tried using mBERT on the English WSJ dataset, but do not see a substantial improvement. We leave how to combine their method with MPoSM as a promising future direction.
6.3 How does modeling long context influence the results?
Taking advantage of the flexibility of our model, we analyze whether modeling long-term context is always better than modeling local context. We compare two models: the MPoSM (full) model is the default model described in Sec. 3, and the MPoSM (width=2) model that replaces the Bi-LSTM network with an MLP and only takes in the neighboring 4 predicted POS tags as the input (i.e., local context). We test our model on 4 languages: English, German, Indonesian, and Korean. These languages are selected to have representative statistics among all the languages in the universal treebank in terms of dataset size and average word frequency (see Table 1). The results are in Table 4. On English and German, the default model is better than the MPoSM (width=2) variant by 1.3 and 3.3 points respectively. However, on Indonesian and Korean, the trend is reversed with the MPoSM (width=2) variant showing 0.6 and 1.4 point advantage respectively. We notice that the languages do not benefit from using a longer context also correlates well with the languages with weak tag-level dependencies. Such property prevents the MPoSM from
Example 1 (for n1-v1): o1 o2 n1 o1 o2 o1 o2 v1  
Example 2 (for n2-v2): o1 o2 n2 o1 o2 o1 o2 v2
Figure 3: Illustration of the tag-level regex for sentences in D(2-4). D(0) sentences can be generated by removing the “ $\circ 1 \circ 2$ ” block between n and v.
benefit from the advantage of dependence modeling on those languages, and consequently using a longer context does not provide additional help. More detailed analysis is in Appendix E.
7 Analysis and Challenges
7.1 Agreement Learning Experiments
Inducing good POS tags requires models to understand what "agreement" is. To match the gold 45-tag set of Penn Treebank, the model needs to distinguish between VBP (Verb, non-3rd person singular present) and VBZ (Verb, 3rd person singular present) tags (see examples in Figure 1). Though local morphological features do provide useful cues for such classification, models should achieve better performance by observing the full picture of agreement in the long context. From the results in Sec. 6, we notice evaluation in real-life datasets may contain many confounding factors. Hence, we design a well-controlled synthetic dataset to examine exactly how well the model learns these agreements. Surprisingly, we find that the limitation of current models is not about leveraging long context, but a common fundamental limitation on using co-dependency to distinguish tags.
Controlled Dataset Design. To provide a simplified and well-controlled environment, our synthetic datasets consist of a small set of words and tags, and simple sentences. Specifically, we use 6 different tags, with 5 unique words correspond to each tag. Our 6 tags are named after nouns (n1, n2), verbs (v1, v2), and other unimportant tokens (o1, o2). In every sentence, n1 always appear before v1, and n2 before v2, analogous to subject-verb agreement in English. We create the synthetic data by first sampling a tag sequence (illustrated in Figure 3) and then randomly select words of each tag in the sequence. We also make sure the two agreements (n1-v1, n2-v2) have ex
<sup>7</sup>We choose language mainly according to the dataset statistics instead of linguistic properties as in preliminary experiments, those statistics are more influential to the performance.
D(0)	D(2-4)
MPoSM (width=1)	99.50 (95%)	87.19 (0%)
MPoSM (width=2)	92.99 (30%)	87.62 (0%)
MPoSM (full)	96.50 (75%)	95.31 (30%)
Stratos (2019) (width=2)	92.99 (30%)	86.56 (0%)
Tran et al. (2016) (full)	80.97 (0%)	82.50 (0%)
Table 5: Oracle M-1 on the synthetic datasets with the percentage of perfect runs (100 M-1) in the bracket.
actly symmetric data. We use the "distance", i.e., the number of tokens, between n and v to control the agreement-learning difficulty. The larger the distance is, the harder the example is. Therefore, we create two subsets with different levels of difficulty, and each contains 40,000 sentences. In the first simpler subset D(0), n and v are adjacent. In the second harder subset D(2-4), n and v are separated by 2-4 words. Complete illustrations, regexes and additional results are in the Appendix H.
Surprising Difficulty of Learning Agreement. Model performances on our synthetic datasets are in Table 5. We report the mean M-1 of 20 runs and the percentage of perfect runs (achieving  $100\mathrm{M - 1}$ ), as models are expected to consistently achieve the perfect score if they really acquire agreement. We include three variants of MPoSM using different contexts (from the width  $= 1$  model only considering the immediate neighboring tokens to the full model considering the whole sentence). We compare with two representative baselines: the SOTA model Stratos (2019) which uses the context with width  $= 2$ , and the neural HMM model (Tran et al., 2016) which leverages unidirectional full context. In Table 5, we first notice the surprising difficulty of learning agreement even in the simple D(0) setting, where the n and v are already adjacent. None of the models can consistently produce the perfect tags in this setting. The best results are from MPoSM with a specific inductive bias of only using the width  $= 1$  local context, but it still fails to achieve the perfect score consistently. Other models using a larger context show substantially lower results. On the harder D(2-4) setting, we see similar observations. Due to the architecture limitation, none of the models using local context can achieve the perfect score even once. Models using the long context also fail to perform well consistently. MPoSM (full) is the single best model that successfully acquires agreement, albeit only  $30\%$  of the time. These observations demonstrate the difficulty of learning agreement in POS induction. As reflected by the results on D(0), such difficulty
Figure 4: Log-scale sizes of the predicted clusters and the gold clusters.
cannot be fully attributed to the long-term issue. We suspect the latent variable-based loss function used in current models can contain many bad local minimums, similar to the optimization difficulty observed in Gaussian Mixture Models (Jin et al., 2016). Models are likely to stuck in one local minimum (e.g., viewing n1 and n2 as the same tag) and never reach the global optimum.
Finally, we want to point out that our findings are not contradictory with recent studies that show the derivation of agreements from MLM-style models (Jawahar et al., 2019; Goldberg, 2019; Lin et al., 2019). One key difference is that they directly measure the word-level agreement, e.g., are should follow areas in Figure 1. However, POS induction focuses on the tag-level agreement, i.e., VBP should follow NNS. Our MPoSM can also be viewed as adding an explicit discretization step in a normal MLM so that we can predict discrete tags. If we remove the POS-factorization step in Eqn 1, and directly predict the word from the word context, our model can also capture the word-level agreement.
7.2 Error Analysis of Predicted Clusters
In Table 3, we notice that performances of different models are saturating around 78 M-1 on the English Penn WSJ dataset. To examine the limitations of current models point out future directions, we manually investigated the clusters learned by our model. Below, we list our main findings on English (see similar findings of Portuguese in Appendix I):
The sizes of predicted clusters are more uniform than gold clusters. Only 1 predicted cluster contains very few (less than 3000) words, while the scale of gold clusters showing a much larger variance, with  $29\%$  of the 45 clusters containing less than 3000 words. A bar plot illustration is in Figure 4. We can see that the size of gold clusters has a much larger range than the predicted clusters. Under the current losses, assigning a small number of words to one tag is likely to make the loss worse, but it hard to match the skewed distribution of natural tags. Johnson (2007) show similar findings on
HMM models trained with EM. These consistent findings may hint at a common limitation of current objectives. Future work should explore different objectives with more suitable inductive biases.
Agreements are not learned well. Similar to the observation in Sec. 7.1, agreements are not learned well in the predicted clusters. For example, the VBP tag (Verb, non-3rd person singular present) is an important tag in the subject-verb agreement. While this tag has 15377 occurrences in the gold annotations, it is not the major tag in any of the predicted clusters. Most VBP words are either mixed with the VB or the VBZ words. We consistently observe models fail to separate these verbs, showing a large room for improvement.
Difficulty in mapping one word to multiple tags. Without using mBERT representations, MPoSM (also applies to many other models, e.g., He et al. (2018); Stratos (2019), etc.) predicts the same tag for one word. However, the same word can have different tags in different contexts. For example, the word 'that' can have gold tags IN, RB, and WDT. Future works should explore directions on capturing the multi-sense phenomenon.
Dataset biases influence predicted clusters. For example, the English WSJ dataset contains many financial news reports, so numerical words (e.g., 'million', 'billion') and related symbols like  $\%$  are very frequent. Since these words always appear in a distinctive context, models will naturally cluster these tokens together. Hence, we encourage future research to explore more diverse datasets.
We propose MPoSM, a POS induction model inspired by MLM and can model complex long-term dependencies between POS tags. Our model shows competitive performance on both English and multilingual datasets. We analyze the effectiveness of using long context compared to local context. Finally, we use synthetic datasets and analyses to point out remaining challenges.
The model proposed in this paper is intended to study how syntax emerge from unsupervised learning objectives. It can also help understand languages with limited annotations. However, as we showed in this paper, the syntax predicted by current models can contain errors and be influenced by
the choice of datasets, so the model's output should be used with caution and examined by experts. Our model has been tested on 10 diverse languages. Our findings on these languages should generalize to languages with similar linguistic properties, but we suggest careful empirical studies before applying our approach to languages distant from those we study in this paper.
We thank the reviewers for their helpful comments. We thank Chao Lou for the help in data preprocessing steps. This work was supported by ONR Grant N00014-18-1-2871, NSF-CAREER Award 1846185, DARPA MCS Grant N66001-19-2-4031, and a Bloomberg Data Science Ph.D. Fellowship. The views are those of the authors and not of the funding agency.
Omri Abend, Roi Reichart, and Ari Rappoport. 2010. Improved unsupervised POS induction through prototype discovery. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1298-1307, Uppsala, Sweden. Association for Computational Linguistics.
Waleed Ammar, Chris Dyer, and Noah A. Smith. 2014. Conditional random field autoencoders for unsupervised structured prediction. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 3311-3319.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2014. Tailoring continuous word representations for dependency parsing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 809-815, Baltimore, Maryland. Association for Computational Linguistics.
Taylor Berg-Kirkpatrick, Alexandre Bouchard-Côté, John DeNero, and Dan Klein. 2010. Painless unsupervised learning with features. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 582-590, Los Angeles, California. Association for Computational Linguistics.
Phil Blunsom and Trevor Cohn. 2011. A hierarchical Pitman-Yor process HMM for unsupervised part of speech induction. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 865-874, Portland, Oregon, USA. Association for Computational Linguistics.
Peter F Brown, Vincent J Della Pietra, Peter V Desouza, Jennifer C Lai, and Robert L Mercer. 1992. Class-based n-gram models of natural language. Computational linguistics, 18(4):467-480.
Christos Christodoulopoulos, Sharon Goldwater, and Mark Steedman. 2010. Two decades of unsupervised POS induction: How far have we come? In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 575-584, Cambridge, MA. Association for Computational Linguistics.
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: pretraining text encoders as discriminators rather than generators. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Dipanjan Das and Slav Petrov. 2011. Unsupervised part-of-speech tagging with bilingual graph-based projections. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 600-609, Portland, Oregon, USA. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Kevin Gimpel and Noah A. Smith. 2012. Concavity and initialization for unsupervised dependency parsing. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 577-581, Montreal, Canada. Association for Computational Linguistics.
Yoav Goldberg. 2019. Assessing bert's syntactic abilities. ArXiv preprint, abs/1901.05287.
Vikram Gupta, Haoyue Shi, Kevin Gimpel, and Mrinmaya Sachan. 2022. Deep clustering of text representations for supervision-free probing of syntax. In Proceedings of the AAAI Conference on Artificial Intelligence.
Aria Haghighi and Dan Klein. 2006. Prototype-driven learning for sequence models. In Proceedings of
the Human Language Technology Conference of the NAACL, Main Conference, pages 320-327, New York City, USA. Association for Computational Linguistics.
Junxian He, Graham Neubig, and Taylor Berg-Kirkpatrick. 2018. Unsupervised learning of syntactic structure with invertible neural projections. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1292-1302, Brussels, Belgium. Association for Computational Linguistics.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Eric Jang, Shixiang Gu, and Ben Poole. 2017. Categorical reparameterization with gumbel-softmax. In 5th International Conference on Learning Representations, ICLR 2017, Toulouse, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Ganesh Jawahar, Benoit Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3651-3657, Florence, Italy. Association for Computational Linguistics.
Yong Jiang, Wenjuan Han, and Kewei Tu. 2016. Unsupervised neural dependency parsing. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 763-771, Austin, Texas. Association for Computational Linguistics.
Chi Jin, Yuchen Zhang, Sivaraman Balakrishnan, Martin J. Wainwright, and Michael I. Jordan. 2016. Local maxima in the likelihood of gaussian mixture models: Structural results and algorithmic consequences. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 4116-4124.
Mark Johnson. 2007. Why doesn't EM find good HMM POS-taggers? In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 296-305, Prague, Czech Republic. Association for Computational Linguistics.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427-431, Valencia, Spain. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations,
ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Chu-Cheng Lin, Waleed Ammar, Chris Dyer, and Lori Levin. 2015. Unsupervised POS induction with word embeddings. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1311-1316, Denver, Colorado. Association for Computational Linguistics.
Yongjie Lin, Yi Chern Tan, and Robert Frank. 2019. Open sesame: Getting inside BERT's linguistic knowledge. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 241-253, Florence, Italy. Association for Computational Linguistics.
Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. 2017. The concrete distribution: A continuous relaxation of discrete random variables. In 5th International Conference on Learning Representations, ICLR 2017, Toulouse, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313-330.
Ryan McDonald, Joakim Nivre, Yvonne Quirmbach-Brundage, Yoav Goldberg, Dipanjan Das, Kuzman Ganchev, Keith Hall, Slav Petrov, Hao Zhang, Oscar Tackstrom, Claudia Bedini, Nuria Bertomeu Castelló, and Jungmee Lee. 2013. Universal Dependency annotation for multilingual parsing. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 92–97, Sofia, Bulgaria. Association for Computational Linguistics.
Marina Meila and David Heckerman. 1998. An experimental comparison of several clustering and initialization methods. UAI '98: Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence, pages 386-395.
Bernard Merialdo. 1994. Tagging English text with a probabilistic model. Computational Linguistics, 20(2):155-171.
Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings.
Ofir Press and Lior Wolf. 2017. Using the output embedding to improve language models. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 157-163, Valencia, Spain. Association for Computational Linguistics.