SlowGuess/NAACL_2022_OCR huggingface dataset:
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Yikang Shen, Yi Tay, Che Zheng, Dara Bahri, Donald Metzler, and Aaron Courville. 2021. StructFormer: Joint unsupervised induction of dependency and constituency structure from masked language modeling. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7196-7209, Online. Association for Computational Linguistics.
Haoyue Shi, Karen Livescu, and Kevin Gimpel. 2020. On the role of supervision in unsupervised constituency parsing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7611-7621, Online. Association for Computational Linguistics.
Benjamin Snyder, Tahira Naseem, Jacob Eisenstein, and Regina Barzilay. 2008. Unsupervised multilingual learning for POS tagging. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 1041-1050, Honolulu, Hawaii. Association for Computational Linguistics.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1958.
Karl Stratos. 2019. Mutual information maximization for simple and accurate part-of-speech induction. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1095-1104, Minneapolis, Minnesota. Association for Computational Linguistics.
Karl Stratos, Michael Collins, and Daniel Hsu. 2016. Unsupervised part-of-speech tagging with anchor hidden Markov models. Transactions of the Association for Computational Linguistics, 4:245-257.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593-4601, Florence, Italy. Association for Computational Linguistics.
Ke M. Tran, Yonatan Bisk, Ashish Vaswani, Daniel Marcu, and Kevin Knight. 2016. Unsupervised neural hidden Markov models. In Proceedings of the Workshop on Structured Prediction for NLP, pages 63-71, Austin, TX. Association for Computational Linguistics.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. 2008. Extracting and composing robust features with denoising autoencoders. In Machine Learning, Proceedings of the Twenty-Fifth International Conference (ICML 2008), Helsinki, Finland, June 5-9, 2008, volume 307 of ACM International Conference Proceeding Series, pages 1096-1103. ACM.
Xuewen Yang, Yingru Liu, Dongliang Xie, Xin Wang, and Niranjan Balasubramanian. 2019. Latent part-of-speech sequences for neural machine translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 780-790, Hong Kong, China. Association for Computational Linguistics.
Hao Zhu, Yonatan Bisk, and Graham Neubig. 2020. The return of lexical dependencies: Neural lexicalized PCFGs. Transactions of the Association for Computational Linguistics, 8:647-661.
A Straight-Through Gubmbel-Softmax Estimator
To allow the gradients from the masked POS construction module to back-propagate to the local POS prediction module, we replace the standard argmax in the local POS prediction module with the straight-through Gumbel-Softmax estimator. Specifically, we follow Jang et al. (2017); Maddison et al. (2017) to calculate the one-hot POS-tag vector using one-hot(softmax  $(g_{j} + \log it_{j},\tau)$ ). In this equation,  $\tau$  is the softmax temperature and  $g_{j}$  are i.i.d. samples drawn from a standard Gumbel distribution, i.e.,  $g_{j} = -\log (-\log (U(0,1)))$  where  $U(0,1)$  is a uniform distribution over the range [0, 1]. Following Jang et al. (2017), we use arg max to discretize the distribution to a one-hot vector in the forward pass, but back-propagate through the continuous Gumbel-softmax. With this technique, the whole MPoSM becomes end-to-end differentiable.
Alternatively, instead of operating on the single POS embedding of the Gumbel-Softmax output, we can also use the weighted sum of all the POS embeddings with weight  $P(z_{i}|x_{i})$ . However, empirically, we notice the weighted sum approach
does not perform well when the number of tags is large (e.g., 45 in the Penn WSJ dataset).
B Dataset Links
The universal treebank dataset is from https://github.com/ryanmcd.uni-dep-tb. The English Penn WSJ dataset can be obtained through LDC.
C Implementation and Hyper-Parameter Details
For initialization, for the English 45-tag Penn WSJ dataset, we use the pretrained word embedding provided in He et al. (2018). For the main results on the universal treebank, we do not use any external resources and always initialize our models using MLM pretraining. Additionally, we also report the results with mBERT contextualized representations on the universal treebank following Gupta et al. (2022). Same to the implementation in Gupta et al. (2022), we also use the average representation over all the subwords and layers as the representation for each word. We apply the "connecting  $\mathrm{P(x|z)}$  and  $\mathrm{P(z|x)}$  technique for all our models not using pretrained word embeddings or pretrained language models. We apply the dataset rechunking technique to all our experiments.
For the hyper-parameters, we train all our model using Adam (Kingma and Ba, 2015) with an initial learning rate 0.001. The batch size is set to 80 and we decay the learning rate with a factor 0.1 the loss stagnates. We set the word embedding dimension to 100, POS embedding dimension to 200, the character embedding dimension to 100, and the hidden vector dimension to 128. We use one layer of MLP in the local POS prediction module and one layer of Bi-LSTM in the masked POS reconstruction module. The masking rate is set to  $15\%$  and the Gumbel softmax temperature is set to 2.0. We set the dropout rate (Srivastava et al., 2014) to 0.5. Specially, with pretraining word embeddings, we tie the input and output embeddings following (Press and Wolf, 2017) and add one more layer in the local POS prediction layer to more effectively convert the pretrained embedding to POS tags following (He et al., 2018). And for our synthetic experiments, since the vocabulary size is small, we use a smaller character embedding dimension of 8. We use the loss as the metric to judge if our model has been converged. In this work, the results on the Penn WSJ dataset are the mean of 5 different runs,
de	en	es	fr	id	it	ja	ko	pt-br	sv
MPoSMOR	71.8	72.3	73.2	73.7	69.4	69.7	76.8	55.2	76.2	63.7
(±2.5)	(±1.7)	(±1.7)	(±1.3)	(±1.8)	(±3.6)	(±1.5)	(±1.3)	(±0.2)	(±2.6)
MPoSM	68.3	70.0	69.7	71.7	67.8	64.2	74.7	52.7	74.7	61.9
(±2.9)	(±2.2)	(±4.5)	(±2.3)	(±0.8)	(±1.9)	(±0.7)	(±0.4)	(±0.7)	(±1.9)
MPoSM + mBERTOR	77.5	72.1	77.0	74.8	72.4	74.8	76.0	56.6	78.1	65.5
(±0.3)	(±1.5)	(±1.5)	(±4.2)	(±5.2)	(±1.0)	(±1.1)	(±1.3)	(±1.7)	(±4.6)
MPoSM + mBERT	75.8	68.5	75.4	73.6	71.0	73.4	73.3	55.1	77.4	64.2
(±0.8)	(±3.6)	(±1.3)	(±5.3)	(±5.2)	(±1.0)	(±1.4)	(±1.6)	(±2.3)	(±3.5)
Stratos (2019)OR	75.4	73.1	73.1	70.4	73.6	67.4	77.9	65.6	70.7	67.1
(±1.5)	(±1.7)	(±1.0)	(±2.9)	(±1.5)	(±3.3)	(±0.4)	(±1.2)	(±2.3)	(±1.5)
Gupta et al. (2022)OR	81.7	76.7	79.5	70.8	76.9	71.8	84.7	69.7	78.9	69.7
Stratos et al. (2016)**	63.4	71.4	74.3	71.9	67.3	60.2	69.4	61.8	65.8	61.0
Berg-Kirkpatrick et al. (2010)**	67.5	62.4	67.1	62.1	61.3	52.9	78.2	60.5	63.2	56.7
(±1.8)	(±3.5)	(±3.1)	(±4.5)	(±3.9)	(±2.9)	(±2.9)	(±3.6)	(±2.2)	(±2.5)
Brown et al. (1992)**	60.0	62.9	67.4	66.4	59.3	66.1	60.3	47.5	67.4	61.9
(±1.8)	(±1.0)	(±1.7)	(±4.0)	(±1.2)	(±3.9)	(±1.2)	(±1.5)	(±1.8)	(±2.8)
Table 6: Performance on the Universal Dependency dataset. Gupta et al. (2022) also leverage pretrained mBERT model. All the other models do not use pretrained models or embeddings. Subscript OR denotes models evaluated by oracle M-1 and ** refers to unspecified model selection.
Context Type Ci	de	en	es	fr	id	it	ja	ko	pt-br	sv
zi-2, zi-1	0.56	0.86	0.65	0.66	0.39	0.57	0.74	0.27	0.59	0.59
zi+2, zi+1	0.56	0.86	0.65	0.67	0.39	0.58	0.70	0.23	0.59	0.58
zi+2, zi+1, zi-1, zi-2	1.30	1.92	1.36	1.38	1.02	1.32	2.09	0.86	1.28	1.51
Table 7: Mutual information between the tag-level context and the tag  ${z}_{i}$  on all 10 languages in the universal treebank.
and the results on the universal treebank are the mean of 3 different runs. The experiments using mBERT can be run on a single RTX A6000 GPU, and all other experiments can be conducted on a single Titan Xp GPU. The time of experiments can take from several hours to several days, depending on the size of the dataset and the models.
D Results and Standard Deviations on the Universal Treebank
The full means and standard deviations (for our model and for previous works that reported this number) are shown in Table 6. We also include the fully unsupervised performance (evaluating the model with the best loss) in the MPoSM row. Our fully unsupervised model is slightly worse than the oracle version of both our model and Stratos (2019), but show comparable or higher performance to other results (Stratos et al., 2016; Berg-Kirkpatrick et al., 2010; Brown et al., 1992).
E Analysis on the Dependency among Gold Tags
In Sec. 6, we notice that MPoSM does not work equally well on all the languages. For example, in Table 4, we can see that out of 4 different languages, using full context instead of local context
only improve 2 of them: English and German. In this section, we provide evidence that these different trends can result from the different strength of dependencies among tags in different languages.
Assuming a tag sequence is  $z_{1}, z_{2}, \ldots, z_{n}$ , we compute the mutual information between a tag-level context of  $z_{i}$  (denoted as  $C_{i}$ ) and the tag  $z_{i}$ . A larger mutual information value can represent stronger dependencies among the gold tags. The results are shown in Table 7. We can see that for all kinds of mutual information calculated in the table, Korean and Indonesian has the two lowest values, both substantially lower than the value of German and English. Notably, Korean and Indonesian are also the worst two languages of the MPoSM model, while German and English and two of the languages with better performances. By its design, our model will induce tags that have strong dependencies among each other (see Sec. 2). Hence, it is not strange that on Korean and Indonesian, the MPoSM model could produce tags different from
Note that using a larger context will always lead to a larger mutual information value due to the property of mutual information. However, directly comparing the mutual information value with a very long context is confounded by many spurious correlations in the dataset. Hence, in this study, we only compare mutual information value in a limited context. Nonetheless, the trend shown in Table .7 is consistent across different context types.
Name	Distance between n and v	Regex
D(0)	0	(o1 o2){1,2}(n1 v1 | n2 v2)
MORPH	0	same as D(0) (+ morph. feature)
D(2-4)	2-4	(o1 o2){1,2}(n1 (o1 o2){1,2} v1 | n2 (o1 o2){1,2} v2)
Table 8: Tag-level regular expressions and the distances between n and v for each synthetic subset.
en (Penn)	ko (uni)
MPoSM (full)	78.6 (±1.7)	55.2 (±1.3)
MPoSM (width=2)	77.3 (±0.3)	56.6 (±1.4)
MPoSM-Word (full)	72.2 (±1.6)	54.1 (±3.3)
MPoSM-Word (width=2)	69.5 (±0.4)	62.2 (±0.7)
Table 9: Oracle M-1 Performance of different context types on English and Korean.
the gold tags and become less effective. And due to the difference between the predicted tags and the gold tags, it is not surprising to see that using a larger context in these two languages does not help the MPoSM model in these two languages.
F Discussion on uni-Korean Results
In Sec. 6, we notice that MPoSM does not work well on the Korean dataset in the universal treebank. Inspired by Stratos (2019) that achieves decent performance on Korean and the observation in Appendix E, we study another modification of MPoSM for the Korean language, MPoSM-word. Instead of using the local POS prediction module to predict a POS tag, the MPoSM-word directly feeds the word embedding to an MLP, and use the output as the input of the masked POS reconstruction module. Finally, we use the predicted tags after the Bi-LSTM layer as the induced tags. The results are shown in Table 9. We do observe that the MPoSM-word (width=2) variant, which is most similar to Stratos (2019), achieves the best result, demonstrating the effectiveness of such inductive biases on this Korean dataset. Nonetheless, this preference is not consistent over languages. In our preliminary study, we notice many other languages still prefer our default model. We show the result on the 45-tag English dataset in Table 9, where the default MPoSM show substantial advantages. These preferences together with the observation in Appendix E suggest different languages (or datasets) can prefer different types of dependency modeling (e.g. tag-tag dependency vs. word-tag dependency) and we encourage further study on this topic.
G Using Inducted Tags for Unsupervised Dependency Parsing
In this section, as a side study, we test whether the performance trends on POS induction can transfer to unsupervised dependency parsing. We choose to use the Neural E-DMV model from Jiang et al. (2016), a commonly used baseline model that uses gold POS tags in the training. In our experiments, we replace the gold tags with the inducted tags from different models to see if the parsing performance correlates with the tag quality measured by M-1 accuracy. Following the convention in unsupervised parsing experiment setups, we train all the models on sections 2–21 of the English Penn WSJ dataset, and use section 22 for validation and section 23 for testing. We remove all the punctuation and only train and test on sentences with length  $\leq 10$  (i.e., following the WSJ10 setting). We compared three different models, our MPoSM (78.6 M-1), the model from Stratos (2019) (78.1 M-1), and the model from Tran et al. (2016) (75.0 M-1). We notice the models are highly sensitive to initialization. Hence, to remove the influence of bad initialization, we train each model ten different times and compare the best run. Using the gold tags, the E-DMV model can reach over 70 DDA (directed dependency accuracy). However, none of the models trained using predicted tags achieve DDA over 45, showing a substantial performance gap between the gold tags and the predicted tags. Surprisingly, while the tags from the Neural HMM model Tran et al. (2016) have lower M-1 accuracy than the other two models, it shows a small advantage in the parsing performance over the over two models. We suspect the different trends may result from a mismatch between the objective optimized in parsing models and tagging models. The DMV objective explicitly models the transition probability between different nodes, hence the neural HMM model may have a slight advantage by using a more similar HMM-style objective.
D(0)	MORPH	D(2-4)
MPoSM (width=1)	99.50 (95%)	95.41 (55%)	87.19 (0%)
-connecting	83.37 (0%)	85.09 (0%)	83.13 (0%)
MPoSM (width=2)	92.99 (30%)	90.02 (30%)	87.62 (0%)
-connecting	83.97 (5%)	84.03 (0%)	83.44 (0%)
MPoSM (full)	96.50 (75%)	93.01 (30%)	95.31 (30%)
-connecting	75.02 (5%)	73.27 (0%)	82.81 (0%)
Stratos (2019) (width=2)	92.99 (30%)	90.52 (30%)	86.56 (0%)
Tran et al. (2016) (full)	80.97 (0%)	82.54 (0%)	82.50 (0%)
Table 10: The Oracle M-1 score of different models on the synthetic dataset. The number in the bracket is the percentage of perfect runs (100 M-1).
Figure 5: Illustration of the tag-level regular expression used to generate sentences for D(0) and MORPH. For MORPH, each word has useful morphological features, while all the characters in every word are randomly generated in D(0).
H Additional Agreement Learning Experiment Design and Results
H.1 Additional Experiment Design
Besides the D(0) and D(2-4) subsets introduced in Sec. 7.1, we add another variant: MORPH. The MORPH setting is a variant of D(0) with additional morphological features. While characters in every word in D(0) are all randomly generated, in MORPH, words with the n1 tags always end with -n1, words with the n2 tags always end with -n2, etc. The tag-level regular expressions of all the subsets are shown in Table 8.
H.2 Illustrations for Each Subset
We provide illustrations of the tag-level regular expression for each subset. The illustration for D(2-4) is in Figure 3. The tag-level regular expressions D(0) and MORPH are the same and the illustration can be seen in Figure 5.
H.3 Additional Results
We show additional results on the synthetic datasets in Table 10. Besides the results of the default MPoSM, we also include an ablation of removing the "Connecting  $P(x|z)$  and  $P(z|x)$ " trick introduced in Sec. 3.2. We can see connecting these two probabilities does bring substantial improvement on this agreement learning task. Surprisingly, adding the morphological features (MORPH) does not help the models learn the agreement. Instead,
Figure 6: Log-scale sizes of the predicted clusters and the gold clusters for pt-br in the universal treebank.
nearly all models perform slightly worse on this variant. We suspect the problem may lies in the specific design of the morphological feature. The current setting provides an additional feature to first cluster n1 and n2 words, v1 and v2 words together since they have a common character 'n' or 'v' in the word, whereas normally we randomly sample characters to form the word. Hence, it can be easier for the models to enter the unideal local minimum.
I Predicted Clusters Analysis on Brazilian Portuguese
We provide additional analysis on the pt-br dataset in the universal treebank and check if our findings on the English dataset can generalize to the other language. Due to the 12-tag annotations on the universal treebank do not contain fine-grained tags, it is difficult to single out an agreement type to conduct a well-controlled analysis (like the subject-verb agreement analysis on English in the main paper), but below we verify all the other findings.
The sizes of predicted clusters are more uniform than gold clusters. Similar to the findings on the English 45-tag dataset, the sizes of predicted clusters are much more uniform than the gold clusters. A bar plot is shown in Figure 6.
Difficulty in mapping one word to multiple tags. Brazilian Portuguese also has words with different senses and POS-tags. For example, the word 'parecido', which means 'similar' in English, has three
possible gold tags in the annotated data, including ADJ, VERB, and ADV. But again, in the model predictions, this word is always paired with the same tag.
Since models on the universal treebank are only required to predict 12 tags, the influence of dataset biases is smaller than the English 45-tag data. However, we still can find some hints about the negative effect of the lack of linguistic diversity in the data. In the predicted clusters, nouns are separated into a number of clusters. Possibly due to the special domains of the data, the corpus includes lots of nouns representing locations and events. These nouns usually appear in a similar context after the ADP tag, hence models are likely to use a single cluster for these nouns, which is not ideal. While one can argue models should learn to separate the correct syntactic property from other spurious statistical properties, small datasets may not contain enough data to represent the complete picture of grammar. Hence, models are more likely to capture the statistical properties that are more common in the presented corpus and unlikely to induce the POS tags that are more well-suited for the general language.
J Additional Experiment Results
We briefly describe several variants we have tried in our preliminary experiments but do not observe significant improvement.
- For the dependency modeling network in the masked POS reconstruction module in the MPoSM model, we have also explored using a Transformer (Vaswani et al., 2017) architecture or adding self-attention to the Bi-LSTM. However, we do not see substantial improvements. On the 45-tag English Penn Treebank dataset, our best Transformer result reaches 77.2 M-1, which is still lower than the average M-1 of the Bi-LSTM counterparts in Table 3. We suspect this trend is due to two reasons: (1) we notice that the Transformer models are more sensitive to initialization and hyper-parameter settings than the LSTM counterparts. Additionally, POS induction datasets are relatively small, which makes it harder to train a good Transformer model. (2) compared to Transformer models, LSTM models have the advantage of a preference of learning
short-term dependencies first while learning long-term dependencies is still possible. This inductive bias could be useful for the POS induction task.
Instead of directly training our model using gradient descent, another way to optimize our model is to view the reconstructed POS as latent variables and use EM-based algorithms to optimize the objective, similar to the method used in Yang et al. (2019). However, in our experiments, we do not observe substantial improvement by using EM-based optimization methods over SGD-based methods.