SlowGuess/NAACL_2022_OCR huggingface dataset:
Xiang Zhou Shiyue Zhang Mohit Bansal
{xzh, shiyue, mbansal}@cs.unc.edu
Previous Part-Of-Speech (POS) induction models usually assume certain independence assumptions (e.g., Markov, unidirectional, local dependency) that do not hold in real languages. For example, the subject-verb agreement can be both long-term and bidirectional. To facilitate flexible dependency modeling, we propose a Masked Part-of-Speech Model (MPoSM), inspired by the recent success of Masked Language Models (MLM). MPoSM can model arbitrary tag dependency and perform POS induction through the objective of masked POS reconstruction. We achieve competitive results on both the English Penn WSJ dataset as well as the universal treebank containing 10 diverse languages. Though modeling the long-term dependency should ideally help this task, our ablation study shows mixed trends in different languages. To better understand this phenomenon, we design a novel synthetic experiment that can specifically diagnose the model's ability to learn tag agreement. Surprisingly, we find that even strong baselines fail to solve this problem consistently in a very simplified setting: the agreement between adjacent words. Nonetheless, MPoSM achieves overall better performance. Lastly, we conduct a detailed error analysis to shed light on other remaining challenges.<sup>1</sup>
Unsupervised Part-Of-Speech (POS) tagging is the task of discovering POS tags from text without any supervision. These unsupervised syntax induction approaches can reduce the effort needed for collecting expensive syntactic annotation, and can bring us insights about what inductive bias leads to the emergence of syntax. Recent POS induction models have made great progress using different frameworks (Christodoulopoulos et al., 2010; Berg-Kirkpatrick et al., 2010; He et al., 2018; Stratos
Figure 1: Two long-term tag dependency examples in English.
et al., 2016; Shi et al., 2020). However, most of them assume certain independence assumptions among POS tags, e.g., Markov (Merialdo, 1994; Berg-Kirkpatrick et al., 2010; Ammar et al., 2014; He et al., 2018), unidirectional (Tran et al., 2016), local dependency (Stratos et al., 2016; Gupta et al., 2022), etc. On the contrary, complex and long-term dependency appear in many real languages and plays an important role in defining the POS tags. For example, in Figure 1, the VBP tag of are and the NNS tag of areas depend on each other, and so do the VBZ tag of is and the NN tag of news. So in this case, models only conditioning the immediate preceding tag (Markov) or 1-2 neighboring words (local) cannot explain the distinction between NNS and NN, or between VBZ and VBP. While unidirectional (e.g., using a unidirectional LSTM (Tran et al., 2016)) models are in theory capable of modeling long-term dependency through optimizing the joint probability of tags, bidirectional architectures still show clear advantage in language modeling literature (Bahdanau et al., 2015; Vaswani et al., 2017; Devlin et al., 2019).
In this work, we present a novel framework for POS induction that is capable of modeling arbitrary long-term bidirectional dependencies: Masked Part-Of-Speech Model (MPoSM), inspired by recent success of Masked Language Models (MLM) (Devlin et al., 2019). Specifically, MPoSM consists of two modules (see Figure 2): a local POS
$^{1}$ Our code is available at https://github.com/owenzx/MPoSM
${}^{2}$  Similar agreements are also common in many other languages. Various other long-term dependencies also exist, e.g., tense consistency, long-distance PP attachment, etc.
Human Language Technologies, pages 1099 - 1114
prediction module that maps each word to its POS tag and a masked POS reconstruction module that masks a certain portion of tags produced from the previous step, and then learns to first predict the masked tags as latent variables and then reconstruct the corresponding words. We use a bidirectional LSTM (Bi-LSTM) to predict the mask tags conditioned on the remaining tags, which grants our model the ability to model complex long-term and bidirectional dependencies among tags. Through the training signal back-propagated from this module, the tags predicted from the local POS prediction module will also be encouraged to have global inter-dependency, which leads to better tags. Since we do not have gold POS tags, at the masked positions, we marginalize over all possible tags and optimize word reconstruction probabilities. Intuitively, the correct induction of POS tags is beneficial for the prediction of the correct masked words. For example, in Figure 1, if we mask the second positions of the two sentences (corresponding to areas and news), inducing two different tags (i.e., NNS for areas and NN for news) correctly will make the word prediction easier than inducing the same tag. From a probabilistic view, our model is conceptually similar to approximately modeling the probability of generating the whole sentence from latent tags using masked loss as a surrogate.
MPoSM achieves competitive performance on both the 45-tag English Penn WSJ dataset (Marcus et al., 1993) and the 12-tag universal treebank (McDonald et al., 2013) containing 10 diverse different languages. It achieves comparable oracle M-1 compared to the SOTA (state-of-the-art) models (Stratos, 2019; Gupta et al., 2022) on Penn WSJ dataset and achieves higher performance than Stratos (2019) on 4 out of 10 languages on the universal treebank. We also show that substantial improvements can be made with the help of contextualized representations in mBERT, similar to Gupta et al. (2022). We conduct an ablation study on multiple languages by replacing the Bi-LSTM architecture with a window-based MLP that models the local dependency of tags. Surprisingly, while modeling the full-sentence context can improve the performance of English and German, modeling local context is better for Indonesian and Korean. Our mutual information analysis indicates that this difference may be resulted from the different degrees of gold-tag dependency of different languages.
Since real-life datasets can contain many con
founding factors, we next design a suite of well-controlled synthetic experiments from the angle of agreement-learning to examine how well the model can take advantage of the long-term dependency. Our synthetic datasets can guarantee enough training signals for the model to capture the agreements. However, we show that all current models fail to consistently solve the agreement learning problems, even with the most basic agreement happening between adjacent words. Nonetheless, our model shows the best performance with the highest percentage of solving these problems in multiple runs. We conjecture that this is relate to the general optimization problems of latent variable inference problems (Jin et al., 2016) (see more discussions in Section 7). Such obstacles prevent models from gaining additional benefits from modeling long-term dependency. Finally, we did error analysis on the predicted clusters for English and Portuguese and identify remaining challenges both from imperfect modeling and lack of data diversity.
In summary, our main contributions are: (1) a novel POS induction architecture with MLM-inspired loss that allows learning arbitrary tag dependencies and reaches close-to-SOTA performance; and (2) examining the effectiveness of using long-term context and providing a suite of synthetic datasets to expose the challenges in agreement learning and pointing out future challenges.
POS Induction. A POS tag is a category of words that share the same grammatical property. A simplified form of these tags will involve commonly known categories such as nouns, verbs, etc. Formally, given a sentence with  $l$  words  $x = \{x_{i}\}_{i=1}^{l}$ , the corresponding POS tags  $z = \{z_{i}\}_{i=1}^{l}$ , then the goal of the POS induction task is to infer  $z$  from  $x$  without supervision from gold tags.
Limitations of Existing POS Induction Models. From the perspective of probabilistic graphical models, POS tags can be viewed as latent variables related to all the observed words. Each tag  $z$  is a latent variable that generates the corresponding word  $x$ . Hence, inducing the POS tag sequence becomes the problem of performing MAP inference of the latent variables. This is a popular and effective view adopted by many previous works. To make such inference tractable, previous works have to add certain assumptions, including adding Markov assumption to the latent variables  $z$  (i.e.,
the current tag only depends on the immediate previous tag) (Merialdo, 1994; Berg-Kirkpatrick et al., 2010; Ammar et al., 2014), only considering local dependencies (Stratos, 2019; Gupta et al., 2022), unidirectional dependencies (Tran et al., 2016), etc. However, dependencies in real language are not constrained by length or direction, as we discussed in Sec. 1. Hence, simplifying and ruling the capability out in the model design is suboptimal. To mitigate this problem, in Sec. 3, we will describe our approach to model long-term dependency.
Why are the Learned Latent Variables Correlated with POS-Tags? Before introducing our method, we discuss why latent variable models can induce POS-tags well. Take the vanilla HMM as an example, the latent variables in the model can be viewed as being optimized towards two objectives: the transition probability  $p(z_{i}|z_{i - 1})$  and the emission probability  $p(x_{i}|z_{i})$ . They characterize two properties respectively: (1) strong ordering dependencies among latent variables; and (2) the strong correlation between latent variables and the observed word. In short, the success of previous latent variable models implies: A word's inherent category that has strong ordering constraints will highly resemble the POS tag. In this work, we follow this assumption, but propose a model that is able to learn arbitrary bidirectional long-term dependencies  $p(z_{i}|\{z_{j}\}_{j\neq i})$  instead of  $p(z_{i}|z_{i - 1})$ .
3 Masked Part-Of-Speech Model
Inspired by the recent success in masked language modeling (MLM) (Devlin et al., 2019), we present Masked Part-Of-Speech Model (MPoSM). Next, we will first describe the model architecture and then introduce several useful additional techniques.
As is shown in Figure 2, our model consists of two parts: a local POS prediction module and a masked POS reconstruction module. The local POS prediction module predicts a POS tag for each word, and the masked POS reconstruction module encourages strong dependencies among these tags.
Local POS Prediction. Given the input word sequence  $\pmb{x} = \{x_{i}\}_{i=1}^{l}$  with length  $l$ , we first get the word embeddings. As morphological features are shown to be useful for POS induction (Christodoulopoulos et al., 2010) to capture inflection (e.g., the 's' suffix for English plu
Figure 2: Illustration of our MPoSM. The model consists of two parts: the local POS prediction module (blue part at the bottom) and the masked POS reconstruction module (green part at the top).
rals), we follow Stratos (2019) to extract character-level representations using a Bi-LSTM. We concatenate word embeddings and char representations to form the final representations for each word,  $\boldsymbol{w} = \{w_{i}\}_{i=1}^{l}$ . Then, we use a single context-independent feed-forward network to predict the POS tags  $z$  out of  $\boldsymbol{w}$ , i.e.,  $z_{i} = \arg \max(\text{Softmax}(\text{FF}(w_{i}))$ . Essentially, this module models  $P(z_{i}|x_{i})$  for every position and predicts the POS tag only conditioned on the word itself without considering its context. We make this design choice as POS tags are the syntactic property of each individual word, so it should be able to be predicted as an attribute of the word.<sup>3</sup> Importantly, in order to make the whole model end-to-end differentiable, we replace the arg max with a straight-through Gumbel-Softmax estimator (Jang et al., 2017; Maddison et al., 2017) (see Appendix A for more details).
Masked POS Reconstruction. After we get all the predicted POS tags  $z = \{z_{i}\}_{i=1}^{l}$  for the previous module, we conduct masked POS reconstruction to encourage modeling strong dependencies among  $z$ . Specifically, we follow Devlin et al. (2019) to mask 15% of the predicted POS tags and replace them with a placeholder MASK tag. Then we map them into POS embeddings and use a Bi-LSTM (Hochreiter and Schmidhuber, 1997) as the dependency-modeling network. This grants flexibility of modeling the long-term and bidirectional dependency among tags without any assumption that the underlying models are fully trained.
However, it gives our model the limitation of only predicting a fixed tag for each word, same as Stratos (2019). Nonetheless, the upper bound M-1 on the 45-tag Penn dataset is 94.6 and the mean upper bound on UD is 95.4, which are both substantially higher than current models.
4We also experimented using the Transformer architecture in our preliminary experiments, but did not observe additional gain. See Appendix J for details.
tions and thus brings us an advantage over traditional HMM-based models. Then, we predict the masked POS tags out of the contextualized representations from the Bi-LSTM, so, essentially, it models  $P(\hat{z}_j|C_j)$ , where  $\hat{z}_j$  is the reconstructed tag at position  $j$  and  $C_j = \{z_i\}_{i \neq j}$  is the context. We treat the predicted  $\hat{z}_j$  as latent variables and maximize the probability of the corresponding word  $x_j$  which can be written out by marginalizing over  $\hat{z}_j$ :
$$
P (x _ {j} | C _ {j}) = \sum_ {\hat {z} _ {j}} P (x _ {j} | \hat {z} _ {j}) P (\hat {z} _ {j} | C _ {j}) \tag {1}
$$
The conditional probability  $P(x_{j}|\hat{z}_{j})$  can be modeled through another feed-forward network with the POS embeddings as the input. Intuitively, predicting the  $P(\hat{z}_j|C_j)$  objective encourages strong dependency among the tags and predicting  $P(x_{j}|\hat{z}_{j})$  reinforces the connection between the words and the tags. Hence, the total loss is the sum of all the log-probabilities at masked positions:
$$
\mathcal {L} _ {\mathrm {M P o S M}} = \sum_ {j \in \text {M a x e d P o s i t i o n s}} \log P \left(x _ {j} \mid C _ {j}\right) \tag {2}
$$
Importantly, the supervision from this module will back-propagate to the local POS prediction module. Therefore, even though it produces POS tags independently, the supervision helps it to capture the interdependency among all the tags.
During testing, we use the output of the local prediction module as the output tags.
3.2 Additional Techniques
Below, we introduce several additional techniques used in our model to achieve good performance.
Careful Initialization. Similar to many other unsupervised learning models (Gimpel and Smith, 2012; Meila and Heckerman, 1998; He et al., 2018), we found our model to be sensitive to initialization in our preliminary experiments. Below, we propose Masked Language Modeling Pretraining (MLMP). We use a two-stage training procedure: (1) we remove the modeling architecture for  $P(x_{j}|\hat{z}_{j})$  and  $P(\hat{z}_j|C_j)$ , and directly apply an MLP to model  $P(x_{j}|C_{j})$  without explicitly predicting the masked tag; (2) we initialize our MPoSM with the pretrained model in (1) and continue training with the loss in Eqn. 2. This procedure trains the bottom layers with a smoother objective and provides a better starting point for optimizing the MPoSM loss.
Besides, the MPoSM model can leverage knowledge from both pretrained embeddings similar to He et al. (2018) and Zhu et al. (2020), or pretrained language models similar to Gupta et al. (2022).
Connecting  $\mathbf{P}(\mathbf{x}|\mathbf{z})$  and  $\mathbf{P}(\mathbf{z}|\mathbf{x})$ . While the local POS prediction module models  $P(z|x)$ , the masked POS reconstruction module has a part that models  $P(x|z)$  (Eqn. 1). These two probabilities can be connected using the Bayes' rule:  $P(x|z) = \frac{P(z|x)P(x)}{\sum xP(z|x)P(x)}$ . If we assume the training set is representative enough of the language, we can approximate  $P(x)$  by the word frequency in the dataset, and then we can compute  $P(x|z)$  directly following the Bayes' rule instead of using a neural network to model it. We notice that binding these two probabilities can usually make the training more stable and improve the performance when training from scratch. Note that we do not adopt this change when using pretrained word embeddings because we can use the pretrained embedding weights at the output layer (Press and Wolf, 2017), which brings additional knowledge for the final word prediction.
Dataset Re chunking. One potential problem of using the full sentence context is the position bias of POS tags. For example, since a large number of English sentences start with the word 'the', position 0 will have a strong bias towards predicting the 'DT' tag. In our experiments, we concatenate all the sentences in the original dataset and re-chunk them randomly. Then we combine the rechunked dataset and the original dataset as our training set. In our preliminary experiments, we find it can improve the stability and the performance of the model.
4 Connections to Related Works
The HMM-based POS induction model (Merialdo, 1994) has many extensions, including using handengineered linguistic features (Berg-Kirkpatrick et al., 2010), pretrained embeddings (Lin et al., 2015), task-specific modifications (Blunsom and Cohn, 2011; Stratos et al., 2016), flow-based transformations (He et al., 2018), external resources (Haghighi and Klein, 2006; Snyder et al., 2008; Das and Petrov, 2011), etc. They all optimize the probability of the sequence,  $P(\pmb{x})$ . However, it requires certain dependency assumptions to be tractable. Our model instead optimizes the sum of conditional word probabilities given the remaining context  $\sum_{i} \log P(x_{i} | \pmb{x}_{1..i-1}, \pmb{x}_{i+1..l})$ , i.e., MLM loss (Devlin et al., 2019). While being different
We can also use the tags predicted in the masked POS reconstruction module  $P(\hat{z}_i|C_i)$  as the output, but we find the output of local prediction module is empirically better.
en (Penn)	de	en (UD)	es	fr	id	it	ja	ko	pt-br	sv
# sentences	49208	15918	43948	16007	16422	5593	7189	9494	6339	11998	6159
# words	1173766	293460	1046829	424425	396511	121923	167873	92033	69690	298323	96319
# vocab	49206	52435	46348	50334	44453	22221	22344	56758	36335	34011	16241
Avg word freq.	23.85	5.60	22.59	8.43	8.92	5.49	7.51	1.62	1.92	8.77	5.93
Tag Mutual Info.	0.85	0.56	0.86	0.65	0.66	0.39	0.57	0.74	0.27	0.59	0.59
Table 1: Dataset statistics. For each row, the language with the largest number are in bold and the language with the smallest number is underlined. Computation details about the tag mutual information is in Appendix E.
from  $P(x)$ , this objective is an effective surrogate and makes modeling complex dependencies possible. There also exist some earlier methods that do not require the Markov assumption. For example, Abend et al. (2010) design a method to directly cluster the embeddings containing distributional and morphological information of the word, and then identify prototype words to facilitate the final POS induction. Tran et al. (2016) propose a neural HMM model. Similar to our model, it can also model long-term dependency (due to the use of LSTM), however, they still constrain the dependence to be uni-directional (due to the HMM nature). Our model does not have such constraints and empirically works better.
Architecture-wise, our model is conceptually similar to a denoising auto-encoder (Vincent et al., 2008), where the masking step can be viewed as adding noises to the POS tags. The idea of using auto-encoder models for unsupervised learning has been explored with CRFs in Ammar et al. (2014). However, they still require Markov independence assumption to make inference on CRF tractable, while our model has the ability to model complex long-term dependencies. Plus, we use an MLM-inspired loss instead of reconstructing Brown clusters (Brown et al., 1992) as Ammar et al. (2014).
Our model also provide additional insight on the relation between MLM and syntax emergence. Such connection has also been explored in previous works. Pretrained transformers using MLM (Devlin et al., 2019; Clark et al., 2020; Raffel et al., 2020) have shown strong syntactic abilities (Tenney et al., 2019; Jawahar et al., 2019; Goldberg, 2019). CBOW and skip-gram embeddings (Mikolov et al., 2013) can be viewed as an MLM with a limited context window (i.e., local context), and have been shown to be useful for syntax induction, especially with small window sizes (Bansal et al., 2014; Lin et al., 2015; He et al., 2018). Some recent POS induction works explore CBOW-style objectives (Stratos, 2019; Gupta et al., 2022). However, using the sentence-level MLM objective for syntax induction is under-explored. The only exception is
the recent work of Shen et al. (2021), which focuses on a different task: unsupervised parsing. The different tasks lead to substantially different focuses and designs in the architecture. They use MLM with a dependency-constrained self-attention mechanism to extract parses, while we extend MLM to the POS-tag level (MPoSM) and explicitly discretizes the latent variables to extract tags.
We evaluate our model on two datasets: the 45-tag English Penn WSJ dataset (Marcus et al., 1993) and the 12-tag universal treebank (McDonald et al., 2013). Following Ammar et al. (2014) and Stratos (2019), we use the v2.0 version<sup>6</sup> containing 10 different languages. Detailed statistics are in Table 1.
Following recent works (Stratos, 2019; Gupta et al., 2022), we use the Many-to-One accuracy (M-1) (Johnson, 2007) as our metric, and train and evaluate our model on the whole dataset. Following Shi et al. (2020), we distinguish between the oracle performance that selects the model with the best M-1 metric  $(\mathrm{M - 1_{OR}})$ , and the fully unsupervised performance that selects the model with the best loss (M-1). However, many previous works used different or unspecified model selection settings. For a fair comparison, we get results under our setting using their official code if possible.
For the English Penn WSJ dataset, we use the pretrained embedding provided in He et al. (2018). For the main results on the universal treebank, we do not use any external resources and use MLMP initialization. Additionally, we also report the results with mBERT contextualized representations on the universal treebank following Gupta et al. (2022), where they show mBERT representations empirically outperforms English BERT representations on POS-tag induction. Same to the implementation in Gupta et al. (2022), we also use the average
$^{6}$ https://github.com/ryanmcd.uni-dep-tb
de	en	es	fr	id	it	ja	ko	pt-br	sv
MPOSMOR	71.8	72.3	73.2	73.7	69.4	69.7	76.8	55.2	76.2	63.7
MPOSM + mBERTOR	77.5	72.1	77.0	74.8	72.4	74.8	76.0	56.6	78.1	65.5
Stratos (2019)OR	75.4	73.1	73.1	70.4	73.6	67.4	77.9	65.6	70.7	67.1
Gupta et al. (2022)OR	81.7	76.7	79.5	70.8	76.9	71.8	84.7	69.7	78.9	69.7
Stratos et al. (2016)**	63.4	71.4	74.3	71.9	67.3	60.2	69.4	61.8	65.8	61.0
Berg-Kirkpatrick et al. (2010)**	67.5	62.4	67.1	62.1	61.3	52.9	78.2	60.5	63.2	56.7
Brown et al. (1992)**	60.0	62.9	67.4	66.4	59.3	66.1	60.3	47.5	67.4	61.9
Table 2: Performance on the universal treebank. Gupta et al. (2022) also leverages pretrained mBERT model. All the other models do not use pretrained models or embeddings. Subscript OR denotes models evaluated by oracle M-1 and \*\* refers to unspecified model selection. Standard deviations and non-oracle numbers are in the Appendix D.
M-1OR	M-1
MPoSM	75.6 (±2.0)	74.5 (±1.4)
MPoSM + emb	78.6 (±1.7)	77.9 (±1.8)
Tran et al. (2016)	-	75.0 (±1.5)
He et al. (2018)	-	75.6 (±2.7)
Stratos (2019)	78.1 (±0.8)	-
Gupta et al. (2022)	-	79.5 (±0.9)
Brown et al. (1992)**	65.6
Berg-Kirkpatrick et al. (2010)**	74.9 (±1.5)
Tran et al. (2016)**	79.1
Abend et al. (2010)**	75.1
He et al. (2018)*	-	80.8 (±1.3)