Venue: NAACL 2021

# PCFGs Can Do Better: Inducing

Probabilistic Context-Free Grammars with Many Symbols

 Songlin Yang\({}^{\spadesuit}\), Yanpeng Zhao\({}^{\diamond}\), Kewei Tu\({}^{\spadesuit}\)1

Footnote 1: Corresponding Author

###### Abstract

Probabilistic context-free grammars (PCFGs) with neural parameterization have been shown to be effective in unsupervised phrase-structure grammar induction. However, due to the cubic computational complexity of PCFG representation and parsing, previous approaches cannot scale up to a relatively large number of (nonterminal and preterminal) symbols. In this work, we present a new parameterization form of PCFGs based on tensor decomposition, which has at most quadratic computational complexity in the symbol number and therefore allows us to use a much larger number of symbols. We further use neural parameterization for the new form to improve unsupervised parsing performance. We evaluate our model across ten languages and empirically demonstrate the effectiveness of using more symbols. 2

Footnote 2: Our code: [https://github.com/sustcsonglin/TN-PCFG](https://github.com/sustcsonglin/TN-PCFG)

\({}^{\blacklozenge}\)School of Information Science and Technology, ShanghaiTech University

Shanghai Engineering Research Center of Intelligent Vision and Imaging

Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences

University of Chinese Academy of Sciences

\({}^{\diamond}\)ILCC, University of Edinburgh

{yangsl,tukw}@shanghaitech.edu.cn

yannzhao.ed@gmail.com

## 1 Introduction

Unsupervised constituency parsing is the task of inducing phrase-structure grammars from raw text without using parse tree annotations. Early work induces probabilistic context-free grammars (PCFGs) via the Expectation Maximization algorithm and finds the result unsatisfactory Lari and Young (1990); Carroll and Charniak (1992). Recently, PCFGs with neural parameterization (i.e., using neural networks to generate rule probabilities) have been shown to achieve good results in unsupervised constituency parsing Kim et al. (2019); Jin et al. (2019); Zhu et al. (2020). However, due to the cubic computational complexity of PCFG representation and parsing, these approaches learn PCFGs with relatively small numbers of nonterminals and preterminals. For example, Jin et al. (2019) use 30 nonterminals (with no distinction between preterminals and other nonterminals) and Kim et al. (2019) use 30 nonterminals and 60 preterminals.

In this paper, we study PCFG induction with a much larger number of nonterminal and preterminal symbols. We are partly motivated by the classic work of latent variable grammars in supervised constituency parsing Matsuzaki et al. (2005); Petrov et al. (2006); Liang et al. (2007); Cohen et al. (2012); Zhao et al. (2018). While the Penn treebank grammar contains only tens of nonterminals and preterminals, it has been found that dividing them into subtypes could significantly improves the parsing accuracy of the grammar. For example, the best model from Petrov et al. (2006) contains over 1000 nonterminal and preterminal symbols. We are also motivated by the recent work of Buhai et al. (2019) who show that when learning latent variable models, increasing the number of hidden states is often helpful; and by Chiu and Rush (2020) who show that a neural hidden Markov model with up to \(2^{16}\) hidden states can achieve surprisingly good performance in language modeling.

A major challenge in employing a large number of nonterminal and preterminal symbols is that representing and parsing with a PCFG requires a computational complexity that is cubic in its symbol number. To resolve the issue, we rely on a new parameterization form of PCFGs based on tensor decomposition, which reduces the computational complexity from cubic to at most quadratic. Furthermore, we apply neural parameterization to the new form, which is crucial for boosting unsupervised parsing performance of PCFGs as shown by Kim et al. (2019).

We empirically evaluate our approach across ten languages. On English WSJ, our best model with 500 preterminals and 250 nonterminals improves over the model with 60 preterminals and 30 nonter

minals by 6.3% mean F1 score, and we also observe consistent decrease in perplexity and overall increase in F1 score with more symbols in our model, thus confirming the effectiveness of using more symbols. Our best model also surpasses the strong baseline Compound PCFGs Kim et al. (2019) by 1.4% mean F1. We further conduct multilingual evaluation on nine additional languages. The evaluation results suggest good generalizability of our approach on languages beyond English.

Our key contributions can be summarized as follows: (1) We propose a new parameterization form of PCFGs based on tensor decomposition, which enables us to use a large number of symbols in PCFGs. (2) We further apply neural parameterization to improve unsupervised parsing performance. (3) We evaluate our model across ten languages and empirically show the effectiveness of our approach.

## 2 Related work

Grammar induction using neural networks:There is a recent resurgence of interest in unsupervised constituency parsing, mostly driven by neural network based methods Shen et al. (2018, 2019); Drozdov et al. (2019, 2020); Kim et al. (2019, 2019); Jin et al. (2019); Zhu et al. (2020). These methods can be categorized into two major groups: those built on top of a generative grammar and those without a grammar component. The approaches most related to ours belong to the first category, which use neural networks to produce grammar rule probabilities. Jin et al. (2019) use an invertible neural projection network (a.k.a. normalizing flow Rezende and Mohamed (2015)) to parameterize the preterminal rules of a PCFG. Kim et al. (2019) use neural networks to parameterize all the PCFG rules. Zhu et al. (2020) extend their work to lexicalized PCFGs, which are more expressive than PCFGs and can model both dependency and constituency parse trees simultaneously.

In other unsupervised syntactic induction tasks, there is also a trend to use neural networks to produce grammar rule probabilities. In unsupervised dependency parsing, the Dependency Model with Valence (DMV) Klein and Manning (2004) has been parameterized neurally to achieve higher induction accuracy Jiang et al. (2016); Yang et al. (2020). In part-of-speech (POS) induction, neurally parameterized Hidden Markov Models (HMM) also achieve state-of-the-art results Tran et al. (2016); He et al. (2018).

Tensor decomposition on PCFGs:Our work is closely related to Cohen et al. (2013) in that both use tensor decomposition to parameterize the probabilities of binary rules for the purpose of reducing the time complexity of the inside algorithm. However, Cohen et al. (2013) use this technique to speed up inference of an existing PCFG, and they need to actually perform tensor decomposition on the rule probability tensor of the PCFG. In contrast, we draw inspiration from this technique to design a new parameterization form of PCFG that can be directly learned from data. Since we do not have a probability tensor to start with, additional tricks have to be inserted in order to ensure validity of the parameterization, as will be discussed later.

## 3 Background

### Tensor form of PCFGs

PCFGs build upon context-free grammars (CFGs). We start by introducing CFGs and establishing notations. A CFG is defined as a 5-tuple \({\cal G}=({\cal S},{\cal N},{\cal P},\Sigma,{\cal R})\) where \({\cal S}\) is the start symbol, \({\cal N}\) is a finite set of nonterminal symbols, \({\cal P}\) is a finite set of preterminal symbols,2\(\Sigma\) is a finite set of terminal symbols, and \({\cal R}\) is a set of rules in the following form:

Footnote 2: Strictly, CFGs do not distinguish nonterminals \({\cal N}\) (constituent labels) from preterminals \({\cal P}\) (part-of-speech tags). They are both treated as nonterminals. \({\cal N},{\cal P},\Sigma\) satisfy \({\cal N}\cap{\cal P}=\emptyset\) and \(({\cal N}\cup{\cal P})\cap\Sigma=\emptyset\).

\[S \to A A A\in{\cal N}\] \[A \mapsto BC, A\in{\cal N}, B,C\in{\cal N}\cup{\cal P}\] \[T \to w, T\in{\cal P},w\in\Sigma\]

PCFGs extend CFGs by associating each rule \(r\in{\cal R}\) with a probability \(\pi_{r}\). Denote \(n\), \(p\), and \(q\) as the number of symbols in \({\cal N}\), \({\cal P}\), and \(\Sigma\), respectively. It is convenient to represent the probabilities of the binary rules in the tensor form:

\[{\bf T}_{h_{A},h_{B},h_{C}}=\pi_{A\to BC}\,,\quad{\bf T}\in{\mathbb{R}^{n \times m\times m}}\,,\]

where \({\bf T}\) is an order-3 tensor, \(m=n+p\), and \(h_{A}\in[0,n)\) and \(h_{B}\), \(h_{C}\in[0,m)\) are symbol indices. For the convenience of computation, we assign indices \([0,n)\) to nonterminals in \({\cal N}\) and \([n,m)\) to preterminals in \({\cal P}\). Similarly, for a preterminal rule we define

\[{\bf Q}_{h_{T},h_{w}}=\pi_{T\to w}\,,\quad{\bf Q}\in{\mathbb{R}^{p\times q}}\,.\] 

Again, \(h_{T}\) and \(h_{w}\) are the preterminal index and the terminal index, respectively. Finally, for a start rule we define

\[\mathbf{r}_{h_{A}}=\pi_{S\to A}\,,\quad\mathbf{r}\in\mathbb{R}^{n}\,.\]

Generative learning of PCFGs involves maximizing the log-likelihood of every observed sentence \(\boldsymbol{w}=w_{1},\ldots,w_{l}\):

\[\log p_{\theta}(\boldsymbol{w})=\log\sum_{t\in T_{\mathcal{G}}(\boldsymbol{ w})}p(t)\,,\]

where \(T_{\mathcal{G}}(\boldsymbol{w})\) contains all the parse trees of the sentence \(\boldsymbol{w}\) under a PCFG \(\mathcal{G}\). The probability of a parse tree \(t\in T_{\mathcal{G}}\) is defined as \(p(t)=\prod_{r\in t_{\mathcal{R}}}\pi_{r}\), where \(t_{\mathcal{R}}\) is the set of rules used in the derivation of t. \(\log p_{\theta}(\boldsymbol{w})\) can be estimated efficiently through the inside algorithm, which is fully differentiable and amenable to gradient optimization methods.

### Tensor form of the inside algorithm

We first pad \(\mathbf{T}\), \(\mathbf{Q}\), and \(\mathbf{r}\) with zeros such that \(\mathbf{T}\in\mathbb{R}^{m\times m\times m}\), \(\mathbf{Q}\in\mathbb{R}^{m\times q}\), \(\mathbf{r}\in\mathbb{R}^{m}\), and all of them can be indexed by both nonterminals and preterminals.

The inside algorithm computes the probability of a symbol \(A\) spanning a substring \(\boldsymbol{w}_{i,j}=w_{i},\dots,w_{j}\) in a recursive manner (\(0\leq i<j<l\)):

\[s_{i,j}^{A}=\sum_{k=i}^{j-1}\sum_{B,C}\pi_{A\to BC}\cdot s_{i, k}^{B}\cdot s_{k+1,j}^{C}\,.\] (1) Base Case: \[s_{i,i}^{T}=\pi_{T\twodot w_{i}}\,,0\leq i<l\,.\]

We use the tensor form of PCFGs to rewrite Equation 1 as:

\[\mathbf{s}_{i,j}^{h_{A}} =\sum_{k=i}^{j-1}\sum_{h_{B},h_{C}}\mathbf{T}_{h_{A},h_{B},h_{C}} \cdot\mathbf{s}_{i,k}^{h_{B}}\cdot\mathbf{s}_{k+1,j}^{h_{C}}\] \[=\sum_{k=i}^{j-1}\big{(}\mathbf{T}_{h_{A}}\cdot\mathbf{s}_{k+1, j}\big{)}\cdot\mathbf{s}_{i,k}\,, \tag{2}\]

where \(\mathbf{s}_{i,j}\), \(\mathbf{s}_{i,k}\), and \(\mathbf{s}_{k+1,j}\) are all \(m\)-dimensional vectors; the dimension \(h_{A}\) corresponds to the symbol \(A\). Thus

\[\mathbf{s}_{i,j}=\sum_{k=i}^{j-1}\big{(}\mathbf{T}\cdot\mathbf{s}_ {k+1,j}\big{)}\cdot\mathbf{s}_{i,k}\,. \tag{3}\]

Equation 3 represents the core computation of the inside algorithm as tensor-vector dot product. It is amenable to be accelerated on a parallel computing device such as GPUs. However, the time and space complexity is cubic in \(m\), which makes it impractical to use a large number of nonterminals and preterminals.

## 4 Parameterizing PCFGs based on tensor decomposition

The tensor form of the inside algorithm has a high computational complexity of \(\mathcal{O}(m^{3}l^{3})\). It hinders the algorithm from scaling to a large \(m\). To resolve the issue, we resort to a new parameterization form of PCFGs based on tensor decomposition (TD-PCFGs) (Cohen et al., 2013). As discussed in Section 2, while Cohen et al. (2013) use a TD-PCFG to approximate an existing PCFG for speedup in parsing, we regard a TD-PCFG as a stand-alone model and learn it directly from data.

The basic idea behind TD-PCFGs is using Kruskal decomposition of the order-3 tensor \(\mathbf{T}\). Specifically, we require \(\mathbf{T}\) to be in the Kruskal form,

\[\mathbf{T}=\sum_{l=1}^{d}\mathbf{T}^{(l)}\,,~{}~{}\mathbf{T}^{( l)}=\mathbf{u}^{(l)}\otimes\mathbf{v}^{(l)}\otimes\mathbf{w}^{(l)}\,, \tag{4}\]

where \(\mathbf{u}^{(l)}\in\mathbb{R}^{n}\) is a column vector of a matrix \(\mathbf{U}\in\mathbb{R}^{n\times d}\); \(\mathbf{v}^{(l)}\), \(\mathbf{w}^{(l)}\in\mathbb{R}^{m}\) are column vectors of matrices \(\mathbf{V}\), \(\mathbf{W}\in\mathbb{R}^{m\times d}\), respectively; \(\otimes\) indicates Kronecker product. Thus \(\mathbf{T}^{(l)}\in\mathbb{R}^{n\times m\times n}\) is an order-3 tensor and

\[\mathbf{T}^{(l)}_{i,j,k}=\mathbf{u}^{(l)}_{i}\cdot\mathbf{v}^{(l )}_{j}\cdot\mathbf{w}^{(l)}_{k}\,.\]

The Kruskal form of the tensor \(\mathbf{T}\) is crucial for reducing the computation of Equation 3. To show this, we let \(\mathbf{x}=\mathbf{s}_{i,k}\), \(\mathbf{y}=\mathbf{s}_{k+1,j}\), and \(\mathbf{z}\) be any summand in the right-hand side of Equation 3, so we have:

\[\mathbf{z}=(\mathbf{T}\cdot\mathbf{y})\cdot\mathbf{x}\,. \tag{5}\]

Substitute \(\mathbf{T}\) in Equation 4 into Equation 5 and consider the \(i\)-th dimension of \(\mathbf{z}\):

\[\mathbf{z}_{i} =(\mathbf{T}_{i}\cdot\mathbf{y})\cdot\mathbf{x}\] \[=\sum_{j=1}^{m}\sum_{k=1}^{m}\sum_{l=1}^{d}\mathbf{T}^{(l)}_{i,j, k}\cdot\mathbf{x}_{j}\cdot\mathbf{y}_{k}\] \[=\sum_{j=1}^{m}\sum_{k=1}^{m}\sum_{l=1}^{d}\mathbf{u}^{(l)}_{i} \cdot\mathbf{v}^{(l)}_{j}\cdot\mathbf{w}^{(l)}_{k}\cdot\mathbf{x}_{j}\cdot \mathbf{y}_{k}\] \[=\sum_{l=1}^{d}\mathbf{u}^{(l)}_{i}\cdot\left(\sum_{j=1}^{m} \mathbf{v}^{(l)}_{j}\cdot\mathbf{x}_{j}\right)\cdot\left(\sum_{k=1}^{m} \mathbf{w}^{(l)}_{k}\cdot\mathbf{y}_{k}\right)\] \[=\sum_{l=1}^{d}\mathbf{u}^{(l)}_{i}\cdot\left(\mathbf{x}^{T} \mathbf{v}^{(l)}\right)\cdot\left(\mathbf{y}^{T}\mathbf{w}^{(l)}\right)\] \[=\left(\mathbf{e}^{T}_{i}\mathbf{U}\right)\cdot\left(\left( \mathbf{V}^{T}\mathbf{x}\right)\odot\left(\mathbf{W}^{T}\mathbf{y}\right) \right)\,,\] (6) 

where \(\odot\) indicates Hadamard (element-wise) product; \({\bf e}_{i}\in\mathbb{R}^{m}\) is a one-hot vector that selects the \(i\)-th row of \({\bf U}\). We have padded \({\bf U}\) with zeros such that \({\bf U}\in\mathbb{R}^{m\times d}\) and the last \(m-n\) rows are all zeros. Thus

\[{\bf z=U}\cdot\left(\left({\bf V}^{T}{\bf x}\right)\odot\left({\bf W}^{T}{\bf y }\right)\right), \tag{7}\]

and accordingly,

\[{\bf s}_{i,j}={\bf U}\cdot\sum_{k=i}^{j-1}\left(\left({\bf V}^{T}{\bf s}_{i, k}\right)\odot\left({\bf W}^{T}{\bf s}_{k+1,j}\right)\right). \tag{8}\]

Equation 8 computes the inside probabilities using TD-PCFGs. It has a time complexity \(\mathcal{O}(md)\). By caching \({\bf V}^{T}{\bf s}_{i,k}\) and \({\bf W}^{T}{\bf s}_{k+1,j}\), the time complexity of the inside algorithm becomes \(\mathcal{O}(dl^{3}+mll^{2})\)Cohen et al. (2013), which is at most quadratic in \(m\) since we typically set \(d=\mathcal{O}(m)\). Interestingly, Equation 8 has similar forms to recursive neural networks Socher et al. (2013) if we treat inside score vectors as span embeddings.

One problem with TD-PCFGs is that, since we use three matrices \({\bf U},{\bf V}\) and \({\bf W}\) to represent tensor \({\bf T}\) of binary rule probabilities, how we can ensure that \({\bf T}\) is non-negative and properly normalized, i.e., for a given left-hand side symbol \(A\), \(\sum_{j,k}{\bf T}_{h_{A},j,k}=1\). Simply reconstructing \({\bf T}\) with \({\bf U},{\bf V}\) and \({\bf W}\) and then performing normalization would take \(\mathcal{O}(m^{3})\) time, thus defeating the purpose of TD-PCFGs. Our solution is to require that the three matrices are non-negative and meanwhile \({\bf U}\) is row-normalized and \({\bf V}\) and \({\bf W}\) are column-normalized Shen et al. (2018b).

**Theorem 1**.: _Given non-negative matrices \({\bf U}\in\mathbb{R}^{n\times d}\) and \({\bf V},{\bf W}\in\mathbb{R}^{m\times d}\), if \({\bf U}\) is row-normalized and \({\bf V}\) and \({\bf W}\) are column-normalized, then \({\bf U}\), \({\bf V}\), and \({\bf W}\) are a Kruskal decomposition of a tensor \({\bf T}\in\mathbb{R}^{n\times m\times m}\) where \({\bf T}_{i,j,k}\in[0,1]\) and \({\bf T}_{i}\) is normalized such that \(\sum_{j,k}{\bf T}_{i,j,k}=1\)._

Proof.: \[\sum_{j=1}^{m}\sum_{k=1}^{m}{\bf T}_{i,j,k} =\sum_{j=1}^{m}\sum_{k=1}^{d}\mathbf{u}_{i}^{(l)}\cdot\mathbf{v}_ {j}^{(l)}\cdot\mathbf{w}_{k}^{(l)}\] \[=\sum_{l=1}^{d}\mathbf{u}_{i}^{(l)}\cdot(\sum_{j=1}^{m}\mathbf{ v}_{j}^{(l)})\cdot(\sum_{k=1}^{m}\mathbf{w}_{j}^{(l)})\] \[=\sum_{l=1}^{d}\mathbf{u}_{i}^{(l)}\cdot 1\cdot 1=1\]

## 5 Neural parameterization of TD-PCFGs

We use neural parameterization for TD-PCFGs as it has demonstrated its effectiveness in inducing PCFGs Kim et al. (2019a). In a neurally parameterized TD-PCFGs, the original TD-PCFG parameters are generated by neural networks, rather than being learned directly; parameters of the neural network will thus be the parameters to be optimized. This modeling approach breaks the parameter number limit of the original TD-PCFG, so we can control the total number of parameters flexibly. When the total number of symbols is small, we can over-parameterize the model as over-parameterization has been shown to ease optimization Arora et al. (2018); Xu et al. (2018); Du et al. (2019). On the other hand, when the total number of symbols is huge, we can decrease the number of parameters to save GPU memories and speed up training.

Specifically, we use neural networks to generate the following set of parameters of a TD-PCFG:

\[\Theta=\left\{{\bf U}\,,~{}{\bf V}\,,~{}{\bf W}\,,~{}{\bf Q}\,,~{}{\bf r} \right\}.\]

The resulting model is referred to as neural PCFGs based on tensor decomposition (TN-PCFGs).

We start with the neural parameterization of \({\bf U}\in\mathbb{R}^{n\times d}\) and \({\bf V},{\bf W}\in\mathbb{R}^{m\times d}\). We use shared symbol embeddings \({\bf E}_{s}\in\mathbb{R}^{m\times k}\) (\(k\) is the symbol embedding dimension) in which each row is the embedding of a nonterminal or preterminal. We first compute an unnormalized \(\tilde{{\bf U}}\) by applying a neural network \(f_{u}(\cdot)\) to symbol embeddings \({\bf E}_{s}\):

\[\tilde{{\bf U}}=f_{u}({\bf E}_{s})=\left({\rm ReLU}\left({\bf E}_{s}{\bf M} _{u}^{(1)}\right)\right){\bf M}_{u}^{(2)}\,,\]

where \({\bf M}_{u}^{(1)}\in\mathbb{R}^{k\times k}\) and \({\bf M}_{u}^{(2)}\in\mathbb{R}^{k\times d}\) are learnable parameters of \(f_{u}(\cdot)\). For simplicity, we omit the learnable bias terms. We compute unnormalized \(\tilde{{\bf V}}\) and \(\tilde{{\bf W}}\) in a similar way. Note that only \({\bf E}_{s}\) is shared in computing the three unnormalized matrices. Then we apply the Softmax activation function to each row of \(\tilde{{\bf U}}\) and to each column of \(\hat{{\bf V}}\) and \(\tilde{{\bf W}}\), and obtain normalized \({\bf U}\), \({\bf V}\), and \({\bf W}\).

For preterminal-rule probabilities \({\bf Q}\in\mathbb{R}^{p\times q}\) and start-rule probabilities \({\bf r}\in\mathbb{R}^{n}\), we follow Kim et al. (2019a) and define them as:

\[{\bf Q}_{h_{T},h_{W}} =\pi_{T\to w}=\frac{\exp({\bf u}_{w}^{T}f_{t}({\bf w}_{T}))}{\sum_{w^{ \prime}\in\Sigma}\exp({\bf u}_{w^{\prime}}^{T}f_{t}({\bf w}_{T}))}\,,\] \[{\bf r}_{h_{A}} =\pi_{S\to A}=\frac{\exp({\bf u}_{M}^{T}f_{s}({\bf w}_{S}))}{\sum_{A^{ \prime}\in{\cal N}}\exp({\bf u}_{A^{\prime}}^{T}f_{s}({\bf w}_{S}))}\,,\] 

where \({\bf w}\) and \({\bf u}\) are symbol embeddings; \(f_{s}(\cdotp)\) and \(f_{t}(\cdotp)\) are neural networks that encode the input into a vector (see details in Kim et al. (2019a)). Note that the symbol embeddings are not shared between preterminal rules and start rules.

## 6 Parsing with TD-PCFGs

Parsing seeks the most probable parse \(t^{\star}\) from all the parses \(T_{\mathcal{G}}(\mathbf{w})\) of a sentence \(\mathbf{w}\):

\[t^{\star}=\arg\max_{t\in T_{\mathcal{G}}(\mathbf{w})}p(t|\mathbf{w})\,. \tag{9}\]

Typically, the CYK algorithm3 can be directly used to solve this problem exactly: it first computes the score of the most likely parse; and then automatic differentiation is applied to recover the best tree structure \(t^{\star}\)Eisner (2016); Rush (2020). This, however, relies on the original probability tensor \({\bf T}\) and is incompatible with our decomposed representation.4 If we reconstruct \({\bf T}\) from \({\bf U}\), \({\bf V}\), \({\bf W}\) and then perform CYK, then the resulting time and space complexity would degrade to \(\mathcal{O}(m^{3}l^{3})\) and become unaffordable when \(m\) is large. Therefore, we resort to Minimum Bayes-Risk (MBR) style decoding because we can compute the inside probabilities efficiently.

Footnote 3: The CYK algorithm is similar to the inside algorithm. The only difference is that it uses Max whenever the inside algorithm performs Sum over \(k\) and \(B,C\) (_cf._ Equation 1).

Our decoding method consists of two stages. The first stage computes the conditional probability of a substring \(\mathbf{w}_{i,j}\) being a constituent in a given sentence \(\mathbf{w}\) (a.k.a. posteriors of spans being a constituent):

\[p(\mathbf{w}_{i,j}|\mathbf{w})=\frac{1}{p(\mathbf{w})}\sum_{t\in T_{\mathcal{G}}(\mathbf{w})} p(t)\cdot\mathbbm{1}_{\left\{\mathbf{w}_{i,j}\in t\right\}}\,.\]

We can estimate the posteriors efficiently by using automatic differentiation after obtaining all the inside probabilities. This has the same time complexity as our improved inside algorithm, which is \(\mathcal{O}(dl^{3}+mdl^{2})\). The second stage uses the CYK algorithm to find the parse tree that has the highest expected number of constituentsSmith and Eisner (2006):

\[t^{\star}=\arg\max_{t\in T_{\mathcal{G}}(\mathbf{w})}\sum_{\mathbf{w}_{i,j}\in t}p( \mathbf{w}_{i,j}|\mathbf{w})\,. \tag{10}\]

The time complexity of the second stage is \(\mathcal{O}(l^{3})\), so the overall time complexity of our decoding method is \(\mathcal{O}(dl^{3}+mdl^{2})\), which is much faster than \(\mathcal{O}(m^{3}l^{3})\) in general.

## 7 Experimental setup

### Datasets

We evaluate TN-PCFGs across ten languages. We use the Wall Street Journal (WSJ) corpus of the Penn Treebank Marcus et al. (1994) for English, the Penn Chinese Treebank 5.1 (CTB) Xue et al. (2005) for Chinese, and the SPRML dataset Seddah et al. (2014) for the other eight morphology-rich languages. We use a unified data preprocessing pipeline5 provided by Zhao and Titov (2021). The same pipeline has been used in several recent papers Shen et al. (2018, 2019); Kim et al. (2019a); Zhao and Titov (2020). Specifically, for every treebank, punctuation is removed from all data splits and the top 10,000 frequent words in the training data are used as the vocabulary.

Footnote 5: [https://github.com/zhaoyanpeng/xcfg](https://github.com/zhaoyanpeng/xcfg).

### Settings and hyperparameters

For baseline models we use the best configurations reported by the authors. For example, we use 30 nonterminals and 60 preterminals for N-PCFGs and C-PCFGs. We implement TN-PCFGs and reimplement N-PCFGs and C-PCFGs using automatic differentiation Eisner (2016) and we borrow the idea of Zhang et al. (2020) to batchify the inside algorithm. Inspired by Kim et al. (2019a), for TN-PCFGs we set \(n/p\), the ratio of the nonterminal number to the preterminal number, to \(1/2\). For \({\bf U}\in\mathbb{R}^{n\times d}\) and \({\bf V},{\bf W}\in\mathbb{R}^{m\times d}\) we set \(d=p\) when there are more than 200 preterminals and \(d=200\) otherwise. The symbol embedding dimension \(k\) is set to 256. We optimize TN-PCFGs using the Adam optimizer Kingma and Ba (2015) with \(\beta_{1}=0.75\), \(\beta_{2}=0.999\), and learning rate 0.001 with batch size 4. We use the unit Gaussian distribution to initialize embedding parameters. We do not use the curriculum learning strategy that is used by Kim et al. (2019a) when training TN-PCFGs.

### Evaluation

Following Kim et al. (2019a), we train a TN-PCFG for each treebank separately. For each setting we run the TN-PCFG and the baselines four times with different random seeds and for ten epochs each 

time. Early stopping is performed based on the perplexity on the development data. The best model in each run is selected according to the perplexity on the development data. We tune model hyperparameters only on the development data of WSJ and use the same model configurations on the other treebanks.6 We report average sentence-level F1 score7 as well as their biased standard deviations.

Footnote 6: Shi et al. (2020) suggest not using the gold parses of the development data for hyperparameter tuning and model selection in unsupervised parsing. Here we still use the gold parses of the WSJ development set for the English experiments in order to conduct fair comparison with previous work. No gold parse is used in the experiments of any other language.

Footnote 7: Following Kim et al. (2019), we remove all trivial spans (single-word spans and sentence-level spans). Sentence-level means that we compute F1 for each sentence and then average over all sentences.

## 8 Experimental results

We evaluate our models mainly on WSJ (Section 8.1-8.3). We first give an overview of model performance in Section 8.1 and then conduct ablation study of TN-PCFGs in Section 8.2. We quantitatively and qualitatively analyze constituent labels induced by TN-PCFGs in Section 8.3. In Section 8.4, we conduct a multilingual evaluation over nine additional languages.

### Main results

Our best TN-PCFG model uses 500 preterminals (\(p=500\)). We compare it with a wide range of recent unsupervised parsing models (see the top section of Table 1). Since we use MBR decoding for TN-PCFGs, which produces higher F1-measure than the CYK decoding Goodman (1996), for fair comparison we also use MBR decoding for our reimplemented N-PCFGs and C-PCFGs (see the middle section of Table 1).

We draw three key observations from Table 1: (1) TN-PCFG (\(p=500\)) achieves the best mean and max F1 score. Notebly, it outperforms the strong baseline model C-PCFG by 1.4% mean F1. Compared with TN-PCFG (\(p=60\)), TN-PCFG (\(p=500\)) brings a 6.3% mean F1 improvement, demonstrating the effectiveness of using more symbols. (2) Our reimplementations of N-PCFGs and C-PCFGs are comparable to those of Kim et al. (2019), (3) MBR decoding indeed gives higher F1 scores (+1.4% mean F1 for N-PCFG and +0.9% mean F1 for C-PCFG).

In Table 1 we also show the results of Constituent test (CT) Cao et al. (2020) and DIORA Drozdov et al. (2019, 2020), two recent state-of-the-art approaches. However, our work is not directly comparable to these approaches. CT relies on pretrained language models (RoBERTa) and DIORA relies on pretrained word embeddings (context insensitive ELMo). In contrast, our model and the other approaches do not use pretrained word embeddings and instead learn word embeddings from scratch. We are also aware of URNNG Kim et al. (2019), which has a max F1 score of 45.4%, but it uses punctuation and hence is not directly comparable to the models listed in the table.

\begin{table}
\begin{tabular}{l c c} \hline \hline Model & \multicolumn{2}{c}{WSJ} \\ \cline{2-3}  & Mean & Max \\ \hline Left Branching & & 8.7 \\ Right Branching & & 39.5 \\ Random Trees & 18.1 & 18.2 \\ \hline \multicolumn{3}{c}{Systems without pretrained word embeddings} \\ \hline PRPN\({}^{\dagger}\) Shen et al. (2018) & 47.3 & 47.9 \\ ON\({}^{\dagger}\) Shen et al. (2019) & 48.1 & 50.0 \\ N-PCFG Kim et al. (2019) & 50.8 & 52.6 \\ C-PCFG Kim et al. (2019) & 55.2 & 60.1 \\ NL-PCFG Zhu et al. (2020) & 55.3 & \\ \hline N-PCFg\({}^{\star}\) & 50.9\({}_{\pm 2.3}\) & 54.6 \\ N-PCfg\({}^{\star}\) w/ MBR & 52.3\({}_{\pm 2.3}\) & 55.8 \\ C-PCFG\({}^{\star}\) & 55.4\({}_{\pm 2.2}\) & 59.0 \\ C-PCFG\({}^{\star}\) w/ MBR & 56.3\({}_{\pm 2.1}\) & 60.0 \\ TN-PCFG \(p=60\) (ours) & 51.4\({}_{\pm 4.0}\) & 55.6 \\ TN-PCFG \(p=500\) (ours) & **57.7\({}_{\pm 4.2}\)** & **61.4** \\ \hline \multicolumn{3}{c}{Systems with pretrained word embeddings} \\ \hline DIORA Drozdov et al. (2019) & & 56.8 \\ S-DIORA Drozdov et al. (2020) & 57.6 & 64.0 \\ CT Cao et al. (2020) & 62.8 & 65.9 \\ \hline Oracle Trees & & 84.3 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Unlabeled sentence-level F1 scores on the WSJ test data. \({}^{\dagger}\) indicates numbers reported by Kim et al. (2019). \({}^{\star}\) indicates our reimplementations of N-PCFGs and C-PCFGs. \(p\) denotes the preterminal number.

\begin{table}
\begin{tabular}{l r l} \hline \hline
**Model** & \multicolumn{2}{c}{**Time (minutes)**} & **Total Parameters (M)** \\ \hline N-PCFG & 19 & 5.3 \\ C-PCFG & 20 & 15.4 \\ TN-PCFG (\(p=60\)) & 13 & 3.6 \\ TN-PCFG (\(p=500\)) & 26 & 4.2 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Average running time per epoch and the parameter number of each model.

 

We report the average running time8 per epoch and the parameter numbers of different models in Table 2. We can see that TN-PCFG (\(p=500\)), which uses a much larger number of symbols, has even fewer parameters and is not significantly slower than N-PCFG.

Footnote 8: We measure the running time on a single Titan V GPU.

### Influence of symbol number

Figure 1 illustrates the change of F1 scores and perplexities as the number of nonterminals and preterminals increase. We can see that, as the symbol number increases, the perplexities decrease while F1 scores tend to increase.

### Analysis on constituent labels

We analyze model performance by breaking down recall numbers by constituent labels (Table 3). We use the top six frequent constituent labels in the WSJ test data (NP, VP, PP, SBAR, ADJP, and ADVP). We first observe that the right-branching baseline remains competitive. It achieves the highest recall on VPs and SBARs. TN-PCFG (\(p=500\)) displays a relatively even performance across the six labels. Specifically, it performs best on NPs and PPs among all the labels and it beats all the other models on ADJPs. Compared with TN-PCFG (\(p\) = 60), TN-PCFG (\(p\) = 500) results in the largest improvement on VPs (+19.5% recall), which are usually long (with an average length of 11) in comparison with the other types of constituents. As NPs and VPs cover about 54% of the total constituents in the WSJ test data, it is not surprising that models which are accurate on these labels have high F1 scores (e.g., C-PCFGs and TN-PCFGs (\(p=500\))).

We further analyze the correspondence between the nonterminals of trained models and gold constituent labels. For each model, we look at all the correctly-predicted constituents in the test set and estimate the empirical posterior distribution of nonterminals assigned to a constituent given the gold label of the constituent (see Figure 2). Compared with the other three models, in TN-PCFG (\(p=500\)), the most frequent nonterminals are more likely to correspond to a single gold label. One possible explanation is that it contains much more nonterminals and therefore constituents of different labels are less likely to compete for the same nonterminal.

Figure 2d (TN-PCFG (\(p=500\))) also illustrates that a gold label may correspond to multiple nonterminals. A natural question that follows is: _do these nonterminals capture different subtypes of the gold label?_ We find it is indeed the case for some nonterminals. Take the gold label NPs (noun phrases), while not all the nonterminals have clear interpretation, we find that NT-3 corresponds to constituents which represent a company name; NT-99 corresponds to constituents which contain a possessive affix (e.g., "'s" in "the market 's decline"); NT-94 represents constituents preceded by an indefinite article. We further look into the gold label PPs (preposition phrases). Interestingly, NT-108, NT-175, and NT-218 roughly divided preposition phrases into three groups starting with 'with, by, from, to', 'in, on, for', and 'of', respectively. See Appendix for more examples.

### Multilingual evaluation

In order to understand the generalizability of TD-PCFGs on languages beyond English, we conduct a multilingual evaluation of TD-PCFGs on CTB and SPMRL. We use the best model configurations obtained on the English development data and do not perform any further tuning on CTB and SPMRL. We compare TN-PCFGs with N-PCFGs and C-PCFGs and use MBR decoding by default. The results are shown in Table 4. In terms of the average F1 over the nine languages, all the three models beat trivial left- and right-branching baselines by a large margin, which suggests they have good generalizability on languages beyond English. Among the three models, TN-PCFG (\(p=500\)) fares best. It achieves the highest F1 score on six out of nine treebanks. On Swedish, N-PCFG is worse than the right-branching baseline (-13.4% F1), while

Figure 1: F1 scores and perplexities w.r.t. the preterminal number \(p\) of TN-PCFGs on the WSJ test data. Recall that the nonterminal number \(n\) is set to half of \(p\).

 

TN-PCFG (\(p=500\)) surpasses the right-branching baseline by 9.6% F1.

## 9 Discussions

In our experiments, we do not find it beneficial to use the compound trick Kim et al. (2019) in TN-PCFGs, which is commonly used in previous work of PCFG induction Kim et al. (2019); Zhao

\begin{table}
\begin{tabular}{r c c c c c c c c|c} \hline \hline  & **Model** & **Chinese** & **Basque** & **German** & **French** & **Hebrew** & **Hungarian** & **Korean** & **Polish** & **Swedish** & **Mean** \\ \hline Left Branching\({}^{\dagger}\) & 7.2 & 17.9 & 10.0 & 5.7 & 8.5 & 13.3 & 18.5 & 10.9 & 8.4 & 11.2 \\ Right Branching\({}^{\ddagger}\) & 25.5 & 15.4 & 14.7 & 26.4 & 30.0 & 12.7 & 19.2 & 34.2 & 30.4 & 23.2 \\ Random Trees\({}^{\ast}\) & 15.2 & 19.5 & 13.9 & 16.2 & 19.7 & 14.1 & 22.2 & 21.4 & 16.4 & 17.6 \\ N-PCFG w MBR & 26.3\({}_{\pm 2.5}\) & 35.1\({}_{\pm 2.0}\) & 42.3\({}_{\pm 1.6}\) & **45.0\({}_{\pm 2.0}\)** & **45.7\({}_{\pm 2.2}\)** & 43.5\({}_{\pm 1.2}\) & 28.4\({}_{\pm 6.5}\) & 43.2\({}_{\pm 2.8}\) & 17.0 \({}_{\pm 9.9}\) & 36.3 \\ C-PCFG w/ MBR & 38.7\({}_{\pm 6.6}\) & 36.0\({}_{\pm 1.2}\) & 43.5\({}_{\pm 1.2}\) & 45.0\({}_{\pm 1.1}\) & 45.2\({}_{\pm 0.5}\) & **44.9\({}_{\pm 1.5}\)** & 30.5\({}_{\pm 4.2}\) & 43.8\({}_{\pm 1.3}\) & 33.0\({}_{\pm 15.4}\) & 40.1 \\ TN-PCFG \(p=500\) & **39.2\({}_{\pm 5.0}\)** & **36.0\({}_{\pm 3.0}\)** & **47.1\({}_{\pm 1.7}\)** & 39.1\({}_{\pm 1.4}\) & 39.2\({}_{\pm 10.7}\) & 43.1\({}_{\pm 1.1}\) & **35.4\({}_{\pm 2.8}\)** & **48.6\({}_{\pm 3.1}\)** & **40.0 \({}_{\pm 4.8}\)** & **40.9** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Sentence-level F1 scores on CTB and SPMRL. \({}^{\dagger}\) denotes results reported by Zhao and Titov (2021).

Figure 2: Correspondence between nonterminals and gold constituent labels. For each gold label, we visualize the proportion of correctly-predicted constituents that correspond to each nonterminal. Nonterminals NT-# are listed in descending order of the prediction frequency (e.g., NT-21 is the most predicted nonterminal in TN-PCFG (\(p=500\)). Only the top 30 frequent nonterminals are listed. We show the seven most frequent gold labels and aggregate the rest (denoted by OTHER).

 

and Titov, 2020; Zhu et al., 2020). We speculate that the additional expressiveness brought by compound parameterization may not be necessary for a TN-PCFG with many symbols which is already sufficiently expressive; on the other hand, compound parameterization makes learning harder when we use more symbols.

We also find neural parameterization and the choice of nonlinear activation functions greatly influence the performance. Without using neural parameterization, TD-PCFGs have only around \(30\%\) S-F1 scores on WSJ, which are even worse than the right-branching baseline. Activation functions other than ReLU (such as tanh and sigmoid) result in much worse performance. It is an interesting open question why ReLU and neural parameterization are crucial in PCFG induction.

When evaluating our model with a large number of symbols, we find that only a small fraction of the symbols are predicted in the parse trees (for example, when our model uses 250 nonterminals, only tens of them are found in the predicted parse trees of the test corpus). We expect that our models can benefit from regularization techniques such as state dropout Chiu and Rush (2020).

## 10 Conclusion

We have presented TD-PCFGs, a new parameterization form of PCFGs based on tensor decomposition. TD-PCFGs rely on Kruskal decomposition of the binary-rule probability tensor to reduce the computational complexity of PCFG representation and parsing from cubic to at most quadratic in the symbol number, which allows us to scale up TD-PCFGs to a much larger number of (nonterminal and preterminal) symbols. We further propose neurally parameterized TD-PCFGs (TN-PCFGs) and learn neural networks to produce the parameters of TD-PCFGs. On WSJ test data, TN-PCFGs outperform strong baseline models; we empirically show that using more nonterminal and preterminal symbols contributes to the high unsupervised parsing performance of TN-PCFGs. Our multilingual evaluation on nine additional languages further reveals the capability of TN-PCFGs to generalize to languages beyond English.

## Acknowledgements

This work was supported by the National Natural Science Foundation of China (61976139).

## References

* Arora et al. (2018) Sanjeev Arora, Nadav Cohen, and Elad Hazan. 2018. On the optimization of deep networks: Implicit acceleration by overparameterization. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmassan, Stockholm, Sweden, July 10-15, 2018, pp. 80 of Proceedings of Machine Learning Research, pages 244-253. PMLR.
* Buhai et al. (2019) Rares-Darius Buhai, Yoni Halpern, Yoon Kim, Andrej Risteski, and David Sontag. 2019. Empirical study of the benefits of overparameterization in learning latent variable models. arXiv preprint arXiv:1907.00030.
* Cao et al. (2020) Steven Cao, Nikita Kitaev, and Dan Klein. 2020. Unsupervised parsing via constituency tests. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online. Association for Computational Linguistics.
* Carroll and Charniak (1992) Glenn Carroll and Eugene Charniak. 1992. _Two experiments on learning probabilistic dependency grammars from corpora_. Department of Computer Science, Univ.
* Chiu and Rush (2020) Justin Chiu and Alexander Rush. 2020. Scaling hidden Markov language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online. Association for Computational Linguistics.
* Cohen et al. (2013) Shay B. Cohen, Giorgio Satta, and Michael Collins. 2013. Approximate PCFG parsing using tensor decomposition. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Atlanta, Georgia. Association for Computational Linguistics.
* Cohen et al. (2012) Shay B. Cohen, Karl Stratos, Michael Collins, Dean P. Foster, and Lyle Ungar. 2012. Spectral learning of latent-variable PCFGs. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Jeju Island, Korea. Association for Computational Linguistics.
* Drozdov et al. (2020) Andrew Drozdov, Subendhu Rongali, Yi-Pei Chen, Tim O'Gorman, Mohit Iyyer, and Andrew McCallum. 2020. Unsupervised parsing with S-DIORA: Single tree encoding for deep inside-outside recursive autoencoders. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online. Association for Computational Linguistics.
* Drozdov et al. (2019) Andrew Drozdov, Patrick Verga, Mohit Yadav, Mohit Iyyer, and Andrew McCallum. 2019. Unsupervised latent tree induction with deep inside-outside recursive auto-encoders. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan 

guage Technologies, Volume 1 (Long and Short Papers)_, pages 1129-1141, Minneapolis, Minnesota. Association for Computational Linguistics.
* Du et al. (2019) Simon S. Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. 2019. Gradient descent provably optimizes over-parameterized neural networks. In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
* Eisner (2016) Jason Eisner. 2016. Inside-outside and forward-backward algorithms are just backprop (tutorial paper). In _Proceedings of the Workshop on Structured Prediction for NLP_, pages 1-17, Austin, TX. Association for Computational Linguistics.
* Goodman (1996) Joshua Goodman. 1996. Parsing algorithms and metrics. In _34th Annual Meeting of the Association for Computational Linguistics_, pages 177-183, Santa Cruz, California, USA. Association for Computational Linguistics.
* He et al. (2018) Junxian He, Graham Neubig, and Taylor BergKirkpatrick. 2018. Unsupervised learning of syntactic structure with invertible neural projections. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 1292-1302, Brussels, Belgium. Association for Computational Linguistics.
* Jiang et al. (2016) Yong Jiang, Wenjuan Han, and Kewei Tu. 2016. Unsupervised neural dependency parsing. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 763-771, Austin, Texas. Association for Computational Linguistics.
* Jin et al. (2019) Lifeng Jin, Finale Doshi-Velez, Timothy Miller, Lane Schwartz, and William Schuler. 2019. Unsupervised learning of PCFGs with normalizing flow. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 2442-2452, Florence, Italy. Association for Computational Linguistics.
* Kim et al. (2019a) Yoon Kim, Chris Dyer, and Alexander Rush. 2019a. Compound probabilistic context-free grammars for grammar induction. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 2369-2385, Florence, Italy. Association for Computational Linguistics.
* Kim et al. (2019b) Yoon Kim, Alexander Rush, Lei Yu, Adhiguna Koncro, Chris Dyer, and Gabor Melis. 2019b. Unsupervised recurrent neural network grammars. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 1105-1117, Minneapolis, Minnesota. Association for Computational Linguistics.
* Kingma and Ba (2015) Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In _Proceedings of the 3rd International Conference on Learning Representations_.
* Klein and Manning (2004) Dan Klein and Christopher Manning. 2004. Corpus-based induction of syntactic structure: Models of dependency and constituency. In _Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04)_, pages 478-485, Barcelona, Spain.
* Lari and Young (1990) Karim Lari and Steve J Young. 1990. The estimation of stochastic context-free grammars using the inside-outside algorithm. _Computer speech & language_, 4(1):35-56.
* Liang et al. (2007) Percy Liang, Slav Petrov, Michael Jordan, and Dan Klein. 2007. The infinite PCFG using hierarchical Dirichlet processes. In _Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL)_, pages 688-697, Prague, Czech Republic. Association for Computational Linguistics.
* Marcus et al. (1994) Mitchell Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasperger. 1994. The Penn Treebank: Annotating predicate argument structure. In _Human Language Technology; Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994_.
* Matsuzaki et al. (2005) Takuya Matsuzaki, Yusuke Miyao, and Jun'ichi Tsuji. 2005. Probabilistic CFG with latent annotations. In _Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL'05)_, pages 75-82, Ann Arbor, Michigan. Association for Computational Linguistics.
* Petrov et al. (2006) Slav Petrov, Leon Barrett, Romain Thibaux, and Dan Klein. 2006. Learning accurate, compact, and interpretable tree annotation. In _Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics_, pages 433-440, Sydney, Australia. Association for Computational Linguistics.
* Rezende and Mohamed (2015) Danilo Jimenez Rezende and Shakir Mohamed. 2015. Variational inference with normalizing flows. In _Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015_, volume 37 of _JMLR Workshop and Conference Proceedings_, pages 1530-1538. JMLR.org.
* Rush (2020) Alexander Rush. 2020. Torch-struct: Deep structured prediction library. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_, pages 335-342, Online. Association for Computational Linguistics.
* Seddah et al. (2014) Djame Seddah, Sandra Kubler, and Reut Tsarfaty. 2014. Introducing the SPMRL 2014 shared task on parsing morphologically-rich languages. In _Proceedings 

of the First Joint Workshop on Statistical Parsing of Morphologically Rich Languages and Syntactic Analysis of Non-Canonical Languages_, pages 103-109, Dublin, Ireland. Dublin City University.
* Shen et al. (2018a) Yikang Shen, Zhouhan Lin, Chin wei Huang, and Aaron Courville. 2018a. Neural language modeling by jointly learning syntax and lexicon. In _International Conference on Learning Representations_.
* Shen et al. (2019) Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron Courville. 2019. Ordered neurons: Integrating tree structures into recurrent neural networks. In _International Conference on Learning Representations_.
* Shen et al. (2018b) Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. 2018b. Efficient attention: Attention with linear complexities. _arXiv preprint arXiv:1812.01243_.
* Shi et al. (2020) Haoyue Shi, Karen Livescu, and Kevin Gimpel. 2020. On the role of supervision in unsupervised constituency parsing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 7611-7621, Online. Association for Computational Linguistics.
* Smith and Eisner (2006) David A. Smith and Jason Eisner. 2006. Minimum risk annealing for training log-linear models. In _ACL 2006, 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, Sydney, Australia, 17-21 July 2006_. The Association for Computer Linguistics.
* Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_, pages 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.
* Tran et al. (2016) Ke M. Tran, Yonatan Bisk, Ashish Vaswani, Daniel Marcu, and Kevin Knight. 2016. Unsupervised neural hidden Markov models. In _Proceedings of the Workshop on Structured Prediction for NLP_, pages 63-71, Austin, TX. Association for Computational Linguistics.
* Xu et al. (2018) Ji Xu, Daniel J. Hsu, and Arian Maleki. 2018. Benefits of over-parameterization with EM. In _Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018, Montreal, Canada_, pages 10685-10695.
* Xue et al. (2005) Naiwen Xue, Fei Xia, Fu-dong Chiou, and Marta Palmer. 2005. The penn chinese treebank: Phrase structure annotation of a large corpus. _Natural Language Engineering_, 11(2):207-238.
* Yang et al. (2020) Songlin Yang, Yong Jiang, Wenjuan Han, and Kewei Tu. 2020. Second-order unsupervised neural dependency parsing. In _COLING_.
* Zhang et al. (2020) Yu Zhang, Houquan Zhou, and Zhenghua Li. 2020. Fast and accurate neural crf constituency parsing. In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20_, pages 4046-4053. International Joint Conferences on Artificial Intelligence Organization Main track.
* Zhao and Titov (2020) Yanpeng Zhao and Ivan Titov. 2020. Visually grounded compound PCFGs. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 4369-4379, Online. Association for Computational Linguistics.
* Zhao and Titov (2021) Yanpeng Zhao and Ivan Titov. 2021. An empirical study of compound pcfgs. _arXiv preprint arXiv:2103.02298_.
* Zhao et al. (2018) Yanpeng Zhao, Liwen Zhang, and Kewei Tu. 2018. Gaussian mixture latent vector grammars. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1181-1189, Melbourne, Australia. Association for Computational Linguistics.
* Zhu et al. (2020) Hao Zhu, Yonatan Bisk, and Graham Neubig. 2020. The return of lexical dependencies: Neural lexicalized pcfgs. _CoRR_, abs/2007.15135.

## Appendix

To evaluate whether the nonterminals of trained models capture meaningful constituent clusters, we predict the unlabeled parse trees in the test set using MBR decoding, then choose the correctly predicted constituents and assign the maximum a posterior (MAP) nonterminal symbols to them. Table 5 lists some representative nonterminals and their example constituents 



\begin{table}
\begin{tabular}{l|c|p{142.3pt}} \hline
**Nonterminal** & **Description** & \\ \hline  & & Quantum Chemical Corp. / Dow Chemical Co. / Union Carbide Corp. / First Boston Corp. / Petroleum Inc. / British Petroleum Co. / Ford Motor Co. / Jaguar PLC / Jaguar shares / General Motors Corp. / Northern Trust Co. / Norwest Corp. / Fidelity Investments / Windsor Fund ’s / Vanguard Group Inc. \\ \hline  & & Quantum ’s shares / Quantum ’s fortunes / year ’s end / the company ’s swap / September ’s steep rise / the stock market ’s plunge / the market ’s decline / this year ’s good inflation news / the nation ’s largest fund company / Fidelity ’s stock funds / Friday ’s close / today ’s closing prices / this year ’s fat profits \\ \hline  & & an acquisition of Petrolane Inc. / a transaction / a crisis / a bid / a minority interest / a role / a 0.7 % gain / an imminent easing of monetary policy / a blip / a steep run-up/ a steeper rise / a substantial rise / a broad-based advance / a one-time event / a buffer / a group / an opportunity / a buying opportunity \\ \hline  & & NP 99.8 million / 34.375 share /85 cents /a year ago /36 \% / 1.18 billion /15 \% / eight members / 0.9 \% / 32.8 billion / 1.9 \% / 1982 = 100 / about 2 billion / 8.8 \% and 9.2 \% / Ten points / 167.7 million shares / 20,000 shares / about 2,000 stores / up to 500 a square foot / 17 each / its 15 offering price / 190.58 points \\ \hline  & & The timing / nearly a quarter / the bottom / the troughs / the chemical building block / only about half / the start / the day / the drawn-out process / the possibility / the actions / the extent / the start / the pace / the beginning / the prices / less than 15 \% / two years / the volume / the chaos / the last hour \\ \hline  & & of plastics / of the chemical industry / of the boom / of 99.8 million or 3.92 a share / of its value / of Quantum ’s competitors / of the heap / of its resources / of the plastics market / of polyethylene / its ethylene needs /of a few cents a pound / of intense speculation / of retail equity trading / of stock / of Securities Dealers \\ \hline  & & to Mr. Hardiman / to pre-crash levels / through the third quarter / from institutions / through the market / to Contel Cellular / with investors / to the junk bond market / with borrowed money / by the House / from the deficit-reduction bill / by the end of the month / to the streamlined bill / by the Senate action / with the streamlined bill / from the Senate bill / to top paid executives / like idiots \\ \hline  & & for the ride / on plastics / for plastics producers / on plastics / for this article / at Oppenheimer \& Co. / on Wall Street / in a Sept. 29 meeting with analysts / in Morris III / in the hospital / between insured and insurer / for Quantum / in the U.S. / in the market ’s decline / top of a 0.7 \% gain in August / in the economy / in the producer price index / in the prices of consumer and industrial goods \\ \hline  & & Within the next year / To make its point / summer / Under the latest offer / At this price / In March / In the same month / Since April / In the third quarter / Right now / As she calculates it / Unlike 1987 course / in both instances / in the October 1987 crash / In 1984 / Under the agreement / In addition / In July / In computer publishing / On the other hand / In personal computers / At the same time / As a result of this trend / Beginning in mid-1987 / In addition \\ \hline  & & in August / in September / in July / for September / after Friday ’s close / in the House-passed bill / in the Senate bill / in fiscal 1990 / in tax increases / over five years / in air conditioners / in Styrofoam / in federal fees / in fiscal 1990 / in Chicago / in Japan / in Europe in Denver / at Salomon Bros / in California / in principle / in August / in operating costs / in Van Nuys / in San Diego / in Texas / in the dollar / in the past two years / in Dutch corporate law / since July / in 1964 / in June / for instance / in France / in Germany / in Palermo / for quotas \\ \hline \end{tabular}
\end{table}
Table 5: Nonterminals and the corresponding example constituents.