Venue: NAACL 2022

# Dynamic Programming in Rank Space: Scaling Structured Inference with Low-Rank HMMs and PCFGs

Songlin Yang, Wei Liu1, Kewei Tu2

School of Information Science and Technology, ShanghaiTech University

Shanghai Engineering Research Center of Intelligent Vision and Imaging {yangsl,liuwei4,tukw}@shanghaitech.edu.cn

Footnote 1: Equal contributions: Songlin Yang formulated the idea and Wei Liu conducted the experiments. \({}^{\dagger}\) Corresponding author.

###### Abstract

Hidden Markov Models (HMMs) and Probabilistic Context-Free Grammars (PCFGs) are widely used structured models, both of which can be represented as factor graph grammars (FGFGs), a powerful formalism capable of describing a wide range of models. Recent research found it beneficial to use large state spaces for HMMs and PCFGs. However, inference with large state spaces is computationally demanding, especially for PCFGs. To tackle this challenge, we leverage tensor rank decomposition (aka. CPD) to decrease inference computational complexities for a subset of FGGs subsuming HMMs and PCFGs. We apply CPD on the factors of an FGG and then construct a new FGG defined in the rank space. Inference with the new FGG produces the same result but has a lower time complexity when the rank size is smaller than the state size. We conduct experiments on HMM language modeling and unsupervised PCFG parsing, showing better performance than previous work. Our code is publicly available at [https://github.com/VPeterV/RankSpace-Models](https://github.com/VPeterV/RankSpace-Models).

+
Footnote †: _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4797 - 4809

July 10-15, 2022 ©2022 Association for Computational Linguistics

## 1 Introduction

Hidden Markov Models (HMMs) and Probabilistic Context-Free Grammars (PCFGs) are widely used structured models in natural language processing. They can both be represented as factor graph grammars (FGGs) [3], which are a powerful tool to describe a wide range of models, allowing exact and tractable inference in most situations of interest.

Over-parameterization has been shown beneficial in facilitating optimization of deep networks [16, 17, 18, 19]. Buhai et al. (2020) found that over-parameterization is also helpful in learning latent variable models by increasing the number of hidden states. Buhai et al. (2020) argued that it is important to study over-parameterization in structured settings because structured latent variable models are more suitable to model real-word phenomena which exhibit complex dependencies. HMMs and PCFGs are typical structured latent variable models, and recently researchers have found it beneficial to use large state spaces for HMMs and PCFGs [15, 19, 20, 21, 22]. However, structured inference with large state spaces is computationally demanding, especially for PCFGs, pushing researchers to develop methods to decrease the computational complexities. Chiu and Rush (2020) propose a neural VL-HMM with \(2^{15}\) states for language modeling, narrowing down the performance gap between HMMs and LSTMs. They follow Dedieu et al. (2019) to impose a strong sparsity constraint (i.e., each hidden state can only generate a small subset of terminal symbols) to decrease the time complexity of the forward algorithm, thus requiring pre-clustering of terminal symbols. Yang et al. (2021b) use a large state space for neural PCFG induction and achieve superior unsupervised constituency parsing performance. They follow Cohen et al. (2013) to use tensor rank decomposition (aka. canonical-polyadic decomposition (CPD) [1]) to decrease the computational complexity of the inside algorithm, but only scale the state size from tens to hundreds because the resulting complexity is still high. Chiu et al. (2021) use tensor matricization and low-rank matrix decomposition to accelerate structured inference on chain and tree structure models. However, their method has an even higher complexity than Yang et al. (2021b) on PCFGs. Recently, Fu and Lapata (2021) propose a family of randomized dynamic programming algorithms to scale structured models to tens of thousands of states, which is orthogonal to the aforementioned low-rank-based

approaches as the former performs approximate inference whereas the latter perform exact inference.

In this work, we propose a new low-rank-based approach to scale structured inference, which can be described by FGG notations intuitively. We first provide an intuitive and unifying perspective toward the work of Yang et al. (2021) and Chiu et al. (2021), showing that their low-rank decomposition-based models can be viewed as decomposing large factors in an FGG--e.g., the binary rule probability tensor in PCFGs-- into several smaller factors connected by new "rank" nodes. Then we target at a subset of FGGs--which we refer to as B-FGGs--subsuming all models considered by Chiu et al. (2021), whereby the inference algorithms can be formulated via B-graphs Gallo et al. (1993); Klein and Manning (2001). We propose a novel framework to support a family of inference algorithms in the rank space for B-FGGs. Within the framework, we apply CPD on the factors of a B-FFG and then construct a new B-FPG defined in the rank space by marginalizing all the state nodes. Inference with the new B-FGG has the same result and a lower time complexity if the rank size is smaller than the state size.

We conduct experiments in unsupervised PCFG parsing and HMM language modeling. For PCFG induction, we manage to use 20 times more hidden states than Yang et al. (2021), obtaining much better unsupervised parsing performance. For HMM language modeling, we achieve lower perplexity and lower inference complexity than Chiu et al. (2021).

## 2 Background

### Factor graph grammar

Factor graphs are fixed-sized and thus incapable of modeling substructures that repeat a variable number of times. Chiang and Riley (2020) propose factor graph grammars (FGFGs) to overcome this limitation, which are expressive enough to subsume HMMs and PCFGs. The main purpose of introducing FGFGs in this work is to facilitate more intuitive presentation of our method, and to enable generalization beyond HMMs and PCFGs.

#### 2.1.1 Basics

We display necessary notations and concepts of FGGs Chiang and Riley (2020, Def. 1,2,5,6,8).

**Definition 1**.: A hypergraph is a tuple \(\left(V,E,att,lab^{V},lab^{E}\right)\) where

* \(V\) and \(E\) are finite set of nodes and hyperedges.
* \(att:E\to V^{\star}\) maps each hyperedge to zero or more (not necessarily distinct) endpoint nodes.
* \(lab^{V}:V\to L^{V}\) assigns labels to nodes.
* \(lab^{E}:E\to L^{E}\) assigns labels to edges.

**Definition 2**.: A factor graph is a hypergraph with mappings \(\Omega\) and \(F\) where

* \(\Omega\) maps node labels to sets of possible values. \(\Omega(v)\triangleq\Omega(lab^{V}(v))\).
* \(F\) maps edge labels to functions. \(F(e)\triangleq F(lab^{E}(e))\) is of type \(\Omega(v_{1})\times\dots\times\Omega(v_{k})\) where \(att(e)=v_{1}\cdots v_{k}\).

In the terminology of factor graphs, a node \(v\) with its domain \(\Omega(v)\) is a _variable_, and an hyperedge \(e\) with \(F(e)\) is a _factor_. We typically use \(\mathbf{T},\mathbf{N},\mathbf{O}\) to denote hidden state, nonterminal state and observation variables for HMMs and PCFGs.

**Definition 3**.: A hypergraph fragment is a tuple \((V,E,att,lab^{V},lab^{E},ext)\) where

* \((V,E,att,lab^{V},lab^{E})\) is a hypergraph.
* \(ext\in V^{\star}\) is a set of zero or more external nodes and each of which can be seen as a connecting point of this hypergraph fragment with another fragment.

**Definition 4**.: A hyperedge replacement graph grammar (HRG) Drewes et al. (1997) is a tuple \((N,T,P,S)\) where

* \(N,T\subset L^{E}\) is finite set of nonterminal and terminal symbols. \(N\cap T=\emptyset\).
* \(P\) is a finite set of rules (\(X\to R\)) where \(X\in N\) and \(R\) is a hypergraph fragment with edge labels in \(N\cup T\)1. Footnote 1: Note that, for the lhs of \(P\)Chiang and Riley (2020) also draw their endpoint nodes using external node notations. We follow this practice.
* \(S\in N\) is the start symbol.

**Definition 5**.: A HRG with mapping \(\Omega,F\) (Def. 2) is referred to as an FGG. In particular, \(F\) is defined on terminal edge labels \(T\) only.

**Notations.**

* [noitemsep]
* \(\mathsf{N}\) : variable node. \(\mathsf{S}\) : external node.
* \(\mathsf{X_{e}}\) : hyperedge \(e\) with label \(X\in N\). indicates zero or more endpoint nodes.

 

* \(F(e)\): factor \(F(e)\).

Fig. 1 illustrates HGG representations of HMM and PCFG.

Generative storyAn FGG starts with , repeatedly selects and uses rule \(X\to R\) from to replace \(e\) with \(R\), until no exists.

#### 2.1.2 Conjunction

The conjunction operation Chiang and Riley (2020); Sec. 4 allows modularizing an FGG into two parts, one defining the model and the other defining a query. In this paper, we only consider querying the observed sentence \(w_{0},\cdots,w_{n-1}\), which is exemplified by the red part of Fig. 1. We sometimes omit the red part without further elaboration.

#### 2.1.3 Inference

Denote \(\xi\) as an assignment of all variables, \(\Xi_{D}\) as the set of all assignments of factor graph \(D\), and \(\mathcal{D}(G)\) as the set of all derivations of an FGG \(G\), i.e., all factor graphs generated by \(G\). an FGG \(G\) assigns a score \(w_{G}(D,\xi)\) to each \(D\in\mathcal{D}(G)\) along with each \(\xi\in\Xi_{D}\). A factor graph \(D\in\mathcal{D}(G)\) assigns a score \(w_{D}(\xi)\) to each \(\xi\in\Xi_{D}\):

\[w_{D}(\xi)=\prod_{e\in D}F(e)(\xi(v_{1}),\ldots,\xi(v_{k})) \tag{1}\]

with \(att(e)=v_{1}\cdots v_{k}\). Notably, \(w_{D}(\xi)\triangleq w_{G}(D,\xi)\). The inference problem is to compute the sum-product of \(G\):

\[Z_{G}=\sum_{D\in\mathcal{D}(G)}\sum_{\xi\in\Xi_{D}}w_{G}(D,\xi) \tag{2}\]

To obtain \(Z_{G}\), the key difficulty is in the marginalization over all derivations, since can be obtained by running standard variable elimination (VE) on factor graph \(D\). To tackle this, Chiang and Riley (2020, Thm. 15) propose an extended VE. For each \(X\in N,\xi\in\Xi_{X}\)2, define \(P^{X}\) as all rules in \(P\) with left-hand side \(X\), and then define:

Footnote 2: \(\Xi_{X}\) is defined as the set of assignments to the endpoints of an edge \(e\) labeled X, so \(\Xi_{X}=\Omega\left(\ell_{1}\right)\times\cdots\times\Omega\left(\ell_{k}\right)\) where \(att(e)=v_{1}\cdots v_{k},lab^{V}(v_{i})=\ell_{i}\).

\[\psi_{X}(\xi)=\sum_{(X\to R)\in P^{X}}\tau_{R}(\xi). \tag{3}\]

for each rhs \(R=(V,E_{N}\cup E_{T},att,lab^{V},lab^{E},ext)\), where \(E_{N},E_{T}\) consist of nonterminal/terminal-labeled edges only, and \(\tau_{R}(\xi)\) is given by:

\[\begin{split}\tau_{R}(\xi)&=\sum_{ \begin{subarray}{c}\xi^{\prime}\in\Xi_{R}\\ \xi^{\prime}(ert)=\xi\end{subarray}}\prod_{e\in E_{T}}F(e)\left(\xi^{\prime}( \mathrm{att}(e))\right)\\ &\qquad\qquad\quad\prod_{e\in E_{N}}\psi_{lab^{E}(e)}\left(\xi^{ \prime}(\mathrm{att}(e))\right)\end{split} \tag{4}\]

This defines a recursive formula for computing \(\psi_{S}\), i.e., \(Z_{G}\). Next, we will show how Eq. 3-4 recover the well-known inside algorithm.

Example: the inside algorithmConsider \(\pi_{6}\) in Fig.1(b). All possible fragments \(R\) (rhs of \(\pi_{6}\)) differs in the value of \(k\), i.e., the splitting point, so we use \(R_{k}\) to distinguish them. Then Eq. 3 becomes:

\[\psi_{X_{i,k}}(\xi)=\sum_{i<k<j}\tau_{R_{k}}(\xi) \tag{5}\]

Figure 1: FGG representations of (a) HMMs and (b) PCFGs. Examples come from Chiang and Riley (2020).

 

Putting values into Eq. 4:

\[\tau_{R_{k}}(\xi)=\sum_{n_{2},n_{3}}p(\xi,n_{2},n_{3})\psi_{X_{i,k}}(n_{2})\psi_{ X_{k,j}}(n_{3}) \tag{6}\]

where \(p\) denotes FGG rule probability \(p(\mathbf{N}_{1}\rightarrow\mathbf{N}_{2}\mathbf{N}_{3})\). It is easy to see that \(\psi_{X_{i,k}}\) is exactly the inside score of span \([i,k)\), and Eq. 5-6 recovers the recursive formula of the inside algorithm.

Remark.Eq. 4 can be viewed as unidirectional (from \(e\in E_{N}\) to external nodes) belief propagation (BP) in the factor graph fragment \(R\), where the incoming message is \(\psi_{lab^{E}}(e)\) for \(e\in E_{N}\), and the outcome of Eq. 4 can be viewed as the message passed to the external nodes. The time complexity of message updates grows exponentially with the number of variables in the factors. Therefore, to decrease inference complexity, one may decompose large factors into smaller factors connected by new nodes, as shown in the next subsection.

### Tensor rank decomposition on factors

Consider a factor \(F(e)\) (Def. 2), it can be represented as an order-\(k\) tensor in \(\mathbb{R}^{N_{1}\times\cdots\times N_{k}}\) where \(N_{i}\triangleq|\Omega(v_{i})|\). We can use tensor rank decomposition (aka. CPD) to decompose \(F(e)\) into a weighted sum of outer products of vectors:

\[F(e)=\sum_{q=1}^{r}\lambda_{q}\textbf{w}_{e_{1}}^{q}\otimes\textbf{w}_{e_{2}} ^{q}\otimes\cdots\otimes\textbf{w}_{e_{k}}^{q}\]

where \(r\) is the rank size; \(\textbf{w}_{e_{k}}^{q}\in\mathbb{R}^{N_{k}};\)\(\otimes\) is outer product; \(\lambda_{q}\) is weight, which can be absorbed into \(\left\{\textbf{w}_{e_{k}}^{q}\right\}\) and we omit it throughout the paper.

Dupty and Lee (2020, Sec. 4.1) show that BP can be written in the following matrix form when applying CPD on factors:

\[\textbf{m}_{ei} =\textbf{W}_{e_{i}}^{T}\left(\odot_{j\in N(e)\backslash i} \textbf{W}_{e_{j}}\textbf{n}_{je}\right) \tag{7}\] \[\textbf{n}_{ie} =\odot_{c\in N(i)\backslash e}\textbf{m}_{ci} \tag{8}\]

where \(\textbf{m}_{ei}\in\mathbb{R}^{N_{i}}\) is factor-to-node message; \(\textbf{n}_{ie}\in\mathbb{R}^{N_{i}}\) is node-to-factor message; \(N(\cdot)\) indicates neighborhood ; \(\textbf{W}_{e_{j}}=[\textbf{w}_{e_{j}}^{1},\cdots,\textbf{w}_{e_{j}}^{r}]^{T} \in\mathbb{R}^{r\times m}\); \(\odot\) is element-wise product. We remark that this amounts to replacing the large factor \(F(e)\) with smaller factors \(\left\{F(e_{i})\right\}\) connected by a new node **R** that represents rank, where each \(F(e_{i})\) can be represented as \(\textbf{W}_{e_{i}}\). Fig. 2 illustrates this intuition. We refer to **R** as rank nodes and others as state nodes thereafter.

## 3 Low-rank structured inference

In this section, we recover the accelerated inside algorithms of TD-PCFG Cohen et al. (2013); Yang et al. (2021b) and LPCFG Chiu et al. (2021) in an intuitive and unifying manner using the FGG notations. The accelerated forward algorithm of LHMM Chiu et al. (2021) can be derived similarly.

Denote \(\textsf{T}\in\mathbb{R}^{m\times m\times m}\) as the tensor representation of \(p(\textbf{N}_{1}\rightarrow\textbf{N}_{2}\textbf{N}_{3})\) , and \(\boldsymbol{\alpha}_{i,j}\in\mathbb{R}^{m}\) as the inside score of span \([i,j)\). Cohen et al. (2013) and Yang et al. (2021b) use CPD to decompose T, i.e., let \(\textsf{T}=\sum_{q=1}^{r}\textbf{u}_{q}\otimes\textbf{v}_{q}\otimes\textbf{ w}_{q}\) where \(\textbf{u}_{q},\textbf{v}_{q},\textbf{w}_{q}\in\mathbb{R}^{m}\). Denote \(\textbf{U},\textbf{V},\textbf{W}\in\mathbb{R}^{r\times m}\) as the resulting matrices of stacking all \(\textbf{u}_{q},\textbf{v}_{q},\textbf{w}_{q}\), Cohen et al. (2013) derived the recursive form:

\[\boldsymbol{\alpha}_{i,j} =\sum_{k=i+1}^{j-1}\textbf{U}^{T}\left((\textbf{V}\boldsymbol{ \alpha}_{i,k})\odot(\textbf{W}\boldsymbol{\alpha}_{k,j})\right) \tag{9}\] \[=\textbf{U}^{T}\sum_{k=i+1}^{j-1}\left((\textbf{V}\boldsymbol{ \alpha}_{i,k})\odot(\textbf{W}\boldsymbol{\alpha}_{k,j})\right) \tag{10}\]

Eq. 9 can be derived automatically by combining Eq. 7 (or Fig. 3 (a)) and Eq. 5-6. Cohen et al. (2013) note that \(\textbf{U}^{\textbf{T}}\) can be extracted to the front of the summation (Eq. 10), and \(\textbf{V}\boldsymbol{\alpha}_{i,k},\textbf{W}\boldsymbol{\alpha}_{k,j}\) can be cached and reused, leading to further complexity reduction. The resulting inside algorithm time complexity is \(O(n^{3}r+n^{2}mr)\).

Figure 3: Representations of the rhs of \(\pi_{6}\) (Fig. 1) after decomposition. (a): TD-PCFG Cohen et al. (2013); Yang et al. (2021b). (b): LPCFG Chiu et al. (2021).

Figure 2: Using CPD to decompose a factor can be seen as adding a new node.

 

Recently, Chiu et al. (2021) use low-rank matrix decomposition to accelerate PCFG inference. They first perform tensor matricization to flatten T to \({\sf T^{\prime}}\in\mathbb{R}^{m\times m^{2}}\) , and then let \({\sf T^{\prime}}={\bf U}^{T}{\bf V}\) where \({\bf U}\in\mathbb{R}^{r\times m},{\bf V}\in\mathbb{R}^{r\times m^{2}}\). By un-flattening \({\bf V}\) to \({\bf V^{\prime}}\in\mathbb{R}^{r\times m\times m}\), their accelerated inside algorithm has the following recursive form:

\[\boldsymbol{\alpha}_{i,j} =\sum_{k=i+1}^{j-1}{\bf U}^{T}\left({\bf V^{\prime}}\cdot \boldsymbol{\alpha}_{k,j}\cdot\boldsymbol{\alpha}_{i,k}\right) \tag{11}\] \[={\bf U}^{T}\sum_{k=i+1}^{j-1}\left({\bf V^{\prime}}\cdot \boldsymbol{\alpha}_{k,j}\cdot\boldsymbol{\alpha}_{i,k}\right) \tag{12}\]

Eq. 11 can be derived by combining Fig. 3 (b) and Eq. 5-6. The resulting inside time complexity is \(O(n^{3}m^{2}r+n^{2}mr)\), which is higher than that of TD-PCFG.

When learning a PCFG and a HMM, there is no need to first learn T and then perform decomposition on T. Instead, one can learn the decomposed matrices (e.g., \({\bf U},{\bf V}\)) to learn T implicitly. During inference, one can follow Eq. 10 or 12 without the need to reconstruct T.

Validity of probability.The remaining problem is to ensure that T is a valid probability tensor (i.e., being nonnegative and properly normalized) when learning it implicitly. Yang et al. (2021) essentially transform Fig. 3(a) into a Bayesian network, adding directed arrows \({\bf N_{1}}\rightarrow{\bf R},{\bf R}\rightarrow{\bf N_{2}},{\bf R} \rightarrow{\bf N_{3}}\). This is equivalent to requiring that \({\bf V},{\bf W}\) are nonnegative and column-wise normalized and \({\bf U}\) is nonnegative and row-wise normalized, as described in Yang et al. (2021, Thm. 1). One can apply the Softmax re-parameterization to enforce such requirement, which is more convenient in end-to-end learning. Chiu et al. (2021) replace the local normalization of Yang et al. (2021) with global normalization, and we refer readers to their paper for more details. We adopt the strategy of Yang et al. (2021) in this work.

## 4 Rank-space modeling and inference

### Rank-space inference with B-FGGs

Interestingly, when applying CPD on factors and if the rank size is smaller than the state size, we can even obtain better inference time complexities for a subset of FGGs which we refer to as B-FGFGs.

We call a hyperedge a B-edge if its head contains exactly one node. B-graphs Gallo et al. (1993) are a subset of directed hypergraphs whose hyperedges are all B-edges. Many dynamic programming algorithms can be formulated through B-graphs Klein and Manning (2001); Huang (2008); Azuma et al. (2017); Chiu et al. (2021); Fu and Lapata (2021), including the inference algorithms of many structured models, e.g., HMMs, Hidden Semi-Markov Models (HSMMs), and PCFGs. We follow the concept of B-graphs to define B-FGFGs.

Definition 6.A hypergraph fragment is a B-hypergraph fragment iff. there is exactly one external node and there is no nonterminal-labeled hyperedge connecting to it. An FGG is a B-FGG iff. all rhs of its rules are B-hypergraph fragments.

It is easy to see that the aforementioned models are subsumed by B-FGGs. We can design a family of accelerated inference algorithms for B-FGFGs based on the following strategy. (1) If there are multiple factors within a hypergraph fragment, merge them into a single factor. Then apply CPD on the single factor, thereby introducing rank nodes. (2) Find repeated substructures that take rank nodes as external nodes. Marginalize all state nodes to derive new rules. (3) Design new inference algorithms that can be carried out in the rank space based on the general-purpose FGG inference algorithm and the derived new rules. We give two examples, the rank-space inside algorithm and the rank-space forward algorithm, in the following two subsections to help readers understand this strategy.

### The rank-space inside algorithm

Consider an B-FGG \(G\) shown in Fig.1(b) and replace the rhs of \(\pi_{6}\) with Fig. 3(a), i.e., we use CPD to decompose binary rule probability tensor. Besides \({\bf U},{\bf V},{\bf W}\in\mathbb{R}^{r\times m}\) defined in Sec. 3, we define the start rule probability vector as \({\bf s}\in\mathbb{R}^{m\times 1}\), and the unary rule probability matrix as \({\bf E}\in\mathbb{R}^{o\times m}\) where \(o\) is the vocabulary size.

Fig. 4(a) is an example (partial) factor graph \(D\) generated by \(G\). We highlight substructures of interest with dashed rectangles. Each substructure consists of a node \({\bf N}\) and two factors connecting to it. \({\bf N}\) is an external node connecting two hypergraph fragments which contain the two factors respectively. For each substructure, we can marginalize the state node \({\bf N}\) out, merging the two factors into a single one. After marginalizing all state nodes, we obtain a (partial) factor graph \(D^{\prime}\) shown in the right of Fig. 4(a) where \({\bf H}={\bf V}{\bf U}^{T},{\bf I}={\bf W}{\bf U}^{T},{\bf J}={\bf VE}^{T},{ \bf K}={\bf WE}^{T}\), \({\bf L}=({\bf Us})^{T}\). We denote this transformation as \(\mathcal{M}(D)=D^{\prime}\). It is worth men 

tioning that \({\bf H},{\bf I},{\bf J},{\bf K},{\bf L}\) are computed only _once_ and then reused multiple times during inference, which is the key to reduce the time complexity.

Then we define a new B-FGG \(G^{\prime}\) with rules shown in Fig 4(b). It is easy to verify that for each \(D\in\mathcal{D}(G)\), we have \(\mathcal{M}(D)\in\mathcal{D}(G^{\prime})\), and vice versa. Moreover, we have:

because marginalizing hidden variables does not affect the result of sum-product inference. Therefore, \(Z_{G}=Z_{G^{\prime}}\) (Eq. 2).

We can easily derive the inference (inside) algorithm of \(G^{\prime}\) by following Eq. 3-4 and Fig. 4(b)3. Let \(\mathbf{\alpha}_{i,j}\in\mathbb{R}^{r}\) denote the rank-space inside score for span \([i,j)\). When \(j>i+2\):

Footnote 3: \(\pi_{6}\) is used for generating sentences of length 1, we do not consider this in the following derivation of the inside algorithm to reduce clutter.

\[\mathbf{\alpha}_{i,j} =\overbrace{\sum_{i+1<k<j-1}({\bf H}\mathbf{\alpha}_{i,k}\odot{\bf I }\mathbf{\alpha}_{k,j})}^{\text{from $\pi_{1}$ of Fig.~{}\ref{fig:eq:eq:eq:eq: eq:eq: 

decoding. Therefore, we follow Yang et al. (2021b) to use Minimum Bayes-Risk decoding (Goodman, 1996). Specifically, we estimate the span marginals using auto-differentiation (Eisner, 2016; Rush, 2020), which has the same complexity as the inside algorithm. Then we use the CYK algorithm to find the final parse with the maximum number of expected spans in \(O(n^{3})\) time, similar to Smith and Eisner (2006).

Implementation.The implementation of the inside algorithm greatly influences the actual running speed. First, \(O(n^{2})\) out of \(O(n^{3})\) can be computed in parallel using parallel parsing techniques (Yi et al., 2011; Canny et al., 2013; Zhang et al., 2020; Rush, 2020). In this work, we adapt the efficient implementation of Zhang et al. (2020) for fast inside computation. Second, we adopt the log-einsum-exp trick (Peharz et al., 2020) to avoid expensive log-sum-exp operations on high-dimensional vectors, which reduces both GPU memory usage and total running time.

### The rank-space forward algorithm

Consider an B-FGG \(G\) shown in Fig. 1 (a). We replace the rhs of \(\pi_{2}\) by the hypergraph fragment in the right of Fig. 5(a), i.e., we merge the factor \(p(\mathbf{T}_{2}\mid\mathbf{T}_{1})\) and \(p(\mathbf{O}_{3}\mid\mathbf{T}_{2})\) into a single factor, which can be represented as \(\mathsf{T}\in\mathbb{R}^{m\times m\times o}\) and can be decomposed into three matrices \(\mathbf{U},\mathbf{V}\in\mathbb{R}^{r\times m},\mathbf{W}\in\mathbb{R}^{r\times o}\) via CPD, where \(m/o/r\) is the state/vocabulary/rank size. Fig. 5(b) gives an example factor graph of HMMs with sentences of length 3. Similar to previous subsection, we marginalize state nodes \(\mathbf{T}\) to construct a new B-FGG \(G^{\prime}\). The rule set of \(G^{\prime}\) can be obtained by replacing all variable nodes \(\mathbf{T}\) with \(\mathbf{R}\) and modifying all factors accordingly, as one can easily infer from Fig. 5(c). Inference with \(G^{\prime}\) simply coincides with the forward algorithm, which has a \(O(nr^{2})\) time complexity and is lower than \(O(nmrr)\) of LHMM (Chiu et al., 2021) when \(r<m\).

### Neural parameterization

We use neural networks to produce probabilities for all factors, which has been shown to benefit learning and unsupervised induction of syntactic structures (Jiang et al., 2016; He et al., 2018; Kim et al., 2019; Han et al., 2019; Jin et al., 2019; Zhu et al., 2020; Yang et al., 2020, 2021b; Zhao and Titov, 2020; Zhang et al., 2021; Chiu and Rush, 2020; Chiu et al., 2021; Kim, 2021). We use the neural parameterization of Yang et al. (2021b) with slight modifications. We show the details in Appd. A and Appd. B.

## 5 Experiments

### Unsupervised parsing with PCFGs

Setting.We evaluate our model on Penn Treebank (PTB) (Marcus et al., 1994). Our implementation is based on the open-sourced code of Yang et al. (2021b)7 and we use the same setting as theirs. For all experiments, we set the ratio of nonterminal number to the preterminal number to 1:2 8 which is the common practise. We set the rank size to 1000. We show other details in Appl. C and D.

Footnote 7: github.com/sustcsonglin/TN-PCFG

Main result.Table 1 shows the result on PTB. Among previous unsupervised PCFG models, TN-PCFG (Yang et al., 2021b) uses the largest number of states (500 perterminals and 250 nonterminals). Our model is able to use much more states thanks to our new inside algorithm with lower time complexity, surpassing all previous PCFG-based models by a large margin and achieving a new state-of-the-art in unsupervised constituency parsing in terms of sentence-level F1 score on PTB.

Figure 5: (a): merge the two factors into a single one, and apply CPD on the resulting factor. (b): factor graph of a HMM for sentences of length 3. (c): the resulting factor graph after marginalizing the state nodes.

 

Ablation study.Fig. 6 shows the change of the sentence-level F1 scores and perplexity with the change of the number of preterminals. As we can see, when increasing the state, the perplexity tends to decrease while the F1 score tends to increase, validating the effectiveness of using large state spaces for neural PCFG induction.

### HMM language modeling

Setting.We conduct the language modeling experiment also on PTB. Our implementation is based on the open-sourced code of Chiu et al. (2021)9. We set the rank size to 4096. See Appd. C and D for more details.

Footnote 9: github.com/justinchiu/low-rank-models

Main result.Table 2 shows the perplexity on the PTB validation and test sets. As discussed earlier, VL-HMM Chiu and Rush (2020) imposes strong sparsity constraint to decrease the time complexity of the forward algorithm and requires preclustering of terminal symbols. Specifically, VL-HMM uses Brown clustering Brown et al. (1992), introducing external information to improve performance. Replacing Brown clustering with uniform clustering leads to a 10 point increase in perplexity on the PTB validation set. LHMM Chiu et al. (2021) and our model only impose low-rank constraint without using any external information and are thus more comparable. Our method outperforms LHMM by \(4.8\) point when using the same state number (i.e., \(2^{14}\)), and it can use more states thanks to our lower inference time complexity.

Ablation study.As we can see in Table 3, the perplexity tends to decrease when increasing the state number, validating the effectiveness of using more states for neural HMM language modeling.

\begin{table}
\begin{tabular}{l c} \hline \hline
**Model** & **S-F1** \\ \hline N-PCFG Kim et al. (2019) & 50.8 \\ C-PCFG Kim et al. (2019) & 55.2 \\ NL-PCFG Zhu et al. (2020) & 55.3 \\ TN-PCFG Yang et al. (2021b) & 57.7 \\ NBL-PCFG Yang et al. (2021a) & 60.4 \\ \hline Ours with 9000 PTs and 4500NTs & **64.1** \\ \hline For reference & \\ \hline Constituency test Cao et al. (2020) & 62.8 \\ S-DIORA Drozdov et al. (2020) & 57.6 \\ StructFormer Shen et al. (2021) & 54.0 \\ DIORA+span constraint Xu et al. (2021) & 61.2 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Results on PTB. S-F1: sentence-level F1. PTs: preterminals. NTs: nonterminals.

Figure 6: The change of F1 scores and perplexities with the change of number of perterminal symbols.

\begin{table}
\begin{tabular}{c c c} \hline \hline
**\#States** & **Val** & **Test** \\ \hline \(2^{12}\) & 149.8 & 139.1 \\ \(2^{13}\) & 143.8 & 133.4 \\ \(2^{14}\) & 149.5 & 137.4 \\ \(2^{15}\) & 141.1 & 131.1 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Perplexity with varying numbers of states. Following Chiu et al. (2021), we fix the rank to 2048 for faster ablation studies.

 

Discussion.It is interesting to note that our HMM model is roughly equivalent to another HMM with interchanged rank and state sizes as can be seen in Fig.5(c). To verify this equivalence, we run LHMM in the original state space with 2048 states and rank \(2^{15}\). The resulting perplexity is 133.49 on average on the PTB test set, which is worse than that of ours (126.4). We leave further experimentation and analyses of this discrepancy for future work.

## 6 Related work

Tensor and matrix decomposition have been used to decrease time and space complexities of probabilistic inference algorithms. Siddiqi et al. (2010) propose a reduced-rank HMM whereby the forward algorithm can be carried out in the rank space, which is similar to our model, but our method is more general. Cohen and Collins (2012); Cohen et al. (2013) use CPD for fast (latent-variable) PCFG parsing, but they do not leverage CPD for fast learning and they need to actually perform CPD on existing probability tensors. Rabusseau et al. (2016) use low-rank approximation method to learn weighted tree automata, which subsumes PCFGs and latent-variable PCFGs. Our method can subsume more models. Yang et al. (2021b,a) propose CPD-based neural parameterizations for (lexicalized) PCFGs. Yang et al. (2021b) aim at scaling PCFG inference. We achieve better time complexity than theirs and hence can use much more hidden states. Yang et al. (2021a) aims to decrease the complexity of lexicalized PCFG parsing, which can also be described within our framework. Chiu et al. (2021) use low-rank matrix decomposition, which can be viewed as CPD on order-2 tensors, to accelerate inference on chain and tree structure models including HMMs and PCFGs. However, their method is only efficient when the parameter tensors are of order 2, e.g., in HMMs and HSMMs. Our method leverages full CPD, thus enabling efficient inference with higher-order factors, e.g., in PCFGs. Our method can be applied to all models considered by Chiu et al. (2021), performing inference in the rank-space with lower complexities.

Besides HMMs and PCFGs, Wrigley et al. (2017) propose an efficient sampling-based junction-tree algorithm using CPD to decompose high-order factors. Dupty and Lee (2020) also use CPD to decompose high-order factors for fast belief propagation. Yang and Tu (2022) use CPD to decompose second-order factors in semantic dependency parsing to accelerate second-order parsing with mean-field inference. Besides CPD, Ducamp et al. (2020) use tensor train decomposition for fast and scalable message passing in Bayesian networks. Bonnevie and Schmidt (2021) leverage matrix product states (i.e., tensor trains) for scalable discrete probabilistic inference. Miller et al. (2021) leverage tensor networks for fast sequential probabilistic inference.

## 7 Conclusion and future work

In this work, we leveraged tensor rank decomposition (CPD) for low-rank scaling of structured inference. We showed that CPD amounts to decomposing a large factor into several smaller factors connected by a new rank node, and gave a unifying perspective towards previous low-rank structured models (Yang et al., 2021b; Chiu et al., 2021). We also presented a novel framework to design a family of rank-space inference algorithms for B-FGGs, a subset of FGGs which subsume most structured models of interest to the NLP community. We have shown the application of our method in scaling PCFG and HMM inference, and experiments on unsupervised parsing and language modeling validate the effectiveness of using large state spaces facilitated by our method.

We believe our framework can be applied to many other models which have high inference time complexity and are subsumed by B-FGGs, including lexicalized PCFGs, quasi-synchronous context-free grammars (QCFGs), etc. A direct application of our method is to decrease the inference complexity of the neural QCFG (Kim, 2021).

## Acknowledgments

This work was supported by the National Natural Science Foundation of China (61976139).

## References

* Arora et al. (2018) Sanjeev Arora, Nadav Cohen, and Elad Hazan. 2018. On the optimization of deep networks: Implicit acceleration by overparameterization. In _Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmassan, Stockholm, Sweden, July 10-15, 2018_, volume 80 of _Proceedings of Machine Learning Research_, pages 244-253. PMLR.
* Azuma et al. (2017) Ai Azuma, Masashi Shimbo, and Yuji Matsumoto. 2017. An algebraic formalization of forward 

and forward-backward algorithms. _CoRR_, abs/1702.06941.
* Bonnevie and Schmidt (2021) Rasmus Bonnevie and Mikkel N. Schmidt. 2021. Matrix product states for inference in discrete probabilistic models. _Journal of Machine Learning Research_, 22(187):1-48.
* Brown et al. (1992) Peter F. Brown, Vincent J. Della Pietra, Peter V. deSouza, Jenifer C. Lai, and Robert L. Mercer. 1992. Class-based \(n\)-gram models of natural language. _Computational Linguistics_, 18(4):467-480.
* Buhai et al. (2020) Rares-Darius Buhai, Yoni Halpern, Yoon Kim, Andrej Risteski, and David A. Sontag. 2020. Empirical study of the benefits of overparameterization in learning latent variable models. In _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_, volume 119 of _Proceedings of Machine Learning Research_, pages 1211-1219. PMLR.
* Buys et al. (2018) Jan Buys, Yonatan Bisk, and Yejin Choi. 2018. Bridging hmms and rnns through architectural transformations.
* Canny et al. (2013) John Canny, David Hall, and Dan Klein. 2013. A multi-teraflop constituency parser using GPUs. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_, pages 1898-1907, Seattle, Washington, USA. Association for Computational Linguistics.
* Cao et al. (2020) Steven Cao, Nikita Kitaev, and Dan Klein. 2020. Unsupervised parsing via constituency tests. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 4798-4808, Online. Association for Computational Linguistics.
* Chiang and Riley (2020) David Chiang and Danceg Riley. 2020. Factor graph grammars. In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
* Chiu et al. (2021) Justin Chiu, Yuntian Deng, and Alexander Rush. 2021. Low-rank constraints for fast inference in structured models. _Advances in Neural Information Processing Systems_, 34.
* Chiu and Rush (2020) Justin Chiu and Alexander Rush. 2020. Scaling hidden Markov language models. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1341-1349, Online. Association for Computational Linguistics.
* Cohen and Collins (2012) Shay B. Cohen and Michael Collins. 2012. Tensor decomposition for fast parsing with latent-variable pcfgs. In _Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of a meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States_, pages 2528-2536.
* Cohen et al. (2013) Shay B. Cohen, Giorgio Satta, and Michael Collins. 2013. Approximate PCFG parsing using tensor decomposition. In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 487-496, Atlanta, Georgia. Association for Computational Linguistics.
* Dedieu et al. (2019) Antoine Dedieu, Nishad Gothoskar, Scott Swingle, Wolfgang Lehrach, Miguel Lazaro-Gredilla, and Dieep George. 2019. Learning higher-order sequential structure with cloned hmms. _CoRR_, abs/1905.00507.
* Drewes et al. (1997) Frank Drewes, Hans-Jorg Kreowski, and Annegret Habel. 1997. Hyperedge replacement, graph grammars. In Grzegorz Rozenberg, editor, _Handbook of Graph Grammars and Computing by Graph Transformations, Volume 1: Foundations_, pages 95-162. World Scientific.
* Drozdov et al. (2020) Andrew Drozdov, Subendhu Rongali, Yi-Pei Chen, Tim O'Gorman, Mohit Iyyer, and Andrew McCallum. 2020. Unsupervised parsing with S-DIORA: Single tree encoding for deep inside-outside recursive autoencoders. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 4832-4845, Online. Association for Computational Linguistics.
* Du et al. (2019) Simon S. Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. 2019. Gradient descent provably optimizes over-parameterized neural networks. In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
* Ducamp et al. (2020) Gaspard Ducamp, Philippe Bonnard, Anthony Nouy, and Pierre-Henri Wuillemin. 2020. An efficient low-rank tensors representation for algorithms in complex probabilistic graphical models. In _International Conference on Probabilistic Graphical Models, PGM 2020, 23-25 September 2020, Aalborg, Hotel Comwell Rebild Bakker, Skorping, Denmark_, volume 138 of _Proceedings of Machine Learning Research_, pages 173-184. PMLR.
* Dupty and Lee (2020) Mohammed Haroon Dupty and Wee Sun Lee. 2020. Neutralizing efficient higher-order belief propagation. _CoRR_, abs/2010.09283.
* Eisner (2016) Jason Eisner. 2016. Inside-outside and forward-backward algorithms are just backprop (tutorial paper). In _Proceedings of the Workshop on Structured Prediction for NLP_, pages 1-17, Austin, TX. Association for Computational Linguistics.
* Eisner and Blatz (2007) Jason Eisner and John Blatz. 2007. Program transformations for optimization of parsing algorithms and other weighted logic programs. In _Proceedings of FG 2006: The 11th Conference on Formal Grammar_, pages 45-85. CSLI Publications.
* Fu and Lapata (2021) Yao Fu and Mirella Lapata. 2021. Scaling structured inference with randomization. _CoRR_, abs/2112.03638 

Giorgio Gallo, Giustino Longo, and Stefano Pallottino. 1993 Directed hypergraphs and applications. _Discret. Appl. Math._, 42(2):177-201.
* Goodman [1996] Joshua Goodman. 1996. Parsing algorithms and metrics. In _34th Annual Meeting of the Association for Computational Linguistics_, pages 177-183, Santa Cruz, California, USA. Association for Computational Linguistics.
* Han et al. [2019] Wenjuan Han, Yong Jiang, and Kewei Tu. 2019. Enhancing unsupervised generative dependency parser with contextual information. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 5315-5325, Florence, Italy. Association for Computational Linguistics.
* He et al. [2018] Junxian He, Graham Neubig, and Taylor Berg-Kirkpatrick. 2018. Unsupervised learning of syntactic structure with invertible neural projections. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 1292-1302, Brussels, Belgium. Association for Computational Linguistics.
* Tutorial notes_, pages 1-18, Manchester, UK. Coling 2008 Organizing Committee.
* Jiang et al. [2016] Yong Jiang, Wenjuan Han, and Kewei Tu. 2016. Unsupervised neural dependency parsing. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 763-771, Austin, Texas. Association for Computational Linguistics.
* Jin et al. [2019] Lifeng Jin, Finale Doshi-Velez, Timothy Miller, Lane Schwartz, and William Schuler. 2019. Unsupervised learning of PCFGs with normalizing flow. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 2442-2452, Florence, Italy. Association for Computational Linguistics.
* Kim [2021] Yoon Kim. 2021. Sequence-to-sequence learning with latent neural grammars. _Advances in Neural Information Processing Systems_, 34.
* Kim et al. [2019] Yoon Kim, Chris Dyer, and Alexander Rush. 2019. Compound probabilistic context-free grammars for grammar induction. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 2369-2385, Florence, Italy. Association for Computational Linguistics.
* Klein and Manning [2001] Dan Klein and Christopher D. Manning. 2001. Parsing and hypergraphs. In _Proceedings of the Seventh International Workshop on Parsing Technologies (IWPT-2001), 17-19 October 2001, Beijing, China_. Tsinghua University Press.
* Marcus et al. [1994] Mitchell Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. 1994. The Penn Treebank: Annotating predicate argument structure. In _Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994_.
* May 3, 2018, Conference Track Proceedings_. OpenReview.net.
* Miller et al. [2021] Jacob Miller, Guillaume Rabusseau, and John Terilla. 2021. Tensor networks for probabilistic sequence modeling. In _The 24th International Conference on Artificial Intelligence and Statistics, AISTATS 2021, April 13-15, 2021, Virtual Event_, volume 130 of _Proceedings of Machine Learning Research_, pages 3079-3087. PMLR.
* Peharz et al. [2020] Robert Peharz, Steven Lang, Antonio Vergari, Karl Stelzner, Alejandro Molina, Martin Trapp, Guy Van den Broeck, Kristin Kersting, and Zoubin Ghahlramani. 2020. Einsum networks: Fast and scalable learning of tractable probabilistic circuits. In _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_, volume 119 of _Proceedings of Machine Learning Research_, pages 7563-7574. PMLR.
* Rabanser et al. [2017] Stephan Rabanser, Oleksandr Shchur, and Stephan Gunnemann. 2017. Introduction to tensor decompositions and their applications in machine learning. _CoRR_, abs/1711.10781.
* Rabusseau et al. [2016] Guillaume Rabusseau, Borja Balle, and Shay B. Cohen. 2016. Low-rank approximation of weighted tree automata. In _Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, AISTATS 2016, Cadiz, Spain, May 9-11, 2016_, volume 51 of _JMLR Workshop and Conference Proceedings_, pages 839-847. JMLR.org.
* Rush [2020] Alexander Rush. 2020. Torch-struct: Deep structured prediction library. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_, pages 335-342, Online. Association for Computational Linguistics.
* Shen et al. [2021] Yikang Shen, Yi Tay, Che Zheng, Dara Bahri, Donald Metzler, and Aaron Courville. 2021. StructFormer: Joint unsupervised induction of dependency and constituency structure from masked language modeling. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 7196-7209, Online. Association for Computational Linguistics.
* Siddiqi et al. [2010] Sajid M. Siddiqi, Byron Boots, and Geoffrey J. Gordon. 2010. Reduced-rank hidden markov models.

 

In _Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2010, Chia Laguna Resort, Sardinia, Italy, May 13-15, 2010_, volume 9 of _JMLR Proceedings_, pages 741-748. JMLR.org.
* Smith and Eisner (2006) David A. Smith and Jason Eisner. 2006. Minimum risk annealing for training log-linear models. In _Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions_, pages 787-794, Sydney, Australia. Association for Computational Linguistics.
* Vieira et al. (2021) Tim Vieira, Ryan Cotterell, and Jason Eisner. 2021. Searching for more efficient dynamic programs. In _Findings of the Association for Computational Linguistics: EMNLP 2021_, pages 3812-3830, Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Wrigley et al. (2017) Andrew Wrigley, Wee Sun Lee, and Nan Ye. 2017. Tensor belief propagation. In _Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017_, volume 70 of _Proceedings of Machine Learning Research_, pages 3771-3779. PMLR.
* Xu et al. (2018) Ji Xu, Daniel J. Hsu, and Arian Maleki. 2018. Benefits of over-parameterization with EM. In _Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018, Montreal, Canada_, pages 10685-10695.
* Xu et al. (2021) Zhiyang Xu, Andrew Drozdov, Jay Yoon Lee, Tim O'Gorman, Subendhu Rongali, Dylan Finkbeiner, Shilpa Suresh, Mohit Iyyer, and Andrew McCallum. 2021. Improved latent tree induction with distant supervision via span constraints. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 4818-4831, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Yang et al. (2020) Songlin Yang, Yong Jiang, Wenjuan Han, and Kewei Tu. 2020. Second-order unsupervised neural dependency parsing. In _Proceedings of the 28th International Conference on Computational Linguistics_, pages 3911-3924, Barcelona, Spain (Online). International Committee on Computational Linguistics.
* Yang and Tu (2022) Songlin Yang and Kewei Tu. 2022. Modeling label correlations for second-order semantic dependency parsing with mean-field inference. _CoRR_, abs/2204.03619.
* Yang et al. (2021a) Songlin Yang, Yanpeng Zhao, and Kewei Tu. 2021a. Neural bi-lexicalized PCFG induction. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 2688-2699, Online. Association for Computational Linguistics.
* Yang et al. (2021b) Songlin Yang, Yanpeng Zhao, and Kewei Tu. 2021b. PCFGs can do better: Inducing probabilistic context-free grammars with many symbols. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1487-1498, Online. Association for Computational Linguistics.
* Yi et al. (2011) Youngmin Yi, Chao-Yue Lai, Slav Petrov, and Kurt Keutzer. 2011. Efficient parallel CKY parsing on GPUs. In _Proceedings of the 12th International Conference on Parsing Technologies_, pages 175-185, Dublin, Ireland. Association for Computational Linguistics.
* Zhang et al. (2021) Songyang Zhang, Linfeng Song, Lifeng Jin, Kun Xu, Dong Yu, and Jiebo Luo. 2021. Video-aided unsupervised grammar induction. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1513-1524, Online. Association for Computational Linguistics.
* Zhang et al. (2020) Yu Zhang, Houquan Zhou, and Zhenghua Li. 2020. Fast and accurate neural CRF constituency parsing. In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020_, pages 4046-4053. ijcai.org.
* Zhao and Titov (2020) Yanpeng Zhao and Ivan Titov. 2020. Visually grounded compound PCFGs. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 4369-4379, Online. Association for Computational Linguistics.
* Zhu et al. (2020) Hao Zhu, Yonatan Bisk, and Graham Neubig. 2020. The return of lexical dependencies: Neural lexicalized PCFGs. _Transactions of the Association for Computational Linguistics_, 8:647-661.

## Appendix A Neural parameterization of PCFGs

In this section, we give the full parameterization of PCFGs. We follow Yang et al. (2021b) with slight modifications for generations of \({\bf U},{\bf V},{\bf W}\in\mathbb{R}^{m\times r}\) in 4.2. We use the same MLPs with two residual layers as Yang et al. (2021b):

\[{\bf s}=\frac{\exp({\bf u}_{S}^{T}h_{1}({\bf w}_{A})}{\sum_{A^{ \prime}\in\mathcal{N}}\exp({\bf u}_{S}^{T}h_{1}({\bf w}_{A^{\prime}}))}\] \[{\bf E}=\frac{\exp({\bf u}_{E}^{T}h_{2}({\bf w}_{t})}{\sum_{E^{ \prime}\in\Sigma}\exp({\bf u}_{E^{\prime}}^{T}h_{2}({\bf w}_{t}))}\] \[{\bf U}=\frac{\exp({\bf u}_{H}^{T}f_{1}({\bf w}_{n})}{\sum_{n^{ \prime}\in\mathcal{N}}\exp({\bf u}_{H}^{T}f_{1}({\bf w}_{n^{\prime}}))}\] \[{\bf V}=\frac{\exp({\bf u}_{H}^{T}f_{2}({\bf w}_{l})}{\sum_{H^{ \prime}\in\mathcal{H}}\exp({\bf u}_{H^{\prime}}^{T}f_{2}({\bf w}_{l}))}\] \[{\bf W}=\frac{\exp({\bf u}_{H}^{T}f_{3}({\bf w}_{l})}{\sum_{H^{ \prime}\in\mathcal{H}}\exp({\bf u}_{H^{\prime}}^{T}f_{3}({\bf w}_{l}))}\] \[h_{i}({\bf x})=g_{i,1}(g_{i,2}(\tilde{\bf W}_{i}{\bf x}))\] \[g_{i,j}({\bf y})=ReLU(\tilde{\bf V}_{i,j}ReLU(\tilde{\bf U}_{i, j}{\bf y}))+{\bf y}\] 

where \(\Sigma\) is the vocabulary set, \({\cal H}\) is the set of rank, \({\cal N}\) is a finite set of nonterminals, \({\bf W}_{l}=[{\bf W}_{n};{\bf W}_{t}],{\bf w}_{l},{\bf w}_{n},{\bf w}_{t}\in{\bf W }_{l},{\bf W}_{n},{\bf W}_{t}.\) The main differences of neural parameterization between ours and previous work are that we make the projection parameter \({\bf u}_{H}\) shared among \({\bf U},{\bf V},~{}\text{and}~{}{\bf U}.\)

## Appendix B Neural parameterization of HMMs

In this section, we give the full parameterization of HMMs, which is similar to PCFGs' parameterization. Define \({\bf s}\) as start probability for HMMs. And the definitions of \({\bf U},{\bf V},{\bf W}\) are same as definitions in \(4.3\):

\[{\bf s} =\frac{\exp({\bf u}^{T}_{P}h_{1}({\bf v}_{s}))}{\sum_{s^{\prime} \in{\cal S}}\exp({\bf u}^{T}_{P}h_{1}({\bf w}_{s^{\prime}}))}\] \[{\bf U} =\frac{\exp({\bf u}^{T}_{H}{\bf w}_{u})}{\sum_{H^{\prime}\in{\cal H }}\exp({\bf u}^{T}_{H^{\prime}}{\bf w}_{u})}\] \[{\bf V} =\frac{\exp({\bf u}^{T}_{H^{\prime}}{\bf w}_{v})}{\sum_{v^{\prime} \in{\cal S}}\exp({\bf u}^{T}_{H^{\prime}}{\bf w}_{v^{\prime}})}\] \[{\bf W} =\frac{\exp({\bf u}^{T}_{W}h_{2}({\bf w}_{w})}{\sum_{w^{\prime} \in{\cal S}}\exp({\bf u}^{T}_{W}h_{2}({\bf w}_{w^{\prime}}))}\] \[h_{i}({\bf x}) =g_{i,1}(g_{i,2}(\tilde{\bf W}_{i}{\bf x}))\] \[g_{i,j}({\bf y}) =ReLU(\tilde{\bf V}_{i,j}ReLU(\tilde{\bf U}_{i,j}{\bf y}))+{\bf y}\]

where \({\cal S}\) is a finite set of states, \({\cal H}\) is the set of rank, \(\Sigma\) is vocabulary set.

## Appendix C Data details

Penn Treebank (PTB) Marcus et al. (1994)10 consists of 929k training words, 73k validation words, and 82k test words, with a vocabulary of size 10k.

Footnote 10: The licence of PTB dataset is LDC User Agreement for Non-Members, which can be seen on [https://catalog.ldc.upenn.edu/LDC99T42](https://catalog.ldc.upenn.edu/LDC99T42)

For PCFGs, we follow Yang et al. (2021b) and use their code to preprocess dataset. This processing discards punctuation and lowercases all tokens with 10k most frequent words as the vocabulary. The splits of the dataset are: 2-21 for training, 22 for validation and 23 for test.

For HMMs, we follow Chiu et al. (2021) and use their code to preprocess dataset. We lowercase all words and substitutes OOV words with UNKs. EOS tokens have been inserted after each sentence.