SlowGuess/NAACL_2021_OCR huggingface dataset:
Songlin Yang\*, Yanpeng Zhao\*, Kewei Tu\*
$\clubsuit$ School of Information Science and Technology, ShanghaiTech University
$\diamond$  ILCC, University of Edinburgh
{yangsl,tukw}@shanghaiitech.edu.cn
yannzhao.ed@gmail.com
Probabilistic context-free grammars (PCFGs) with neural parameterization have been shown to be effective in unsupervised phrase-structure grammar induction. However, due to the cubic computational complexity of PCFG representation and parsing, previous approaches cannot scale up to a relatively large number of (nonterminal and preterminal) symbols. In this work, we present a new parameterization form of PCFGs based on tensor decomposition, which has at most quadratic computational complexity in the symbol number and therefore allows us to use a much larger number of symbols. We further use neural parameterization for the new form to improve unsupervised parsing performance. We evaluate our model across ten languages and empirically demonstrate the effectiveness of using more symbols.
Unsupervised constituency parsing is the task of inducing phrase-structure grammars from raw text without using parse tree annotations. Early work induces probabilistic context-free grammars (PCFGs) via the Expectation Maximization algorithm and finds the result unsatisfactory (Lari and Young, 1990; Carroll and Charniak, 1992). Recently, PCFGs with neural parameterization (i.e., using neural networks to generate rule probabilities) have been shown to achieve good results in unsupervised constituency parsing (Kim et al., 2019a; Jin et al., 2019; Zhu et al., 2020). However, due to the cubic computational complexity of PCFG representation and parsing, these approaches learn PCFGs with relatively small numbers of nonterminals and preterminals. For example, Jin et al. (2019) use 30
nonterminals (with no distinction between preterminals and other nonterminals) and Kim et al. (2019a) use 30 nonterminals and 60 preterminals.
In this paper, we study PCFG induction with a much larger number of nonterminal and preterminal symbols. We are partly motivated by the classic work of latent variable grammars in supervised constituency parsing (Matsuzaki et al., 2005; Petrov et al., 2006; Liang et al., 2007; Cohen et al., 2012; Zhao et al., 2018). While the Penn treebank grammar contains only tens of nonterminals and preterminals, it has been found that dividing them into subtypes could significantly improve the parsing accuracy of the grammar. For example, the best model from Petrov et al. (2006) contains over 1000 nonterminal and preterminal symbols. We are also motivated by the recent work of Buhai et al. (2019) who show that when learning latent variable models, increasing the number of hidden states is often helpful; and by Chiu and Rush (2020) who show that a neural hidden Markov model with up to  $2^{16}$  hidden states can achieve surprisingly good performance in language modeling.
A major challenge in employing a large number of nonterminal and preterminal symbols is that representing and parsing with a PCFG requires a computational complexity that is cubic in its symbol number. To resolve the issue, we rely on a new parameterization form of PCFGs based on tensor decomposition, which reduces the computational complexity from cubic to at most quadratic. Furthermore, we apply neural parameterization to the new form, which is crucial for boosting unsupervised parsing performance of PCFGs as shown by Kim et al. (2019a).
We empirically evaluate our approach across ten languages. On English WSJ, our best model with 500 preterminals and 250 nonterminals improves over the model with 60 preterminals and 30 nonter
$^{1}$ Our code: https://github.com/sustcsonglin/TN-PCFG
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1487-1498 June 6-11, 2021. ©2021 Association for Computational Linguistics
minals by  $6.3\%$  mean F1 score, and we also observe consistent decrease in perplexity and overall increase in F1 score with more symbols in our model, thus confirming the effectiveness of using more symbols. Our best model also surpasses the strong baseline Compound PCFGs (Kim et al., 2019a) by  $1.4\%$  mean F1. We further conduct multilingual evaluation on nine additional languages. The evaluation results suggest good generalizability of our approach on languages beyond English.
Our key contributions can be summarized as follows: (1) We propose a new parameterization form of PCFGs based on tensor decomposition, which enables us to use a large number of symbols in PCFGs. (2) We further apply neural parameterization to improve unsupervised parsing performance. (3) We evaluate our model across ten languages and empirically show the effectiveness of our approach.
Grammar induction using neural networks: There is a recent resurgence of interest in unsupervised constituency parsing, mostly driven by neural network based methods (Shen et al., 2018a, 2019; Drozdov et al., 2019, 2020; Kim et al., 2019a,b; Jin et al., 2019; Zhu et al., 2020). These methods can be categorized into two major groups: those built on top of a generative grammar and those without a grammar component. The approaches most related to ours belong to the first category, which use neural networks to produce grammar rule probabilities. Jin et al. (2019) use an invertible neural projection network (a.k.a. normalizing flow (Rezende and Mohamed, 2015)) to parameterize the preterminal rules of a PCFG. Kim et al. (2019a) use neural networks to parameterize all the PCFG rules. Zhu et al. (2020) extend their work to lexicalized PCFGs, which are more expressive than PCFGs and can model both dependency and constituency parse trees simultaneously.
In other unsupervised syntactic induction tasks, there is also a trend to use neural networks to produce grammar rule probabilities. In unsupervised dependency parsing, the Dependency Model with Valence (DMV) (Klein and Manning, 2004) has been parameterized neurally to achieve higher induction accuracy (Jiang et al., 2016; Yang et al., 2020). In part-of-speech (POS) induction, neurally parameterized Hidden Markov Models (HMM) also achieve state-of-the-art results (Tran et al., 2016; He et al., 2018).
Tensor decomposition on PCFGs: Our work is closely related to Cohen et al. (2013) in that both use tensor decomposition to parameterize the probabilities of binary rules for the purpose of reducing the time complexity of the inside algorithm. However, Cohen et al. (2013) use this technique to speed up inference of an existing PCFG, and they need to actually perform tensor decomposition on the rule probability tensor of the PCFG. In contrast, we draw inspiration from this technique to design a new parameterization form of PCFG that can be directly learned from data. Since we do not have a probability tensor to start with, additional tricks have to be inserted in order to ensure validity of the parameterization, as will be discussed later.
3.1 Tensor form of PCFGs
PCFGs build upon context-free grammars (CFGs). We start by introducing CFGs and establishing notations. A CFG is defined as a 5-tuple  $\mathcal{G} = (S, \mathcal{N}, \mathcal{P}, \Sigma, \mathcal{R})$  where  $S$  is the start symbol,  $\mathcal{N}$  is a finite set of nonterminal symbols,  $\mathcal{P}$  is a finite set of preterminal symbols,  $\Sigma$  is a finite set of terminal symbols, and  $\mathcal{R}$  is a set of rules in the following form:
$$
S \to A A \in \mathcal {N}
$$
$$
A \to B C, \qquad A \in \mathcal {N}, \quad B, C \in \mathcal {N} \cup \mathcal {P}
$$
$$
T \to w, T \in \mathcal {P}, w \in \Sigma
$$
PCFGs extend CFGs by associating each rule  $r \in \mathcal{R}$  with a probability  $\pi_r$ . Denote  $n, p,$  and  $q$  as the number of symbols in  $\mathcal{N}, \mathcal{P},$  and  $\Sigma$ , respectively. It is convenient to represent the probabilities of the binary rules in the tensor form:
$$
\mathbf {T} _ {h _ {A}, h _ {B}, h _ {C}} = \pi_ {A \to B C}, \quad \mathbf {T} \in \mathbb {R} ^ {n \times m \times m},
$$
where  $\mathbf{T}$  is an order-3 tensor,  $m = n + p$ , and  $h_A \in [0,n)$  and  $h_B, h_C \in [0,m)$  are symbol indices. For the convenience of computation, we assign indices  $[0,n)$  to nonterminals in  $\mathcal{N}$  and  $[n,m)$  to preterminals in  $\mathcal{P}$ . Similarly, for a preterminal rule we define
$$
\mathbf {Q} _ {h _ {T}, h _ {w}} = \boldsymbol {\pi} _ {T \rightarrow w}, \quad \mathbf {Q} \in \mathbb {R} ^ {p \times q}.
$$
${}^{2}$  Strictly, CFGs do not distinguish nonterminals  $\mathcal{N}$  (constituent labels) from preterminals  $\mathcal{P}$  (part-of-speech tags). They are both treated as nonterminals.  $\mathcal{N},\mathcal{P},\Sigma$  satisfy  $\mathcal{N} \cap  \mathcal{P} = \varnothing$  and  $\left( {\mathcal{N} \cup  \mathcal{P}}\right)  \cap  \Sigma  = \varnothing$  .
Again,  $h_T$  and  $h_w$  are the preterminal index and the terminal index, respectively. Finally, for a start rule we define
$$
\mathbf {r} _ {h _ {A}} = \pi_ {S \rightarrow A}, \quad \mathbf {r} \in \mathbb {R} ^ {n}.
$$
Generative learning of PCFGs involves maximizing the log-likelihood of every observed sentence  $\boldsymbol{w} = w_{1},\dots ,w_{l}$ :
$$
\log p _ {\theta} (\pmb {w}) = \log \sum_ {t \in T _ {\mathcal {G}} (\pmb {w})} p (t),
$$
where  $T_{\mathcal{G}}(\boldsymbol{w})$  contains all the parse trees of the sentence  $\boldsymbol{w}$  under a PCFG  $\mathcal{G}$ . The probability of a parse tree  $t \in T_{\mathcal{G}}$  is defined as  $p(t) = \prod_{r \in t_{\mathcal{R}}} \pi_r$ , where  $t_{\mathcal{R}}$  is the set of rules used in the derivation of  $t$ .  $\log p_{\theta}(\boldsymbol{w})$  can be estimated efficiently through the inside algorithm, which is fully differentiable and amenable to gradient optimization methods.
3.2 Tensor form of the inside algorithm
We first pad  $\mathbf{T}$ ,  $\mathbf{Q}$ , and  $\mathbf{r}$  with zeros such that  $\mathbf{T} \in \mathbb{R}^{m \times m \times m}$ ,  $\mathbf{Q} \in \mathbb{R}^{m \times q}$ ,  $\mathbf{r} \in \mathbb{R}^m$ , and all of them can be indexed by both nonterminals and preterminals.
The inside algorithm computes the probability of a symbol  $A$  spanning a substring  $w_{i,j} = w_i, \ldots, w_j$  in a recursive manner  $(0 \leq i < j < l)$ :
$$
s _ {i, j} ^ {A} = \sum_ {k = i} ^ {j - 1} \sum_ {B, C} \pi_ {A \rightarrow B C} \cdot s _ {i, k} ^ {B} \cdot s _ {k + 1, j} ^ {C}. \tag {1}
$$
Base Case:  $s_{i,i}^{T} = \pi_{T \to w_{i}}$ ,  $0 \leq i < l$ .
We use the tensor form of PCFGs to rewrite Equation 1 as:
$$
\begin{array}{l} \mathbf {s} _ {i, j} ^ {h _ {A}} = \sum_ {k = i} ^ {j - 1} \sum_ {h _ {B}, h _ {C}} \mathbf {T} _ {h _ {A}, h _ {B}, h _ {C}} \cdot \mathbf {s} _ {i, k} ^ {h _ {B}} \cdot \mathbf {s} _ {k + 1, j} ^ {h _ {C}} \\ = \sum_ {k = i} ^ {j - 1} \left(\mathbf {T} _ {h _ {A}} \cdot \mathbf {s} _ {k + 1, j}\right) \cdot \mathbf {s} _ {i, k}, \tag {2} \\ \end{array}
$$
where  $\mathbf{s}_{i,j}$ ,  $\mathbf{s}_{i,k}$ , and  $\mathbf{s}_{k+1,j}$  are all  $m$ -dimensional vectors; the dimension  $h_A$  corresponds to the symbol  $A$ . Thus
$$
\mathbf {s} _ {i, j} = \sum_ {k = i} ^ {j - 1} \left(\mathbf {T} \cdot \mathbf {s} _ {k + 1, j}\right) \cdot \mathbf {s} _ {i, k}. \tag {3}
$$
Equation 3 represents the core computation of the inside algorithm as tensor-vector dot product. It is amenable to be accelerated on a parallel computing device such as GPUs. However, the time and space complexity is cubic in  $m$ , which makes it impractical to use a large number of nonterminals and preterminals.
4 Parameterizing PCFGs based on tensor decomposition
The tensor form of the inside algorithm has a high computational complexity of  $\mathcal{O}(m^3 l^3)$ . It hinders the algorithm from scaling to a large  $m$ . To resolve the issue, we resort to a new parameterization form of PCFGs based on tensor decomposition (TD-PCFGs) (Cohen et al., 2013). As discussed in Section 2, while Cohen et al. (2013) use a TD-PCFG to approximate an existing PCFG for speedup in parsing, we regard a TD-PCFG as a stand-alone model and learn it directly from data.
The basic idea behind TD-PCFGs is using Kruskal decomposition of the order-3 tensor  $\mathbf{T}$ . Specifically, we require  $\mathbf{T}$  to be in the Kruskal form,
$$
\mathbf {T} = \sum_ {l = 1} ^ {d} \mathbf {T} ^ {(l)}, \mathbf {T} ^ {(l)} = \mathbf {u} ^ {(l)} \otimes \mathbf {v} ^ {(l)} \otimes \mathbf {w} ^ {(l)}, (4)
$$
where  $\mathbf{u}^{(l)}\in \mathbb{R}^n$  is a column vector of a matrix  $\mathbf{U}\in \mathbb{R}^{n\times d}$  .  $\mathbf{v}^{(l)},\mathbf{w}^{(l)}\in \mathbb{R}^{m}$  are column vectors of matrices  $\mathbf{V},\mathbf{W}\in \mathbb{R}^{m\times d}$  , respectively;  $\otimes$  indicates Kronecker product. Thus  $\mathbf{T}^{(l)}\in \mathbb{R}^{n\times m\times m}$  is an order-3 tensor and
$$
\mathbf {T} _ {i, j, k} ^ {(l)} = \mathbf {u} _ {i} ^ {(l)} \cdot \mathbf {v} _ {j} ^ {(l)} \cdot \mathbf {w} _ {k} ^ {(l)}.
$$
The Kruskal form of the tensor  $\mathbf{T}$  is crucial for reducing the computation of Equation 3. To show this, we let  $\mathbf{x} = \mathbf{s}_{i,k},\mathbf{y} = \mathbf{s}_{k + 1,j}$ , and  $\mathbf{z}$  be any summand in the right-hand side of Equation 3, so we have:
$$
\mathbf {z} = (\mathbf {T} \cdot \mathbf {y}) \cdot \mathbf {x}. \tag {5}
$$
Substitute  $\mathbf{T}$  in Equation 4 into Equation 5 and consider the  $i$ -th dimension of  $\mathbf{z}$ :
$$
\begin{array}{l} \mathbf {z} _ {i} = \left(\mathbf {T} _ {i} \cdot \mathbf {y}\right) \cdot \mathbf {x} \\ = \sum_ {j = 1} ^ {m} \sum_ {k = 1} ^ {m} \sum_ {l = 1} ^ {d} \mathbf {T} _ {i, j, k} ^ {(l)} \cdot \mathbf {x} _ {j} \cdot \mathbf {y} _ {k} \\ = \sum_ {j = 1} ^ {m} \sum_ {k = 1} ^ {m} \sum_ {l = 1} ^ {d} \mathbf {u} _ {i} ^ {(l)} \cdot \mathbf {v} _ {j} ^ {(l)} \cdot \mathbf {w} _ {k} ^ {(l)} \cdot \mathbf {x} _ {j} \cdot \mathbf {y} _ {k} \\ = \sum_ {l = 1} ^ {d} \mathbf {u} _ {i} ^ {(l)} \cdot \left(\sum_ {j = 1} ^ {m} \mathbf {v} _ {j} ^ {(l)} \cdot \mathbf {x} _ {j}\right) \cdot \left(\sum_ {k = 1} ^ {m} \mathbf {w} _ {k} ^ {(l)} \cdot \mathbf {y} _ {k}\right) \\ = \sum_ {l = 1} ^ {d} \mathbf {u} _ {i} ^ {(l)} \cdot \left(\mathbf {x} ^ {T} \mathbf {v} ^ {(l)}\right) \cdot \left(\mathbf {y} ^ {T} \mathbf {w} ^ {(l)}\right) \\ = \left(\mathbf {e} _ {i} ^ {T} \mathbf {U}\right) \cdot \left(\left(\mathbf {V} ^ {T} \mathbf {x}\right) \odot \left(\mathbf {W} ^ {T} \mathbf {y}\right)\right), \tag {6} \\ \end{array}
$$
where  $\odot$  indicates Hadamard (element-wise) product;  $\mathbf{e}_i\in \mathbb{R}^m$  is a one-hot vector that selects the  $i$ -th row of  $\mathbf{U}$ . We have padded  $\mathbf{U}$  with zeros such that  $\mathbf{U}\in \mathbb{R}^{m\times d}$  and the last  $m - n$  rows are all zeros. Thus
$$
\mathbf {z} = \mathbf {U} \cdot \left(\left(\mathbf {V} ^ {T} \mathbf {x}\right) \odot \left(\mathbf {W} ^ {T} \mathbf {y}\right)\right), \tag {7}
$$
and accordingly,
$$
\mathbf {s} _ {i, j} = \mathbf {U} \cdot \sum_ {k = i} ^ {j - 1} \left(\left(\mathbf {V} ^ {T} \mathbf {s} _ {i, k}\right) \odot \left(\mathbf {W} ^ {T} \mathbf {s} _ {k + 1, j}\right)\right). \tag {8}
$$
Equation 8 computes the inside probabilities using TD-PCFGs. It has a time complexity  $\mathcal{O}(md)$ . By caching  $\mathbf{V}^T\mathbf{s}_{i,k}$  and  $\mathbf{W}^T\mathbf{s}_{k+1,j}$ , the time complexity of the inside algorithm becomes  $\mathcal{O}(dl^3 + mdl^2)$  (Cohen et al., 2013), which is at most quadratic in  $m$  since we typically set  $d = \mathcal{O}(m)$ . Interestingly, Equation 8 has similar forms to recursive neural networks (Socher et al., 2013) if we treat inside score vectors as span embeddings.
One problem with TD-PCFGs is that, since we use three matrices  $\mathbf{U},\mathbf{V}$  and  $\mathbf{W}$  to represent tensor  $\mathbf{T}$  of binary rule probabilities, how we can ensure that  $\mathbf{T}$  is non-negative and properly normalized, i.e., for a given left-hand side symbol  $A$ $\sum_{j,k}\mathbf{T}_{h_A,j,k} = 1$ . Simply reconstructing  $\mathbf{T}$  with  $\mathbf{U},\mathbf{V}$  and  $\mathbf{W}$  and then performing normalization would take  $\mathcal{O}(m^3)$  time, thus defeating the purpose of TD-PCFGs. Our solution is to require that the three matrices are non-negative and meanwhile  $\mathbf{U}$  is row-normalized and  $\mathbf{V}$  and  $\mathbf{W}$  are column-normalized (Shen et al., 2018b).
Theorem 1. Given non-negative matrices  $\mathbf{U} \in \mathbb{R}^{n \times d}$  and  $\mathbf{V}, \mathbf{W} \in \mathbb{R}^{m \times d}$ , if  $\mathbf{U}$  is row-normalized and  $\mathbf{V}$  and  $\mathbf{W}$  are column-normalized, then  $\mathbf{U}, \mathbf{V}$ , and  $\mathbf{W}$  are a Kruskal decomposition of a tensor  $\mathbf{T} \in \mathbb{R}^{n \times m \times m}$  where  $\mathbf{T}_{i,j,k} \in [0,1]$  and  $\mathbf{T}_i$  is normalized such that  $\sum_{j,k} \mathbf{T}_{i,j,k} = 1$ .
$$
\begin{array}{l} \sum_ {j = 1} ^ {m} \sum_ {k = 1} ^ {m} {\bf T} _ {i, j, k} = \sum_ {j = 1} ^ {m} \sum_ {k = 1} ^ {m} \sum_ {l = 1} ^ {d} {\bf u} _ {i} ^ {(l)} \cdot {\bf v} _ {j} ^ {(l)} \cdot {\bf w} _ {k} ^ {(l)} \\ = \sum_ {l = 1} ^ {d} \mathbf {u} _ {i} ^ {(l)} \cdot (\sum_ {j = 1} ^ {m} \mathbf {v} _ {j} ^ {(l)}) \cdot (\sum_ {k = 1} ^ {m} \mathbf {w} _ {j} ^ {(l)}) \\ = \sum_ {l = 1} ^ {d} \mathbf {u} _ {i} ^ {(l)} \cdot 1 \cdot 1 = 1 \\ \end{array}
$$
5 Neural parameterization of TD-PCFGs
We use neural parameterization for TD-PCFGs as it has demonstrated its effectiveness in inducing PCFGs (Kim et al., 2019a). In a neurally parameterized TD-PCFGs, the original TD-PCFG parameters are generated by neural networks, rather than being learned directly; parameters of the neural network will thus be the parameters to be optimized. This modeling approach breaks the parameter number limit of the original TD-PCFG, so we can control the total number of parameters flexibly. When the total number of symbols is small, we can overparameterize the model as over-parameterization has been shown to ease optimization (Arora et al., 2018; Xu et al., 2018; Du et al., 2019). On the other hand, when the total number of symbols is huge, we can decrease the number of parameters to save GPU memories and speed up training.
Specifically, we use neural networks to generate the following set of parameters of a TD-PCFG:
$$
\Theta = \left\{\mathbf {U}, \mathbf {V}, \mathbf {W}, \mathbf {Q}, \mathbf {r} \right\}.
$$
The resulting model is referred to as neural PCFGs based on tensor decomposition (TN-PCFGs).
We start with the neural parameterization of  $\mathbf{U} \in \mathbb{R}^{n \times d}$  and  $\mathbf{V}, \mathbf{W} \in \mathbb{R}^{m \times d}$ . We use shared symbol embeddings  $\mathbf{E}_s \in \mathbb{R}^{m \times k}$  ( $k$  is the symbol embedding dimension) in which each row is the embedding of a nonterminal or preterminal. We first compute an unnormalized  $\tilde{\mathbf{U}}$  by applying a neural network  $f_u(\cdot)$  to symbol embeddings  $\mathbf{E}_s$ :
$$
\tilde {\mathbf {U}} = f _ {u} (\mathbf {E} _ {s}) = \left(\mathrm {R e L U} \left(\mathbf {E} _ {s} \mathbf {M} _ {u} ^ {(1)}\right)\right) \mathbf {M} _ {u} ^ {(2)},
$$
where  $\mathbf{M}_u^{(1)}\in \mathbb{R}^{k\times k}$  and  $\mathbf{M}_u^{(2)}\in \mathbb{R}^{k\times d}$  are learnable parameters of  $f_{u}(\cdot)$ . For simplicity, we omit the learnable bias terms. We compute unnormalized  $\tilde{\mathbf{V}}$  and  $\tilde{\mathbf{W}}$  in a similar way. Note that only  $\mathbf{E}_s$  is shared in computing the three unnormalized matrices. Then we apply the Softmax activation function to each row of  $\tilde{\mathbf{U}}$  and to each column of  $\tilde{\mathbf{V}}$  and  $\tilde{\mathbf{W}}$ , and obtain normalized  $\mathbf{U}$ ,  $\mathbf{V}$ , and  $\mathbf{W}$ .
For preterminal-rule probabilities  $\mathbf{Q} \in \mathbb{R}^{p \times q}$  and start-rule probabilities  $\mathbf{r} \in \mathbb{R}^n$ , we follow (Kim et al., 2019a) and define them as:
$$
\begin{array}{l} \mathbf {Q} _ {h _ {T}, h _ {w}} = \pi_ {T \rightarrow w} = \frac {\exp (\mathbf {u} _ {w} ^ {T} f _ {t} (\mathbf {w} _ {T}))}{\sum_ {w ^ {\prime} \in \Sigma} \exp (\mathbf {u} _ {w ^ {\prime}} ^ {T} f _ {t} (\mathbf {w} _ {T}))}, \\ \mathbf {r} _ {h _ {A}} = \boldsymbol {\pi} _ {S \rightarrow A} = \frac {\exp (\mathbf {u} _ {A} ^ {T} f _ {s} (\mathbf {w} _ {S}))}{\sum_ {A ^ {\prime} \in \mathcal {N}} \exp (\mathbf {u} _ {A ^ {\prime}} ^ {T} f _ {s} (\mathbf {w} _ {S}))}, \\ \end{array}
$$
where  $\mathbf{w}$  and  $\mathbf{u}$  are symbol embeddings;  $f_{s}(\cdot)$  and  $f_{t}(\cdot)$  are neural networks that encode the input into a vector (see details in Kim et al. (2019a)). Note that the symbol embeddings are not shared between preterminal rules and start rules.
6 Parsing with TD-PCFGs
Parsing seeks the most probable parse  $t^{\star}$  from all the parses  $T_{\mathcal{G}}(\boldsymbol{w})$  of a sentence  $\boldsymbol{w}$ :
$$
t ^ {\star} = \arg \max  _ {t \in T _ {\mathcal {G}} (\boldsymbol {w})} p (t | \boldsymbol {w}). \tag {9}
$$
Typically, the CYK algorithm<sup>3</sup> can be directly used to solve this problem exactly: it first computes the score of the most likely parse; and then automatic differentiation is applied to recover the best tree structure  $t^{\star}$  (Eisner, 2016; Rush, 2020). This, however, relies on the original probability tensor  $\mathbf{T}$  and is incompatible with our decomposed representation.<sup>4</sup> If we reconstruct  $\mathbf{T}$  from  $\mathbf{U}, \mathbf{V}, \mathbf{W}$  and then perform CYK, then the resulting time and space complexity would degrade to  $\mathcal{O}(m^{3}l^{3})$  and become unaffordable when  $m$  is large. Therefore, we resort to Minimum Bayes-Risk (MBR) style decoding because we can compute the inside probabilities efficiently.
Our decoding method consists of two stages. The first stage computes the conditional probability of a substring  $\boldsymbol{w}_{i,j}$  being a constituent in a given sentence  $\boldsymbol{w}$  (a.k.a. posteriors of spans being a constituent):
$$
p (\pmb {w} _ {i, j} | \pmb {w}) = \frac {1}{p (\pmb {w})} \sum_ {t \in T _ {\mathcal {G}} (\pmb {w})} p (t) \cdot \mathbb {1} _ {\{\pmb {w} _ {i, j} \in t \}}.
$$
We can estimate the posteriors efficiently by using automatic differentiation after obtaining all the inside probabilities. This has the same time complexity as our improved inside algorithm, which is  $\mathcal{O}(dl^3 + mdl^2)$ . The second stage uses the CYK algorithm to find the parse tree that has the highest expected number of constituents (Smith and Eisner, 2006):
$$
t ^ {\star} = \arg \max  _ {t \in T _ {\mathcal {G}} (\boldsymbol {w})} \sum_ {\boldsymbol {w} _ {i, j} \in t} p \left(\boldsymbol {w} _ {i, j} \mid \boldsymbol {w}\right). \tag {10}
$$
The time complexity of the second stage is  $\mathcal{O}(l^3)$  so the overall time complexity of our decoding method is  $\mathcal{O}(dl^3 + mdl^2)$ , which is much faster than  $\mathcal{O}(m^3 l^3)$  in general.
7 Experimental setup
We evaluate TN-PCFGs across ten languages. We use the Wall Street Journal (WSJ) corpus of the Penn Treebank (Marcus et al., 1994) for English, the Penn Chinese Treebank 5.1 (CTB) (Xue et al., 2005) for Chinese, and the SPRML dataset (Seddah et al., 2014) for the other eight morphology-rich languages. We use a unified data preprocessing pipeline provided by Zhao and Titov (2021). The same pipeline has been used in several recent papers (Shen et al., 2018a, 2019; Kim et al., 2019a; Zhao and Titov, 2020). Specifically, for every treebank, punctuation is removed from all data splits and the top 10,000 frequent words in the training data are used as the vocabulary.
7.2 Settings and hyperparameters