SlowGuess/NAACL_2022_OCR huggingface dataset:
Songlin Yang*, Wei Liu*, Kewei Tu†
School of Information Science and Technology, ShanghaiTech University
{yangsl, liuwei4, tukw}@shanghaiitech.edu.cn
Hidden Markov Models (HMMs) and Probabilistic Context-Free Grammars (PCFGs) are widely used structured models, both of which can be represented as factor graph grammars (FGGs), a powerful formalism capable of describing a wide range of models. Recent research found it beneficial to use large state spaces for HMMs and PCFGs. However, inference with large state spaces is computationally demanding, especially for PCFGs. To tackle this challenge, we leverage tensor rank decomposition (aka. CPD) to decrease inference computational complexities for a subset of FGGs subsuming HMMs and PCFGs. We apply CPD on the factors of an FGG and then construct a new FGG defined in the rank space. Inference with the new FGG produces the same result but has a lower time complexity when the rank size is smaller than the state size. We conduct experiments on HMM language modeling and unsupervised PCFG parsing, showing better performance than previous work. Our code is publicly available at https://github.com/VPeterV/RankSpace-Models.
Hidden Markov Models (HMMs) and Probabilistic Context-Free Grammars (PCFGs) are widely used structured models in natural language processing. They can both be represented as factor graph grammars (FGGs) (Chiang and Riley, 2020), which are a powerful tool to describe a wide range of models, allowing exact and tractable inference in most situations of interest.
Over-parameterization has been shown beneficial in facilitating optimization of deep networks (Arora et al., 2018; Xu et al., 2018; Du et al., 2019). Buhai et al. (2020) found that overparameterization is also helpful in learning latent
variable models by increasing the number of hidden states. Buhai et al. (2020) argued that it is important to study over-parameterization in structured settings because structured latent variable models are more suitable to model real-word phenomena which exhibit complex dependencies. HMMs and PCFGs are typical structured latent variable models, and recently researchers have found it beneficial to use large state spaces for HMMs and PCFGs (Dedieu et al., 2019; Chiu and Rush, 2020; Yang et al., 2021b; Chiu et al., 2021). However, structured inference with large state spaces is computationally demanding, especially for PCFGs, pushing researchers to develop methods to decrease the computational complexities. Chiu and Rush (2020) propose a neural VL-HMM with  $2^{15}$  states for language modeling, narrowing down the performance gap between HMMs and LSTMs. They follow Dedieu et al. (2019) to impose a strong sparsity constraint (i.e., each hidden state can only generate a small subset of terminal symbols) to decrease the time complexity of the forward algorithm, thus requiring pre-clustering of terminal symbols. Yang et al. (2021b) use a large state space for neural PCFG induction and achieve superior unsupervised constituency parsing performance. They follow Cohen et al. (2013) to use tensor rank decomposition (aka. canonical-polyadic decomposition (CPD) (Rabanser et al., 2017)) to decrease the computational complexity of the inside algorithm, but only scale the state size from tens to hundreds because the resulting complexity is still high. Chiu et al. (2021) use tensor matricization and low-rank matrix decomposition to accelerate structured inference on chain and tree structure models. However, their method has an even higher complexity than Yang et al. (2021b) on PCFGs. Recently, Fu and Lapata (2021) propose a family of randomized dynamic programming algorithms to scale structured models to tens of thousands of states, which is orthogonal to the aforementioned low-rank-based
*Equal contributions: Songlin Yang formulated the idea and Wei Liu conducted the experiments. † Corresponding author.
Human Language Technologies, pages 4797 - 4809
approaches as the former performs approximate inference whereas the latter perform exact inference.
In this work, we propose a new low-rank-based approach to scale structured inference, which can be described by FGG notations intuitively. We first provide an intuitive and unifying perspective toward the work of Yang et al. (2021b) and Chiu et al. (2021), showing that their low-rank decomposition-based models can be viewed as decomposing large factors in an FGG—e.g., the binary rule probability tensor in PCFGs—into several smaller factors connected by new "rank" nodes. Then we target at a subset of FGGs—which we refer to as B-FGGs—subsuming all models considered by Chiu et al. (2021), whereby the inference algorithms can be formulated via B-graphs (Gallo et al., 1993; Klein and Manning, 2001). We propose a novel framework to support a family of inference algorithms in the rank space for B-FGGs. Within the framework, we apply CPD on the factors of a B-FGG and then construct a new B-FGG defined in the rank space by marginalizing all the state nodes. Inference with the new B-FGG has the same result and a lower time complexity if the rank size is smaller than the state size.
We conduct experiments in unsupervised PCFG parsing and HMM language modeling. For PCFG induction, we manage to use 20 times more hidden states than Yang et al. (2021b), obtaining much better unsupervised parsing performance. For HMM language modeling, we achieve lower perplexity and lower inference complexity than Chiu et al. (2021).
2.1 Factor graph grammar
Factor graphs are fixed-sized and thus incapable of modeling substructures that repeat a variable number of times. Chiang and Riley (2020) propose factor graph grammars (FGGs) to overcome this limitation, which are expressive enough to subsume HMMs and PCFGs. The main purpose of introducing FGGs in this work is to facilitate more intuitive presentation of our method, and to enable generalization beyond HMMs and PCFGs.
2.1.1 Basics
We display necessary notations and concepts of FGGs (Chiang and Riley, 2020, Def. 1,2,5,6,8).
Definition 1. A hypergraph is a tuple  $(V,E,att,\mathsf{lab}^V,\mathsf{lab}^E)$  where
-  $V$  and  $E$  are finite set of nodes and hyperedges.
-  $att: E \to V^{\star}$  maps each hyperedge to zero or more (not necessarily distinct) endpoint nodes.
-  $lab^V: V \to L^V$  assigns labels to nodes.
-  $lab^{E}: E \to L^{E}$  assigns labels to edges.
Definition 2. A factor graph is a hypergraph with mappings  $\Omega$  and  $F$  where
-  $\Omega$  maps node labels to sets of possible values.  $\Omega(v) \triangleq \Omega(lab^V(v))$ .
-  $F$  maps edge labels to functions.  $F(e) \triangleq F(lab^{E}(e))$  is of type  $\Omega(v_{1}) \times \dots \times \Omega(v_{k})$  where  $att(e) = v_{1} \cdots v_{k}$ .
In the terminology of factor graphs, a node  $v$  with its domain  $\Omega(v)$  is a variable, and an hyperedge  $e$  with  $F(e)$  is a factor. We typically use  $\mathbf{T}, \mathbf{N}, \mathbf{O}$  to denote hidden state, nonterminal state and observation variables for HMMs and PCFGs.
Definition 3. A hypergraph fragment is a tuple  $(V,E,att,\mathsf{lab}^V,\mathsf{lab}^E,\mathsf{ext})$  where
-  $(V, E, \text{att}, \text{lab}^V, \text{lab}^E)$  is a hypergraph.
-  $ext \in V^{\star}$  is a set of zero or more external nodes and each of which can be seen as a connecting point of this hypergraph fragment with another fragment.
Definition 4. A hyperedge replacement graph grammar (HRG) (Drewes et al., 1997) is a tuple  $(N, T, P, S)$  where
-  $N, T \subset L^{E}$  is finite set of nonterminal and terminal symbols.  $N \cap T = \emptyset$ .
-  $P$  is a finite set of rules  $(X \to R)$  where  $X \in N$  and  $R$  is a hypergraph fragment with edge labels in  $N \cup T^1$ .
-  $S \in  N$  is the start symbol.
Definition 5. A HRG with mapping  $\Omega, F$  (Def. 2) is referred to as an FGG. In particular,  $F$  is defined on terminal edge labels  $T$  only.
Notations.
-  $\widehat{\mathbf{N}}$ : variable node.  $\widehat{\mathbf{N}}$ : external node.
-  $\bigcirc - \boxed{\mathbf{X}_{\mathbf{e}}}$ : hyperedge  $e$  with label  $X \in N$ .
- indicates zero or more endpoint nodes.
<sup>1</sup>Note that, for the lhs of  $P$ , Chiang and Riley (2020) also draw their endpoint nodes using external node notations. We follow this practice.
Figure 1: FGG representations of (a) HMMs and (b) PCFGs. Examples come from Chiang and Riley (2020).
$F(e)$  : factor  $F(e)$
Fig. 1 illustrates HGG representations of HMM and PCFG.
Generative story. An FGG starts with  $\boxed{\mathbf{S}}$ , repeatedly selects  $-\boxed{\mathbf{X}_{\mathrm{e}}}$  and uses rule  $X \to R$  from  $P$  to replace  $e$  with  $R$ , until no  $-\boxed{\mathbf{X}_{\mathrm{e}}}$  exists.
2.1.2 Conjunction
The conjunction operation (Chiang and Riley, 2020, Sec. 4) allows modularizing an FGG into two parts, one defining the model and the other defining a query. In this paper, we only consider querying the observed sentence  $w_{0}, \dots, w_{n-1}$ , which is exemplified by the red part of Fig. 1. We sometimes omit the red part without further elaboration.
2.1.3 Inference
Denote  $\xi$  as an assignment of all variables,  $\Xi_D$  as the set of all assignments of factor graph  $D$ , and  $\mathcal{D}(G)$  as the set of all derivations of an FGG  $G$ , i.e., all factor graphs generated by  $G$ . An FGG  $G$  assigns a score  $w_G(D, \xi)$  to each  $D \in \mathcal{D}(G)$  along with each  $\xi \in \Xi_D$ . A factor graph  $D \in \mathcal{D}(G)$  assigns a score  $w_D(\xi)$  to each  $\xi \in \Xi_D$ :
$$
w _ {D} (\xi) = \prod_ {e \in D} F (e) (\xi (v _ {1}), \dots , \xi (v _ {k})) \quad (1)
$$
with  $\text{att}(e) = v_1 \cdots v_k$ . Notably,  $w_D(\xi) \triangleq w_G(D, \xi)$ . The inference problem is to compute the sum-product of  $G$ :
$$
Z _ {G} = \sum_ {D \in \mathcal {D} (G)} \sum_ {\xi \in \Xi_ {D}} w _ {G} (D, \xi) \tag {2}
$$
To obtain  $Z_{G}$ , the key difficulty is in the marginalization over all derivations, since  $\sum_{\xi \in \Xi_D} w_D(\xi)$  can be obtained by running standard variable elimination (VE) on factor graph  $D$ . To tackle this, Chiang and Riley (2020, Thm. 15) propose an extended VE. For each  $X \in N$ ,  $\xi \in \Xi_X^2$ , define  $P^X$  as all rules in  $P$  with left-hand side  $X$ , and then define:
$$
\psi_ {X} (\xi) = \sum_ {(X \rightarrow R) \in P ^ {X}} \tau_ {R} (\xi). \tag {3}
$$
for each  $\mathrm{rhs} R = (V, E_N \cup E_T, \text{att}, \text{lab}^V, \text{lab}^E, \text{ext})$ , where  $E_N, E_T$  consist of nonterminal/terminal-labeled edges only, and  $\tau_R(\xi)$  is given by:
$$
\begin{array}{l} \tau_ {R} (\xi) = \sum_ {\substack {\xi^ {\prime} \in \Xi_ {R} \\ \xi^ {\prime} (e x t) = \xi}} \prod_ {e \in E _ {T}} F (e) \left(\xi^ {\prime} (\operatorname {att}(e))\right) \tag{4} \\ \prod_ {e \in E _ {N}} \psi_ {l a b ^ {E} (e)} \left(\xi^ {\prime} (\mathrm {a t t} (e))\right) \\ \end{array}
$$
This defines a recursive formula for computing  $\psi_S$ , i.e.,  $Z_G$ . Next, we will show how Eq. 3-4 recover the well-known inside algorithm.
Example: the inside algorithm. Consider  $\pi_6$  in Fig.1(b). All possible fragments  $R$  (rhs of  $\pi_6$ ) differs in the value of  $k$ , i.e., the splitting point, so we use  $R_k$  to distinguish them. Then Eq. 3 becomes:
$$
\psi_ {X _ {i, k}} (\xi) = \sum_ {i <   k <   j} \tau_ {R _ {k}} (\xi) \tag {5}
$$
$^2 \Xi_X$  is defined as the set of assignments to the endpoints of an edge  $e$  labeled  $\mathbf{X}$ , so  $\Xi_{X} = \Omega (\ell_{1}) \times \dots \times \Omega (\ell_{k})$  where  $att(e) = v_{1} \cdots v_{k}, lab^{V}(v_{i}) = \ell_{i}$ .
Putting values into Eq. 4:
$$
\tau_ {R _ {k}} (\xi) = \sum_ {n _ {2}, n _ {3}} p \left(\xi , n _ {2}, n _ {3}\right) \psi_ {X _ {i, k}} \left(n _ {2}\right) \psi_ {X _ {k, j}} \left(n _ {3}\right) \tag {6}
$$
where  $p$  denotes FGG rule probability  $p(\mathbf{N}_1 \to \mathbf{N}_2\mathbf{N}_3)$ . It is easy to see that  $\psi_{X_{i,k}}$  is exactly the inside score of span  $[i, k)$ , and Eq. 5-6 recovers the recursive formula of the inside algorithm.
Remark. Eq. 4 can be viewed as unidirectional (from  $e \in E_N$  to external nodes) belief propagation (BP) in the factor graph fragment  $R$ , where the incoming message is  $\psi_{lab^E}(e)$  for  $e \in E_N$ , and the outcome of Eq. 4 can be viewed as the message passed to the external nodes. The time complexity of message updates grows exponentially with the number of variables in the factors. Therefore, to decrease inference complexity, one may decompose large factors into smaller factors connected by new nodes, as shown in the next subsection.
2.2 Tensor rank decomposition on factors
Consider a factor  $F(e)$  (Def. 2), it can be represented as an order-  $k$  tensor in  $\mathbb{R}^{N_1\times \dots \times N_k}$  where  $N_{i}\triangleq |\Omega (v_{i})|$ . We can use tensor rank decomposition (aka. CPD) to decompose  $F(e)$  into a weighted sum of outer products of vectors:
$$
F (e) = \sum_ {q = 1} ^ {r} \lambda_ {q} \mathbf {w} _ {e _ {1}} ^ {q} \otimes \mathbf {w} _ {e _ {2}} ^ {q} \otimes \dots \otimes \mathbf {w} _ {e _ {k}} ^ {q}
$$
where  $r$  is the rank size;  $\mathbf{w}_{e_k}^q\in \mathbb{R}^{N_k}$ $\otimes$  is outer product;  $\lambda_{q}$  is weight, which can be absorbed into  $\{\mathbf{w}_{e_k}^q\}$  and we omit it throughout the paper.
Dupty and Lee (2020, Sec. 4.1) show that BP can be written in the following matrix form when applying CPD on factors:
$$
\mathbf {m} _ {e i} = \mathbf {W} _ {e i} ^ {T} \left(\odot_ {j \in N (e) \backslash i} \mathbf {W} _ {e j} \mathbf {n} _ {j e}\right) \tag {7}
$$
$$
\mathbf {n} _ {i e} = \odot_ {c \in N (i) \backslash e} \mathbf {m} _ {c i} \tag {8}
$$
where  $\mathbf{m}_{ei} \in \mathbb{R}^{N_i}$  is factor-to-node message;  $\mathbf{n}_{ie} \in \mathbb{R}^{N_i}$  is node-to-factor message;  $N(\cdot)$  indicates neighborhood;  $\mathbf{W}_{e_j} = [\mathbf{w}_{e_j}^1, \dots, \mathbf{w}_{e_j}^r]^T \in \mathbb{R}^{r \times m}$ ;  $\odot$  is element-wise product. We remark that this amounts to replacing the large factor  $F(e)$  with smaller factors  $\{F(e_i)\}$  connected by a new node  $\mathbf{R}$  that represents rank, where each  $F(e_i)$  can be represented as  $\mathbf{W}_{e_i}$ . Fig. 2 illustrates this intuition. We refer to  $\mathbf{R}$  as rank nodes and others as state nodes thereafter.
Figure 2: Using CPD to decompose a factor can be seen as adding a new node.
(b)
Figure 3: Representations of the  $\mathbf{r}\mathbf{h}\mathbf{s}$  of  $\pi_6$  (Fig. 1) after decomposition. (a): TD-PCFG (Cohen et al., 2013; Yang et al., 2021b). (b): LPCFG (Chiu et al., 2021).
3 Low-rank structured inference
In this section, we recover the accelerated inside algorithms of TD-PCFG (Cohen et al., 2013; Yang et al., 2021b) and LPCFG (Chiu et al., 2021) in an intuitive and unifying manner using the FGG notations. The accelerated forward algorithm of LHMM (Chiu et al., 2021) can be derived similarly.
Denote  $\mathsf{T} \in \mathbb{R}^{m \times m \times m}$  as the tensor representation of  $p(\mathbf{N}_1 \to \mathbf{N}_2\mathbf{N}_3)$ , and  $\alpha_{i,j} \in \mathbb{R}^m$  as the inside score of span  $[i,j)$ . Cohen et al. (2013) and Yang et al. (2021b) use CPD to decompose  $\mathsf{T}$ , i.e., let  $\mathsf{T} = \sum_{q=1}^{r} \mathbf{u}_q \otimes \mathbf{v}_q \otimes \mathbf{w}_q$  where  $\mathbf{u}_q, \mathbf{v}_q, \mathbf{w}_q \in \mathbb{R}^m$ . Denote  $\mathbf{U}, \mathbf{V}, \mathbf{W} \in \mathbb{R}^{r \times m}$  as the resulting matrices of stacking all  $\mathbf{u}_q, \mathbf{v}_q, \mathbf{w}_q$ , Cohen et al. (2013) derived the recursive form:
$$
\begin{array}{l} \boldsymbol {\alpha} _ {i, j} = \sum_ {k = i + 1} ^ {j - 1} \mathbf {U} ^ {T} \left(\left(\mathbf {V} \boldsymbol {\alpha} _ {i, k}\right) \odot \left(\mathbf {W} \boldsymbol {\alpha} _ {k, j}\right)\right) (9) \\ = \mathbf {U} ^ {T} \sum_ {k = i + 1} ^ {j - 1} \left(\left(\mathbf {V} \boldsymbol {\alpha} _ {i, k}\right) \odot \left(\mathbf {W} \boldsymbol {\alpha} _ {k, j}\right)\right) (10) \\ \end{array}
$$
Eq. 9 can be derived automatically by combining Eq. 7 (or Fig. 3 (a)) and Eq. 5-6. Cohen et al. (2013) note that  $\mathbf{U}^{\mathbf{T}}$  can be extracted to the front of the summation (Eq. 10), and  $\mathbf{V}\alpha_{i,k},\mathbf{W}\alpha_{k,j}$  can be cached and reused, leading to further complexity reduction. The resulting inside algorithm time complexity is  $O(n^{3}r + n^{2}mr)$ .
Recently, Chiu et al. (2021) use low-rank matrix decomposition to accelerate PCFG inference. They first perform tensor matricization to flatten  $\mathsf{T}$  to  $\mathsf{T}' \in \mathbb{R}^{m \times m^2}$ , and then let  $\mathsf{T}' = \mathbf{U}^T\mathbf{V}$  where  $\mathbf{U} \in \mathbb{R}^{r \times m}$ ,  $\mathbf{V} \in \mathbb{R}^{r \times m^2}$ . By un-flattening  $\mathbf{V}$  to  $\mathbf{V}' \in \mathbb{R}^{r \times m \times m}$ , their accelerated inside algorithm has the following recursive form:
$$
\begin{array}{l} \boldsymbol {\alpha} _ {i, j} = \sum_ {k = i + 1} ^ {j - 1} \mathbf {U} ^ {T} \left(\mathbf {V} ^ {\prime} \cdot \boldsymbol {\alpha} _ {k, j} \cdot \boldsymbol {\alpha} _ {i, k}\right) (11) \\ = \mathbf {U} ^ {T} \sum_ {k = i + 1} ^ {j - 1} \left(\mathbf {V} ^ {\prime} \cdot \boldsymbol {\alpha} _ {k, j} \cdot \boldsymbol {\alpha} _ {i, k}\right) (12) \\ \end{array}
$$
Eq. 11 can be derived by combining Fig. 3 (b) and Eq. 5-6. The resulting inside time complexity is  $O(n^{3}m^{2}r + n^{2}mr)$ , which is higher than that of TD-PCFG.
When learning a PCFG and a HMM, there is no need to first learn  $\mathsf{T}$  and then perform decomposition on  $\mathsf{T}$ . Instead, one can learn the decomposed matrices (e.g.,  $\mathbf{U},\mathbf{V}$ ) to learn  $\mathsf{T}$  implicitly. During inference, one can follow Eq. 10 or 12 without the need to reconstruct  $\mathsf{T}$ .
Validity of probability. The remaining problem is to ensure that  $\mathsf{T}$  is a valid probability tensor (i.e., being nonnegative and properly normalized) when learning it implicitly. Yang et al. (2021b) essentially transform Fig. 3(a) into a Bayesian network, adding directed arrows  $\mathbf{N}_1 \rightarrow \mathbf{R}, \mathbf{R} \rightarrow \mathbf{N}_2, \mathbf{R} \rightarrow \mathbf{N}_3$ . This is equivalent to requiring that  $\mathbf{V}, \mathbf{W}$  are nonnegative and column-wise normalized and  $\mathbf{U}$  is nonnegative and row-wise normalized, as described in Yang et al. (2021b, Thm. 1). One can apply the Softmax re-parameterization to enforce such requirement, which is more convenient in end-to-end learning. Chiu et al. (2021) replace the local normalization of Yang et al. (2021b) with global normalization, and we refer readers to their paper for more details. We adopt the strategy of Yang et al. (2021b) in this work.
4 Rank-space modeling and inference
4.1 Rank-space inference with B-FGGs
Interestingly, when applying CPD on factors and if the rank size is smaller than the state size, we can even obtain better inference time complexities for a subset of FGGs which we refer to as B-FGGs.
We call a hyperedge a B-edge if its head contains exactly one node. B-graphs (Gallo et al., 1993) are a subset of directed hypergraphs whose hyperedges
are all B-edges. Many dynamic programming algorithms can be formulated through B-graphs (Klein and Manning, 2001; Huang, 2008; Azuma et al., 2017; Chiu et al., 2021; Fu and Lapata, 2021), including the inference algorithms of many structured models, e.g., HMMs, Hidden Semi-Markov Models (HSMMs), and PCFGs. We follow the concept of B-graphs to define B-FGGs.
Definition 6. A hypergraph fragment is a B-hypergraph fragment iff. there is exactly one external node and there is no nonterminal-labeled hyperedge connecting to it. An FGG is a B-FGG iff. all rhs of its rules are B-hypergraph fragments.
It is easy to see that the aforementioned models are subsumed by B-FGGs. We can design a family of accelerated inference algorithms for B-FGGs based on the following strategy. (1) If there are multiple factors within a hypergraph fragment, merge them into a single factor. Then apply CPD on the single factor, thereby introducing rank nodes. (2) Find repeated substructures that take rank nodes as external nodes. Marginalize all state nodes to derive new rules. (3) Design new inference algorithms that can be carried out in the rank space based on the general-purpose FGG inference algorithm and the derived new rules. We give two examples, the rank-space inside algorithm and the rank-space forward algorithm, in the following two subsections to help readers understand this strategy.
4.2 The rank-space inside algorithm
Consider an B-FGG  $G$  shown in Fig.1(b) and replace the  $\mathrm{rs}$  of  $\pi_6$  with Fig. 3(a), i.e., we use CPD to decompose binary rule probability tensor. Besides  $\mathbf{U},\mathbf{V},\mathbf{W}\in \mathbb{R}^{r\times m}$  defined in Sec. 3, we define the start rule probability vector as  $\mathbf{s}\in \mathbb{R}^{m\times 1}$  and the unary rule probability matrix as  $\mathbf{E}\in \mathbb{R}^{o\times m}$  where  $o$  is the vocabulary size.
Fig. 4(a) is an example (partial) factor graph  $D$  generated by  $G$ . We highlight substructures of interest with dashed rectangles. Each substructure consists of a node  $\mathbf{N}$  and two factors connecting to it.  $\mathbf{N}$  is an external node connecting two hypergraph fragments which contain the two factors respectively. For each substructure, we can marginalize the state node  $\mathbf{N}$  out, merging the two factors into a single one. After marginalizing all state nodes, we obtain a (partial) factor graph  $D'$  shown in the right of Fig. 4(a) where  $\mathbf{H} = \mathbf{V}\mathbf{U}^T$ ,  $\mathbf{I} = \mathbf{W}\mathbf{U}^T$ ,  $\mathbf{J} = \mathbf{V}\mathbf{E}^T$ ,  $\mathbf{K} = \mathbf{W}\mathbf{E}^T$ ,  $\mathbf{L} = (\mathbf{U}\mathbf{s})^T$ . We denote this transformation as  $\mathcal{M}(D) = D'$ . It is worth men
(b)
Figure 4: (a): illustration of marginalizing state nodes N. (b): rule set of the new FGG.  $\pi_1$  can be applied when  $k \neq i + 1$  and  $k + 1 \neq j$ ;  $\pi_2$  and  $\pi_3$  can be applied when  $i \neq j - 1$ ;  $\pi_4$  can be applied when  $j = i + 2$ .
tioning that  $\mathbf{H},\mathbf{I},\mathbf{J},\mathbf{K},\mathbf{L}$  are computed only once and then reused multiple times during inference, which is the key to reduce the time complexity.
Then we define a new B-FGG  $G'$  with rules shown in Fig 4(b). It is easy to verify that for each  $D \in \mathcal{D}(G)$ , we have  $\mathcal{M}(D) \in \mathcal{D}(G')$ , and vice versa. Moreover, we have:
$$
\sum_ {\xi \in \Xi_ {D}} w _ {G} (D, \xi) = \sum_ {\xi \in \Xi_ {\mathcal {M} (D)}} w _ {G ^ {\prime}} (\mathcal {M} (D), \xi)
$$
because marginalizing hidden variables does not affect the result of sum-product inference. Therefore,  $Z_{G} = Z_{G^{\prime}}$  (Eq. 2).
We can easily derive the inference (inside) algorithm of  $G'$  by following Eq. 3-4 and Fig. 4(b)  $^3$ . Let  $\alpha_{i,j} \in \mathbb{R}^r$  denote the rank-space inside score for span  $[i,j)$ . When  $j > i + 2$ :
$$
\begin{array}{l} \boldsymbol {\alpha} _ {i, j} = \overbrace {\sum_ {i + 1 <   k <   j - 1} (\mathbf {H} \boldsymbol {\alpha} _ {i , k} \odot \mathbf {I} \boldsymbol {\alpha} _ {k , j})} ^ {\text {f r o m} \pi_ {1} \text {o f F i g . 4 (b)}} \\ + \underbrace {\mathbf {J} _ {: , w _ {i}} \odot \mathbf {I} \boldsymbol {\alpha} _ {i + 1 , j}} _ {\mathrm {f r o m} \pi_ {2}} + \underbrace {\mathbf {H} \boldsymbol {\alpha} _ {i , j - 1} \odot \mathbf {K} _ {: , w _ {j - 1}}} _ {\mathrm {f r o m} \pi_ {3}} \\ \end{array}
$$
and when  $j = i + 2$ ,  $\pmb{\alpha}_{i,j} = \mathbf{J}_{:,w_i}\odot \mathbf{K}_{:,w_{i + 1}}$  (from  $\pi_4$ ).  $w_{j}$  is the index of the  $j$ -th word of the input sentence in the vocabulary;  $\mathbf{A}_{:,j}$  indicates the  $j$ -th column of  $\mathbf{A}$ .
We note that, similar to Cohen et al. (2013), we can cache  $\mathbf{H}\alpha_{i,k},\mathbf{I}\alpha_{k,j}$  and reuse them to further
accelerate inference  $^{4}$ . Denote  $\alpha_{i,j}^{L},\alpha_{i,j}^{R}\in \mathbb{R}^{r}$  as the inside scores of span  $[i,j)$  serving as a left/right child of a larger span. Then we have:
$$
\pmb {\alpha} _ {i, i + 1} ^ {L} = \mathbf {K} _ {:, i} \qquad \pmb {\alpha} _ {i, i + 1} ^ {R} = \mathbf {J} _ {:, i}
$$