SlowGuess/NAACL_2022_OCR huggingface dataset:
$$
\boldsymbol {\alpha} _ {i, j} ^ {L} = \mathbf {H} \boldsymbol {\alpha} _ {i, j} \quad \alpha_ {i, j} ^ {R} = \mathbf {I} \boldsymbol {\alpha} _ {i, j}
$$
$$
\boldsymbol {\alpha} _ {i, j} = \sum_ {i <   k <   j} \left(\boldsymbol {\alpha} _ {i, k} ^ {L} \odot \boldsymbol {\alpha} _ {k, j} ^ {R}\right)
$$
and finally,  $Z_{G'} = \mathbf{L}\alpha_{0,n}$ . We minimize - log  $Z_{G'}$  using mini-batch gradient descent for unsupervised learning. The resulting inference complexity is  $O(n^{3}r + n^{2}r^{2})^{5}$ , which is lower than  $O(n^{3}r + n^{2}mr)$  of TD-PCFG when  $r < m$ , enabling the use of a large state space for PCFGs in the low-rank setting.
The key difference between the rank-space inference and the original state-space inference is that they follow different variable elimination orders. The former marginalizes all state nodes before performing inference and marginalizes rank nodes from bottom up during inference; whereas the later marginalizes both state and rank nodes alternately from bottom up during inference.
Parsing. Low-rank inference does not support the Viterbi semiring<sup>6</sup>, inhibiting the use of CYK
4In fact, this is a typical application of the unfold-refold transformation (Eisner and Blatz, 2007; Vieira et al., 2021).
This does not take into account the one-time cost of computing  $\mathbf{H},\mathbf{I},\mathbf{J},\mathbf{K}$  before inference.
The Viterbi semiring is also known as the max-product semiring. Chiu et al. (2021, Appd. C) and Yang et al. (2021b, Sec. 6) have discussed this issue.
$^3\pi_6$  is used for generating sentences of length 1, we do not consider this in the following derivation of the inside algorithm to reduce clutter.
(b)
Figure 5: (a): merge the two factors into a single one, and apply CPD on the resulting factor. (b): factor graph of a HMM for sentences of length 3. (c): the resulting factor graph after marginalizing the state nodes.
decoding. Therefore, we follow Yang et al. (2021b) to use Minimum Bayes-Risk decoding (Goodman, 1996). Specifically, we estimate the span marginals using auto-differentiation (Eisner, 2016; Rush, 2020), which has the same complexity as the inside algorithm. Then we use the CYK algorithm to find the final parse with the maximum number of expected spans in  $O(n^3)$  time, similar to Smith and Eisner (2006).
Implementation. The implementation of the inside algorithm greatly influences the actual running speed. First,  $O(n^{2})$  out of  $O(n^{3})$  can be computed in parallel using parallel parsing techniques (Yi et al., 2011; Canny et al., 2013; Zhang et al., 2020; Rush, 2020). In this work, we adapt the efficient implementation of Zhang et al. (2020) for fast inside computation. Second, we adopt the log-einsum-exp trick (Peharz et al., 2020) to avoid expensive log-sum-exp operations on high-dimensional vectors, which reduces both GPU memory usage and total running time.
4.3 The rank-space forward algorithm
Consider an B-FGG  $G$  shown in Fig. 1 (a). We replace the  $\mathbf{r}$ s of  $\pi_2$  by the hypergraph fragment in the right of Fig. 5(a), i.e., we merge the factor  $p(\mathbf{T}_2\mid \mathbf{T}_1)$  and  $p(\mathbf{O}_3\mid \mathbf{T}_2)$  into a single factor, which can be represented as  $\mathsf{T}\in \mathbb{R}^{m\times m\times o}$  and can be decomposed into three matrices  $\mathbf{U},\mathbf{V}\in \mathbb{R}^{r\times m},\mathbf{W}\in \mathbb{R}^{r\times o}$  via CPD, where  $m / o / r$  is the state/vocabulary/rank size. Fig. 5(b) gives an example factor graph of HMMs with sentences of length 3. Similar to previous subsection, we marginalize state nodes  $\mathbf{T}$  to construct a new B-FGG  $G^{\prime}$ . The rule set of  $G^{\prime}$  can be obtained by replacing all variable nodes  $\mathbf{T}$  with  $\mathbf{R}$  and modifying all factors accordingly, as one can easily infer from Fig. 5(c). Inference with  $G^{\prime}$  simply coincides with the forward algorithm, which has a  $O(nr^2)$  time complexity and is lower than  $O(nmr)$  of LHMM (Chiu et al., 2021) when  $r < m$ .
4.4 Neural parameterization
We use neural networks to produce probabilities for all factors, which has been shown to benefit learning and unsupervised induction of syntactic structures (Jiang et al., 2016; He et al., 2018; Kim et al., 2019; Han et al., 2019; Jin et al., 2019; Zhu et al., 2020; Yang et al., 2020, 2021b; Zhao and Titov, 2020; Zhang et al., 2021; Chiu and Rush, 2020; Chiu et al., 2021; Kim, 2021). We use the neural parameterization of Yang et al. (2021b) with slight modifications. We show the details in Appd. A and Appd. B.
5.1 Unsupervised parsing with PCFGs
Setting. We evaluate our model on Penn Treebank (PTB) (Marcus et al., 1994). Our implementation is based on the open-sourced code of Yang et al. (2021b)<sup>7</sup> and we use the same setting as theirs. For all experiments, we set the ratio of nonterminal number to the preterminal number to  $1:2^{8}$  which is the common practise. We set the rank size to 1000. We show other details in Appd. C and D.
Main result. Table 1 shows the result on PTB. Among previous unsupervised PCFG models, TN-PCFG (Yang et al., 2021b) uses the largest number of states (500 perterminals and 250 nonterminals). Our model is able to use much more states thanks to our new inside algorithm with lower time complexity, surpassing all previous PCFG-based models by a large margin and achieving a new state-of-the-art in unsupervised constituency parsing in terms of sentence-level F1 score on PTB.
$^{7}$ github.com/sustcsonglin/TN-PCFG
Although we did not explicitly distinguish between nonterminal and preterminal symbols previously, in our implementation, we follow Kim et al. (2019) to make such distinction, in which terminal words can only be generated by preterminal symbols, and binary rules can only be invoked by nonterminal symbols.
Model	S-F1
N-PCFG (Kim et al., 2019)	50.8
C-PCFG (Kim et al., 2019)	55.2
NL-PCFG (Zhu et al., 2020)	55.3
TN-PCFG (Yang et al., 2021b)	57.7
NBL-PCFG (Yang et al., 2021a)	60.4
Ours with 9000 PTs and 4500 NTs	64.1
For reference
Constituency test (Cao et al., 2020)	62.8
S-DIORA (Drozdov et al., 2020)	57.6
StructFormer (Shen et al., 2021)	54.0
DIORA+span constraint (Xu et al., 2021)	61.2
(b)
Figure 6: The change of F1 scores and perplexities with the change of number of perterminal symbols.
Ablation study. Fig. 6 shows the change of the sentence-level F1 scores and perplexity with the change of the number of preterminals. As we can see, when increasing the state, the perplexity tends to decrease while the F1 score tends to increase, validating the effectiveness of using large state spaces for neural PCFG induction.
5.2 HMM language modeling
Setting. We conduct the language modeling experiment also on PTB. Our implementation is based
Table 1: Results on PTB. S-F1: sentence-level F1. PTs: preterminals. NTs: nonterminals.
Model	Val	Test
VL-HMM (215 states, Brown)	125.0	116.0
VL-HMM (214 states, Brown)†	136	-
VL-HMM (214 states, Uniform)†	146	-
LHMM (214 states)	141.4	131.8
Ours (214 states)	135.6	127.0
Ours (215 states)	137.0	126.4
For reference
HMM+RNN (Buys et al., 2018)	142.3	-
AWD-LSTM (Merit et al., 2018)	60.0	57.3
Table 2: Resulting perplexity on PTB validate set and test set. VL-HMM: (Chiu and Rush, 2020). LHMM: (Chiu et al., 2021). † denotes results reported by ablation study of Chiu and Rush (2020).
#States	Val	Test
212	149.8	139.1
213	143.8	133.4
214	149.5	137.4
215	141.1	131.1
Table 3: Perplexity with varying numbers of states. Following Chiu et al. (2021), we fix the rank to 2048 for faster ablation studies.
on the open-sourced code of Chiu et al. (2021) $^{9}$ . We set the rank size to 4096. See Appd. C and D for more details.
Main result. Table 2 shows the perplexity on the PTB validation and test sets. As discussed earlier, VL-HMM (Chiu and Rush, 2020) imposes strong sparsity constraint to decrease the time complexity of the forward algorithm and requires preclustering of terminal symbols. Specifically, VL-HMM uses Brown clustering (Brown et al., 1992), introducing external information to improve performance. Replacing Brown clustering with uniform clustering leads to a 10 point increase in perplexity on the PTB validation set. LHMM (Chiu et al., 2021) and our model only impose low-rank constraint without using any external information and are thus more comparable. Our method outperforms LHMM by 4.8 point when using the same state number (i.e.,  $2^{14}$ ), and it can use more states thanks to our lower inference time complexity.
Ablation study. As we can see in Table 3, the perplexity tends to decrease when increasing the state number, validating the effectiveness of using more states for neural HMM language modeling.
$^{9}$ github.com/justinchiu/low-rank-models
Discussion. It is interesting to note that our HMM model is roughly equivalent to another HMM with interchanged rank and state sizes as can be seen in Fig.5(c). To verify this equivalence, we run LHMM in the original state space with 2048 states and rank  $2^{15}$ . The resulting perplexity is 133.49 on average on the PTB test set, which is worse than that of ours (126.4). We leave further experimentation and analyses of this discrepancy for future work.
Tensor and matrix decomposition have been used to decrease time and space complexities of probabilistic inference algorithms. Siddiqi et al. (2010) propose a reduced-rank HMM whereby the forward algorithm can be carried out in the rank space, which is similar to our model, but our method is more general. Cohen and Collins (2012); Cohen et al. (2013) use CPD for fast (latent-variable) PCFG parsing, but they do not leverage CPD for fast learning and they need to actually perform CPD on existing probability tensors. Rabusseau et al. (2016) use low-rank approximation method to learn weighted tree automata, which subsumes PCFGs and latent-variable PCFGs. Our method can subsume more models. Yang et al. (2021b,a) propose CPD-based neural parameterizations for (lexicalized) PCFGs. Yang et al. (2021b) aim at scaling PCFG inference. We achieve better time complexity than theirs and hence can use much more hidden states. Yang et al. (2021a) aims to decrease the complexity of lexicalized PCFG parsing, which can also be described within our framework. Chiu et al. (2021) use low-rank matrix decomposition, which can be viewed as CPD on order-2 tensors, to accelerate inference on chain and tree structure models including HMMs and PCFGs. However, their method is only efficient when the parameter tensors are of order 2, e.g., in HMMs and HSMMs. Our method leverages full CPD, thus enabling efficient inference with higher-order factors, e.g., in PCFGs. Our method can be applied to all models considered by Chiu et al. (2021), performing inference in the rank-space with lower complexities.
Besides HMMs and PCFGs, Wrigley et al. (2017) propose an efficient sampling-based junction-tree algorithm using CPD to decompose high-order factors. Dupty and Lee (2020) also use CPD to decompose high-order factors for fast belief propagation. Yang and Tu (2022) use CPD to
decompose second-order factors in semantic dependency parsing to accelerate second-order parsing with mean-field inference. Besides CPD, Ducamp et al. (2020) use tensor train decomposition for fast and scalable message passing in Bayesian networks. Bonnevie and Schmidt (2021) leverage matrix product states (i.e., tensor trains) for scalable discrete probabilistic inference. Miller et al. (2021) leverage tensor networks for fast sequential probabilistic inference.
In this work, we leveraged tensor rank decomposition (CPD) for low-rank scaling of structured inference. We showed that CPD amounts to decomposing a large factor into several smaller factors connected by a new rank node, and gave a unifying perspective towards previous low-rank structured models (Yang et al., 2021b; Chiu et al., 2021). We also presented a novel framework to design a family of rank-space inference algorithms for B-FGGs, a subset of FGGs which subsume most structured models of interest to the NLP community. We have shown the application of our method in scaling PCFG and HMM inference, and experiments on unsupervised parsing and language modeling validate the effectiveness of using large state spaces facilitated by our method.
We believe our framework can be applied to many other models which have high inference time complexity and are subsumed by B-FGGs, including lexicalized PCFGs, quasi-synchronous context-free grammars (QCFGs), etc. A direct application of our method is to decrease the inference complexity of the neural QCFG (Kim, 2021).
Sanjeev Arora, Nadav Cohen, and Elad Hazan. 2018. On the optimization of deep networks: Implicit acceleration by overparameterization. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 244-253. PMLR.
Ai Azuma, Masashi Shimbo, and Yuji Matsumoto. 2017. An algebraic formalization of forward
and forward-backward algorithms. CoRR, abs/1702.06941.
Rasmus Bonnevie and Mikkel N. Schmidt. 2021. Matrix product states for inference in discrete probabilistic models. Journal of Machine Learning Research, 22(187):1-48.
Peter F. Brown, Vincent J. Della Pietra, Peter V. deSouza, Jenifer C. Lai, and Robert L. Mercer. 1992. Class-based  $n$ -gram models of natural language. Computational Linguistics, 18(4):467-480.
Rares-Darius Buhai, Yoni Halpern, Yoon Kim, Andrej Risteski, and David A. Sontag. 2020. Empirical study of the benefits of overparameterization in learning latent variable models. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 1211-1219. PMLR.
Jan Buys, Yonatan Bisk, and Yejin Choi. 2018. Bridging hmms and rnns through architectural transformations.
John Canny, David Hall, and Dan Klein. 2013. A multiteraflop constituency parser using GPUs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1898-1907, Seattle, Washington, USA. Association for Computational Linguistics.
Steven Cao, Nikita Kitaev, and Dan Klein. 2020. Unsupervised parsing via constituency tests. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4798-4808, Online. Association for Computational Linguistics.
David Chiang and Darcey Riley. 2020. Factor graph grammars. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Justin Chiu, Yuntian Deng, and Alexander Rush. 2021. Low-rank constraints for fast inference in structured models. Advances in Neural Information Processing Systems, 34.
Justin Chiu and Alexander Rush. 2020. Scaling hidden Markov language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1341-1349, Online. Association for Computational Linguistics.
Shay B. Cohen and Michael Collins. 2012. Tensor decomposition for fast parsing with latent-variable pcfgs. In Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of a meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States, pages 2528-2536.
Shay B. Cohen, Giorgio Satta, and Michael Collins. 2013. Approximate PCFG parsing using tensor decomposition. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 487-496, Atlanta, Georgia. Association for Computational Linguistics.
Antoine Dedieu, Nishad Gothoskar, Scott Swingle, Wolfgang Lehrach, Miguel Lázaro-Gredilla, and Dileep George. 2019. Learning higher-order sequential structure with cloned hmms. CoRR, abs/1905.00507.
Frank Drewes, Hans-Jörg Kreowski, and Annegret Habel. 1997. Hyperedge replacement, graph grammars. In Grzegorz Rozenberg, editor, Handbook of Graph Grammars and Computing by Graph Transformations, Volume 1: Foundations, pages 95-162. World Scientific.
Andrew Drozdov, Subendhu Rongali, Yi-Pei Chen, Tim O'Gorman, Mohit Iyyer, and Andrew McCallum. 2020. Unsupervised parsing with S-DIORA: Single tree encoding for deep inside-outside recursive autoencoders. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4832-4845, Online. Association for Computational Linguistics.
Simon S. Du, Xiyu Zhai, Barnabás Póczos, and Aarti Singh. 2019. Gradient descent provably optimizes over-parameterized neural networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Gaspard Ducamp, Philippe Bonnard, Anthony Nouy, and Pierre-Henri Wuillemin. 2020. An efficient low-rank tensors representation for algorithms in complex probabilistic graphical models. In International Conference on Probabilistic Graphical Models, PGM '2020, 23-25 September 2020, Aalborg, Hotel Comwell Rebild Bakker, Skørping, Denmark, volume 138 of Proceedings of Machine Learning Research, pages 173-184. PMLR.
Mohammed Haroon Dupty and Wee Sun Lee. 2020. Neuralizing efficient higher-order belief propagation. CoRR, abs/2010.09283.
Jason Eisner. 2016. Inside-outside and forward-backward algorithms are just backprop (tutorial paper). In Proceedings of the Workshop on Structured Prediction for NLP, pages 1-17, Austin, TX. Association for Computational Linguistics.
Jason Eisner and John Blatz. 2007. Program transformations for optimization of parsing algorithms and other weighted logic programs. In Proceedings of FG 2006: The 11th Conference on Formal Grammar, pages 45-85. CSLI Publications.
Yao Fu and Mirella Lapata. 2021. Scaling structured inference with randomization. CoRR, abs/2112.03638.
Giorgio Gallo, Giustino Longo, and Stefano Pallottino. 1993. Directed hypergraphs and applications. Discret. Appl. Math., 42(2):177-201.
Joshua Goodman. 1996. Parsing algorithms and metrics. In 34th Annual Meeting of the Association for Computational Linguistics, pages 177-183, Santa Cruz, California, USA. Association for Computational Linguistics.
Wenjuan Han, Yong Jiang, and Kewei Tu. 2019. Enhancing unsupervised generative dependency parser with contextual information. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5315-5325, Florence, Italy. Association for Computational Linguistics.
Junxian He, Graham Neubig, and Taylor Berg-Kirkpatrick. 2018. Unsupervised learning of syntactic structure with invertible neural projections. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1292-1302, Brussels, Belgium. Association for Computational Linguistics.
Liang Huang. 2008. Advanced dynamic programming in semiring and hypergraph frameworks. In *Coling 2008: Advanced Dynamic Programming in Computational Linguistics: Theory, Algorithms and Applications - Tutorial notes*, pages 1-18, Manchester, UK. Coling 2008 Organizing Committee.
Yong Jiang, Wenjuan Han, and Kewei Tu. 2016. Unsupervised neural dependency parsing. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 763-771, Austin, Texas. Association for Computational Linguistics.
Lifeng Jin, Finale Doshi-Velez, Timothy Miller, Lane Schwartz, and William Schuler. 2019. Unsupervised learning of PCFGs with normalizing flow. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2442-2452, Florence, Italy. Association for Computational Linguistics.
Yoon Kim. 2021. Sequence-to-sequence learning with latent neural grammars. Advances in Neural Information Processing Systems, 34.
Yoon Kim, Chris Dyer, and Alexander Rush. 2019. Compound probabilistic context-free grammars for grammar induction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2369-2385, Florence, Italy. Association for Computational Linguistics.
Dan Klein and Christopher D. Manning. 2001. Parsing and hypergraphs. In Proceedings of the Seventh International Workshop on Parsing Technologies (IWPT-2001), 17-19 October 2001, Beijing, China. Tsinghua University Press.
Mitchell Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. 1994. The Penn Treebank: Annotating predicate argument structure. In Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994.
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018. Regularizing and optimizing LSTM language models. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.
Jacob Miller, Guillaume Rabusseau, and John Terilla. 2021. Tensor networks for probabilistic sequence modeling. In The 24th International Conference on Artificial Intelligence and Statistics, AISTATS 2021, April 13-15, 2021, Virtual Event, volume 130 of Proceedings of Machine Learning Research, pages 3079-3087. PMLR.
Robert Peharz, Steven Lang, Antonio Vergari, Karl Stelzner, Alejandro Molina, Martin Trapp, Guy Van den Broeck, Kristian Kersting, and Zoubin Ghahramani. 2020. Einsum networks: Fast and scalable learning of tractable probabilistic circuits. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 7563-7574. PMLR.
Stephan Rabanser, Oleksandr Shchur, and Stephan Gunnemann. 2017. Introduction to tensor decompositions and their applications in machine learning. CoRR, abs/1711.10781.
Guillaume Rabusseau, Borja Balle, and Shay B. Cohen. 2016. Low-rank approximation of weighted tree automata. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, AISTATS 2016, Cadiz, Spain, May 9-11, 2016, volume 51 of JMLR Workshop and Conference Proceedings, pages 839-847. JMLR.org.
Alexander Rush. 2020. Torch-struct: Deep structured prediction library. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 335-342, Online. Association for Computational Linguistics.
Yikang Shen, Yi Tay, Che Zheng, Dara Bahri, Donald Metzler, and Aaron Courville. 2021. StructFormer: Joint unsupervised induction of dependency and constituency structure from masked language modeling. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7196-7209, Online. Association for Computational Linguistics.
Sajid M. Siddiqi, Byron Boots, and Geoffrey J. Gordon. 2010. Reduced-rank hidden markov models.