SlowGuess/NAACL_2022_OCR huggingface dataset:
In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2010, Chia Laguna Resort, Sardinia, Italy, May 13-15, 2010, volume 9 of JMLR Proceedings, pages 741-748. JMLR.org.
David A. Smith and Jason Eisner. 2006. Minimum risk annealing for training log-linear models. In Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions, pages 787-794, Sydney, Australia. Association for Computational Linguistics.
Tim Vieira, Ryan Cotterell, and Jason Eisner. 2021. Searching for more efficient dynamic programs. In *Findings of the Association for Computational Linguistics: EMNLP* 2021, pages 3812-3830, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Andrew Wrigley, Wee Sun Lee, and Nan Ye. 2017. Tensor belief propagation. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 3771-3779. PMLR.
Ji Xu, Daniel J. Hsu, and Arian Maleki. 2018. Benefits of over-parameterization with EM. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018, Montréal, Canada, pages 10685-10695.
Zhiyang Xu, Andrew Drozdov, Jay Yoon Lee, Tim O'Gorman, Subendhu Rongali, Dylan Finkbeiner, Shilpa Suresh, Mohit Iyyer, and Andrew McCallum. 2021. Improved latent tree induction with distant supervision via span constraints. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4818-4831, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Songlin Yang, Yong Jiang, Wenjuan Han, and Kewei Tu. 2020. Second-order unsupervised neural dependency parsing. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3911-3924, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Songlin Yang and Kewei Tu. 2022. Modeling label correlations for second-order semantic dependency parsing with mean-field inference. CoRR, abs/2204.03619.
Songlin Yang, Yanpeng Zhao, and Kewei Tu. 2021a. Neural bi-lexicalized PCFG induction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2688-2699, Online. Association for Computational Linguistics.
Songlin Yang, Yanpeng Zhao, and Kewei Tu. 2021b. PCFGs can do better: Inducing probabilistic context-free grammars with many symbols. In Proceedings
of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1487-1498, Online. Association for Computational Linguistics.
Youngmin Yi, Chao-Yue Lai, Slav Petrov, and Kurt Keutzer. 2011. Efficient parallel CKY parsing on GPUs. In Proceedings of the 12th International Conference on Parsing Technologies, pages 175-185, Dublin, Ireland. Association for Computational Linguistics.
Songyang Zhang, Linfeng Song, Lifeng Jin, Kun Xu, Dong Yu, and Jiebo Luo. 2021. Video-aided unsupervised grammar induction. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1513–1524, Online. Association for Computational Linguistics.
Yu Zhang, Houquan Zhou, and Zhenghua Li. 2020. Fast and accurate neural CRF constituency parsing. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pages 4046-4053. ijcai.org.
Yanpeng Zhao and Ivan Titov. 2020. Visually grounded compound PCFGs. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4369-4379, Online. Association for Computational Linguistics.
Hao Zhu, Yonatan Bisk, and Graham Neubig. 2020. The return of lexical dependencies: Neural lexicalized PCFGs. Transactions of the Association for Computational Linguistics, 8:647-661.
A Neural parameterization of PCFGs
In this section, we give the full parameterization of PCFGs. We follow Yang et al. (2021b) with slight modifications for generations of  $\mathbf{U},\mathbf{V},\mathbf{W}\in$ $\mathbb{R}^{m\times r}$  in 4.2. We use the same MLPs with two residual layers as Yang et al. (2021b):
$$
\begin{array}{l} \mathbf {s} = \frac {\exp \left(\mathbf {u} _ {S} ^ {T} h _ {1} (\mathbf {w} _ {A}) \right.}{\sum_ {A ^ {\prime} \in \mathcal {N}} \exp \left(\mathbf {u} _ {S} ^ {T} h _ {1} (\mathbf {w} _ {A ^ {\prime}})\right)} \\ \mathbf {E} = \frac {\exp (\mathbf {u} _ {E} ^ {T} h _ {2} (\mathbf {w} _ {t})}{\sum_ {E ^ {\prime} \in \Sigma} \exp (\mathbf {u} _ {E ^ {\prime}} ^ {T} h _ {2} (\mathbf {w} _ {t}))} \\ \mathbf {U} = \frac {\exp \left(\mathbf {u} _ {H} ^ {T} f _ {1} \left(\mathbf {w} _ {n}\right) \right.}{\sum_ {n ^ {\prime} \in \mathcal {N}} \exp \left(\mathbf {u} _ {H} ^ {T} f _ {1} \left(\mathbf {w} _ {n ^ {\prime}}\right)\right)} \\ \mathbf {V} = \frac {\exp (\mathbf {u} _ {H} ^ {T} f _ {2} (\mathbf {w} _ {l})}{\sum_ {H ^ {\prime} \in \mathcal {H}} \exp (\mathbf {u} _ {H ^ {\prime}} ^ {T} f _ {2} (\mathbf {w} _ {l}))} \\ \mathbf {W} = \frac {\exp (\mathbf {u} _ {H} ^ {T} f _ {3} (\mathbf {w} _ {l})}{\sum_ {H ^ {\prime} \in \mathcal {H}} \exp (\mathbf {u} _ {H ^ {\prime}} ^ {T} f _ {3} (\mathbf {w} _ {l}))} \\ h _ {i} (\mathbf {x}) = g _ {i, 1} \left(g _ {i, 2} \left(\tilde {\mathbf {W}} _ {i} \mathbf {x}\right)\right) \\ g _ {i, j} (\mathbf {y}) = R e L U \left(\tilde {\mathbf {V}} _ {i, j} R e L U \left(\tilde {\mathbf {U}} _ {i, j} \mathbf {y}\right)\right) + \mathbf {y} \\ \end{array}
$$
where  $\Sigma$  is the vocabulary set,  $\mathcal{H}$  is the set of rank,  $\mathcal{N}$  is a finite set of nonterminals,  $\mathbf{W}_l = [\mathbf{W}_n; \mathbf{W}_t], \mathbf{w}_l, \mathbf{w}_n, \mathbf{w}_t \in \mathbf{W}_l, \mathbf{W}_n, \mathbf{W}_t$ . The main differences of neural parameterization between ours and previous work are that we make the projection parameter  $\mathbf{u}_H$  shared among  $\mathbf{U}, \mathbf{V}$ , and  $\mathbf{U}$ .
B Neural parameterization of HMMs
In this section, we give the full parameterization of HMMs, which is similar to PCFGs' parameterization. Define  $\mathbf{s}$  as start probability for HMMs. And the definitions of  $\mathbf{U},\mathbf{V},\mathbf{W}$  are same as definitions in 4.3:
$$
\mathbf {s} = \frac {\exp (\mathbf {u} _ {P} ^ {T} h _ {1} (\mathbf {w} _ {s}))}{\sum_ {s ^ {\prime} \in \mathcal {S}} \exp (\mathbf {u} _ {P} ^ {T} h _ {1} (\mathbf {w} _ {s ^ {\prime}}))}
$$
$$
\mathbf {U} = \frac {\exp (\mathbf {u} _ {H} ^ {T} \mathbf {w} _ {u})}{\sum_ {H ^ {\prime} \in \mathcal {H}} \exp (\mathbf {u} _ {H ^ {\prime}} ^ {T} \mathbf {w} _ {u})}
$$
$$
\mathbf {V} = \frac {\exp (\mathbf {u} _ {H} ^ {T} \mathbf {w} _ {v})}{\sum_ {v ^ {\prime} \in \mathcal {S}} \exp (\mathbf {u} _ {H} ^ {T} \mathbf {w} _ {v ^ {\prime}})}
$$
$$
\mathbf {W} = \frac {\exp (\mathbf {u} _ {W} ^ {T} h _ {2} (\mathbf {w} _ {w})}{\sum_ {w ^ {\prime} \in \Sigma} \exp (\mathbf {u} _ {W} ^ {T} h _ {2} (\mathbf {w} _ {w ^ {\prime}}))}
$$
$$
h _ {i} (\mathbf {x}) = g _ {i, 1} (g _ {i, 2} (\tilde {\mathbf {W}} _ {i} \mathbf {x}))
$$
$$
g _ {i, j} (\mathbf {y}) = R e L U (\tilde {\mathbf {V}} _ {i, j} R e L U (\tilde {\mathbf {U}} _ {i, j} \mathbf {y})) + \mathbf {y}
$$
where  $S$  is a finite set of states,  $\mathcal{H}$  is the set of rank,  $\Sigma$  is vocabulary set.
C Data details
Penn Treebank (PTB) (Marcus et al., 1994) $^{10}$  consists of 929k training words, 73k validation words, and 82k test words, with a vocabulary of size 10k.
For PCFGs, we follow Yang et al. (2021b) and use their code to preprocess dataset. This processing discards punctuation and lowercase tokens with 10k most frequent words as the vocabulary. The splits of the dataset are: 2-21 for training, 22 for validation and 23 for test.
For HMMs, we follow Chiu et al. (2021) and use their code to preprocess dataset. We lowercase all words and substitutes OOV words with UNKs. EOS tokens have been inserted after each sentence.
For PCFGs, we use Xavier normal initialization to initialize the weights in  $h_i$  and  $f_i$ . We optimize our model using Adam optimizer with  $\beta_1 = 0.75$ ,  $\beta_2 = 0.999$ , and the learning rate 0.002, setting the dimension of all embeddings to 256.
For HMMs, we initialize all parameters by Xavier normal initialization except for  $\mathbf{w}_s$  and  $\mathbf{w}_w$ . We use AdamW optimizer with  $\beta_{1} = 0.99$ ,  $\beta_{2} = 0.999$ , and the learning rate 0.001, and a max grad norm of 5. We use dropout rate of 0.1 to dropout  $\mathbf{w}_s$  and  $\mathbf{U}$ ,  $\mathbf{V}$  in HMMs. We train for 30 epochs with a max batch size of 256 tokens, and reduce the learning by multiplying  $\frac{1}{2}$  if the validation perplexity fails to improve after 2 evaluations. Evaluations are performed one time per epoch. We follow Chiu et al. (2021) to shuffle sentences and leverage bucket iterator, where batch of sentences are drawn from buckets containing sentences of similar lengths to minimizing padding.
We run all experiments on NVIDIA TITAN RTX and NVIDIA RTX 2080ti and all experimental results are averaged from four runs.
10The licence of PTB dataset is LDC User Agreement for Non-Members, which can be seen on https://catalog.ldc.upenn.edu/LDC99T42