SlowGuess/ACL_2020_OCR huggingface dataset:
Hidetaka Kamigaito, Katsuhiko Hayashi, Tsutomu Hirao, Hiroya Takamura, Manabu Okumura, and Masaaki Nagata. 2017. Supervised attention for sequence-to-sequence constituency parsing. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 7-12, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. Cite arxiv:1412.6980Comment: Published as a conference paper at the 3rd International Conference for Learning Representations, San Diego, 2015.
Nikita Kitaev and Dan Klein. 2018. Constituency parsing with a self-attentive encoder. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 2675-2685.
Jiangming Liu and Yue Zhang. 2017a. Encoder-decoder shift-reduce syntactic parsing. In Proceedings of the 15th International Conference on Parsing Technologies, IWPT 2017, Pisa, Italy, September 20-22, 2017, pages 105-114.
Jiangming Liu and Yue Zhang. 2017b. In-order transition-based constituent parsing. Transactions of the Association for Computational Linguistics, 5:413-424.
Jiangming Liu and Yue Zhang. 2017c. Shift-reduce constituent parsing with neural lookahead features. TACL, 5:45-58.
Lemao Liu, Muhua Zhu, and Shuming Shi. 2018. Improving sequence-to-sequence constituency parsing. In AAAI Conference on Artificial Intelligence.
Chunpeng Ma, Lemao Liu, Akihiro Tamura, Tiejun Zhao, and Eiichiro Sumita. 2017. Deterministic attention for sequence-to-sequence constituent parsing. In AAAI Conference on Artificial Intelligence.
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19:313-330.
D. J. Rosenkrantz and P. M. Lewis. 1970. Deterministic left corner parsing. In Proceedings of the 11th Annual Symposium on Switching and Automata Theory (Swat 1970), SWAT '70, pages 139-152, Washington, DC, USA. IEEE Computer Society.
Mitchell Stern, Jacob Andreas, and Dan Klein. 2017. A minimal span-based neural constituency parser. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, pages 818-827.
Jun Suzuki, Sho Takase, Hidetakaka Kamigaito, Makoto Morishita, and Masaaki Nagata. 2018. An empirical study of building a strong baseline for constituency parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 612-618, Melbourne, Australia. Association for Computational Linguistics.
Oriol Vinyals, Lukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. 2015. Grammar as a foreign language. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2, NIPS'15, pages 2773-2781, Cambridge, MA, USA. MIT Press.
Taro Watanabe and Eiichiro Sumita. 2015. Transition-based neural constituent parsing. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics, ACL 2015, 26-31 July 2015, Beijing, China, Volume 1: Long Papers, pages 1169-1179.
Muhua Zhu, Yue Zhang, Wenliang Chen, Min Zhang, and Jingbo Zhu. 2013. Fast and accurate shift-reduce constituent parsing. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, ACL 2013, 4-9 August 2013, Sofia, Bulgaria, Volume 1: Long Papers, pages 434-443.
A.1 Top-down Transition System
In the top-down transition system defined by Dyer et al. (2016), parser configurations have the form  $c = \langle \Sigma, B \rangle$ , where  $\Sigma$  is a stack of constituents and  $B$  is the buffer that contains words from the input sentence. The top-down algorithm also provides three transitions (described in Figure 3) that can be used on the stack and the buffer (that initially contains the whole unparsed sentence) to build the final constituent tree. Concretely:
- a Shift transition is used to push words from the buffer into the stack,
- a Non-Terminal- $X$  transition to push a nonterminal node  $X$  into the stack,
- and a Reduce transition to pop elements from the stack until a non-terminal node is found and create a new subtree with all these elements as its children, pushing this new constituent into the stack.
A.2 In-order Transition System
Liu and Zhang (2017b) define a transition system that builds a phrase structure tree in an in-order traversal order: the non-terminal node on top of
Shift:  $\langle \Sigma, w_i | B \rangle \Rightarrow \langle \Sigma | w_i, B \rangle$
$$
\mathrm {N T - X :} \quad \langle \Sigma , B \rangle \Rightarrow \langle \Sigma | X, B \rangle
$$
Reduce:  $\langle \Sigma |X|s_k|\ldots |s_0,B\rangle \Rightarrow \langle \Sigma |X_{s_k\ldots s_0},B\rangle$
Figure 3: Transitions available in a top-down transition system (NT-X = Non-Terminal-X).
Shift:  $\langle \Sigma, w_i | B, \text{false} \rangle \Rightarrow \langle \Sigma | w_i, B, \text{false} \rangle$
NT-X:  $\langle \Sigma, B, \text{false} \rangle \Rightarrow \langle \Sigma | s_0 | X, B, \text{false} \rangle$
Reduce:  $\langle \Sigma |s_k|X|s_{k - 1}|\ldots |s_0,B,f a l s e\rangle$
$$
\Rightarrow \langle \Sigma | X _ {s _ {k} \dots s _ {0}}, B, f a l s e \rangle
$$
Finish:  $\langle \Sigma, B, \text{false} \rangle \Rightarrow \langle \Sigma, B, \text{true} \rangle$
Figure 4: Transitions available in an in-order transition system (NT-X = Non-Terminal-X).
the tree being built is only considered after the first child node is completed in the stack, building each subtree in a bottom-up manner, but choosing the non-terminal node on top before the new constituent is reduced. This transition system has parser configurations with the stack-buffer form  $c = \langle \Sigma, B \rangle$  and uses the following actions (described in Figure 4):
- a Shift transition to move words from the buffer to the stack,
- a Non-Terminal- $X$  transition to push a non-terminal node  $X$  into the stack as long as the first child of the future constituent is on top of the stack,
- a Reduce transition to pop all elements from the stack until the first non-terminal node is found, which is also popped together with the preceding element in the stack to build a new constituent with all of them as children of the non-terminal node,
and, finally, a Finish transition to terminate the parsing process.
The in-order transition system is a combination of the classic bottom-up and the new top-down algorithms, providing advantages of both of them: the access to information from partial parses from the bottom-up approach, and the non-local outlook of the top-down approach.
Hyper-parameters
BiLSTM encoder layers	2
BiLSTM encoder input size	100
BiLSTM encoder hidden size	200
LSTM decoder layers	1
LSTM decoder hidden size	400
POS tag embedding dimension	6
Pretrained word embedding dimension	100
Word embedding dimension	64
Label embedding dimension	20
Action embedding dimension	40
Attention hidden size	50
Initial learning rate	0.001
β1, β2	0.9
λ	10-6
Table 3: Model hyper-parameters.
A.3 Data and Settings
Following common practice, we test the proposed approaches on the Wall Street Journal sections of the English Penn Treebank (Marcus et al., 1993) with standard splits: sections 2-21 are used as training data, section 22 for development and section 23 for testing.
We adopt stochastic gradient descent with Adam (Kingma and Ba, 2014) and hyper-parameter selection as (Liu and Zhang, 2017a), detailed in Table 3. In addition, we use predicted POS tags and pretrained word embeddings (generated on the AFP portion of English Gigaword) as (Dyer et al., 2016; Liu and Zhang, 2017a,b).
All neural models are trained by minimizing the following cross-entropy loss objective with an  $l_{2}$  regularization term:
$$
\mathcal {L} (\theta) = - \sum_ {i} \sum_ {j} \log p _ {y _ {i j}} + \frac {\lambda}{2} | | \theta | | ^ {2}
$$
where  $\theta$  is the set of parameters,  $p_{y_{ij}}$  is the probability of the  $j$ th token in the  $i$ th training example given by the model and  $\lambda$  is a regularization hyperparameter. For further details about the neural architecture, the reader can refer to (Liu and Zhang, 2017a).
For our executions, we report the average accuracy and speed over 3 runs with random initialization and on a single CPU core.