SlowGuess/NAACL_2021_OCR huggingface dataset:
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Ashwin K Vijayakumar, Michael Cogswell, Ramprasath R Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2016. Diverse beam search: Decoding diverse solutions from neural sequence models. arXiv preprint arXiv:1610.02424.
David Vilares and Carlos Gomez-Rodriguez. 2018. A transition-based algorithm for unrestricted amr parsing. arXiv preprint arXiv:1805.09007.
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In Advances in neural information processing systems, pages 2692-2700.
Chuan Wang and Nianwen Xue. 2017. Getting the most out of amr parsing. In Proceedings of the 2017 conference on empirical methods in natural language processing, pages 1257-1268.
Chuan Wang, Nianwen Xue, and Sameer Pradhan. 2015. A transition-based algorithm for amr parsing. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 366-375.
Dongqin Xu, Junhui Li, Muhua Zhu, Min Zhang, and Guodong Zhou. 2020. Improving amr parsing with sequence-to-sequence pre-training. arXiv preprint arXiv:2010.01771.
Sheng Zhang, Xutai Ma, Kevin Duh, and Benjamin Van Durme. 2019a. Amr parsing as sequence-to-graph transduction. arXiv preprint arXiv:1905.08704.
Sheng Zhang, Xutai Ma, Kevin Duh, and Benjamin Van Durme. 2019b. Broad-coverage semantic parsing as transduction. arXiv preprint arXiv:1909.02607.
Zhirui Zhang, Shujie Liu, Mu Li, Ming Zhou, and Enhong Chen. 2017. Stack-based multi-layer attention for transition-based dependency parsing. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1677-1682, Copenhagen, Denmark. Association for Computational Linguistics.
Jiawei Zhou and Alexander M Rush. 2019. Simple unsupervised summarization by contextual matching. arXiv preprint arXiv:1907.13337.
Jie Zhu, Junhui Li, Muhua Zhu, Longhua Qian, Min Zhang, and Guodong Zhou. 2019. Modeling graph structure in transformer for better amr-to-text generation. arXiv preprint arXiv:1909.00136.
A A More Detailed Example of Action-Pointer Transitions
We present a step-by-step walk-through of our actions on a less trivial example for generating the AMR in Figure 6. The sentence contains a named entity which also demonstrates the MERGE and SUBGRAPH usage of our transition system.
B Action-Pointer Oracle
For a given sentence, at every oracle step, apply the actions in the order listed below. Continue until the source cursor moves past the last token.
1. If cursor is on a non-final token of a span aligned to a node, keep moving cursor (and collecting tokens) with MERGE till it reaches the final token of the span.
2. If there is a matching pattern for current token(s) in SUBGRAPH() action dictionary:
- Apply matching SUBGRAPH() action.
- Generate edges between the root of the sub-graph and past nodes through LA(), RA(). Generate closer edges first.
Otherwise, for all nodes aligned to the current token, in top-down order:
- Generate node through COPY (lemma or first sense), and if not possible then through  $\mathrm{PRED}()$ .
- Generate edges between the last nodes and past nodes through LA(), RA(). Generate closer edges first.
3. If no action performed at step 2, move cursor with REDUCE otherwise, move cursor with SHIFT.
C Action-Pointer Decoding
We outline the decoding algorithm for our model in Algorithm 1, to combine the actions with pointers, as well as taking in parsing states and graph structures for the model during the decoding steps. Detailed beam search process is ignored. Although tackling the specific problem of AMR graph generation with pointers, our constraint decoding process is a modified beam search algorithm with different components and step-wise controls, among others (Vijayakumar et al., 2016; Zhou and Rush, 2019).
D Number of Parameters
Our model is a single Transformer (Vaswani et al., 2017) model. The pointer distribution, action-source alignment encoding from parsing state, and structural graph embedding are all contained in certain attention layers and heads, without introducing any extra parameters on original Transformer. We fix our model size and all the embedding size to be 256, and the feedforward hidden size in Transformer as 512. And they are the same for our base model with 6 layers and 4 heads and our small model with 3 layers and 4 heads, both for encoder and decoder.
We use pre-trained RoBERTa embeddings for the source token embeddings. The embeddings are extracted in pre-processing and fixed. The RoBERTa model parameters are fixed and not trained with our model. We have a projection layer to project the RoBERTa embedding size 1024/768 to our model size 256.
The target side dictionary is built from all the oracle actions without pointers on training data. The dictionary size for AMR 1.0 is 4640, for AMR 2.0 is 9288, and for AMR 3.0 is 11680. We build the target action embeddings along with the model for the action prediction on top of Transformer decoder. The dictionary embedding size is fixed at 256.
Overall, the total number of parameters for our 6 layer base model is 14,852,096 on AMR 1.0, 21,438,464 on AMR 2.0, and 25,550,848 on AMR 3.0 (difference is in target dictionary embedding size). The total number of parameter for our 3 layer small model is 10,898,432 for AMR 1.0 and 17,484,800 on AMR 2.0 (difference is in target dictionary embedding size).
Figure 6: Step-by-step actions based on our action-pointer transition system. We illustrate the use of MERGE and SUBGRAPH with the named entity of a person's name in this example. The source cursor after the action is applied is pointing at words underlined, and the partially built graph is shown in the right-most column.
Algorithm 1: Constrained beam search for action-pointer decoding
Input: Initial token  $a_0 = \langle \rangle$  , beam size  $k$  max step  $T_{max}$  , action dictionary  $D$  without pointers, model  $M$  that outputs both distribution over  $D$  and the pointer distribution from self-attention
Output: Decoded results
$$
y _ {1} = \left(a _ {1}, p _ {1}\right), y _ {2} =
$$
$$
(a _ {2}, p _ {2}), \dots , y _ {T} = (a _ {T}, p _ {T})
$$
initialization: step  $t = 1$ ,  $k$  state machines
while  $t <   = T_{max}$  do
1) Get the valid action dictionary  $D_{t} \subset D$ , previous node action positions  $N_{t} \subset \{0, 1, 2, \dots, t\}$ , current token cursor  $c_{t}$ , and current graph  $G_{t}$  (all from the corresponding state machines);
2) Input prefix  $a_0, a_1, \dots, a_{t-1}$  and  $D_t, c_t, G_t$  into model, get output distribution  $\mathbf{P}(a_t | \mathbf{y}_{<t})$ , and the self-attention distribution  $Q(p)$  from pointer head with  $p$  over  $\{0, 1, \dots, t\}$ ;
3) Take the most likely valid pointer value, with  $p^* = \operatorname{argmax}_{p \in N_t} Q(p)$ , and its score  $q^* = \max_{p \in N_t} Q(p)$ ;
for each possible action  $a$  from  $D$  do
if  $a$  is an edge action then combine the action probability with pointer probability  $\mathbf{P}(y_t) = \mathbf{P}(a_t|\mathbf{y}_{< t})\cdot q^*$  ,with  $y_{t} = (a,p^{*})$
else set  $\mathbf{P}(y_t) = \mathbf{P}(a_t|\mathbf{y}_{<  t})$  ,with  $y_{t} = (a,null)$
Do beam search with  $\mathbf{P}(y_t)$  over  $y_{t}$  to get  $k$  decoded results;
Apply the corresponding actions with the  $k$  state machines to update parser states and partial graphs for each beam candidate.