SlowGuess/NAACL_2022_OCR huggingface dataset:
Yijia Liu, Wanxiang Che, Bo Zheng, Bing Qin, and Ting Liu. 2018. An AMR aligner tuned by transition-based parser. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2422-2430, Brussels, Belgium. Association for Computational Linguistics.
Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412-1421, Lisbon, Portugal. Association for Computational Linguistics.
Chunchuan Lyu, Shay B. Cohen, and Ivan Titov. 2021. A differentiable relaxation of graph segmentation and alignment for AMR parsing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9075-9091, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Chunchuan Lyu and Ivan Titov. 2018. AMR parsing as graph prediction with latent alignment. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 397-407, Melbourne, Australia. Association for Computational Linguistics.
Andriy Mnih and Danilo Jimenez Rezende. 2016. Variational inference for monte carlo objectives. In Proceedings of ICML.
Tahira Naseem, Abhishek Shah, Hui Wan, Radu Florian, Salim Roukos, and Miguel Ballesteros. 2019. Rewarding Smatch: Transition-based AMR parsing with reinforcement learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4586-4592, Florence, Italy. Association for Computational Linguistics.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
Nima Pourdamghani, Yang Gao, Ulf Hermjakob, and Kevin Knight. 2014. Aligning English strings with Abstract Meaning Representation graphs. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 425-429, Doha, Qatar. Association for Computational Linguistics.
Colin Raffel, Minh-Thang Luong, Peter J. Liu, Ron J. Weiss, and Douglas Eck. 2017. Online and linear-time attention by enforcing monotonic alignments. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 2837-2846. PMLR.
Shiv Shankar, Siddhant Garg, and Sunita Sarawagi. 2018. Surprisingly Easy Hard-Attention for Sequence to Sequence Learning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 640–645, Brussels, Belgium. Association for Computational Linguistics.
Shiv Shankar and Sunita Sarawagi. 2019. Posterior attention models for sequence to sequence learning. In Proceedings of ICLR.
Stephanie Strassel and Jennifer Tracey. 2016. LORELEI language packs: Data, tools, and resources for technology development in low resource languages. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 3273-3280, Portož, Slovenia. European Language Resources Association (ELRA).
Shijie Wu and Ryan Cotterell. 2019. Exact hard monotonic attention for character-level transduction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1530-1537, Florence, Italy. Association for Computational Linguistics.
Shijie Wu, Pamela Shapiro, and Ryan Cotterell. 2018. Hard non-monotonic attention for character-level transduction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4425-4438, Brussels, Belgium. Association for Computational Linguistics.
Qingrong Xia, Zhenghua Li, Rui Wang, and Min Zhang. 2021. Stacked AMR parsing with silver data. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4729-4738, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. In Proceedings of ICML.
Pengcheng Yin, Hao Fang, Graham Neubig, Adam Pauls, Emmanouil Antonios Platanios, Yu Su, Sam Thomson, and Jacob Andreas. 2021. Compositional Generalization for Neural Semantic Parsing via Span-level Supervised Attention. In Proceedings of NAACL.
Lei Yu, Jan Buys, and Phil Blunsom. 2016a. Online segment to segment neural transduction. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1307-1316, Austin, Texas. Association for Computational Linguistics.
Lei Yu, Jan Buys, and Phil Blunsom. 2016b. Online Segment to Segment Neural Transduction. In Proceedings of EMNLP.
Thomas Zenkel, Joern Wuebker, and John DeNero. 2020. End-to-End Neural Word Alignment Outperforms GIZA++. In Proceedings of ACL.
Sheng Zhang, Xutai Ma, Kevin Duh, and Benjamin Van Durme. 2019a. AMR parsing as sequence-tograph transduction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 80-94, Florence, Italy. Association for Computational Linguistics.
Sheng Zhang, Xutai Ma, Kevin Duh, and Benjamin Van Durme. 2019b. Broad-coverage semantic parsing as transduction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3786-3798, Hong Kong, China. Association for Computational Linguistics.
Jiawei Zhou, Tahira Naseem, Ramón Fernandez Astudillo, and Radu Florian. 2021a. AMR parsing with action-pointer transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5585-5598, Online. Association for Computational Linguistics.
Jiawei Zhou, Tahira Naseem, Ramón Fernandez Astudillo, Young-Suk Lee, Radu Florian, and Salim Roukos. 2021b. Structure-aware fine-tuning of sequence-to-sequence transformers for transition-based AMR parsing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6279-6290, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
A StructBART Implementation Details
Inference in a transition-based parser corresponds to the usual decoding of with a sequence to sequence model
$$
\hat {a} = \arg \max _ {a} \{p (a \mid w) \}
$$
to obtain the graph  $\hat{g} = M(\hat{a},w)$ . The model  $p(a\mid w)$  is nowadays parametrized with a neural network. The state machine  $M(a,w)$  is defined by
- the position of a cursor  $k \in [|w|] = 1 \cdots |w|$ , moving left to right over  $w$
- a stack of nodes of  $g$  that is initially empty
- the partial graph of  $g$
Furthermore, each action is decoupled into normal and pointer actions  $a_{t} = (b_{t},r_{t})$  with
$$
\begin{array}{l} p \left(a _ {t} \mid a _ {<   t}, w; \theta\right) = \\ p \left(r _ {t} \mid b _ {t}, a _ {<   t}, w; \theta\right) p \left(b _ {t} \mid a _ {<   t}, w; \theta\right) \\ \end{array}
$$
$$
\begin{array}{l} p \left(b _ {t} \mid a _ {<   t}, w; \theta\right) = \\ \operatorname {s o f t m a x} \left(\mathrm {N N} _ {\theta} (a _ {<   t}, w) + m (a _ {<   t}, w)\right) _ {b _ {t}} \\ \end{array}
$$
where  $\mathrm{NN}_{\theta}(a_{< t},w)\in \mathbb{R}^{|V_b|}$  are the logits coming from a neural network model and  $m(a_{< t},w)\in \{-\infty ,0\}^{V_b}$  is a mask forbidding invalid state machine actions e.g. shifting at sentence end. This mask is deterministically computed given the machine state.  $V_{b}$  is the vocabulary of normal actions (all actions minus pointer information). The pointer network is given by
$$
\begin{array}{l} p \left(r _ {t} \mid b _ {t}, a _ {<   t}, w; \theta\right) = \\ \operatorname {s o f t m a x} \left(\mathrm {D S A} _ {\theta} (a _ {<   t}, w) + m 2 (a _ {<   t}, w)) _ {b _ {t}}\right) \\ \end{array}
$$
for  $b_{t}$  executing arc LA/RA actions and 1 otherwise. The network  $\mathrm{DSA}_{\theta}(a_{< t}, w) \in \mathbb{R}^{|a_{< t}|}$  is the decoder's self-attention encoding of action history  $a_{< t}$  (last layer). The mask  $m2$  prevents pointing to any action that is not a node generating action, or has been reduced.
B Parser Training Details
Training with argmax alignments takes approximately 12 hours on a single GPU (2080ti), and longer when sampling alignments proportional to the number of samples. When training with samples, we use gradient accumulation to avoid out of memory problems. We choose accumulation steps roughly proportional to number of samples. Otherwise, training hyperparameters exactly match those from Zhou et al. (2021b).
C Aligner Implementation Details
Comment on sequence length. The alignment model in Wu et al. (2018) demonstrates strong results for character-level translation, which involves translating a single word from one language to another character-by-character. They state the following in reference to their alignment model: the exact marginalization scheme is practically unworkable for machine translation. As a point of reference, we looked at the inflection dataset — in any of the training splits across the 51 languages,  $99\%$  of the sequences are less than 29 tokens long,  $90\%$  are less than 23 tokens, and  $50\%$  are less than 15. In contrast, sentence lengths for AMR3.0 are often considerably longer — the 99/90/50 percentiles for sentence token lengths are 62/35/15 and the AMR token lengths are 45/26/11. Nonetheless, we found their results encouraging and our implementation of the alignment model to still be fast enough despite our using longer sequences.[11]
D Aligner Training Details
We use single layer LSTMs with size 200 hidden dimension, dropout 0.1, learning rate 0.0001, and train with the Adam Optimizer. We use batch size 32 and accumulate gradient over 4 steps (for an effective batch size of 128). For 200 epochs, training takes approximately 1-day on a single GPU (2080ti). We train a new aligner for each parsing experiment. In Table 2 we report alignment results from our highest Smatch parsing experiment.
E Alternative text for Figure 1
Shown are the baseline's point estimate alignment and our aligner's alignment posterior. There are instances of ambiguity where our alignment distribution is preferred to the point estimate.
11We use number of AMR nodes to represent token length, since this is what bounds the computation.