SlowGuess/NAACL_2022_OCR huggingface dataset:
The comparison implies that SRL and DP are better auxiliary tasks for AMR Parsing even under the circumstance where their counterparts exploit far more data (40k v.s. 400k). In fact, the performance of MT drops while introducing more data, which contradicts with Xu et al. (2020) 's findings that more MT data can lead to better result when pretraining the raw Transformer model. However, this is not surprising under the background of Intermediate-task Learning where we already have a pretrained model with large-scale pretraining. Whether the intermediate tasks' form fits for the target task is far more important than the amount of data in the intermediate-task as also revealed by Poth et al. (2021). According to their observation, tasks with the most data (QQP 363k, MNLI 392k) perform far worse (-97.4% relative performance degradation at most) on some target tasks compared with tasks having much smaller datasets (CommonsenseQA 9k, SciTail 23k) which on the contrary give a positive influence.
In conclusion, our findings suggest that the selection of intermediate task is important and should be closely related to AMR parsing in form, otherwise it would even lead to a performance drop for AMR parsing.
Figure 5: The distance distribution of sentences representation. SRL and DP consistently provide more similar sentence representation to AMR than Translation. The computation is illustrated in Figure 7 in appendix.
4.1 More Similar Sentence Representation
To examine how different auxiliary tasks affect AMR parsing, we collect the sentences' representation from different tasks' trained encoders<sup>2</sup>. We use the average hidden state of the encoder's output as the sentence representation. We compute the Cosine Similarity and L2 distance between auxiliary tasks' representation and AMR's representation for same sentence. The test split of AMR 2.0 is used for evaluation. Finally, We apply Gaussian distribution to fit the distribution of distances and draw the probability distribution function curves as shown in Figure 5. It turns out that under both distance metrics, SRL/DP consistently provide more similar sentence representation to AMR than Translation and SRL/DP are more similar to AMR parsing. It empirically justifies our hypothesis that semantically or formally related tasks can lead to a better initialization for AMR parsing.
4.2 Ablation Study on AMRization Methods
As shown in Table 3, we conduct ablation study on how different AMRization methods affect the performance AMR parsing. For both SRL and DP, jointly adopting our AMRization techniques can further improve the performance of AMR parsing significantly, comparing to trivial linearization. The imperfect reentrancy restoration method leads to a significant improvement in terms of both the Topology and Concept related scores. It reveals that transformation of structure to mimic the feature of AMR can better the knowledge transfer between shallow and deep semantics.
As shown in Table 8, compared with jointly using the two techniques, it is worth noting that model
2The computing process of sentences representation distance is illustrated in Figure 7 in appendix
Model	SMATCH	Conc.	Topo.
Ours (w/ Semantic Role Labeling)	84.5	85.5	81.6
- w/ Arg. Reduction(AR)	84.8	85.6	81.9
- w/ Reen. Restoration(RR)	85.0	86.1	82.5
- w/ AR+RR	85.1	85.8	82.2
- w/ AR+RRD	85.2	86.2	82.1
Ours (w/ Dependency Parsing)	84.4	84.7	81.7
- w/ Redundant Relation Removal (RRR)	84.5	85.2	81.8
- w/ Lemmatization (Lemma)	84.7	85.5	81.7
- w/ RRR + Lemma	85.0	85.9	82.0
Table 3: We report the average scores of Concept-related scores and Topology-related scores. The full scores are listed in Table 8. The improvement of involving all techniques against trivial linearization is significant with  $p < 0.005$  for both SRL and DP.
Model	Extra	SMATCH
Ours (w/ Intermediate)
- w/ DP	40k	85.0
- w/ SRL	40k	85.1
- w/ DP,SRL	80k	84.7
Ours (w/ Multitask)
- w/ DP	40k	83.7
- w/ SRL	40k	83.6
- w/ DP,SRL	80k	83.5
Ours (w/ Multi. + Inter.)
- w/ DP	40k	84.1
- w/ SRL	40k	84.1
- w/ DP,SRL	80k	83.9
with solely Reentrancy Restoration reaches highest fine-grained scores in especially on Reentrancy and SRL scores. To explore the reason why it surpasses adopting both techniques, we analyse the number of restored reentrancy. The result shows that about 10k more reentrancies are added when Argument Reduction (AR) is previously executed. It's expected since AR replaces the token span to the root token. Compared with token span, single token is more likely to be recognized as the correference variable according to the Reentrancy Restoration (RR) algorithm, thus generating more reentrancy, which might include bias to the model. This explains why solely using RR can lead to better results on SRL and Reen.
4.3 ITL Outweighs MTL
We report the result of different fine-tuning paradigms in Table 4. It justifies our assumption that classic multitask learning with task tag as previously applied in Xu et al. (2020); Damonte and Monti (2021) does not compare with intermediate training paradigm for AMR Parsing task.
Figure 6: The loss curve on development set of AMR 2.0 for different training paradigms.
Table 4: Analysis on Training Paradigms. Intermediate-task training is more suitable for AMR parsing than Multitask training
Model	BIO	TLP	News3
SPRING	59.7	77.3	73.7
SPRING + silver	59.5	77.5	71.8
SPRINGE	60.5	77.9	74.7
Ours	61.2	78.9	75.4
Table 5: Analysis on OOD data.  ${}^{E}$  denotes result given by the ensembling of models. Our model exploits SRL as the intermediate task.
As shown in Figure 6, Intermediate-task training provides a faster and better converging process than MTL. We assume this is due to the huge gap between AMR parsing and auxiliary tasks which may harm the optimization process of MTL. The process of optimizing all auxiliary tasks simultaneously may introduce noise to AMR Parsing.
We also find that under the setting of ITL, sequentially training SRL and DP tasks did not bring further improvement to AMR parsing. We guess this is due to the catastrophic forgetting problem. Further regularization during training might help the model progressively learn from different auxiliary tasks and relieve catastrophic forgetting.
4.4 Exploration in Out-of-Distribution Generalization
Following Bevilacqua et al. (2021); Lam et al. (2021), we assess the performance of our models when trained on out-of-distribution (OOD) data. The models trained solely on AMR 2.0 training data are used to evaluate out-of-distribution performance on the BIO, the TLP and the News3 dataset.
Table 7 shows the result of our out-of-distribution experiments. Our model surpass other models even the ensembled one(Lam et al., 2021), creating new state-of-the-art for single model.
Model	BOLT	LORELEI	DFA
Dev	SPRING	30.8	72.3	73.5
Ours	56.0	73.9	76.1
Test	SPRING	34.6	73.8	71.1
Ours	59.4	74.5	74.3
Table 6: Model Smatch scores in the low-resource setting. There are 1061, 4441, 6455 examples in the training set of BOLT, LORELEI and DFA, respectively. The model exploits SRL as the intermediate task.
4.5 Exploration in Low Resources Setting
Since the annotation of AMR is both time and labor consuming, it raises our interests if we can improve the learning ability of AMR Parser under low resources setting.
We set three low resources benchmarks BOLT, LORELEI, DFA for AMR parsing based on the different sufficient degree of training examples. Detail of the datasets is described in Appendix D. Compared with the AMR2.0 dataset which has 36521 training samples, the number of training samples in BOLT, LORELEI, DFA are  $2.9\%$ ,  $12.2\%$  and  $17.7\%$  of the number of AMR2.0. Table 6 reports the result. Our model surpasses the SPRING model by a real large margin (about 25 Smatch) in the BOLT dataset which is the most insufficient in data and gains a consistent improvement on all datasets, suggesting that our pretraining method is effective under low resources conditions.
AMR Parsing AMR parsing is a challenging semantic parsing task, since AMR is a deep semantic representation and consists of many separate annotations (Banarescu et al., 2013) (e.g., semantic relations, named entities, co-reference and so on). There are four major methods to do AMR Parsing currently, sequence-to-sequence approaches (Ge et al., 2019; Xu et al., 2020; Bevilacqua et al., 2021; Wang et al., 2022), tree-based approaches (Zhang et al., 2019b,a), graph-based approaches (Lyu and Titov, 2018; Cai and Lam, 2020) and transition-based approaches (Naseem et al., 2019; Lee et al., 2020; Zhou et al., 2021a).
There are two ways to incorporate other tasks to AMR Parsing. Goodman et al. (2016) builds AMR graph directly from dependency trees while (Ge et al., 2019) parse directly from linearized syntactic tree. Xu et al. (2020) introduces Machine Translation, Constituency Parsing as pretraining tasks
for Seq2Seq AMR parsing and Wu et al. (2021) introduces Dependency Parsing for transition-based AMR parsing. However all of them do not take care of the semantic and formal gap between the auxiliary tasks and AMR parsing.
Multitask & Intermediate-task Learning Multi-task Learning (MTL) (Caruana, 1997) aims to jointly train multiple related tasks to improve the performance of all tasks. Different from MTL, Intermediate-task Learning (ITL) is proposed to enhance pretrained models e.g. BERT by training on intermediate task before fine-tuning on the target task. Recent studies(Pruksachatkun et al., 2020; Poth et al., 2021) on ITL expose that choosing right intermediate tasks is important. Tasks that don't match might even bring negative effect to the target even if it has far more data.
Xu et al. (2020); Damonte and Monti (2021); Procopio et al. (2021) utilize auxiliary tasks in a MTL fashion with specific task tags. Bevilacqua et al. (2021); Zhou et al. (2021b) adopt sliver training data in a ITL paradigm. However, there is no work comparing ITL and MTL when introducing auxiliary tasks to enhance PTM-based AMR parser.
In this paper, We find that semantically or formally related tasks, e.g. SRL and DP are better auxiliary tasks for AMR parsing and can further improve the performance by proper AMRization methods to bridge the gap between tasks. And Intermediate-task Learning is more effective in introducing auxiliary tasks compared with Multitask Learning. Extensive experiments and analyses show the effectiveness and priority of our proposed methods.
We thank all reviewers for their valuable advice. This paper is supported by the National Key Research and Development Program of China under Grant No.2020AAA0106700, the National Science Foundation of China under Grant No.61936012 and 61876004.
8 Ethics Consideration
We collect our data from public datasets that permit academic use and buy the license for the datasets that are not free. The open-source tools we use for training and evaluation are freely accessible online without copyright conflicts.
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffith, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In Proceedings of the 7th linguistic annotation workshop and interoperability with discourse, pages 178-186.
Michele Bevilacqua, Rexhina Blloshmi, and Roberto Navigli. 2021. One spring to rule them both: Symmetric amr semantic parsing and generation without a complex pipeline. In Proceedings of the Thirty-Fifth AAAI Conference on Artificial Intelligence.
Steven Bird. 2006. NLTK: The Natural Language Toolkit. In Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions, pages 69-72, Sydney, Australia. Association for Computational Linguistics.
Claire Bonial, L. Donatelli, Mitchell Abrams, Stephanie M. Lukin, Stephen Tratz, Matthew Marge, Ron Artstein, David R. Traum, and Clare R. Voss. 2020. Dialogue-amr: Abstract meaning representation for dialogue. In LREC.
Deng Cai and Wai Lam. 2020. AMR parsing via graph sequence iterative inference. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1290-1301, Online. Association for Computational Linguistics.
Shu Cai and Kevin Knight. 2013. Smatch: an evaluation metric for semantic feature structures. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 748-752, Sofia, Bulgaria. Association for Computational Linguistics.
Sirui Cai, Yuchun Fang, and Zhengyan Ma. 2017. Will outlier tasks deteriorate multitask deep learning? In Neural Information Processing, pages 246-255. Springer International Publishing.
Rich Caruana. 1997. Multitask learning. Machine learning, 28(1):41-75.
Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang. 2021. DialogSum: A real-life scenario dialogue summarization dataset. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 5062-5074, Online. Association for Computational Linguistics.
Marco Damonte, Shay B. Cohen, and Giorgio Satta. 2017. An incremental parser for Abstract Meaning Representation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 536-546, Valencia, Spain. Association for Computational Linguistics.
Marco Damonte and Emilio Monti. 2021. One semantic parser to parse them all: Sequence to sequence
multi-task learning on semantic parsing datasets. In Proceedings of *SEM 2021: The Tenth Joint Conference on Lexical and Computational Semantics*, pages 173-184, Online. Association for Computational Linguistics.
Ramón Fernandez Astudillo, Miguel Ballesteros, Tahira Naseem, Austin Blodgett, and Radu Florian. 2020. Transition-based parsing with stack-transformers. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1001-1007, Online. Association for Computational Linguistics.
DongLai Ge, Junhui Li, Muhua Zhu, and Shoushan Li. 2019. Modeling source syntax and semantics for neural amr parsing. In *IJCAI*, pages 4975-4981.
James Goodman, Andreas Vlachos, and Jason Naradowsky. 2016. Noise reduction and targeted exploration in imitation learning for Abstract Meaning Representation parsing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1-11, Berlin, Germany. Association for Computational Linguistics.
Hardy Hardy and Andreas Vlachos. 2018. Guided neural language generation for abstractive summarization using abstract meaning representation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 768-773.
Karl Moritz Hermann, Tomás Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In NIPS.
Hoang Thanh Lam, Gabriele Picco, Yufang Hou, Young-Suk Lee, Lam M. Nguyen, Dzung T. Phan, Vanessa López, and Ramon Fernandez Astudillo. 2021. Ensemble graph predictions for amr parsing.
Young-Suk Lee, Ramón Fernandez Astudillo, Tahira Naseem, Revanth Gangi Reddy, Radu Florian, and Salim Roukos. 2020. Pushing the limits of amr parsing with self-learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 3208-3214.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Belinda Z. Li, Sewon Min, Srinivasan Iyer, Yashar Mehdad, and Wen-tau Yih. 2020. Efficient one-pass end-to-end entity linking for questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6433-6441, Online. Association for Computational Linguistics.
Kexin Liao, Logan Lebanoff, and Fei Liu. 2018. Abstract meaning representation for multi-document summarization. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1178-1190.
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. 2019. On the variance of the adaptive learning rate and beyond. arXiv preprint arXiv:1908.03265.
Chunchuan Lyu and Ivan Titov. 2018. AMR parsing as graph prediction with latent alignment. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 397-407, Melbourne, Australia. Association for Computational Linguistics.
Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David McClosky. 2014. The stanford corenlp natural language processing toolkit. In ACL.
Mitchell Marcus, Beatrice Santorini, Mary Marcinkiewicz, and Ann Taylor. 1999. Penn treebank 3.
Arindam Mitra and Chitta Baral. 2016. Addressing a question answering challenge by combining statistical methods with inductive rule learning and reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30.
Tahira Naseem, Abhishek Shah, Hui Wan, Radu Florian, Salim Roukos, and Miguel Ballesteros. 2019. Rewarding smatch: Transition-based amr parsing with reinforcement learning. arXiv preprint arXiv:1905.13370.
Clifton Poth, Jonas Pfeiffer, Andreas Rückle, and Iryna Gurevych. 2021. What to pre-train on? Efficient intermediate task selection. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10585-10605, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Luigi Procopio, Rocco Tripodi, and Roberto Navigli. 2021. SGL: Speaking the graph languages of semantic parsing via multilingual translation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 325-337, Online. Association for Computational Linguistics.
Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel Bowman. 2020. Intermediate-task transfer learning with pretrained language models: When and why does it work? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5231-5247.
Sudha Rao, Daniel Marcu, Kevin Knight, and Hal Daumé III. 2017. Biomedical event extraction using abstract meaning representation. In BioNLP 2017, pages 126-135.
Mrinmaya Sachan and Eric Xing. 2016. Machine comprehension using rich semantic representations. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 486-492.
Peiyi Wang, Liang Chen, Tianyu Liu, Damai Dai, Yunbo Cao, Baobao Chang, and Zhifang Sui. 2022. Hierarchical curriculum learning for amr parsing. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL).
Yanshan Wang, Sijia Liu, Majid Rastegar-Mojarad, Liwei Wang, Feichen Shen, Fei Liu, and Hongfang Liu. 2017. Dependency and amr embeddings for drug-drug interaction extraction from biomedical literature. In Proceedings of the 8th acm international conference on bioinformatics, computational biology, and health informatics, pages 36-43.
Ralph M. Weischedel, Eduard H. Hovy, Mitchell P. Marcus, and Martha Palmer. 2017. Ontonotes: A large training corpus for enhanced processing.
Taizhong Wu, Junsheng Zhou, Weiguang Qu, Yanhui Gu, Bin Li, Huilin Zhong, and Yunfei Long. 2021. Improving amr parsing by exploiting the dependency parsing as an auxiliary task. Multim. Tools Appl., 80:30827-30838.
Dongqin Xu, Junhui Li, Muhua Zhu, Min Zhang, and Guodong Zhou. 2020. Improving amr parsing with sequence-to-sequence pre-training. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2501-2511.
Sheng Zhang, Xutai Ma, Kevin Duh, and Benjamin Van Durme. 2019a. AMR parsing as sequence-tograph transduction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 80-94, Florence, Italy. Association for Computational Linguistics.
Sheng Zhang, Xutai Ma, Kevin Duh, and Benjamin Van Durme. 2019b. Broad-coverage semantic parsing as transduction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3786-3798, Hong Kong, China. Association for Computational Linguistics.
Yu Zhang and Qiang Yang. 2021. A survey on multitask learning. IEEE Transactions on Knowledge and Data Engineering, pages 1-1.
Zhisong Zhang, Emma Strubell, and Eduard Hovy. 2021. Comparing span extraction methods for semantic role labeling. In Proceedings of the 5th Workshop on Structured Prediction for NLP (SPNLP 2021), pages
67-77, Online. Association for Computational Linguistics.
Zixuan Zhang and Heng Ji. 2021. Abstract meaning representation guided graph encoding and decoding for joint information extraction. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 39-49.
Jiawei Zhou, Tahira Naseem, Ramón Fernandez Astudillo, and Radu Florian. 2021a. AMR parsing with action-pointer transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5585-5598, Online. Association for Computational Linguistics.
Jiawei Zhou, Tahira Naseem, Ramón Fernandez Astudillo, Young-Suk Lee, Radu Florian, and Salim Roukos. 2021b. Structure-aware fine-tuning of sequence-to-sequence transformers for transition-based AMR parsing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6279-6290, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Algorithm 1 Reentrancy Restoration for SRL
Input: Treenode:T
Output: Graph:G
Description: T is root node of the original SRL after node ROOT is added to form tree structure. G is the output graph with possible reentrancy restored.
Global Variables: Dict:  $\mathrm{V} = \{\}$ . Here Dict is the official data structure of Python's dictionary.
1: for predicate in T.sons do
2: for son in predicate.sons() do
3: if son.name in V.keys() then
4: son = V[son.name]
5: # restore reentrancy
6: else
7: V[son.name] = son
8: return T
B Ensemble Models' Methods
Graphene-4SE Lam et al. (2021) make use of 4 SPRING models from different random seeds and their proposed graph ensemble algorithm to do the ensembling. They also include another ensemble model named Graphene All which includes four checkpoints from models of different architectures, SPRING(Bevilacqua et al., 2021), APT(Zhou et al., 2021a), T5, and Cai&Lam(Cai and Lam, 2020). We do not report the score of Graphene All since it aggregates models with different inductive bias while our ensemble model only use models from one structure. It is out of the scope for fair comparison.
Structure-aware $^{E}$  Zhou et al. (2021b) use ensemble results from 3 models' combination to generate the ensemble model.
Ours  $(\mathbf{w} / \mathbf{SRL})^E$  We use the setting the same as Zhou et al. (2021b), we use the average of three models' parameters as the ensemble model.
C Auxiliary Datasets Description
C.1 Summarization
CNN/DM(Hermann et al., 2015) The CNN / DailyMail Dataset is an English-language dataset containing news articles as written by journalists at CNN and the Daily Mail. The dataset is widely
accepted as benchmark to test models' performance of summarizing.
DIALOGSUM(Chen et al., 2021) The Real-Life Scenario Dialogue Summarization (DIALOGSUM), is a large-scale summarization dataset for dialogues. Unlike CNN/DM which focuses on monologue news summarization, DIALOGSUM covers a wide range of daily-life topics in the form of spoken dialogue. We use all the training data (13k) to conduct the intermediate training.
C.2 Translation
WMT14 EN-DE We select the first 40k,80k,200k and 400k training examples from WMT14 EN-DE training set to form EN-DE translation intermediate tasks.
C.3 Dependency Parsing