SlowGuess/NAACL_2022_OCR huggingface dataset:
Liang Chen $^{1*}$ , Peiyi Wang $^{1*}$ , Runxin Xu $^{1}$ , Tianyu Liu $^{2}$ , Zhifang Sui $^{1}$ , Baobao Chang $^{1\dagger}$
leo.liang.chen@outlook.com; wangpeiyi9979@gmail.com runxinxu@gmail.com; rogertyliu@tencent.com szf, chbb@pku.edu.cn
As Abstract Meaning Representation (AMR) implicitly involves compound semantic annotations, we hypothesize auxiliary tasks which are semantically or formally related can better enhance AMR parsing. We find that 1) Semantic role labeling (SRL) and dependency parsing (DP), would bring more performance gain than other tasks e.g. MT and summarization in the text-to-AMR transition even with much less data. 2) To make a better fit for AMR, data from auxiliary tasks should be properly "AMRized" to PseudoAMR before training. Knowledge from shallow level parsing tasks can be better transferred to AMR Parsing with structure transform. 3) Intermediate-task learning is a better paradigm to introduce auxiliary tasks to AMR parsing, compared to multitask learning. From an empirical perspective, we propose a principled method to involve auxiliary tasks to boost AMR parsing. Extensive experiments show that our method achieves new state-of-the-art performance on different benchmarks especially in topology-related scores. Code and models are released at https://github.com/PKUnlp-icler/ATP.
Abstract Meaning Representation (AMR) (Banarescu et al., 2013) parsing aims to translate a sentence to a directed acyclic graph, which represents the relations among abstract concepts as shown in Figure 1. AMR can be applied to many downstream tasks, such as information extraction (Rao et al., 2017; Wang et al., 2017; Zhang and Ji, 2021), text summarization, (Liao et al., 2018; Hardy and Vlachos, 2018) question answering (MiTRA and Baral, 2016; Sachan and Xing, 2016) and dialogue modeling (Bonial et al., 2020).
Recently, AMR Parsing with the sequence-to-sequence framework achieves most promising re
Figure 1: The Abstract Meaning Representation (AMR), Semantic Role Labeling (SRL), and Dependency Parsing (DP) structure of the sentence "The boy wants to leave."
sults (Xu et al., 2020; Bevilacqua et al., 2021). Comparing with transition-based or graph-based methods, sequence-to-sequence models do not require tedious data processing and is naturally compatible with auxiliary tasks (Xu et al., 2020) and powerful pretrained encoder-decoder models (Bevilacqua et al., 2021). Previous work (Xu et al., 2020; Wu et al., 2021) has shown that the performance of AMR parser can be effectively boosted through co-training with certain auxiliary tasks, e.g. Machine Translation or Dependency Parsing.
However, when introducing auxiliary tasks to enhance AMR parsing, we argue that three important issues still remain under-explored in the previous work. 1) How to choose auxiliary task? The task selection is important since loosely related tasks may even impede the AMR parsing according to Damonte and Monti (2021). However, in literature there are no principles or consensus on how to choose the proper auxiliary tasks for AMR parsing. Though previous work achieves noticeable performance gain through multi-task learning, they do not provide explainable insights on why certain task outperforms others or in which aspects the auxiliary tasks benefit the AMR parser.
Findings of the Association for Computational Linguistics: NAACL 2022, pages 2482 - 2496
Figure 2: Illustration of methodology in this paper. We proposed a principled method to select, transform and train the auxiliary tasks.
2) How to bridge the gap between tasks? The gaps between AMR parsing and auxiliary tasks are non-negligible. For example, Machine Translation generates text sequence while Dependency Parsing (DP) and Semantic Role Labeling (SRL) produces dependency trees and semantic role forests respectively as shown in Figure 1. Prior studies (Xu et al., 2020; Wu et al., 2021; Damonte and Monti, 2021) do not attach particular importance to the gap, which might lead the auxiliary tasks to outlier-task (Zhang and Yang, 2021; Cai et al., 2017) in the Multitask Learning, deteriorating the performance of AMR parsing. 3) How to introduce auxiliary tasks more effectively? After investigating different training paradigms to combine the auxiliary task training with the major objective (AMR parsing), we figure out that, although all baseline models (Xu et al., 2020; Wu et al., 2021; Damonte and Monti, 2021) choose to jointly train the auxiliary tasks and AMR parsing with Multitask Learning (MTL), Intermediate-task Learning (ITL) is a more effective way to introduce the auxiliary tasks for pretrained models. Our observation is also consistent with (Pruksachatkun et al., 2020; Poth et al., 2021), which improve other NLP tasks with enhanced pretrained models.
In response to the above three issues, we summarize a principled method to select, transform and train the auxiliary tasks (Figure 2) to enhance AMR parsing from extensive experiments. 1) Auxiliary Task Selection. We choose auxiliary tasks by estimating their similarities with AMR from the semantics and formality perspectives. AMR is recognized as a deep semantic parsing task which encompasses multiple semantic annotations, e.g. semantic roles, name entities and co-references. As a direct semantic-level sub-task of AMR parsing, we select SRL as one auxiliary task. Traditionally,
formal semantics views syntactic parsing a precursor to semantic parsing, leading to the mapping between syntactic and semantic relations. Hence we introduce dependency parsing, a syntactic parsing task as another auxiliary task. 2) AMRization. Despite being highly related, the output formats of SRL, DP and AMR are distinct from each other. To this end, we introduce transformation rules to "AMRize" SRL and DP to PseudoAMR, intimating the feature of AMR. Specifically, through Reentrancy Restoration we transform the structure of SRL to a graph and restore the reentrancy within arguments, which mimics AMR structure. Through Redundant Relation Removal we conduct transformation in dependency trees and remove relations that are far from semantic relations in AMR graph. 3) Training Paradigm Selection. We find that ITL makes a better fit for AMR parsing than MTL since it allows model progressively transit to the target task instead of learning all tasks simultaneously, which benefits knowledge transfer (Zhang and Yang, 2021).
1. Semantically or formally related tasks, e.g., SRL and DP, are better auxiliary tasks for AMR parsing compared with distantly related tasks, e.g. machine translation and machine reading comprehension.
2. We propose task-specific rules to AMRize the structured data to PseudoAMR. SRL and DP with properly transformed output format further improve AMR parsing.
3. ITL outperforms classic MTL methods when introducing auxiliary tasks to AMR Parsing. We show that ITL derives a steadier and better converging process during training.
Extensive experiments show that our method (PseudoAMR + ITL) achieves the new state-of-the-art of single model on in-distribution (85.2 Smatch score on AMR 2.0, 83.9 on AMR 3.0), out-of-distribution benchmarks. Specifically we observe that AMR parser gains larger improvement on the SRL(+3.3), Reentrancy(+3.1) and NER(+2.0) metrics*, due to higher resemblance with the selected auxiliary tasks.
Figure 3: Illustration of AMRization methods and Graph Linearization. The source sentence is "The boy wants to leave."
As shown in Figure 2, in this paper, we propose a principled method to select auxiliary tasks (Section 2.1), AMRize them into PseudoAMR (Section 2.2) and train PseudoAMR and AMR effectively (Section 2.3) to boost AMR parsing. We formulate both PseudoAMR and AMR parsing as the sequence-to-sequence generation problem. Given a sentence  $x = [x_{i}]_{1\leq i\leq N}$ , the model aims to generate a linearized PseudoAMR or AMR graph  $y = [y_{i}]_{1\leq i\leq M}$  (the right part of Figure 3) with a product of conditional probability:
$$
P (y | x) = \prod_ {i = 1} ^ {M} p \left(y _ {i} \mid \left(y _ {1}, y _ {2}, \dots , y _ {i - 1}\right), x\right)
$$
2.1 Auxiliary Task Selection
When introducing auxiliary tasks for AMR parsing, the selected tasks should be formally or semantically related to AMR, thus the knowledge contained in them can be transferred to AMR parsing. Based on this principle of relevance, we choose semantic role labeling (SRL) and dependency parsing (DP) as our auxiliary tasks. We involve Translation and Summarization tasks for comparison.
Semantic Role Labeling SRL aims to recover the predicate-argument structure of a sentence, which can enhance AMR parsing, because: (1) Recovering the predicate-argument structure is also a sub-task of AMR parsing. As illustrated in Figure
3(a,b), both AMR and SRL locate the predicates ("want", "leave") of the sentence and conduct word-sense disambiguation. Then they both capture the multiple arguments of center predicate. (2) SRL and AMR are known as shallow and deep semantic parsing, respectively. It is reasonable to think that the shallow level of semantic knowledge in SRL is useful for deep semantic parsing.
Dependency Parsing DP aims to parse a sentence into a tree structure, which represents the dependency relation among tokens. The knowledge of DP is useful for AMR parsing, since: (1) Linguistically, DP (syntax parsing task) can be the precursor task of AMR (semantic parsing). (2) The dependency relation of DP is also related to semantic relation of AMR, e.g., as illustrated in Figure 1(c), "NSUBJ" in DP usually represents ".ARG0" in AMR. Actually, they both correspond to the agent-patient relations in the sentence. (3) DP is similar to AMR parsing from the perspective of edge prediction, because both of them need to capture the relation of nodes (tokens/concepts) in the sentence.
2.2 AMRization
Although SRL and DP are highly related to AMR parsing, there still exists gaps between them, e.g., SRL annotations may be disconnected, while AMR is always a connected graph. To bridge these gaps, we transform them into PseudoAMR, which we call AMRization.
*Computed on AMR 2.0 and 3.0 dataset.
2.2.1 Transform SRL to PseudoAMR
We summarize typical gaps between SRL and AMR as: (1) Connectivity. AMR is a connected directed graph while the structure of SRL is a forest. (2) Span-Concept Gap. Nodes in AMR graph represent concepts (e.g., "boy") while that of SRL are token spans (e.g., "the boy", "that boy"). Actually all the mentioned token spans correspond to the same concept. (3) Reentrancy. Reentrancy is an important feature of AMR as shown in Figure 3(a), the instance boy is referenced twice as ARG0. The feature can be applied to conduct coreference resolution. However, there is no reentrancy in SRL. To bridge such gaps, we propose Connectivity Formation, Argument Reduction and Reentrancy Restoration to transform SRL into PseudoAMR.
Connectivity Formation To address the connectivity gap, we need to merge all SRL trees into a connective graph. Note that the merging doesn't guarantee correctness in semantic level. As shown in Figure 3(b-1), we first add a virtual root node, then generating a directed edge from the virtual root to each root of SRL trees, thus the SRL annotation becomes a connected graph.
Argument Reduction To address the Span-Concept Gap, as shown in Figure 3(b-2), if the argument of current predicate is a span with more than one token, we will replace this span with its head token in its dependency structure. Thus token spans "the boy", "that boy" will be transformed to "boy", more similar to the corresponding concept. Similar method has been to applied by (Zhang et al., 2021) to find the head of token spans of argument.
Reentrancy Restoration For the reentrancy gap, we design a heuristic algorithm based on DFS to restore reentrancy in SRL. As shown in Figure 3(b-3), the core idea of the restoration is that we create a variable when the algorithm first sees a node. If the DFS procedure meets node with the same name, the destination of current edge will be redirected to the variable we have created at first. Please refer to Appendix A for the pseudo code of the reentrancy restoration.
Dependency Guided Restoration The previous restoration algorithm can not guarantee the merging of nodes agrees to the meaning of reentrancy in AMR since it merges concept according to their appearance order in the SRL structure. And it does not handle the merging of predicates. As shown
3 Predicate Merging
4 Argument Reduction
Figure 4: Illustration of Dependency Guided Restoration. In step 2, leaf-nodes "The boy" are merged. In step 3, none-leaf node "leave-01" is merged with leaf-node "to leave" since "leave-01" appears in word span "to leave" and word "leave" depends on word "want".
in Figure 3(b-3), the node "leave" and "leave-01" should be merged, however we can't get this information directly from the SRL annotations. We therefore propose another restoration method based on the dependency structure of the corresponding sentence of the SRL as illustrated in Figure 4
This restoration algorithm takes the result of previous Connectivity Formation as input. It first merges the leaf-nodes corresponding to the same token. This step is accurate since leaf-nodes' merging will not bring divergence. The second step is to merge predicate nodes. For all sub-trees of the root node, it first check whether one predicate appear in others' argument span and whether the predicate directly depend on the span's predicate. If both two conditions are satisfied, the algorithm will merge the predicate and the span to one node. Last, it removes the root node and root-edges if the graph remains connected after removing.
2.2.2 Transform Dependency Structure to PseudoAMR
We summarize the gaps between Dependency Tree and AMR as: (1) Redundant Relation. Some relations in dependency parsing focus on syntax, e.g., “:PUNCT” and “:DET”, which are far from semantic relations in AMR. (2) Token-Concept Gap. The basic element of dependency structure is token while that of AMR is the concept, which captures deeper syntax-independent semantics. We use Redundant Relation Removal and Token Lemmatization to transform the dependency structure to
PseudoAMR to handle the gaps.
Redundant Relation Removal For the Redundant Relation Gap, we remove some relations which are far from the sentence's semantics most of the time, such as "PUNCT" and "DET". As illustrated in Figure 3(c-1), by removing some relations of the dependence, the parsing result becomes more compact compared with original DP tree, forcing the model to ignore some semantics-unrelated tokens during seq2seq training.
Token Lemmatization As shown in Figure 3(c-2), for Token-Concept Gap, we conduct lemmatization on the node of dependency tree based on the observation that the affixes of single word do not affect the concept it corresponds to. Together with the smart-initialization (Bevilacqua et al., 2021) by setting the concept token's embedding as the average of the subword constituents, the embedding vector of lemmatized token ('want') becomes closer to the vector concept ('want-01') in the embedding matrix, therefore requiring the model to capture deeper semantic when conducting DP task.
2.2.3 Linearization
After all AMRization steps, the graph structure of SRL/DP also should be linearized before doing seq2seq training. As depicted in the right part of Figure 3, we linearize the graph by the DFS-based travel, and use special tokens  $< R0>$ , ...,  $< Rk>$  to indicate variables, and parentheses to mark the depth, which is the best AMR linearization method of Bevilacqua et al. (2021).
2.3 Training Paradigm Selection
After task selection and AMRization, we still need to choose an appropriate training paradigm to train PseudoAMR and AMR effectively. We explore three training paradigms as follows:
Multitask training Following Xu et al. (2020); Damonte and Monti (2021), we use classic schema in sequence-to-sequence multitask training by adding special task tag at the beginning of input sentence and training all tasks simultaneously. The validation of best model is conducted only on the AMR parsing sub-task.
Intermediate training Similar to Pruksachatkun et al. (2020), we first fine-tune the pretrained model on the intermediate task (PseudoAMR parsing), followed by fine-tuning on the target AMR parsing task under same training setting.
Multitask & Intermediate training We apply a joint paradigm to further explore how different paradigms affect AMR parsing. We first conduct multitask training, followed by fine-tuning on AMR parsing. Under this circumstance, Multitask training plays the role as the intermediate task.
AMR Datasets We conducted out experiment on two AMR benchmark datasets, AMR 2.0 and AMR 3.0. AMR2.0 contains 36521, 1368 and 1371 sentence-AMR pairs in training, validation and testing sets, respectively. AMR 3.0 has 55635, 1722 and 1898 sentence-AMR pairs for training validation and testing set, respectively. We also conducted experiments in out-of-distribution datasets (BIO,TLP,News3) and low-resources setting.
Auxiliary Task Datasets Apart from DP/SRL, we choose NLG tasks including summarization and translation to evaluate the contributions of auxiliary tasks. Description of datasets is listed Appendix C.
We use the Smatch scores (Cai and Knight, 2013) and further the break down scores (Damonte et al., 2017) to evaluate the performance.
To fully understand the aspects where auxiliary tasks improve AMR parsing, we divide the fine-grained scores to two categories: 1) Concept-Related including Concept, NER and Negation scores, which care more about concept centered prediction. 2) Topology-Related including Unlabeled, Reentrancy and SRL scores, which focus on edge and relation prediction. NoWSD and Wikification are listed as isolated scores because NoWSD is highly correlated with Smatch score and wikification relies on external entity linker system.
3.3 Experiment Setups
Model Setting We use current state-of-the-art Seq2Seq AMR Paring model SPRING (Bevilacqua et al., 2021) as our main baseline model and apply BART-Large (Lewis et al., 2020) as our pretrained model. Blink (Li et al., 2020) is used to add wiki tags to the predicted AMR graphs. We do not apply re-category methods and other post-processing methods are the same with Bevilacqua et al. (2021) to restore AMR from token sequence. Please refer to Section E from appendix for more training details.
Model	Extra Data	SMATCH	NoWSD	Wiki	Concept-related	Topology-related
Conc.	NER	Neg.	Unll.	Reen.	SRL
AMR 2.0	Cai and Lam (2020)	N	78.7	79.2	81.3	88.1	87.1	66.1	81.5	63.8	74.5
Fernandez Astudillo et al. (2020)	N	80.2	80.7	78.8	88.1	87.5	64.5	84.2	70.3	78.2
Zhou et al. (2021a)	70k	81.8	82.3	78.8	88.7	88.5	69.7	85.5	71.1	80.8
SPRING (Bevilacqua et al., 2021)	N	83.8	84.4	84.3	90.2	90.6	74.4	86.1	70.8	79.6
SPRING (w/ silver) (Bevilacqua et al., 2021)	200k	84.3	84.8	83.1	90.8	90.5	73.6	86.7	72.4	80.5
SPRING (Ours)	N	84.0	84.3	83.5	89.9	91.8	75.1	87.1	71.3	81.3
ATP (w/ DP)	40k	85.0	85.4	84.1	90.4	92.5	74.7	88.2	74.7	83.1
ATP (w/ SRL)	40k	85.1	85.6	83.6	90.4	91.4	75.7	88.2	75.0	83.5
ATP (w/ SRLD)	40k	85.2	85.6	84.2	90.7	93.1	74.9	88.3	74.7	83.3
Graphene 4SE (Lam et al., 2021)	200k	84.8	85.3	83.9	90.6	92.2	75.2	88.0	71.4	83.5
Structure-awareE(Zhou et al., 2021b)	47k	84.9	-	-	-	-	-	-	-	-
ATP (w/ SRL)E	40k	85.3	85.7	83.9	90.7	92.2	75.0	88.4	75.0	83.6
ATP (w/ SRLD)E	40k	85.3	85.7	84.0	90.8	92.7	74.7	88.4	75.1	83.6
AMR 3.0	Bevilacqua et al. (2021) (w/ silver)	200k	83.0	83.5	82.7	89.8	87.2	73.0	85.4	70.4	78.9
ATP (w/ DP)	40k	83.9	84.3	81.6	89.7	89.2	73.0	87.0	73.7	82.3
ATP (w/ SRL)	40k	83.9	84.3	81.0	89.7	88.4	73.9	87.0	73.9	82.5
Graphene 4SE (Lam et al., 2021)	200k	83.8	84.2	81.9	90.1	88.3	74.6	86.9	70.2	82.5
Structure-awareE(Zhou et al., 2021b)	47k	83.1	-	-	-	-	-	-	-	-
ATP (w/ SRL)E	40k	84.0	84.5	80.7	90.0	88.9	73.1	87.1	73.9	82.6
Table 1: SMATCH and fine-grained F1 scores on AMR 2.0 and 3.0.  ${}^{D}$  denotes model using Dependency Guided Restoration.  ${}^{E}$  denotes result with model ensemble (the details of the ensembling models are described in Appendix B). We conduct ensembling by averaging the models from three random seeds following Zhou et al. (2021b).
AMRization Setting For SRL, we explore four AMRization settings. 1) Trivial. Concept :multisentence and relation :snt are used to represent the virtual root and its edges to each of the SRL trees. 2) With Argument Reduction. We use dependency parser from Stanford CoreNLP Toolkit (Manning et al., 2014) to do the argument reduction. 3) With Reentrancy Restoration 4) All techniques.
For DP, we apply four AMRization settings 1) Trivial. Extra relations in dependency tree are added to the vocabulary of BART 2) With Lemmatization. We use NLTK (Bird, 2006) to conduct token lemmatization 3) With Redundant Relation Removal. We remove PUNCT, DET, MARK and ROOT relations. 4) All techniques.
We report the result (ITL + All AMRization Techniques) on benchmark AMR 2.0 and 3.0 in Table 1. On AMR 2.0, our models with DP or SRL as intermediate task gains consistent improvement over the SPRING model by a large margin (1.2 Smatch) and reach new state-of-the-art for single model (85.2 Smatch). Compared with SPRING with 200k extra data, our models achieve higher performance with much less extra data (40k vs. 200k), suggesting the effectiveness of our intermediate tasks. We also compare our models with contemporary work (Lam et al., 2021; Zhou et al., 2021b). It turns out that our ensemble model beats its counterpart with less extra data, reaching a higher performance (85.3
Smatch). In fact, even without ensembling, our model still performs better than those ensembling models and the model using Dependency Guided Restoration method achieves higher performance than the trivial one, showing the effectiveness of our methods.
On AMR 3.0, Our models consistently outperform other models under both single model (83.9 Smatch) and ensembling setting (84.0 Smatch). Same as AMR 2.0, our single model reaches higher Smatch score than those ensembling models, revealing the effectiveness of our proposed methods.
Fine-grained Performance To better analyse how the AMR parser benefits from the intermediate training and how different intermediate tasks affect the overall performance. We report the fine-grained score as shown in Table 1. We can tell that by incorporating intermediate tasks, considerable increases on most sub-metrics, especially on the Topology-related terms, are observed. On both AMR 2.0 and 3.0 our single model with SRL as intermediate task achieves the highest score in Unlabeled, Reentrancy and SRL metrics, suggesting that SRL intermediate task improves our parser's capability in Coreference and SRL.
DP leads to consistent improvement in topology-related metrics, which also derives better result on NER sub-task (92.5 on AMR 2.0, 89.2 on AMR 3.0). We suppose that the ".nn" relation which signifies multi-word name entities in dependency parsing helps the AMR parser recognize multi-word
Model	Extra	SMATCH	Conc.	Topo.
Ours (w/ NLG)
- w/ DialogSum	13k	84.5	85.5	81.5
- w/ CNNDM	40k	84.4	85.5	81.7
- w/ CNNDM	80k	84.2	85.1	81.4
- w/ EN-DE	40k	84.4	85.3	81.5
- w/ EN-DE	80k	84.4	85.4	81.4
- w/ EN-DE	200k	84.2	84.6	81.2
- w/ EN-DE	400k	83.6	84.9	80.6
Ours (w/ Parsing)
- w/ DP	40k	85.0	85.9	82.0
- w/ SRL	40k	85.1	85.8	82.2
Table 2: Result of Task Selection. We first train BART on different auxiliary tasks for 10 epochs before AMR Parsing. We also report the average scores of Concept-related (Conc.) and Topology-related metrics (Topo.)
name entities. Generally speaking, AMR parser gains large improvement in Topology-related subtasks and NER by incorporating our intermediate tasks in terms of the Smatch scores.
3.5 Exploration in Auxiliary Task Selection
We explore how different tasks affect AMR parsing apart from DP and SRL. We involve two classic conditional NLG tasks, Summarization and Translation for comparison as shown in Table 2.