AMR Parsing with Causal Hierarchical Attention and Pointers
Authors: Chao Lou
Kewei Tu
Venue: EMNLP 2023 Main (2023)
Keywords: semantic parsing
AMR parsing
hierarical attention
pointer mechanism

Abstract: Translation-based AMR parsers have recently gained popularity due to their simplicity and effectiveness. They predict linearized graphs as free texts, avoiding explicit structure modeling. However, this simplicity neglects structural locality in AMR graphs and introduces unnecessary tokens to represent coreferences. In this paper, we introduce new target forms of AMR parsing and a novel model, CHAP, which is equipped with causal hierarchical attention and the pointer mechanism, enabling the integration of structures into the Transformer decoder. We empirically explore various alternative modeling options. Experiments show that our model outperforms baseline models on four out of five benchmarks in the setting of no additional data.

AMR Parsing with Causal Hierarchical Attention and Pointers
Chao Lou, Kewei Tu
School of Information Science and Technology, ShanghaiTech University Shanghai Engineering Research Center of Intelligent Vision and Imaging {louchao, tukw))@sanghaii tech. edu. cn
Abstract
even superior performance (Konstas et al., 2017: Xu al., 2020; Bevilacqua et al., 2021; Lee et al., 2023). This finding has spurred a wave of sub- sequent efforts to design more effective training strategies that maximize the potential of pretrained decoders (Bai et al., 2022; Cheng et al., 2022; Wang et al., 2022; Chen et al., 2022), thereby sidelining the exploration of more suitable de- coders for graph generation. Contrary to preceding translation based models, we contend that explicit structure modeling within pretrained decoders re- mains beneficial in AMR parsing. To our knowl- edge, the Ancestor parser (Yu and Gildea, 2022) is the only translation-basee model contributing to explicit structure modeling, which introduces shortcuts to access ancestors in the graph. How- ever, AMR graphs contain more information than just ancestors, such siblings and coreferences. resulting in suboptimal modeling.
Translation- AMR parsers have recently gained popularity due to their simplicity and effectiveness. They predict linearized graphs free texts, avoiding explicit structure mod- eling. However, this simplicity neglects struc- tural locality in AMR graphs and introduces unnecessary tokens to represent coreferences. this paper, we introduce new target forms of AMR parsing and novel model, CHAP, which is equipped with gausal hierarchical attention and the pointer mechanism, enabling the inte- gration of structures into the Transformer de- coder. We empirically explore various alter- native modeling options. Experiments show that our model outperforms haseline models on four out of five benchmarks in the setting of no additional data.
1 Introduction
Abstract Mcaning Representation (Banarescu et al., 2013) is a semantic representation of natural lan- guage sentences typically depicted directed acyelic graphs, illustrated in Fig. 1a. This rep- resentation is both readable and broad-cov,rage, attracting considerable research attention across various domains, including information extrac- tion (Zhang and Ji, 2021; Xu et al., 2022), sum- marization (Hardy and Vlachos, 2018; Liao et al., 2018), and vision-language understanding (Schus- ter et al., 2015; Choi et al., 2022). However, the inherent flexibility of graph structures makes AMR parsing i.e., translating natural language sentences into AMR graphs, a challenging task.
In this paper, we propose CHAP, novel translation-based AMR parser distinguished by three innovations. Firstly, we introduce new tar- get forms of AMR parsing. As demonstrated Tab. 1.c-e, we use multiple layers to capture dif- ferent semantics, such that each layer is simple and concise. Particularly, the base layer, which en- capsulates all meanings except for coreferences (or reentrancies), is tree-structured representation, abling more convenient structure modeling than the graph structure of AMR. Meanwhile, coreferences are presented through pointers, circumventing sev- eral shortcomings associated with the variable- based coreference representation (See Sec. 3 for more details) used in all previous translation- based models. Secondly, we propose Causal Hierarchical Attention (CHA), the core mechanism of our incre- mental structure modeling, inspired by Transformer Grammars (Sartran et al., 2022). CHA describes a procedure of continuously composing child nodes to their parent nodes and encoding new nodes with all uncomposed nodes, as illustrated Fig. 2. Un-
The development of AMR parsers has been boosted by recent research on pretrained sequence- to-sequence (seq2seq) models. Several studies, cat- egorized as translation-basel models, show that fine-tuning pretrained seq2seq models to predict linearized graphs as if they are free texts (e.g.. ex- amples in 1ab.1.ab) can achieve competitive or
*oorresponding Author

---

like 01
ID
Name
Representation
ungo
arga
PM
( / alpha :argē ( b beta :argl ( ( ganma :arg2 b
porson
tour-O1
b S-DFS <RG> alpha :agg ( <R1> beta :argl <R2> ganma
:arg2 <R1>
and
argl
I sgl.
alpha vagg ( beta ) :argt
garma
:arg2 beta
coiploy-01
I dbl.
( alpha targe beta ) )> argg ( garma
:arg2 beta )1 )> )1 )2
(a) Standard AMR graph
ft
like-01
alpha arggo beta argg ganma
:arg2 beta
orgo
argi
person
Table Graph representations. PM and S-DFS denote the PENMAN form and the SPRINGprS form, the DFS-base linearization proposed by Bevilacqua al.
ang I-al
anpfi
any
(2021), respectively. (c)-() are our proposed representations. (c) is for Usingle(c) (Fig. 3c). (d) is for (double(c) (Fig. 3b). (e) for i (Fig. 3d), Red pointers 4,
employ-01
person
city
(b) Replicate nodes and add coref pointers (red).
which represent coreferences, constitute the coref layer. Blue pointerst. which point
to the left boundaries of subtrees, constitute the auxiliary structure layer. Texts,
which encapsulate all other meanings, constitute the base layer.
( / like-01 :argo (
/ person :argt-of employ-01 :arg1 (
Property
Translation-basse Transition-based
Factor-based
Ours
/ tour-01 :arga p :arg1 c city ) )
Trainability
require alignments
/
Structure modeling
X
/
(c) PENMAN notation
Pretrained decoder
x
/
Variable-frce
/
/
Figure AMR of Employees
liked their city tour.
Table 2: Strengths, shortcomings of different types of models.
like the causal attention in translation-based mod- cls, which allows a token to interact with all its preceding tokens, CHA incorporates a strong in- ductive bias of recursion, composition, and graph topology. Thirdly, deriving from transition- AMR parsers (Zhou et al., 2021a.b). we introduce pointer encoder for encoding histories and a pointer net for predicting coreferences, which is proven be an effective solution for generalizing to a variable-size output space (Vinyals et al., 2015; See et al., 2017).
2020; Zhou et al., 2021a,b) and translation-based models (Konstas et al., 2017; Xu et al., 2020; Bevilacqua et al., 2021; Lee et al., 2023) epito- mize local models as they are trained with teacher forcing, optimizing only next-step predictions, and rely on greedy decoding algorithms, such as greedy search and beam search. Particularly, transition- based models predict actions permitted by transi- tion system, while translation-based models pre- diet AMR graph tokens as free texts. Some factorization- based models are also local (Cai and Lam, 2019, 2020), sequentially composing sub- graphs into bigger ones. We discern differences in four properties among previous local models and model in Tab. 2:
We propose various alternative modeling op- tions of CHA and strategies for integrating CHA with existing pretrained seq2seq models and in- vestigate them via extensive experiments. Ulti- mately, our model CHAP achieves superior per- formance on two in-distribution and three out-of- distribution benchmarks. Our code is available at amm_ammrpaa parser.
Trainability Whether additional information is required for training. Transition- based models rely word-node alignment to define the gold action sequence.
2
Related Work
Structure modeling Whether structures mod- eled explicitly in the decoder. Transition-basee models encode action histories like texts without considering graph structures. Besides, translation- based models opt for compatibility with pretrained decoders, prioritizing this over explicit structure
2.1 AMR Parsing
Most recent AMR parsing models generate AMR graphs via a series of local decisions. Transition- based models (Ballesteros and Al-Onaizan, 2017;
Naseem et al., 2019; Fernandez. Astudillo et al.,

---

alpha
alpha
al alpha
alpha
argO
argO
argO
ang
argO
arg
beta
beta
beta
gamma
beta
gamma
argO
arg
argO
argl
argO
argl
angO
argl
argO
argl
delta
epsilon
delta
epsilon
delta
epsilon
delta
epsilon
zeta
eta
theta
(a) Expand
(b) Compose
(c) Expand
(d) Expand
( (alpha :arg beta :argO (delta) targ
(epsilon
argl
gamma (6)
targO (zeta targO (cta) zrgg
theta
(d)
(c) The base layer in the Usingle form.
Figure 2: Demonstration of Causal Hierarchical Attention. We draw the aggregation on graphs performed at four steps in (a)-(d) and highlight the corresponding token for each step in (e) with green boxes. the The generation order is depth-first and left-to-right: alpha -> beta - del ta + epsilon -> gamma -> zeta -> eta -> theta. The node interest at each step highlighted blue, gathering information from all solid nodes. Gray dashed nodes, on the other hand, are invisible.
modeling.
to as ddouble in this paper and we will present tech- nical details in Sec. 3.3 along with other variants.
Pretrained decoder coders can be leveraged.
Whether pretrained de-
Structure Modeling
Variable-free Whether there are variable tokens in the target representation. Transition-basee mod- cls, factorization-based models and ours generate coreference pointers, obviating the need of intro- ducing variables.
We primarily highlight two advantages of incorpo- rating structured modeling into the decoder. Firstly, the sequential order and adjacence of previous lin- carized form mismatch the locality of real graph structures, making the Transformer decoder hard to understand graph data. Specifically, adjacent nodes in an AMR graph exhibit strong semantic relationships, but they could be distant in the lin- earized form (e.g.. person and tour-01 in Fig. la). Conversely, tokens closely positioned in the lin- earized form may be far apart in the AMR graph (e.g., employ-01 and tour-01 in Fig. Ia). Sec- ondly, previous models embed variables into the lincarized form (e.g.. b in Tab. La and <R1> in Tab. 1.b) and represent coreferences (or reentran- cies) by reusing the same variables. However, the literal value of variables is inconsequential. For ex- ample, in the PENMAN form, (a alpha :argo (b / beta)) conveys the same meaning (n1 / alpha :arg0 (n2 beta)). Furthermore, the usage of variables brings up problems regard- ing generalization (Wong and Mooney, 2007; Poel- man et al., 2022). For instance, in the SPRINGDES form, <RØ> invariably comes first and appears in all training samples, while <R1 00> is considerably less frequent.
2.2 Transformer Grammar
Transformer Grammars (TGs; Sartran et al., 2022) are a novel class of language models that simulta- ncously generate sentences and constituency parse trees, in the fashion of transition-based parsing. The base layer of Tab. 1.c can be viewed as an example action sequence. There are three types of actions in TG: (1) the token "(" represents the action ONT, opening nonterminal; (2) the token ")" represents the action CNT, closing the nearest open nonterminal; and (3) all other tokens (e.g., and argg) represent the action generating a ter- minal. TG carries out top-down generation, where a nonterminal is allocated before its children. We will also explore bottom-up variant in Sec. 3.4. Several studies have already attempted to generate syntax-augments sequences (Aharoni and Gold- berg, 2017: Qian et al., 2021). However, TG differ- entiates itself from prior research through its unique simulation of stack operations in transition-bant based parsing, which implemented by enforcing a spe- cific instance of CHA. A TG-like CHA is referred

---

X2
x3
child node, and 3.4.
x1
3.2 Causal Hierarchical Attention
ROOT
<
Causal Hierarchical Attention (CHA) is situated in the decoder and maintains structures during gen- eration. For each token, CHA performs one of the two actions, namely compose and expand, demonstrated in Fig. The compose operation is performed once all children a parent node have been generated. It aggregates these children to obtain a comprehensive representation of the subtree under the parent node, subsequently set- ting the children invisible in future attention. On the other hand, the expand operation aggregates all visible tokens derive the representation of the subsequent token.
x2
x1
X3
x2
x3
(a) An example tree.
denotes non-leaf node.
(b) Udouble
x2
x3
x1
X2
x3
x3
We note the subtle distinction between the term parent in the DAG representation (e.g.. Fig. la) and in target forms (e.g.. Tab. and Fig. 3a). Recall that TG uses "(" (ONT) indicate minal, which is the parent node of all following tokens before a matched ")" (CNT). This implies that alpha Tab, 1.c considered
(c) Usingle
(d) 1
Figure 3: The target forms and the attention mask of the three variants of CHA for the tree in (a). Orange cells represent the compose operation, while blue cells with plaid represent the expand operation. White cells are masked out. The vertical and horizontal axis represent attending and attended tokens, respectively.
nonter-
being sibling of argo, rather than a parent node governing them. This discrepancy does not impact our modeling because we can treat labels of non- terminals as particular type of child nodes, which are absorbed into the parent node when drawing the DAG representation. The tree of target forms are illustrated in Appx. A.1.
3.1 Multi-layer Target Form
As shown in Tab. 1.c, we incorporate new layer, named the coreference (coref) layer, on top of the conventional one produced by DFS lincarization, named the base layer'. The coref layer serves to represent coreferences, in which a pointer points from mention its nearest preceding mention, and the base layer encapsulates other meanings. From a graph perspective, a referent is replicated to new nodes with an amount equal to the reference count that are linked by newly introduced coref pointers, as illustrated in Fig. 1b. We argue that forms more promising because forms can avoid meaningless tokens (i.e., variables) from cluttering up the base layer, yielding several bene- ficial byproducts: (1) shortens the representation length; (2) aligns the representation more closely with natural language; and (3) allows the base layer to be interpreted as trees, a vital characteristic for our structure modeling.
The two actions can be implemented by modi- fying attention masks conveniently. Specially, the compose operation masks out attention to tokens that are not destined to be composed, as depicted in the fourth row of Fig. 3c. Moreover, the expand operation masks out attention to tokens that have been composed in previous steps, as depicted in the top three rows and the fifth row Fig. 3c.
In subsequent sections, we will explore two classes of generation procedures that utilize dif- ferent target forms and variants of CHA, akin top-down (Dyer et al., 2016; Nguyen al., 2021; Sartran et al., 2022) and bottom-up (Yang and Tu, 2022) parsing.
3.3 Top-down generation
Most prior studies utilize paired brackets de- note nested structures, as shown in Tab. l.add Sec- tion 2.2 outlines that, in left-to-right decoding, due to the prefix "(", this type of representation results in top-down tree generation.
Tab. l.d and are two variants Tab. 1.c. These three forms are designed to support different vari- ants of CHA, which will be introduced in Sec. 3.3
We use the DFS order provided in the AMR datascts.

---

We consider two modeling options of top-down generation, Usingle (Fig. 3c and Udouble (Fig. 3b), varying on actions triggered by ">". More precisely, upon seeing a ")", Jsingle executes a compose, whereas Ildobble executes an additional expand af- ter the compose. For other tokens, both Usingle and Ildouble execute an expand operation. Because the decoder performs one attention for each token, in Udoubll, each ")" is duplicated to represent pose and expand respectively, i.e., ")" becomes ")1 )z". We detail the procedure of generating MAнa for Usingle in Alg. The procedure for ddouble can be found in Sartran et al.'s (2022) Alg. I.
Algorithm 1: MCнa for Usingle.
Data: sequence of token t with length N
Result: attention mask Mcha E RNX $ 5 []
>> Empty stack
MCHA f -00
for < 1 to N do
if t1] = ')' then
b compose
while {li] + '( do MCHA [ij] 0 j { S.pop() end
McHA + 0 S.push(i)
The motivation for the two variants is follows. strict leaf-to-root information aggregation pro- cedure, which is adopted in many studies on tree encoding (Tai et al., 2015; Drozdov et al., 2019; Hu et al., 2021; Zhou et al., 2022), a parent node only aggregates information from its children, re- maining unaware of other generated structures (e.g.. beta is unaware of alpha in Fig. 2b). However, when new nodes are being expanded, utilizing all available information could be a more reasonable approach (e.g., gamma in Fig. 2c). Thus, an expand process is introduced to handle this task. The sit- uation with CHA becomes more flexible. Recall that all child nodes are encoded with the expand ac- tion, which aggregates information from all visible nodes, such that information of non-child nodes is leaked the parent node during composition. Usingle relies on the neural network's capability to encode all necessary information through this leakage, while double employs an explicit expand to allow models to directly revisit their histories.
else
S.push(i) for j E S do McHa[tj] <-
t> expand
end
end end return MCHA
but we utilize pointers in place of left brackets to determine the left boundaries of subtrees. The ex- act procedure can be found in Appx. B.
Parsing Model
Our parser is based on BART (Lewis et al., 2020), pretrained seq2seq model. We make three mod- ifications BART: (1) we add a new module in the decoder to encode generated pointers, (2) we enhance decoder layers with CHA, and (3) we use the pointer net to predict pointers.
3.4 Bottom-up generation
4.1 Encoding Pointers
In the bottom-up generation, the parent node is al- located after all child nodes have been generated. This process enables the model review all yet-to- be-composed tokens before deciding which ones should be composed into subtree, in contrast to the top-down generation, where the model is re- quired to predict the existence of a parent node without seeing its children. The corresponding target form, as illustrated in Tab. l.e, contains no brackets. Instead, special token is placed after the rightmost child node of each parent node, with a pointer pointing the lefumost child node. We execute the compose operation for and the ex- pand operation for other tokens. The generation of the attention mask (Fig. 3d) is analogous to Usingle,
The target form can be represented as tuple of (t, p)2, where and p are the sequence of the basc layer and the coref layer, respectively, such that each pt is the index of the pointed token. We define P4 = if there is no pointer at index i. In the BART model, is encoded using the token embedding. However, no suitable module exists for encoding p. To address this issue, we introduce multi-layer perceptron, denoted ML.,,, which takes the token and position embeddings of the pointed tokens and then outputs the embedding
For the sake of simplicity, we only discuss the target form of top-down generation. The additional struet layer in the hottom-up generation can be modeled similarly to p.

---

of p. Notably, if p; 1, the embedding is set O. All embeddings, including that of t, p and positions, are added together before being fed into subsequential modules.
where a scalar hyperparameter. For decoding, the probability of a hypothesis is the product the probabilities of the base layer, the coref sequence, and the optional struct layer. We enforce a constraint during decoding to ensure the validity of MCHA: the number of ) should not surpass the number of (, and two constraints ensure the well-formedness of pointer: (1) coref- erence pointers can only point to positions with the same token, and (2) left boundary pointers in bottom-up generation cannot point to AMR rela- tions (e.g., ARGO).
4.2 Augmenting Decoder with CHA
We explore three ways to integrate CHA in the de- coder layer, as shown in Fig. 4. The inplace archi- tecture replaces the attention mask of some atten- tion heads with MCHA in the original self-attention module without introducing new parameters. How- ever, this affects the normal functioning of the re- placed heads such that the pretrained model is dis- rupted.
Experiment
Alternatively, we can introduce adapters into de- coder layers (Houlsby et al., 2019). In the parallel architecture, an adapter is introduced in parallel to the original self-attention module. In contrast, adapter is positioned subsequent to the original module in the pipeline architecture. Our adapter is defined as follows:
5.1 Setup
Datasets We conduct experiments on in- distribution benchmarks: (1) AMR 2.0 (Knight et al., 2017), which contains 36521, 1368 and 1371 samples in the training, development and test set, and (2) AMR 3.0 (Knight et al., 2020), which has 55635, 1722 and 1898 samples in the train- ing, development and test set, as well three out- of-distribution benchmarks: (1) The Little Prince (TLP), (2) BioAMR and (3) New3. Besides, we also explore the effects of using silver training data following previous work. To obtain silver data, we sample 200k sentences from the One Billion Word Benchmark data (Chelba et al., 2014) and use a trained CHAP parser to annotate AMR graphs.
21 FFN, (h₂);
22 R,, W K 3ı, WVn. Fı, McnA), x2)),
h. FFN,
where Wº, W Wy are query/key/value projec- tion matrices, FFN1/FFF are down/up projection, h4 is the input hidden states and h, the output hidden states.
4.3 Predicting Pointer
Metrics We report the Smatch score (Cai and Knight, 2013) and other fine- grained metrics (Da- monte et al., 2017) averaged over three runs with different random seeds?. All these metries are in- variant different graph linearizations and exhibit better performance when they are higher. Addition- ally, to provide a more accurate comparison, we include the standard deviation (std dev) if Smatch scores are close.
Following previous work (Vinyals et al., 2015; Zhou et al., 2021b), we reinterpret decoder self- attention heads as a pointer net. However, unlike the previous work, we use the average attention probabilities from multiple heads as the pointer probabilities instead of relying Our preliminary experiments indicate that this mod- ification results slight improvement.
single head.
cross-entropy loss between the predicted pointer probabilities and the ground truth point- ers is used for training. We disregard the associated loss at positions that do not have pointers and ex- clude their probabilities when calculating the entire pointer sequence's 's probability.
Pre-Ipost-processing Owing to the sparsity of wiki tags* in the training set, we follow previ- ous work to remove wiki tags from AMR graphs in the pre- pre-processing, and use the BLINK entity linker (Wu al., 2020) to add wiki tags in the post- processing'. In the post-processing, we also use
4.4 Training and Inference
"We the ain--vallution-enhaaree software pute scores, which is available https: // gith Thunchuattv/arr-evvlluut
We optimize the sum of the standard sequence gen- eration loss and the pointer loss:
tool-enhanced.
gith com amr amrisi amr guidelines/ blob/maste aabbmastte/anmr wwHHamed-entttiee
"https: : /
L Treg2seg a Ipointer:
*We do not add wiki tags in analytical experiments.

---

Premained decoder layer
Forward
Premeinno daas Layor
Feed Forward
Food Forward
Pretriieed decoder
Feed Furward
Prociim Sapperr
emsss-attnntion
Feed Forward
cross-atlention
CHA
CHA
cross-attention
Causal head
adapter
sell--ateettinn
Feed Forward
self-attention
Food Forward
self-attention
CHA head
(a) Parallel
(b) Pipeline
(c) Inplace
Figure 4: Three architectures applying CHA to pretrained decoder layers. Residual connections layernorms omitted.
the lib software to ensure graph validity.
CHA
Smatch
Usingle
82.60
Implementation details We use the BART-base model in analytical experiments and the BART- large model in comparison with baselines. We mod- ify all decoder layers when using the BART-base model, while only modifying the top two layers when using the BART-large model?. For the paral- lel and pipeline architectures, attention modules in adapters have four heads and a hidden size 512. For the inplace architecture, four attention heads are set to perform CHA. We reinterpret four self-attention heads of the top decoder layer weight for the pointer loss is set to 0.075. We use zero initialization for FFN2 and MLP,, such that the modified models are equivalent to the original BART model the beginning of training. More details are available at Appx.
expand -> causal
82.53
compose + expand
82.47
doubbl
82.63
f
82.57
Table The influence of different CHA.
Architecture
Smatch
std dev
Parallel
82.63
0.02
Pipeline
82.59
0.05
pointer net. The
Inplace
82.43
0.04
w/o CHA
82.38
0.12
Table 4: The influence of different architectures.
5.2 Results on Alternative Modeling Options
Baselines SPRING (Bevilacqua et al., 2021) is
Structural modeling We report the results of dif- ferent CHA options in Tab. 3. Jldouble exhibits a slightly better performance than ft and 4single. Be- sides, find that breaking structural localities, i.c.. (I) allowing parent nodes to attend nodes other than their immediate children (row 00.13) and (2) allowing non-parent nodes to attend to nodes that have been composed (row 2, -0.07), nega- tively impacts the performance. We present the attention masks of these two cases in Appx. A.2.
BART model fine-tuned with augmented cabulary and improved graph representations (as shown in Tab. 1.b). Ancestor (Yu and Gildea, 2022) enhances the decoder of SPRING by in- corporating ancestral information of graph nodes. BiBL (Cheng et al., 2022) and AMRBART (Bai et al.. 2022) augment SPRING with supplemen- tary training losses. LeakDistill (Vasylenko et al., 2023)8 trains a SPRING using leaked information and then distills it into standard SPRING.
All these baselines are translation- based models. Transition- bascd and factorization- based models are not included due to their inferior performance.
Architecture Tab. 4, we can see that the in- place architecture has little improvement over the baseline, w/o CHA. This suggests that changing the functions of pretrained heads can be harmful. We also observe that the parallel architecture performs slightly better than the pipeline architecture.
"https: //gi cww/bjascob/ amrlib "The training becomes unstable modify decoder layers the BART-large model. *Contemporary work.
Based on the above results, we present CHAP, which adopts the parallel adapter and uses [Idouble.

---

Model
Extra Data
Smatch NoWSD Wiki. Conc.
NER Neg.
Unlab Reent. SRL
AMR 2.0
SPRING
83.8
84.4
84.3
90.2
90.6
74.4
86.1
70.8
79.6
Ancestor
84.8
85.3
84.1
90.5
91.8
74.0
88.1
75.1
83.4
BiBL
84.6
85.1
83.6
90.3
92.5
73.9
87.8
74.4
83.1
LeakDistill
85.7
86.2
83.9
91.0
91.1
76.8
88.6
74.2
81.8
CHAP (ours)
85.1
85.6
86.4
90.9
90.4
73.4
88.0
73.0
81.0
AMRBART
200K
85.4
85.8
81.4
91.2
91.5
74.0
88.3
73.5
81.5
LeakDistill
A, 140K
86.1
86.5
83.9
91.4
91.6
76.6
88.8
75.1
82.4
CHAP (ours)
200K
85.8
86.1
86.3
91.4
80.4
78.3
88.6
73.9
81.8
AMR 3.0
SPRING
83.0
83.5
82.7
89.8
87.2
73.0
85.4
70.4
78.9
Ancestor
83.5
84.0
81.5
89.5
88.9
72.6
86.6
74.2
82.2
BiBL
83.9
84.3
83.7
89.8
93.2
68.1
87.2
73.8
81.9
LeakDistill
A
84.5
84.9
80.7
90.5
88.5
73.7
87.5
73.1
80.7
CHAP (ours)
84.4"
84.8
84.7
90.5
87.9
73.5
87.3
72.6
80.1
AMRBART
200K
84.2
84.6
78.9
90.2
88.5
72.1
87.1
72.4
80.3
LeakDistill
A, 140K
84.6
84.9
81.3
90.7
87.8
73.0
87.5
73.4
80.9
CHAP (ours)
200K
84.6
85.0
84.5
90.7
88.4
75.2
87.5
73.1
80.7
Table 5: Fine-grained Smatch scores on in-domain benehmarks. Bold and underlined numbers represent the best and the second-best results, respectively. "A" the Extra Data column denotes alignment. *Std 0.04.
Model
Extra Data
TLP
Bio
New3
AMR 3.0 and three for AMR 2.0. Compared to pre- vious work, which uses alignment, CHAP matches LeakDistill AMR 3.0 but falls behind it on AMR 2.0. One possible reason is that alignment addi- tional data is particularly valuable for a relatively small training set of AMR 2.0. We note that the contribution LeakDistill is orthogonal to ours, and we can expect an enhanced performance by in- tegrating their method with our parser. When using silver data, the performance of CHAP on AMR 2.0 can be significantly improved, achieving similar performance to LeakDistill. This result supports the above conjecture. However, on AMR 3.0, the gain from silver data is marginal as in previous work, possibly because AMR 3.0 sufficiently large to train a model based on BART-large.
SPRING
77.3
59.7
73.7
BiBL
78.6
61.0
75.4
BiBL
200K
78.3
61.1
75.4
AMRBART
200K
76.9
63.2
76.9
LeakDistill
A, 140K
82.6
64.5
CHAP (ours)
79.0
62.7
74.8
CHAP (ours)
200K
79.8
63.5
75.1
CHAP (ours)
81.8
65.1
CHAP (ours)
200K
82.7a
66.1
Table Test results on out-of-distribution benchmarks. The scores represented in grey cells derive from model trained AMR 2.0, whereas the remaining scores come from a model trained on AMR 3.0. -: New3 is part of AMR 3.0, these settings excluded from OOD evaluation. "Std dev on TLP is 041 #Std dev on Bio is 0.46.
In out-of out-of--istribbee distrribbttoon evaluation, CHAP is com- petitive with all baselines both TLP and Bio. shown in Tab. 6, indicating CHAP's strong ability of generalization thanks to the explicit structure modeling.
5.3 Main Results
Tab. 5 shows results in-distribution benchmarks. In the setting of no additional data (such that LeakDistill excluded), CHAP outperforms all previous models by 0.3 Smatch score on AMR 2.0 and 0.5 on AMR 3.0. Regarding fine-grained metrics, CHAP performs best on five metrics for
5.4 Ablation Study
An ablation study is presented in Table 7. The first four rows demonstrate that, for model based on BART-base, when we exclude the encoding pointers, the CHA adapters, and the separation

---

Base AMR3
Large
sentation Structures. In these tasks, prior research often employs the PENMAN notation as the tar- get form. These methodologies could also poten- tially benefit from our innovative target form and structural modeling. Although we do not conduct experiments within these tasks, results garnered from broader range of tasks could provide more compelling conclusions.
EP Adapter CL
AMR3
TLP
Bio
CHA
82.91
84.44
81.8
65.1
X
CHA
82.82
84.17
81.7
64.8
X
82.75
84.28
CHA
X
82.63
84.09
82.1
64.6
Causal
X
82.44
83.79
x
X
82.38
83.84
81.6
64.3
Acknowledgments
Table 7: Ablation study. EP: Encoding Pointers. CL: Coreference Layer. Base(Large): BART-base(large).
We thank the anonymous reviewers for their con- structive comments. This work was supported by the National Natural Science Foundation of China (61 (61976139).
of coreferences on AMR 3.0, there are declines in Smatch scores of -0.09, -0.16, and -0228, respectively and 0.277, -0.16, and -0.35 for a model based on Haree Additionally, we sub- stitute CHA in adapters with standard causal atten- tion to ascertain whether the improvement mainly arises from an increased parameter count. From the last three rows, it is evident that CHA has a greater contribution (+0.25) than adding parame- ters (++0.06).
References
Roee Aharoni and Yoav Goldberg. 2017. Towards string-to-tret neural machine translation. In Proceed- ings of the 55th Annual Meeting of the Association for Computational Linguisties (Volume 2: Short Papers), pages 132-140, Vancouver, Canada. Association for Computational Linguistics.
The two rightmost columns in Tab. present the ablation study results on OOD benchmarks. We find that the three proposed components contribute variably across different benchmarks. Specifically, CHA consistently enhances generalization, while the other two components slightly reduce perfor- mance on TLP.
Xuefeng Bai, Yulong Chen, and Yue Zhang. 2022. Graph pre-training for AMR parsing and generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume
1: Long Papers), pages 6001-6015, Dublin, Ireland. Association for Computational Linguisties.
Miguel Ballesteros and Yaser Al-Onaizan. 2017. AMR parsing using stack-LSTMs. In Proceedings of the 2017 Conference on Empirical Methods in Natu- ral Language Processing, pages 1269-1275, Copen- hagen, Denmark. Association for Computational Lin- guistics.
6 Conclusion
We have presented an AMR parser with new target forms of AMR graphs, explicit structure modeling with causal hierarchical attention, and the integra- tion of pointer nets. We discussed and empirically compared multiple modeling options of CHA and the integration of CHA with the Transformer de- coder. Eventually, CHAP outperforms all previous models in-distribution benchmarks in the set- ting of no additional data, indicating the benefit of structure modeling.
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract Meaning Representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Dis- course, pages 178,186, Sofia, Bulgaria. Association for Computational Linguistics.
Michele Bevilacqua. Rexhina Blloshmi, and Roberto Navigli. 2021. One spring to rule them both: Sym- metric amr semantic parsing and generation without complex pipeline. Proceedings of the AAAI Confer- ence on Artificial Intelligence, 35(14):12564-122
Limitations
We focus exclusively on training our models to pre- dict AMR graphs using natural language sentences. Nevertheless, various studies suggest incorporating additional training objectives and strategies, such BiBL and LeakDistill, enhance performance. These methods also can be applied to our model.
Deng Cai and Wai Lam. 2019. Core semantic first: A top-down approach for AMR parsing. In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Join Conference on Natural Language Pro- cessing TEMNLP-IJCNLL pages 3799-3809, Hong Kong, China. Association for Computational Linguis- tics.
There exist numerous other paradigms for se- mantic graph parsing, including Discourse Repre-

---

Deng Cai and Wai Lam, 2020. AMR parsing via graph- sequence iterative inference. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages 1290-1301, Online. Asso- ciation for Computational Linguisties.
Technologies. pages 199-209, San Diego. California. Association for Computational Linguistics.
Ramón Fernandez Astudillo, Miguel Ballesteros, Tahira Naseem, Austin Blodgett, and Radu Florian. 2020. Transition-base parsing with stack-transformers. In Findings of the Association for Computational Lin- guistics: EMNLP 2020, pages 1001-1007, Online. Association for Computational Linguistics.
Shu Cai and Kevin Knight. 2013. Smatch: an evaluation metric for semantic feature structures. In Proceed- ings of the S1st Annual Meeting of the Association for Computational Linguisties (Volume 2: Short Pa- pers). pages 148-752, Sofia, Bulgaria. Association for Computational Linguisties.
Hardy Hardy and Andreas Vlachos. 2018. Guided neu- ral language generation for abstractive summariza- tion using Abstract Meaning Representation. In Pro- ceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing, pages 768-773, Brussels, Belgium. Association for Computational Linguistics.
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants. Phillipp Kochn, and Tony Robinson. 2014. One billion word benehmark for measuring progress in statistical language modeling.
Liang Chen, Peiyi Wang, Runxin Xu, Tianyu Liu, Zhi- fang Sui, and Baobao Chang. 2022. ATP: AMRize then parse! enhancing AMR parsing with Pseu- doAMRs. In Findings of the Association for Compu- lational Linguistics: NAACL 2022, pages 2482-2496, Seattle, United States. Association for Computational Linguistics.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-sfficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR.
Ziming Cheng, Zuchao Li, and Hai Zhao. 2022. BiBL: AMR parsing and generation with bidirectional Bayesian learning. In Proceedings of the 29th Inter- national Conference on Computational Linguistics, pages 5461-5475, Gyeongju, Republic of Korea. In- ternational Committee on Computational Linguistics.
Xiang Hu, Haitao Mi, Zujie Wen, Yafang Wang. Yi Su, Jing Zheng, and Gerard de Melo. 2021. R2D2: Re- cursive transformer based on differentiable tree for interpretable hicrarchical language modeling. In Pro- ceedings of the 59th Annual Meeting of the Associa- tion for Computational Linguistics and the 1 11th Inter- national Joint Conference on Natural Language Pro- cessing (Volume 1: Long Papers), pages 1897-4908. Online. Association for Computational Linguistics.
Woo Suk Choi, Yu-Jung Heo, Dharani Punithan, and Byoung-Tak Zhang. 2022. Scene graph parsing via Abstract Meaning Representation in pre-trained lan- guage models. In Proceedings of the 2nd Workshop on Deep Learning on Gruphs for Natural Language Processing (DI.G4NIPP 2022), pages 30-35, Seattle, Washington. Association for Computational Linguis- tics.
Kevin Knight, Bianca Badarau, Laura Baranescu, Claire Bonial, Madalina Bardocz, Kira Griffitt, Ulf Herm- jakob, Daniel Marcu, Martha Palmer, Tim O' Gorman, and Nathan Schneider. 2017. Abstract Meaning Rep- resentation (AMR) Annotation Release 2.0.
Marco Damonte, Shay B. Cohen, and Giorgio Satta. 2017. An incremental parser for Abstract Meaning Representation. In Proceedings of the 15th Con- ference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Pa- pers, pages 536-546, Valencia, Spain. Association for Computational Linguistics.
Kevin Knight, Bianca Badarau, Laura Baranescu, Claire Bonial, Kira Griffitt, Ulf Hermjakob, Daniel Marcu, Tim O' Gorman, Martha Palmer, Nathan Schneider, and Madalina Bardocz. 2020. Abstract Mcaning Rep- resentation (AMR) Annotation Release 3.0.
Ioannis Konstas, Srinivasan Iyer, Mark Yatskar, Yejin Choi, and Luke Zettlemoyer. 2017. Neural AMR: Sequence-to-seeqeence models for parsing and gener- ation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers). pages 146-157, Vancouver, Canada. Association for Computational Linguistics.
Andrew Drozdov, Patrick Verga, Mohit Yadav, Mohit Iyyer, and Andrew McCallum. 2019. Unsupervised latent tree induction with deep inside-outside recur- sive auto-encoders. In Proceedings of the 2019 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, Volume I (Long and Short Pa- pers), pages 129-1141, Minneapolis, Minnesota. Association for Computational Linguistics.
Young-Suk Lee, Ramón Fernandez Astudillo, Radu Flo- rian, Tahira Naseem, and Salim Roukos. 2023. Amr parsing with instruction fine-tuned pre-trained lan- guage models.
Chris Dyer. Adhiguna Kuncoro, Miguel Ballesteros, and Noah A. Smith. 2016. Recurrent neural network grammars, In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020.

---

BART: Denoising sequence-tor se-ueeeeeee pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computa- tional Linguisties.
Sebastian Schuster, Ranjay Krishna, Angel Chang Li Fei-Fei, and Christopher D. Manning. 2015. Gen- erating semantically precise scene graphs from tex- tual descriptions for improved image retrieval. In Proceedings of the Fourth Workshop on Vision and Language, pages 70-80, Lishon, Portugal. Associa- tion for Computational Linguistics.
Kexin Liao, Logan Lebanoff, and Fei Liu. 2018. Ab- stract Meaning Representation for multi-document summarization. In Proceedings of the 27th Inter- national Conference on Computational Linguistics, pages 1178 1190, Santa Fe, New Mexico, USA. As- sociation for Computational Linguistics.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer- generator networks. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 1073- 1083, Vancouver. Canada. Association for Computa- tional Linguisties.
Ilya Loshchilov and Frank Hutter. 2017. SGDR:
Stochastic gradient descent with warm restarts. In In- ternational Conference on Learning Representations.
Kai Sheng Tai, Richard Socher, and Christopher D. Man- ning. 2015. Improved semantic representations from tree-structured long short-term memory networks, In Proceedings of the 53rd Annual Meeting of the As- sociation for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pages 1556- 1566, Beijing, China. Association for Computational Linguisties.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled 7th International Conference on Learning Representations, ICIR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRe- view.net.
weight decay regularization.
Tahira Naseem, Abhishek Shah. Hui Wan, Radu Flo- rian, Salim Roukos, and Miguel Ballesteros. 2019. Rewarding Smatch: Transition-basee AMR parsing with reinforeement learning. In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics. pages 4586-4592 Florence, Italy. Association for Computational I Linguistics.
Pavlo Vasylenko, Pere Lluis Huguet Cabot, Abelardo Carlos Martínez Lorenzo, and Roberto Navigli. 2023. Incorporating graph information in transformer-based AMR parsing. Findings of the Association for Computational Linguistics: ACL 2023, pages 1995-2011, Toronto, Canada. Association for Computational Linguisties.
Thanh-Tung Nguyen, Xuan-Phi Nguyen, Shafiq Joty, and Xiaoli Li. 2021. A conditional splitting frame- work for efficient constituency parsing. In Proceed- ings the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 5795-5807, Online. Association for Computational Linguistics.
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc.
Peiyi Wang, Liang Chen, Tianyu Liu, Damai Dai. Yunbo Cao, Baobao Chang, and Zhifang Sui. 2022. Hier- archical curriculum learning for AMR parsing. In Proceedings of the 60th Anmual Meeting of the As- sociation for Computational Linguistics (Volume 2: Short Papers), pages 333-339, Dublin, Ireland. As- sociation for Computational Linguistics.
Wessel Poelman, Rik van Noord, and Johan Bos. 2022. Transparent semantic parsing with Universal Depen- dencies using graph transformations. In Proceedings of the 29th International Conference on Computa- tional Linguistics, pages 4186 4192, Gyeongju, Re- public of Korea. International Committee on Compu- tational Linguistics.
Yuk Wah Wong and Raymond Mooney, 2007. Learn- ing synchronous grammars for semantic parsing with lambda calculus. In Proceedings of the 45th Annual Meeting of the Association of Computational Lin- guisties, pages 960 967, Prague. Czech Republic. Association for Computational Linguistics.
Peng Qian, Tahira Naseem, Roger Levy, and Ramón Fernandez. Astudillo. 2021. Structural guidance for transformer language models. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol- ume 1: Long Papers). pages 7735- Online. As- sociation for Computational Linguistics.
Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, and Luke Zettlemoyer. 2020. Scalable zero- shot entity linking with dense entity retrieval. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6397-6407, Online. Association for Computa- tional Linguisties.
Laurent Sartran, Samuel Barrett, Adhiguna Kuncoro, Miloš Stanojevic, Phil Blunsom, and Chris Dyer. 2022. Transformer Grammars: Augmenting Trans- former Language Models with Syntactic Inductive Biases at Scale. Transactions of the Association for Computational Linguistics, 10:1 423-1439.
Dongqin Xu, Junhui Li, Muhua Zhu, Min Zhang, and Guodong Zhou. 2020. Improving AMR parsing with sequence-to-sequence pre-training. In Proceedings

---

of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2501- 2511, Online. Association for Computational Lin- guistics.
A More Plots
A.1 Tree View of the Base Layer
Fig. 6 demonstrates the tree structure of the base layer. There are two main differences from the DAG representation of AMR: (1) Non-leaf nodes have no label. Instead, the label a child node. (2) relations are presented as nodes instead of labeled edges.
Runxin Xu, Peiyi Wang, Tianyu Liu, Shuang Zeng. Baobao Chang, and Zhifang Sui. 2022. A two-stream AMR-enhanced model for document-lever event ar- Proceedings of the 2022 Con- ference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies, pages 5025-5036, Seattle, United States. Association for Computational Lin- guistics.
gument extraction.
A.2 Attention Mask with Broken Structural Locality
Fig. 5 shows the two variants of breaking structural localities, which is discussed in Sec 5.2.
Songlin Yang and Kewei Tu. 2022. Bottom-up con- stituency parsing nested named entity recogni- tion with pointer networks. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume Long Pupers), pages 2403-2416, Dublin, Treland. Association for Compu- tational Linguistics.
B Generate Attention Mask for Bottom-up Generation
Alg. 2 shows the procedure of generating McHa for ft
Chen Yu and Daniel Gildea. 2022. Sequence-to- sequence AMR parsing with ancestor information. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume
C Metrics
We report following fine-graing metrics:
Short Papers), pages 571-577, Dublin, Ireland. Association for Computational Linguistics.
No WSD: This process computes while disre- garding PropBank senses.
Zixuan Zhang and Heng Ji. 2021. Abstract Meaning Representation guided graph encoding and decoding for joint information extraction. the 2021 Conference of the North American Chap- ler of the Association for Computational Linguisties: Human Language Téchnologies, pages 39-49, Online. Association for Computational Linguistics.
Proceedings of
Wikification: This denotes the F-score related to the Wikification task.
Concepts: This signifies the F-score for the
Concept Identification task.
Hao Zhou, Gongshen Liu, and Kewei Tu. 2022. Improv- ing constituent representation with hypertree neural networks. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies. pages 1682-1692, Seattle, United States. Association for Computational Linguistics.
NER: This pertains to the F-score associated with the Named Entity Recognition task.
Negations: This involves the F-score for the Negation Detection task.
Unlabeled: This involves computations on the predicted graphs after all edge labels have been removed.
Jiawei Zhou, Tahira Naseem, Ramón Fernandez As- tudillo, and Radu Florian. 2021a. AMR parsing with action- pointer transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 5585-5598, On- line. Association for Computational Linguistics.
Reentrancy: This is computed solely reen- trant edges.
Semantic Role Labeling (SRL): This is com- puted only for 'ARG-i roles..
Jiawei Zhou, Tahira Naseem, Ramón Fernandez As- tudillo. Young-Sur Suk Lee, Radu Florian, and Salim Roukos. b021b. Structure-aware fine-tuning of sequence-to-seee touuuunncceee transformers for transition- based AMR parsing. In Proceedings of the 2021 Conference on Empirical Methods Natural Lan- guage Processing, pages 6279-6290 Online and Punta Cana, Dominican Republic, Association for Computational Linguistics.
D Implementation Details
We use the BART-base and BART-large check- points downloaded from the transformer library to initialize our models. An augmented vocabulary is used, which includes additional AMR relation

---

x1 X2 X3
x1
x2
x3
(a) compose + expand
(h) expand + causal
Figure 5: The target forms and the attention mask of two variants siingle. Black squares mean that cell is changed.
tokens, such as aarg0. Unlike other translation- based models, we do not add predicates (e.g.. have-condi tion-91) into the vocabulary because they have ingorable effects performance accord- ing to our preliminary experiments.
Algorithm 2: MCUA for ft
Data: sequence of token / with length N, sequence of pointers with length
Result: attention mask MCHa E NNN S ++[]
>> Empty stack
models for 50,000 steps, using batch size of 16. This amounts approximately 22 epochs on AMR 2.0 and around 15 epochs on AMR 3.0. We use an Adam optimizer (Loshchilov and Hutter, 2019), accompanied by a cosine learning rate scheduler (Loshchilov and Hutter, 2017) with a warm-up up phase of 5,000 steps. The peak learning rate is set at 5 x 10-5 for base models and 3 x 10-5 for large models. We use one NVIDIA TITAN V train models based on BART base, costing about 6 hours, and based on BART large, costing about hours.
We train
MCHA <- -00
for <- to N do
if [[] then
compose
while
j s[i] do McH + 0 j x S.pop()
end
MCHA [is] 0 S.push(i)
NVIDIA A40 to train models
else
(i) for E S do
expand
McHa[ij] 0 end
end end
return MeUa

---

ROOT
alpha
carg
arg
beta
crrg
carg
gamma
crrg
arr
delta
epsilon
zeta
carg
theta
eta
(a) The full tree in Fig. 2 without auxiliary tokens.
ROOT
gamma
:arg
carg
zeta
:argO
theta
eta
(b) The tree related to garma in Fig. 2 in the ljsingle form.
ROOT
gamma
:arg0
arg
zeta
crrg
theta
(c) The related to gamma Fig. 1t form.
Figure 6: The structure modeled in the base layer.


Reviews:
[{"review_id": "yyUCwCZAsj", "reviewer": "Reviewer_2oGz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review and we appreciate your time in reading the paper.\n\n> **Reason To Reject 1** Somewhat narrow applicability. No other applications beyond AMR parsing are discussed.\n\nWe have discussed this in the Limitation section. However, we would like to point out that improving performance on one task is a common type of contributions in \\*ACL conferences and the ACL23 review policy explicitly argues against regarding this as a weakness:\n\n    \"A main track paper may well make a big contribution to a narrow subfield.\"\n\nWe hope you could reconsider your evaluation based on the review policy.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Sound and clean approach.\n- State-of-the-art results.", "reasons_to_reject": "- Somewhat narrow applicability. No other applications beyond AMR parsing are discussed.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a novel AMR parser. The authors present to augment a translation-based approach with a decoder that can model structure. The goal is to be able to benefit from pre-trained models while exploiting the structure in the output space.\nThe paper is well written and clear and the authors show that their approach matches the recently proposed state-of-the-art LeakDistill approach in terms of accuracy.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jiuNfiTS0G", "reviewer": "Reviewer_9uSY", "summary": "", "questions": "A. 425: How did you pick this value for $\\alpha$?\n\nB. 430: Could you elaborate a little on how these models work, and what category they fall under according to Table 2?\n\nC. Tab 4, Tab 6: Some of these results are pretty close. Can you include std devs or significance testing?\n\nD. 476: What about these datasets makes them out of distribution? Is it the size of the graphs? The subject matter? How do we know CHA's structure modeling is what's helping?\n\nE. Can you describe how LeakDistill works a little? What about it makes it so much better than CHAP?\n\nF. Because edges in a graph are unordered, a single AMR graph can have multiple ways of turning it into a tree using DFS. How do you pick which one to use? What is your algorithm for translating AMR graphs into trees? Do you foresee any generalization issues because of this?\n\nG. In the big picture, it seems like the ideal decoder architecture for AMRs would be some sort of graph NN, rather than a tree + pointer mechanism. Is the main selling point of decomposing the decoder pipeline into tree and pointer phases that it makes it easier to use with a pretrained transformer decoder? Or would this be possible with a graph NN too?\n\nH. I don't understand the distinction between the \"pointer encoder\" and \"pointer net\" mentioned at 86. Are there two different places where pointers are used (aside from the pointers used in bottom-up parsing)? Same question for points (1) and (3) at 313.\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your encouraging words and constructive comments. Our point-to-point responses to your comments are given below.\n\n> **Reason To Reject 1** Expect std devs or significance testing.\n>\n> **Question C. Tab 4, Tab 6** Some of these results are pretty close. Can you include std devs or significance testing?\n\nIn general, our models have a low std dev. We will report std devs in our revised paper. We report some std devs below (new scores are bolded):\n\nBased on Tab. 4: The influence of different architectures.\n\n| Architecture | Smatch | Std dev  |\n| ------------ | :----: | :------: |\n| Parallel     | 82.63  | **0.02** |\n| Pipeline     | 82.59  | **0.05** |\n| Inplace      | 82.43  | **0.04** |\n| w/o CHA      | 82.38  | **0.12** |\n\nBased on Tab. 5: Fine-grained Smatch scores on in-domain benchmarks (trained on AMR 3.0 without silver data).\n\n| Model       |  Smatch   | Std dev  |\n| ----------- | :-------: | :------: |\n| Ancestor    |   83.5    |    -     |\n| BiBL        |   83.9    |    -     |\n| LeakDistill |   84.5    |    -     |\n| CHAP        | **84.37** | **0.04** |\n\nBased on Tab. 6: Fine-grained Smatch scores on in-domain benchmarks (trained on AMR 3.0 with silver data).\n\n| Model       | TLP: Smatch | TLP: Std dev | Bio: Smatch | Bio: Std dev |\n| ----------- | :---------: | :----------: | :---------: | :----------: |\n| LeakDistill |    82.6     |      -       |    64.5     |      -       |\n| CHAP        |    82.7     |   **0.14**   |    66.1     |   **0.46**   |\n\n\n> **Reason To Reject 2** The claim that CHA is the cause of better generalization requires a lot more justification.\n>\n> **Question D. 476** What about these datasets make them out of distribution? How do we know CHA's structure modeling is what's helping?\n\nThe texts in these datasets are of different subject matters. However, these datasets share the same label set and annotation guidelines.\n\nWe agree that the claim at line 478 should be justified because the reported results only show CHAP's generalization ability. Therefore, we evaluate the gains from each component (i.e., CHA, pointer net, pointer encoder) and report the results below for the setting of Tab. 7 Ablation Study. New scores are bolded. Although models with these components outperform the baseline, we do not find a clear cause or consistent trend in the results. Therefore, we will claim CHAP rather than CHA helps generalization in our revised paper.\n\n| Encoding pointers | Adapter | Coref. layer | TLP      | Bio      |            |\n| ----------------- | ------- | ------------ | -------- | -------- | ---------- |\n| yes               | CHA     | yes          | 81.8     | 65.1     |            |\n| no                | CHA     | yes          | **81.7** | **64.8** |            |\n| no                | CHA     | no           | **82.1** | **64.6** |            |\n| no                | no      | no           | **81.6** | **64.3** | (baseline) |\n\n\n> **Question A. 425** How did you pick this value for $\\alpha$?\n\nWe run a grid search over the set $\\{0.01, 0.05, 0.075, 0.1, 1\\}$ and choose the value resulting in the best validation performance.\n\n> **Question B. 430** Could you elaborate a little on how these models work, and what category they fall under according to Table 2?\n>\n> **Question E** Can you describe how LeakDistill works? What about it makes it so much better than CHAP?\n\nAll these models are translation-based models. We will explain them in the revised paper. Here is a brief introduction:\n\n1. SPRING is a finetuned BART.\n2. Ancestor enhances the decoder of SPRING with ancestors in AMR graphs.\n3. BiBL and AMRBART enhance SPRING with auxiliary training losses.\n4. LeakDistill trains a SPRING with leaked information and then distills it to a standard SPRING. We have discussed the success of LeakDistill at lines 460-464.\n\nTransition-based and factorization-based models have weaker performance, so we do not list their scores in our paper.\n\n> **Question F** Because edges in a graph are unordered, a single AMR graph can have multiple ways of turning it into a tree using DFS. How do you pick which one to use? What is your algorithm for translating AMR graphs into trees? Do you foresee any generalization issues because of this?\n\nFollowing previous work, we use the DFS order provided in the AMR datasets. We believe that the DFS algorithm does not raise a generalization issue:\n\n* Regarding training, previous work[1] suggests that \"any depth first traversal of the AMR is an effective linearization.\"\n* Regarding evaluation, the Smatch metric is invariant to different DFS algorithms.\n\n[1] Neural AMR: Sequence-to-Sequence Models for Parsing and Generation (Konstas et al., ACL 2017)\n\n> **Question G** It seems like the ideal decoder architecture for AMRs would be some sort of graph NN, rather than a tree + pointer mechanism. Is the main selling point of decomposing the decoder pipeline into tree and pointer phases that it makes it easier to use with a pretrained transformer decoder? Or would this be possible with a graph NN too?\n\nPrevious translation-based models (e.g., SPRING) can be easily used with a pretrained decoder by linearizing AMR graphs to texts. Our decomposition aims to use a pretrained decoder better (discussed in Sec 3 lines 161-164 and 171-174).\n\nThe decoding procedure involves a growing graph, making the application of GNNs non-trivial. We are open to the idea of utilizing GNNs and expect an exploration in future work.\n\n> **Question H** I don't understand the distinction between the \"pointer encoder\" and \"pointer net\" mentioned at 86. Are there two different places where pointers are used (aside from the pointers used in bottom-up parsing)? Same question for points (1) and (3) at 313.\n\n\"Pointer net\" (or \"pointer mechanism\"), introduced in Sec 4.3, is used to predict pointers. In contrast, a \"pointer encoder\", introduced in Sec 4.1, encodes previously predicted pointers (just like the word embedding layer in BART's decoder encoding previously predicted tokens).\n\n> **Typos ... C. 111** Since factorization-based models are a whole column in Tab 2, could you spend a little time explaining what they are?\n\nFactorization-based models factorize the score of a graph into parts for smaller subgraphs. They operate graphs directly, so their decoding is generally much more complicated. For example, [2] predicts a subgraph for each token and then composes them to obtain the final AMR graph. [3] predicts an AMR graph by incrementally and iteratively predicting new edges and nodes.\n\n[2] AMR dependency parsing with a typed semantic algebra (Groschwitz et al., ACL 2018) \n\n[3] AMR Parsing via Graph-Sequence Iterative Inference (Cai & Lam, ACL 2020)\n\n\n> **Typos ... F. 408** What is a wiki tag?\n\nAll named entities in AMR graphs are linked to a Wikipedia page and have a wiki tag containing the page name.\n\n> **Typos ... H** Is higher bettter for Smatch?\n\nYes.\n\n> **Typos .. M** Is there a result missing from Tab 7?\n\nYes, the missing value is `83.79`.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written, well-organized, and well-motivated, and for the most part it was enjoyable to read. Sec 2.1 nicely lays out the advantages of their method compared to prior work. As a result, I actually have less to say than usual in terms of praise and criticism. This paper would clearly be a good fit at EMNLP.\n\nThe method itself seems to be an innovative improvement over prior work. The authors compare against an excellent variety of recent baselines on multiple AMR parsing datasets. I don't see any serious issues with the experimental methodology.\n\nIt's also very nice that the authors explored multiple alternatives for CHA, justifying the architecture they eventually settled on.\n", "reasons_to_reject": "My biggest complaint about this paper is that in the experimental section, a lot of the SMATCH scores are very close, and I would like to see std devs or significance testing. More than three runs might also bolster confidence in these scores. Without this, it's hard to have confidence in some of the experimental results.\n\n478: It's not clear to me that CHA is the cause of better generalization, and I think this claim requires a lot more justification.\n", "questions_to_authors": "A. 425: How did you pick this value for $\\alpha$?\n\nB. 430: Could you elaborate a little on how these models work, and what category they fall under according to Table 2?\n\nC. Tab 4, Tab 6: Some of these results are pretty close. Can you include std devs or significance testing?\n\nD. 476: What about these datasets makes them out of distribution? Is it the size of the graphs? The subject matter? How do we know CHA's structure modeling is what's helping?\n\nE. Can you describe how LeakDistill works a little? What about it makes it so much better than CHAP?\n\nF. Because edges in a graph are unordered, a single AMR graph can have multiple ways of turning it into a tree using DFS. How do you pick which one to use? What is your algorithm for translating AMR graphs into trees? Do you foresee any generalization issues because of this?\n\nG. In the big picture, it seems like the ideal decoder architecture for AMRs would be some sort of graph NN, rather than a tree + pointer mechanism. Is the main selling point of decomposing the decoder pipeline into tree and pointer phases that it makes it easier to use with a pretrained transformer decoder? Or would this be possible with a graph NN too?\n\nH. I don't understand the distinction between the \"pointer encoder\" and \"pointer net\" mentioned at 86. Are there two different places where pointers are used (aside from the pointers used in bottom-up parsing)? Same question for points (1) and (3) at 313.\n", "paper_topic_and_main_contributions": "This paper introduces a new method for AMR parsing called CHAP. In contrast to recent work which has focused on using pretrained transformer decoders to output linearized AMR graphs, CHAP attempts to add graph structure to the decoder architecture, while still maintaining compatibility with pretrained transformer decoders (namely BART). The new decoder architecture has three major modifications: (1) a new hierarchical attention mechanism based on Transformer Grammars (Sartran et al. 2022) is used in some decoder layers; (2) the transformer decoder generates a linearized tree representing a depth-first traversal of the AMR (\"base layer\"); (3) a \"coreference layer\" then uses an attention-based pointer mechanism to connect nodes in the tree that refer to the same thing, forming a graph with reentrancies. A benefit of using pointers is that it eliminates arbitrarily-named variables from the output.\n\nThe main experimental results are:\n1. They test multiple variants of TG-style hierarchical attention and find that the original one works best.\n2. They test multiple ways of incorporating hierarchical attention into pretrained decoder layers and settle on an approach that uses it in parallel with the standard attention mechanism.\n3. They show that CHAP performs better than several baselines on the AMR 2.0 and AMR 3.0 datasets, except for a concurrent model called LeakDistill, which uses additional alignment data.\n4. They show that CHAP has better performance on 2 of 3 out-of-distribution benchmarks.\n5. An ablation study showing that only the combination of all aspects of CHAP results in the best performance on AMR 3.0.\n\nOverall, this paper proposes a novel architecture for AMR parsing and shows that it outperforms all recent baselines except for a concurrent project called LeakDistill, which uses some additional training data.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "A. Fig 1: The example sentence isn't grammatically correct. Shouldn't it be \"The employee liked his (or her) city tour\"?\n\nB. Fig 2: I think it would be more helpful to use a sentence in plain English as the running example, or at least explain why it's not necessary to use a plain English example.\n\nC. 111: Since factorization-based models are a whole column in Tab 2, could you spend a little time explaining what they are?\n\nD. 151: researches -> research\n\nE. 276: unaware -> unaware of\n\nF. 408: What is a wiki tag?\n\nG. 435: Structual -> Structural\n\nH. Is higher better for Smatch? Adding an arrow by Smatch would help.\n\nI. 464: worthy -> valuable\n\nJ. It might be helpful to remind the reader, early on in the paper, that the task of AMR parsing is to receive a sequence as input and produce an AMR graph as output. This would help explain why the changes are being made to the decoder.\n\nK. Footnote 6: works -> work\n\nL. The way Table 6 is organized makes it hard to read. Maybe group models based on training data. The shading makes it seem like the shaded results don't matter or aren't as important.\n\nM. Is there a result missing from Tab 7?\n"}}, {"review_id": "FCSws7DeKB", "reviewer": "Reviewer_MHyS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments and suggestions, and they are exceedingly helpful for us to improve our paper. \n\n> 2\\. Tab. 1 references a figure in the appendix and I don't understand why. Was this meant to ref. to Fig. 3?\n\nSorry for the confusion. All references to Fig. 5 in the main text should reference Fig. 3 instead.\n\n> 5\\. In Alg. 1: Something seems off.\n\nYou are right. `S.push` should be performed in both cases. We will also fix the issue of iteration range.\n\n> 6\\. Sec. 3.4 describes the target encoding $\\Uparrow$\n as bottom-up generation. I disagree, as all nodes are still introduced in top-down fashion, they are only completed bottom-up.\n\nThe direction is defined with respect to the syntax tree of linearized AMR, instead of the AMR graph. In a syntax tree, all tokens in an AMR graph (e.g., `alpha` and `:arg0`) are leaves and their compositions are non-leaf nodes. In our bottom-up generation, we first generate tokens and then their compositions (represented by `■` and blue pointers in Tab 1.e). We compare syntax trees and AMR graphs in lines 223-236 and Appendix A.1. We will use a more precise presentation in our revised paper.\n\nThe `top-down fashion` in your question actually refers to `the pre-order DFS` graph linearization of AMR graphes, which is orthogonal to the syntax tree generation direction defined above.\n\n\n> 7\\. line 401: Which trained model do you use to obtain silver data?\n\nWe train CHAP three times. Then, the silver data is obtained using the 2nd best model according to the metric on the validation set. We only generate the silver data once due to the high computational cost of predicting AMR graphs on a massive corpus.\n\n> 1\\. figure placement;\n>\n> 3\\. accessible visual effects;\n>\n> 4\\. the figures illustrate different AMR graphs;\n>\n> 8\\. entangled writing;\n>\n> 9\\. too little explanation in the introduction;\n>\n> 10\\. wrong pre- and surnames in the citations;\n>\n> 11\\. typo.\n\nThank you for bringing up these issues and we will revise our paper accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The contributed representations seem very sensible to me to eliminate variables, specifically the last one follows a trend to remove output representation complexity and modelling relations within the output explicitly. Their ablation study suggests that each single of their contributed additions to the parsing model did benefit the prediction accuracy.", "reasons_to_reject": "I have no serious reasons to reject the paper, but stumbled upon some unclarities and issues in the presentation (listed below). I'd suggest they can be ironed out within an iteration to a camera-ready version.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper focuses on AMR genration using a neural translation architecture extended by a pointer mechanism. The authors contribute\n* new AMR representations/linearizations that involve pointers to dismiss variables, and one that also uses pointers instead of parentheses to encode parent-child-relations\n* an extension of the decoding process with hierarchical attention to enforce local contributions from child node encodings to parents", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. The figure placement seems a little off to me. Most of them are not on the same page they are referenced at and it's somewhat irritating to constantly move between pages.\n2. Tab. 1 references a figure in the appendix and I don't understand why. Was this meant to ref. to Fig. 3?\n3. Tab. 1 and Fig. 3 rely partially on colored highlighting. I'd suggest to replace that in favor of accessible means.\n4. I found the linearizations in tab. 1 irritating, because they are next to an illustration of a different AMR graph.\n5. In Alg. 1: Something seems off. The stack S should contain all indices that refer to sibling or unfinished nodes, right? Then the push to S should be in both cases, not just for closing parentheses. You could also mention that the attention mask contains $-\\inf$ if the index is not used and 0 otherwise, or use denote it with a boolean representation to be more clear. Lastly (to be nitpicky), I understand a loop from 0 to $N$ as $N+1$ iterations, but that does not fit to your $N\\times N$ matrix $M_{\\text{CHA}}$.\n6. Sec. 3.4 describes the target encoding $\\Uparrow$ as bottom-up generation. I disagree, as all nodes are still introduced in top-down fashion, they are only completed bottom-up.\n7. line 401: Which trained model do you use to obtain silver data?\n8. Sec. 5.2 describes the AMR representations as CHA options (as does tab. 3). I understand that these two are entangled, but would prefer to distinguish between these two.\n9. Tab. 1 refers to parts of the output representations as layers. This is only roughly described in the introduction (ln. 64), but should be a little more formalized imho.\n10. The two citations for AMR are mixing the authors' pre- and surnames.\n11. ln 109/110: reply -> rely\n"}}]

Author rebuttal:

Meta-review: As reviewers indicate, this paper introduces a new method for AMR parsing called CHAP. In contrast to recent work which has focused on using pretrained transformer decoders to output linearized AMR graphs, CHAP attempts to add graph structure to the decoder architecture, while still maintaining compatibility with pretrained transformer decoders (namely BART). The new decoder architecture has three major modifications: (1) a new hierarchical attention mechanism based on Transformer Grammars (Sartran et al. 2022) is used in some decoder layers; (2) the transformer decoder generates a linearized tree representing a depth-first traversal of the AMR ("base layer"); (3) a "coreference layer" then uses an attention-based pointer mechanism to connect nodes in the tree that refer to the same thing, forming a graph with reentrancies. A benefit of using pointers is that it eliminates arbitrarily-named variables from the output.

The method itself seems to be an innovative improvement over prior work.The main experimental results pointed out by reviewers are the following: 

    They test multiple variants of TG-style hierarchical attention and find that the original one works best.
    They test multiple ways of incorporating hierarchical attention into pretrained decoder layers and settle on an approach that uses it in parallel with the standard attention mechanism.
    They show that CHAP performs better than several baselines on the AMR 2.0 and AMR 3.0 datasets, except for a concurrent model called LeakDistill, which uses additional alignment data. The authors explored multiple alternatives for CHA, justifying the architecture they eventually settled on.
    They show that CHAP has better performance on 2 of 3 out-of-distribution benchmarks.
    An ablation study showing that only the combination of all aspects of CHAP results in the best performance on AMR 3.0.

Additionally, the  paper is well written and clear, well-organized, and well-motivated.

According to reviewers there are not many reasons to reject the paper.  

    A reviewer indicates that in the experimental section, a lot of the SMATCH scores are very close, and  std devs or significance testing is recommended. The authors addressed this in the rebuttal. 
    Somewhat narrow applicability. No other applications beyond AMR parsing are discussed.
    There are some unclarities an issues pointed out by Reviewer 3. Tthey can be ironed out within an iteration to a camera-ready version.

All in all, it seems that the authors could address the shortcomings in the final version of the paper.
Decision: Accept-Main