SlowGuess/ACL_2020_OCR huggingface dataset:
Qiji Zhou $^{1}$ , Yue Zhang $^{2,3}$ , Donghong Ji $^{1*}$ , Hao Tang $^{1}$
<sup>1</sup>Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering, Wuhan University, China
{qiji.zhou, dhji, tanghaopro}@whu.edu.cn
$^{2}$ School of Engineering, Westlake University
$^{3}$ Institute of Advanced Technology, Westlake Institute for Advanced Study
yue.zhang@wias.org.cn
Abstract Meaning Representations (AMRs) capture sentence-level semantics structural representations to broad-coverage natural sentences. We investigate parsing AMR with explicit dependency structures and interpretable latent structures. We generate the latent soft structure without additional annotations, and fuse both dependency and latent structure via an extended graph neural networks. The fused structural information helps our experiments results to achieve the best reported results on both AMR 2.0 (77.5% Smatch F1 on LDC2017T10) and AMR 1.0 (71.8% Smatch F1 on LDC2014T12).
Abstract Meaning Representations (AMRs) (Banarescu et al., 2013) model sentence level semantics as rooted, directed, acyclic graphs. Nodes in the graph are concepts which represent the events, objects and features of the input sentence, and edges between nodes represent semantic relations. AMR introduces re-entrance relation to depict the node reuse in the graphs. It has been adopted in downstream NLP tasks, including text summarization (Liu et al., 2015; Dohare and Karnick, 2017), question answering (Mitra and Baral, 2016) and machine translation (Jones et al., 2012; Song et al., 2019).
AMR parsing aims to transform natural language sentences into AMR semantic graphs. Similar to constituent parsing and dependency parsing (Nivre, 2008; Dozat and Manning, 2017), AMR parsers mainly employ two parsing techniques: transition-based parsing (Wang et al., 2016; Damonte et al., 2017; Wang and Xue, 2017; Liu et al., 2018; Guo and Lu, 2018) use a sequence of transition actions
Figure 1: An example of AMR graph and its corresponding syntactic dependencies for the sentence "The boy came and left". The dashed lines denote the connected relations in the syntactic dependencies but not appear in the AMR graph.
to incrementally construct the graph, while graph-based parsing (Flanigan et al., 2014; Lyu and Titov, 2018; Zhang et al., 2019a; Cai and Lam, 2019) divides the task into concept identification and relation extraction stages and then generate a full AMR graph with decoding algorithms such as greedy and maximum spanning tree (MST). Additionally, reinforcement learning (Naseem et al., 2019) and sequence-to-sequence (Konstas et al., 2017) have been exploited in AMR parsing as well.
Previous works (Wang et al., 2016; Artzi et al., 2015) shows that structural information can bring benefit to AMR parsing. Illustrated by Figure 1, for example syntactic dependencies can convey the main predicate-argument structure. However, dependency structural information may be noisy due to the error propagation of external parsers. Moreover, AMR concentrates on semantic relations, which can be different from syntactic dependencies. For instance, in Figure 1, AMR prefers to select the coordination (i.e. "and") as the root, which is different from syntactic dependencies (i.e. "came").
Given the above observations, we investigate the effectiveness of latent syntactic dependencies for
†Part of work was done when the author was visiting Westlake University
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4306-4319 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
AMR parsing. Different from existing work (Wang et al., 2016), which uses a dependency parser to provide explicit syntactic structures, we make use of a two-parameter distribution (Bastings et al., 2019) to induce latent graphs, which is differentiable under reparameterization (Kingma and Welling, 2014). We thus build a end-to-end model for AMR parsing with induced latent dependency structures as a middle layer, which is tuned in AMR training and thus can be more aligned to the need of AMR structure.
For better investigating the correlation between induced and gold syntax, and better combine the strengths, we additionally consider fusing gold and induced structural dependencies into an align-free AMR parser (Zhang et al., 2019a). Specifically, we first obtain the input sentence's syntactic dependencies<sup>1</sup> and treat the input sentence as prior of the probabilistic graph generator for inferring the latent graph. Second, we propose an extended graph neural network (GNN) for encoding above structural information. Subsequently we feed the encoded structural information into a two stage align-free AMR parser (Zhang et al., 2019a) for promoting AMR parsing.
To our knowledge, we are the first to incorporate syntactic latent structure in AMR parsing. Experimented results show that our model achieves  $77.5\%$  and  $71.8\%$  SMATCH F1 on standard AMR benchmarks LDC2017T10 and LDC2014T12, respectively, outperforming all previous best reported results. Beyond that, to some extent, our model can interpret the probabilistic relations between the input words in AMR parsing by generating the latent graph<sup>2</sup>.
2 Baseline: Align-Free AMR Parsing
We adopt the parser of Zhang et al. (2019a) as our baseline, which treats AMR parsing as sequence-to-graph transduction.
2.1 Task Formalization
Our baseline splits AMR parsing into a two-stage procedure: concept identification and edge prediction. The first task aims to identify the concepts (nodes) in AMR graph from input tokens, and the second task is designed to predict semantic relations between identified concepts.
Formally, for a given input sequence of words  $\pmb{w} = \langle w_1,\dots,w_n\rangle$ , the goal of concept identification in our baseline is sequentially predicting the concept nodes  $\pmb{u} = \langle u_{1},\dots,u_{m}\rangle$  in the output AMR graph, and deterministically assigning corresponding indices  $\pmb{d} = \langle d_1,\dots,d_m\rangle$ .
$$
P (\boldsymbol {u}) = \prod_ {i = 1} ^ {m} P (u _ {i} \mid u _ {<   i}, d _ {<   i}, \boldsymbol {w}),
$$
After identifying the concept nodes  $c$  and their corresponding indices  $d$ , we predict the semantic relations in the searching space  $\mathcal{R}(u)$ .
$$
\operatorname {Predict}(\boldsymbol {u}) = \underset {r\in \mathcal{R}(\boldsymbol {u})}{\arg \max}\sum_{(u_{i},u_{j})\in r}\operatorname {score}(u_{i},u_{j}),
$$
where  $r = \{(u_i, u_j) \mid 1 \leq i, j \leq m\}$  is a set of directed relations between concept nodes.
2.2 Align-Free Concept Identification
Our baseline extends the pointer-generator network with self-copy mechanism for concept identification (See et al., 2017; Zhang et al., 2018a). The extended model can copy the nodes not only from the source text, but also from the previously generated list of nodes on the target side.
The concept identifier firstly encodes the input sentence into concatenated vector embeddings with GloVe (Pennington et al., 2014), BERT (Devlin et al., 2019), POS (part-of-speech) and character-level (Kim et al., 2016) embeddings. Subsequently, we encode the embedded sentence by a two-layer bidirectional LSTM (Schuster and Paliwal, 1997; Hochreiter and Schmidhuber, 1997):
$$
h _ {i} ^ {l} = [ \overrightarrow {f} ^ {l} (h _ {i} ^ {l - 1}, h _ {i - 1} ^ {l}); \overleftarrow {f} ^ {l} (h _ {i} ^ {l - 1}, h _ {i + 1} ^ {l}) ],
$$
where  $h_i^l$  is the  $l$ -th layer encoded hidden state at the time step  $i$  and  $h_i^0$  is the embedded token  $w_i$ .
Different from the encoding stage, the decoder does not use pre-trained BERT embeddings, but employs a two-layer LSTM to generate the decoding hidden state  $s_t^l$  at each time step:
$$
s _ {t} ^ {l} = f ^ {l} (s _ {t} ^ {l - 1}, s _ {t - 1} ^ {l}),
$$
where  $s_t^{l-1}$  and  $s_{t-1}^l$  are hidden states from last layer and previous time step respectively, and  $s_0^l$  is the concatenation of the last bi-directional encoding hidden states. In addition,  $s_t^0$  is generated from the concatenation of the previous node  $u_{t-1}$  embedding and the attention vector  $\widetilde{s}_{t-1}$ , which combine both source and target information:
1We employ Stanford CoreNLP (Manning et al., 2014) to get the dependencies. 2Our code will be available at: https://github. com/zhouqiji/ACL2020_AMR_Parsing.
$$
\widetilde {s} _ {t} = \tanh (W _ {c} [ c _ {t}; s _ {t} ^ {l} ] + b _ {c}),
$$
where  $W_{c}$  and  $b_{c}$  are trainable parameters,  $c_{t}$  is the context vector calculated by the attention weighted encoding hidden states and the source attention distribution  $a_{\mathrm{src}}^{t}$  following Bahdanau et al. (2015)
The produced attention vector  $\widetilde{s}$  is used to generate the vocabulary distribution:
$$
P _ {\mathrm {v o c a b}} = \mathrm {s o f t m a x} (W _ {\mathrm {v o c a b}} \widetilde {s} _ {t} + b _ {\mathrm {v o c a b}}),
$$
as well as the target attention distribution:
$$
\begin{array}{l} e _ {\mathrm {t g t}} ^ {t} = v _ {\mathrm {t g t}} ^ {\top} \operatorname {t a n h} \left(W _ {\mathrm {t g t}} \widetilde {s} _ {1: t - 1} + U _ {\mathrm {t g t}} \widetilde {s} _ {t} + b _ {\mathrm {t g t}}\right), \\ a _ {\mathrm {t g t}} ^ {t} = \operatorname {s o f t m a x} \left(e _ {\mathrm {t g t}} ^ {t}\right), \\ \end{array}
$$
The source-side copy probability  $p_{\mathrm{src}}$ , target-side copy probability  $p_{\mathrm{tgt}}$  and generation probability  $p_{\mathrm{gen}}$  are calculated by  $\widetilde{s}$ , which can be treated as generation switches:
$$
\left[ p _ {\mathrm {s r c}}, p _ {\mathrm {t g t}}, p _ {\mathrm {g e n}} \right] = \operatorname {s o f t m a x} \left(W _ {\text {s w i t c h}} \widetilde {s} _ {t} + b _ {\text {s w i t c h}}\right),
$$
The final distribution is defined below, if  $v_{t}$  is copied from existing nodes:
$$
P ^ {(\mathrm {n o d e})} (u _ {t}) = p _ {\mathrm {t g t}} \sum_ {i: u _ {i} = u _ {t}} ^ {t - 1} a _ {\mathrm {t g t}} ^ {t} [ i ],
$$
otherwise:
$$
P ^ {(\text {n o d e})} (u _ {t}) = p _ {\text {g e n}} P _ {\text {v o c a b}} (u _ {t}) + p _ {\text {s r c}} \sum_ {i: w _ {i} = u _ {t}} ^ {n} a _ {\text {s r c}} ^ {t} [ i ],
$$
where  $a^t [i]$  is the  $i$ -th element of  $a^t$ , and then deterministically assigned the existing indices to the identified nodes based on whether the node is generated from the target-side distribution.
2.3 Edge Prediction
Our baseline employs a deep biaffine attention classifier for semantic edge prediction (Dozat and Manning, 2017), which have been widely used in graph-based structure parsing (Peng et al., 2017; Lyu and Titov, 2018; Zhang et al., 2019a).
For a node  $u_{t}$ , the probability of  $u_{k}$  being the head node of  $u_{t}$  and the probability of edge  $(u_{k}, u_{t})$  are defined below:
$$
\begin{array}{l} P _ {t} ^ {\left(\text {h e a d}\right)} \left(u _ {k}\right) = \frac {\exp \left(\operatorname {s c o r e} _ {k , t} ^ {\left(\text {e d g e}\right)}\right)}{\sum_ {j = 1} ^ {m} \exp \left(\operatorname {s c o r e} _ {j , t} ^ {\left(\text {e d g e}\right)}\right)}, \\ P _ {k, t} ^ {\left(\text {l a b e l}\right)} (l) = \frac {\exp \left(\operatorname {s c o r e} _ {k , t} ^ {\left(\text {l a b e l}\right)} [ l ]\right)}{\sum_ {l ^ {\prime}} \exp \left(\operatorname {s c o r e} _ {k , t} ^ {\left(\text {l a b e l}\right)} [ l ^ {\prime} ]\right)}, \\ \end{array}
$$
where score  $(\mathrm{score})$  and label  $(\mathrm{edge})$  are calculated via bi-affine attentions.
The overall structure of our model is shown in Figure 2. First, we use an external dependency parser (Manning et al., 2014) to obtain the explicit structural information, and obtain the latent structural information via a probabilistic latent graph generator. We then combine both explicit and latent structural information by encoding the input sentence through an extended graph neural network. Finally, we incorporate our model with an align-free AMR parser for parsing AMR graphs with the benefit of structural information.
3.1 Latent Graph Generator
We generate the latent graph of input sentence via the HardKuma distribution (Bastings et al., 2019), which has both continuous and discrete behaviours. HardKuma can generate samples from the closed interval [0, 1] probabilistically. This feature allows us to predict soft connections probabilities between input words, which can be seen as a latent graph. Specifically, we treat embedded input words as a prior of a two-parameters distribution, and then sample a soft adjacency matrix between input words for representing a dependency.
HardKuma Distribution The HardKuma distribution is derived from the Kumaraswamy distribution (Kuma) (Kumaraswamy, 1980), which is a two-parameters distribution over an open interval  $(0,1)$ , i.e.,  $K \sim \mathrm{Kuma}(a,b)$ , where  $a \in \mathbb{R}_{>0}$  and  $b \in \mathbb{R}_{>0}$ . The Kuma distribution is similar to Beta distribution, but its CDF function has a simpler analytical solution and inverse of the CDF is:
$$
C _ {K} ^ {- 1} (u; \boldsymbol {a}, \boldsymbol {b}) = \left(1 - (1 - u) ^ {1 / b}\right) ^ {1 / a},
$$
We can generate the samples by:
$$
C _ {K} ^ {- 1} (U; \boldsymbol {\alpha}, \boldsymbol {\beta}) \sim \operatorname {K u m a} (\boldsymbol {\alpha}, \boldsymbol {\beta}),
$$
where  $U \sim \mathcal{U}(0,1)$  is the uniform distribution, and we can reconstruct this inverse CDF function by the reparameterizing fashion (Kingma and Welling, 2014; Nalisnick and Smyth, 2017).
In order to include the two discrete points 0 and 1, HardKuma employs a stretch-and-rectify method with support (Louizos et al., 2017), which leads
Figure 2: Stretch of the model which has four main components: (1) A latent graph generator for producing the soft-connected latent graph ( $\S\S3.1$ ); (2) An extended syntactic graph convolutional network for encoding the structural information ( $\S\S3.2$ ); (3) An align-free concept identification for concept node generation ( $\S\S2.2$ ); (4) A deep biaffine classifier for relation edge prediction ( $\S\S2.3$ ).
the variable  $T\sim \mathrm{Kuma}(\pmb {a},\pmb {b},l,r)$  to be sampled from Kuma distribution with an open interval  $(l,r)$  where  $l < 0$  and  $r > 0$ . The new CDF is:
$$
C _ {T} (t; \boldsymbol {a}, \boldsymbol {b}, l, r) = C _ {K} \left(^ {(t - l)} / (r - l); \boldsymbol {a}, \boldsymbol {b}\right),
$$
We pass the stretched variable  $T \sim \mathrm{Kuma}(a, b, l, r)$  through a hard-sigmoid function (i.e.,  $h = \min(1, \max(0, t))$ ) to obtain the rectified variable  $H \sim \mathrm{HardKuma}(a, b, l, r)$ . Therefore, the rectified variable covers the closed interval [0, 1]. Note that all negative values of  $t$  are deterministically mapped to 0. In contrast, all samples  $t > 1$  are mapped to  $1^3$ . Because the rectified variable is sampled based on Kuma distribution, HardKuma first sample a uniform variable over open interval (0, 1) from uniform distribution  $U \sim \mathcal{U}(0, 1)$ , and then generate a Kuma variable through inverse CDF:
$$
k = C _ {K} ^ {- 1} (u; \pmb {a}, \pmb {b}),
$$
Second, we transform the Kuma variable for covering the stretched support:
$$
t = l + (r - l) k,
$$
Finally, we rectify the stretched variable including closed interval [0, 1] via a hard-sigmoid function:
$$
h = \min  (1, \max  (0, t)).
$$
Latent Graph We generate the latent graph of input words  $w$  by sampling from HardKuma distribution with trained parameters  $a$  and  $b$ . We first calculate the prior  $c$  of  $(a, b)$  by employing multi-head self-attention (Vaswani et al., 2017):
$$
\boldsymbol {c} _ {a} = \operatorname {T r a n s f o m e r} _ {a} (\boldsymbol {v}),
$$
$$
\boldsymbol {c} _ {b} = \operatorname {T r a n s f o m e r} _ {b} (\boldsymbol {v}),
$$
where  $\pmb{v} = \langle v_1, \dots, v_n \rangle$  is the embedded input words. Subsequently, we compute  $\pmb{a}$  and  $\pmb{b}$  as:
$$
\boldsymbol {a} = \operatorname {N o r m} \left(\boldsymbol {c} _ {a} \boldsymbol {c} _ {a} ^ {T}\right),
$$
$$
\boldsymbol {b} = \operatorname {N o r m} \left(\boldsymbol {c} _ {b} \boldsymbol {c} _ {b} ^ {T}\right),
$$
where  $\mathbf{a}_i = \langle a_{i1},\dots,a_{in}\rangle$  and  $\pmb {b}_i = \langle b_{i1},\dots,b_{in}\rangle$ ,  $\pmb {c}_a,\pmb {c}_b\in \mathbb{R}^{n\times n}$  and  $\mathrm{Norm}(x)$  is the normalization function. Hence, the latent graph  $L$  is sampled via learned parameters  $\pmb{a}$  and  $\pmb{b}$ :
$$
l _ {i j} \sim \mathrm {H a r d K u m a} (a _ {i j}, b _ {i j}, l, r).
$$
<sup>3</sup>Details of derivations can be found at (Bastings et al., 2019).
3.2 Graph Encoder
For a syntactic graph with  $n$  nodes, the cell  $A_{ij} = 1$  in the corresponding adjacent matrix represents that an edge connects word  $w_i$  to word  $w_j$ . An L-layer syntactic GCN of  $l$ -th layer can be used to represent A, where the hidden vector for each word  $w_i$  at the  $l - th$  layer is:
$$
h _ {i} ^ {(l)} = \sigma (\sum_ {j = 1} ^ {n} \tilde {A} _ {i j} W ^ {(l)} h _ {j} ^ {(l - 1)} / d _ {i} + b ^ {(l)}),
$$
where  $\tilde{A} = A + I$  with the  $n\times n$  identity matrix  $I$ ,  $d_{i} = \sum_{j = 1}^{n}\tilde{A}_{ij}$  is the degree of word  $w_{i}$  in the graph for normalizing the activation to avoid the word representation with significantly different magnitudes (Marcheggiani and Titov, 2017; Kipf and Welling, 2017), and  $\sigma$  is a nonlinear activation function.
In order to take benefits from both explicit and latent structural information in AMR parsing, we extend the Syntactic-GCN (Marcheggiani and Titov, 2017; Zhang et al., 2018b) with a graph fusion layer and omit labels in the graph (i.e. we only consider the connected relation in GCN). Specifically, we propose to merge the parsed syntactic dependencies and sampled latent graph through a graph fusion layer:
$$
\boldsymbol {F} = \pi \boldsymbol {L} + (1 - \pi) \boldsymbol {D}
$$
where  $\pi$  is trainable gate variables are calculated via the sigmoid function,  $D$  and  $L$  are the parsed syntactic dependencies and generated latent graph respectively, and  $F$  represent the fused soft graph. Furthermore,  $F$  is a  $n\times n$  adjacent matrix for the input words  $\boldsymbol{w}$ , different from the sparse adjacent matrix  $A$ ,  $F_{ij}$  denote a soft connection degree from word  $w_{i}$  to word  $w_{j}$ . We adapt syntactic-GCN with a fused adjacent matrix  $F$ , and employ a gate mechanism:
$$
\begin{array}{l} h _ {i} ^ {(l)} = G E L U ( \\ L _ {n o r m} (\sum_ {j = 1} ^ {n} G _ {j} (F _ {i j} W ^ {(l)} h _ {j} ^ {(l - 1)} + b ^ {(l)})), \\ \end{array}
$$
We use GELU (Hendrycks and Gimpel, 2016) as the activation function, and apply layer normalization  $L_{norm}$  (Ba et al., 2016) before passing the results into GELU. The scalar gate  $G_{j}$  is calculated by each edge-node pair:
$$
G _ {j} = \mu (h _ {j} ^ {(l - 1)} \cdot \hat {v} ^ {(l - 1)} + \hat {b} ^ {(l - 1)}),
$$
where  $\mu$  is the logistic sigmoid function,  $\hat{v}$  and  $\hat{b}$  are trainable parameters.
Similar to our baseline (Zhang et al., 2019a), we linearize the AMR concepts nodes by a pre-order traversal over the training dataset. We obtain gradient estimates of  $\mathcal{E}(\phi, \theta)$  through Monte Carlo sampling from:
$$
\begin{array}{l} \mathcal {E} (\phi , \theta) = \mathbb {E} _ {\mathcal {U} (0, I)} [ \log P (n o d e | u _ {t}, g _ {\phi} (\boldsymbol {u}, \boldsymbol {w}), \theta) \\ + \log P _ {t} (\text {h e a d} | u _ {k}, g _ {\phi} (\boldsymbol {u}, \boldsymbol {w}), \theta) \\ + \log P _ {k, t} (l a b e l | l, g _ {\phi} (\boldsymbol {u}, \boldsymbol {w}), \theta) ] \\ + \lambda \text {c o v l o s s} _ {t} \\ \end{array}
$$
where  $u_{t}$  is the reference node at time step  $t$  with reference head  $u_{k}$  and  $l$  is the reference edge label between  $u_{k}$  and  $u_{j}$ . The form  $g_{\phi}(\mathbf{u},\mathbf{w})$  is short for the latent graph samples from uniform distribution to HardKuma distribution (§§3.1).
Different from Bastings et al. (2019), we do not limit the sparsity of sampled latent graphs, i.e. we do not control the proportion of zeros in the latent graph, because we prefer to retain the probabilistic connection information of each word in  $w$ . Finally, we introduce coverage loss into our estimation due to reduce duplication of node generation (See et al., 2017).
3.4 Parsing
We directly generate the latent graph by the PDF function of HardKuma distribution with trained parameters  $\mathbf{a}$  and  $\mathbf{b}$ . In the concept identification stage, we decode the node from the final probability distribution  $P^{(\mathrm{node})}(u_t)$  at each time step, and apply beam search for sequentially generating the concept nodes  $\mathbf{u}$  and deterministically assigning corresponding indices  $\mathbf{d}$ . For edge prediction, we use a bi-affine classifier to calculate the edge scores under the generated nodes  $\mathbf{u}$  and indices  $\mathbf{d}$ :
$$
\boldsymbol {S} = \left\{\text {s c o r e} _ {i, j} ^ {\text {(e d g e)}} \mid 0 \leq i, j \leq m \right\}.
$$
Similar to Zhang et al. (2019a), we apply a maximum spanning tree (MST) algorithm (Chu, 1965; Edmonds, 1967) to generate complete AMR graph and restore the re-entrance relations by merging the receptive nodes via their indices.
We use two standard AMR corpora: AMR1.0 (LDC2014T12) and AMR 2.0 (LDC2017T10). AMR 1.0 contains 13051 sentences in total. AMR
Data	Parser	F1(%)
AMR 2.0	Cai and Lam (2019)	73.2
Lyu and Titov (2018)	74.4±0.2
Lindemann et al. (2019)	75.3±0.1
Naseem et al. (2019)	75.5
Zhang et al. (2019a)	76.3±0.1
- w/o BERT	74.6
Zhang et al. (2019b)	77.0±0.1
Ours	77.5±0.2
- w/o BERT	75.5±0.2
AMR 1.0	Flanigan et al. (2016)	66.0
Pust et al. (2015)	67.1
Wang and Xue (2017)	68.1
Guo and Lu (2018)	68.3±0.4
Zhang et al. (2019a)	70.2±0.1
- w/o BERT	68.8
Zhang et al. (2019b)	71.3±0.1
Ours	71.8±0.2
- w/o BERT	70.0±0.2
2.0 is larger which is split into 36521, 1368 and 1371 sentences in training, development and testing sets respectively. We treat in AMR 2.0 as the main dataset in our experiments since it is larger.
We tune hyperparameters on the development set, and store the checkpoints under best development results for evaluation. We employ the pre-processing and post-processing methods from Zhang et al. (2019a), and get the syntactic dependencies via Stanford Corenlp (Manning et al., 2014). We train our model jointly with the Adam optimizer (Kingma and Ba, 2015). The learning rate is decayed based on the results of development set in training. Training takes approximately 22 hours on two Nivida GeForce GTX 2080 Ti.
Main Results We compare the SMATCH F1 scores (Cai and Knight, 2013) against previous best reported models and other recent AMR parsers. Table 1 summarizes the results on both AMR 1.0 and AMR 2.0 data sets. For AMR 2.0, with the benefit from the fused structural information, we improve our baseline (Zhang et al., 2019a) by  $1.2\%$  F1 in the full model, and  $0.9\%$  F1
Table 1: Main results of SMATCH F1 on AMR 2.0 (LDC2017T10) and 1.0 (LDC2014T12) test sets. Results are evaluated over 3 runs.
Metric	N'19	Z'19a	Z'19b	Ours
SMATCH	75.5	76.3	77	77.5
Unlabeled	80	79	80	80.4
No WSD	76	77	78	78.2
Reentrancies	56	60	61	61.1
Concepts	86	85	86	85.9
Named Ent.	83	78	79	78.8
Wikification	80	86	86	86.5
Negation	67	75	77	76.1
SRL	72	70	71	71.0
Table 2: Fine-grained F1 scores on the AMR 2.0 (LDC2017T10) test set. N'18 is Naseem et al. (2019);  $Z^{\prime}19_{a}$  is Zhang et al. (2019a);  $Z^{\prime}19_{b}$  is Zhang et al. (2019b)
is gained without pre-trained BERT embeddings<sup>4</sup>. In addition, our model outperforms the best reported model (Zhang et al., 2019b) by  $0.5\%$  F1. On AMR 1.0, there are only about 10k sentences for training. We outperform the best results by  $0.5\%$  Smatch F1. We observe that for the smaller data set, our model has a greater improvement of  $1.6\%$  F1 than for the larger data set ( $1.2\%$  F1 comparing with our baseline.)