SlowGuess/NAACL_2022_OCR huggingface dataset:
Ce Zheng $^{1}$ , Xudong Chen $^{1,2}$ , Runxin Xu $^{1}$ , Baobao Chang $^{1*}$
<sup>1</sup>The MOE Key Laboratory of Computational Linguistics, Peking University, China
$^{2}$ School of Software and Microelectronics, Peking University, China
{zce1112zslx,xdc}@pku.edu.cn
runxinxu@gmail.com; chbb@pku.edu.cn
Frame semantic parsing is a fundamental NLP task, which consists of three subtasks: frame identification, argument identification and role classification. Most previous studies tend to neglect relations between different subtasks and arguments and pay little attention to ontological frame knowledge defined in FrameNet. In this paper, we propose a Knowledge-guided Incremental semantic parser with Double-graph (KID). We first introduce Frame Knowledge Graph (FKG), a heterogeneous graph containing both frames and FEs (Frame Elements) built on the frame knowledge so that we can derive knowledge-enhanced representations for frames and FEs. Besides, we propose Frame Semantic Graph (FSG) to represent frame semantic structures extracted from the text with graph structures. In this way, we can transform frame semantic parsing into an incremental graph construction problem to strengthen interactions between subtasks and relations between arguments. Our experiments show that KID outperforms the previous state-of-the-art method by up to 1.7 F1-score on two FrameNet datasets. Our code is availavle at https://github.com/PKUnlp-icler/KID.
The frame semantic parsing task (Gildea and Jurafsky, 2002; Baker et al., 2007) aims to extract frame semantic structures from sentences based on the lexical resource FrameNet (Baker et al., 1998). As shown in Figure 1, given a target in the sentence, frame semantic parsing consists of three subtasks: frame identification, argument identification and role classification. Frame semantic parsing can also contribute to downstream NLP tasks such as machine reading comprehension (Guo et al., 2020), relation extraction (Zhao et al., 2020) and dialogue generation (Gupta et al., 2021).
Figure 1: Given the target receive in this sentence, the frame identification is to identify the frame Receiving evoked by it; the argument identification is to find the arguments (He, the book, ...) of this target; the role classification is to assign frame elements (Recipient, Theme, ...) as semantic roles to these arguments.
FrameNet is an English lexical database, which defines more than one thousand hierarchically-related frames to represent situations, objects or events, and nearly 10 thousand FEs (Frame Elements) as frame-specific semantic roles with more than 100,000 annotated exemplar sentences. In addition, FrameNet defines ontological frame knowledge for each frame such as frame semantic relations, FE mappings and frame/FE definitions. The frame knowledge plays an important role in frame semantic parsing. Most previous approaches (Kshirsagar et al., 2015; Yang and Mitchell, 2017; Peng et al., 2018) only use exemplar sentences and ignore the ontological frame knowledge. Recent researches (Jiang and Riloff, 2021; Su et al., 2021) introduce frame semantic relations and frame definitions into the subtask frame identification. Different from previous work, we construct a heterogeneous graph named Frame Knowledge Graph (FKG) based on frame knowledge to model multiple semantic relations between frames and frames, frames and FEs, as well as FEs and FEs. Furthermore, we apply FKG to all subtasks of frame semantic parsing, which can fully inject frame knowledge into frame semantic parsing. The knowledge-enhanced representations of frames and FEs are learned in a unified vector space and this can also strengthen interactions between frame identification and other subtasks.
Most previous systems neglect interactions be-
Human Language Technologies, pages 4998 - 5011
Figure 2: An example of how frame knowledge contributes to frame semantic parsing. The frame semantic relations and FE mappings guide inter-frame reasoning (from left sentence to right); and the FE definitions help with intra-frame reasoning (Theme to Role and Role to Donor).
tween subtasks, they either focus on one or two subtasks (Hermann et al., 2014; FitzGerald et al., 2015; Marcheggiani and Titov, 2020) of frame semantic parsing or treat all subtasks independently (Das et al., 2014; Peng et al., 2018). Furthermore, in argument identification and role classification, previous approaches process each argument separately with sequence labeling strategy (Yang and Mitchell, 2017; Bastianelli et al., 2020) or span-based graphical models (Tackstrom et al., 2015; Peng et al., 2018). In this paper, we propose Frame Semantic Graph (FSG) to represent frame semantic structures and treat frame semantic parsing as a process to construct this graph incrementally. With graph structure, historical decisions of parsing can guide the current decision of argument identification and role classification, which highlights interactions between subtasks and arguments.
Based on two graphs mentioned above, we propose our framework KID (Knowledge-guided Incremental semantic parser with Double-graph). FKG provides a static knowledge background for encoding frames and FEs while FSG represents dynamic parsing results in frame semantic parsing and highlights relations between arguments.
Overall, our contributions are listed as follow:
- We build FKG based on the ontological frame knowledge in FrameNet. FKG incorporates frame semantic parsing with structured frame knowledge, which can get knowledge-enhanced representations of frames and FEs.
- We propose FSG to represent the frame semantic structures. We treat frame semantic parsing as a process to construct the graph incrementally. This graph focuses on the targetargument and argument-argument relations.
We evaluate the performance of KID on two FrameNet datasets: FN 1.5 and FN 1.7, the results show that the KID achieves state-of-the-art on these datasets by increasing up to 1.7 points on F1-score. Our extensive experiments also verify the effectiveness of these two graphs.
2 Ontological Frame Knowledge
Frame semantics relates linguistic semantics to encyclopedic knowledge and advocates that one cannot understand the semantic meaning of one word without essential frame knowledge related to the word (Fillmore and Baker, 2001). The frame knowledge of a frame contains frame/FE definitions, frame semantic relations and FE mappings. FrameNet defines 8 kinds of frame semantic relations such as Inheritance, Perspective_on and Using; for any two related frames, the FrameNet defines FE mappings between their FEs. For example, the frame Receiving inherits from Getting and the FE Donor of Receiving is mapped to the FE Source of Getting. Each frame or FE has its own definition and may mention other FEs.
We propose two ways of reasoning about frame semantic parsing: inter-frame reasoning and intra-frame reasoning in Figure 2. Frame knowledge mentioned above can guide both ways of reasoning. The frame semantic relation between Receiving and Getting and FE mappings associated with it allow us to learn from the left sentence when parsing the right sentence because similar argument spans of two sentences will have related FEs as their roles. The FE definitions reflect dependencies between arguments. The definition of Role in frame Receiving mentions Theme and Donor, which reflects dependencies between argument the book and argument as a gift.
3 Task Formulation
Given a sentence  $S = w_{0}, \ldots, w_{n-1}$  with a target span  $t$  in  $S$ , the frame semantic parsing aims to extract the frame semantic structure of  $t$ . Suppose that there are  $k$  arguments of  $t$  in  $S$ :  $a_{0}, \ldots, a_{k-1}$ , subtasks can be formulated as follow:
- Frame identification: finding an  $f \in \mathcal{F}$  evoked by target  $t$ , where  $\mathcal{F}$  denotes the set of all frames in the FrameNet.
- Argument identification: finding the boundaries  $i_{\tau}^{s}$  and  $i_{\tau}^{e}$  for each argument  $a_{\tau} = w_{i_{\tau}^{s}}, \ldots, w_{i_{\tau}^{e}}$ .
- Role classification: assigning an FE  $r_{\tau} \in \mathcal{R}_f$  to each  $a_{\tau}$ , where  $\mathcal{R}_f$  denotes the set of all FEs of frame  $f$ .
KID encodes all frames and FEs to knowledge-enhanced representations via frame knowledge graph encoder (section 4.1). For a sentence with a target, contextual representations of tokens are derived from the sentence encoder (section 4.2). Frame semantic parsing is regarded as a process to build FSG incrementally from an initial target node to complete FSG. Frame identification finds a frame evoked by the target and combines the target with its frame into the initial node of FSG (section 4.3.1). Argument identification (section 4.3.2) and role classification (section 4.3.3) for each argument is based on the current snapshot of partial FSG considering all historical decisions. Section 4.4 tells how frame semantic graph decoder encodes partial FSG to its representation and how it expands FSG incrementally, which is also shown in Figure 4.
4.1 Frame Knowledge Graph Encoder
FKG is an undirected multi-relational heterogeneous graph, and Figure 3 shows a subgraph of FKG. Its nodes contain both frames and FEs and there are four kinds of relations in FKG: frame-FE, frame-frame, inter-frame FE-FE and intra-frame FE-FE relations. The following will show how we extract these relations from frame knowledge:
Frame-FE: we connect a frame with its FEs so that we can learn representations of frames and FEs in a unified vector space to strengthen interactions between frame identification and other subtasks.
Frame-frame and inter-frame FE-FE: these two kinds of relations are frame semantic relations and
Figure 3: A subgraph of FKG. We only show intra-frame and inter-frame FE-FE relations in the frame Receiving. Inside the solid rectangular box are a frame and its FEs.
FE mappings respectively and here we ignore relation types of frame semantic relations. They can both guide inter-frame reasoning in Figure 2.
Intra-frame FE-FE: If the definition of an FE mentions another FE in the same frame, they will have intra-frame FE-FE relations with each other. This relation can help with intra-frame reasoning and strengthen interactions between arguments.
The frame knowledge graph encoder aims to get knowledge-enhanced representations of nodes in FKG via an RGCN (Schlichtkrull et al., 2018) module. We use  $\mathcal{F}$  to represent all frames in FrameNet and  $\mathcal{R}_f$  to represent all FEs of frame  $f$ . In addition, we use  $\mathcal{R} = \bigcup_{f\in \mathcal{F}}\mathcal{R}_f$  to represent all FEs in the FrameNet. Let  $0,\ldots ,|\mathcal{F}| - 1$  denote all frames and  $|\mathcal{F}|,\dots ,|\mathcal{F}| + |\mathcal{R}| - 1$  denote all FEs. Moreover, we introduce a special dummy node indexing  $|\mathcal{F}| + |\mathcal{R}|$  into FKG. So the vectors  $y_0,\ldots ,y_M\in \mathbb{R}^{d_n}$  denote the representations of all nodes in FKG, where  $M = |\mathcal{F}| + |\mathcal{R}|$ .
For each node  $i$ , we take a randomly initialized embedding  $y_{i}^{(0)} \in \mathbb{R}^{d_{n}}$  as the input feature of the RGCN module. Then we can get representations of all frames and FEs:
$$
y _ {0}, \dots , y _ {M} = \operatorname {R G C N} \left(y _ {0} ^ {(0)}, \dots , y _ {M} ^ {(0)}\right) \tag {1}
$$
The RGCN module models four kinds of relations: Frame-FE, intra-frame FE-FE, frame-frame and inter-frame FE-FE. For better modeling inter-frame relations between FEs, we also fuse name information into representations of FEs. The FEs whose names are the same will share the same embeddings, i.e. for  $i,j\geq |\mathcal{F}|,y_i^{(0)} = y_j^{(0)}$  if the name of  $i$  is the same as  $j$ .
Figure 4: Based on the representation  $g_{\tau}$  of partial  $G_{\tau}$ , frame semantic graph decoder identifies new argument as a gift with pointer networks, and label it with FE Role.  $G_{\tau}$  will be updated to  $G_{\tau +1}$  with (as a gift, Role).
4.2 Sentence Encoder
The sentence encoder converts tokens of the sentence  $S = w_{0}, \ldots, w_{n-1}$  to their representations  $h_{0}, \ldots, h_{n-1} \in \mathbb{R}^{d_{h}}$ .
We use LSTM (Hochreiter and Schmidhuber, 1997) and GCN (Kipf and Welling, 2016) to model both sequential structure and dependency structure:
$$
\alpha_ {0}, \dots , \alpha_ {n - 1} = \operatorname {B i L S T M} \left(e _ {0}, \dots , e _ {n - 1}\right) \tag {2}
$$
$$
\beta_ {0}, \dots , \beta_ {n - 1} = \operatorname {G C N} \left(\alpha_ {0}, \dots , \alpha_ {n - 1}, \mathcal {T}\right) \tag {3}
$$
$e_i$  denotes the embedding of word  $w_i$ . We get contextual representations  $h_i$  by adding  $\beta_i$  to  $\alpha_i$ . We follow previous studies (Marcheggiani and Titov, 2020; Bastianelli et al., 2020) to use syntax structures like dependency tree  $\mathcal{T}$  of  $S$  here.
Furthermore, we use boundary information (Wang and Chang, 2016; Cross and Huang, 2016; Ouchi et al., 2018) to represent spans like  $s = w_{i}, \ldots, w_{j}$  based on token representations because we need to embed spans into the vector space of FKG in frame identification and role classification:
$$
Q (i, j) = \operatorname {F F N} \left(\left(h _ {j} - h _ {i}\right) \oplus \left(h _ {j} + h _ {i}\right)\right) \tag {4}
$$
The dimension of  $Q(i,j)$  is  $d_{n}$ . The  $\oplus$  denotes concatenation operation and FFN denotes Feed Forward Network.
4.3 Frame Semantic Parsing
4.3.1 Frame Identification
A frame  $f \in \mathcal{F}$  will be identified based on the target  $t$ , representations of tokens  $h_0, \ldots, h_{n-1}$
and representations of frames  $y_0,\ldots ,y_{|\mathcal{F}| - 1}$  with a scoring module. The target  $t = w_{i_t^s},\dots ,w_{i_t^e}$  will be embedded to the vector space of all frames as  $\gamma_t\in \mathbb{R}^{d_n}$ . We can calculate dot product normalized similarities between  $\gamma_{t}$  and all frames  $Y_{\mathcal{F}} = (y_0,\dots ,y_{|\mathcal{F}| - 1})\in \mathbb{R}^{d_n\times |\mathcal{F}|}$  to get the probability distribution of  $f$ . For short, we use  $\pi_t$  to denote  $Q(i_t^s,i_t^e)$ :
$$
\gamma_ {t} = \tanh  (\operatorname {F F N} (\pi_ {t})) \tag {5}
$$
$$
P (f | S, t) = \operatorname {s o f t m a x} \left(Y _ {\mathcal {F}} ^ {\top} \cdot \gamma_ {t}\right) \tag {6}
$$
4.3.2 Argument Identification
Based on  $g_{\tau}$ , the representation of current snapshot of FSG  $G_{\tau}$ , we need to find an argument  $a_{\tau} = w_{i_{\tau}^{s}}, \ldots, w_{i_{\tau}^{e}}$ . We use pointer networks (Vinyals et al., 2015) to identify its start and end positions  $i_{\tau}^{s}$  and  $i_{\tau}^{e}$  separately via an attention mechanism, which is more efficient than traditional span-based model (Chen et al., 2021). Take  $i_{\tau}^{s}$  as example:
$$
\rho_ {\tau} ^ {s} = \operatorname {F F N} \left(g _ {\tau}\right) \tag {7}
$$
$$
P \left(i _ {\tau} ^ {s} \mid S, G _ {\tau}\right) = \operatorname {s o f t m a x} \left(H ^ {\top} \cdot \rho_ {\tau} ^ {s}\right) \tag {8}
$$
$H = (h_0,\dots ,h_{n - 1})\in \mathbb{R}^{d_h\times n}$  represents the output of the sentence encoder, and  $\rho_{\tau}^{s}\in \mathbb{R}^{d_{h}}$  is used to find the start position of argument span  $a_{\tau}$
4.3.3 Role Classification
Based on  $g_{\tau}$  and  $a_{\tau}$ , we embed  $a_{\tau}$  into the vector space of FEs as  $\gamma_{a_{\tau}} \in \mathbb{R}^{d_n}$ . Similar to frame identification, we calculate dot product normalized similarities between  $\gamma_{a_{\tau}}$  and all FEs  $Y_{\mathcal{R}} =$
$(y_{|\mathcal{F}|},\dots ,y_{|\mathcal{F}| + |\mathcal{R}|})\in \mathbb{R}^{d_n\times (|\mathcal{R}| + 1)}$  to get the conditional probability distribution of  $r$  given  $a_{\tau}$  and  $G_{\tau}$
$$
\gamma_ {a _ {\tau}} = \operatorname {F F N} \left(\pi_ {a _ {\tau}} \oplus g _ {\tau}\right) \tag {9}
$$
$$
P \left(r _ {\tau} \mid S, G _ {\tau}, a _ {\tau}\right) = \operatorname {s o f t m a x} \left(Y _ {\mathcal {R}} ^ {\top} \cdot \gamma_ {a _ {\tau}}\right) \tag {10}
$$
4.4 Frame Semantic Graph Decoder
We propose FSG to represent the frame semantic structure of  $t$  in the sentence  $S$  and we treat the frame semantic parsing as a process to construct FSG incrementally. Intermediate results of FSG are partial FSGs representing all historical decisions, which highlights interactions between arguments. Suppose that there are  $k$  arguments of target  $t$ :  $a_0, \ldots, a_{k-1}$  with their roles  $r_0, \ldots, r_{k-1}$ . For  $\tau$ -th snapshot of FSG  $G_{\tau}$ , it contains  $\tau + 1$  nodes: one target node  $(t, f)$  and  $\tau$  argument nodes (if exist)  $(a_0, r_0), \ldots, (a_{\tau - 1}, r_{\tau - 1})$ . The target node will be connected with all argument nodes. The indices of nodes in  $G_{\tau}$  depend on the order in which they are added into the graph, 0 denotes the target node and 1, ...,  $\tau$  denotes  $(a_0, r_0), \ldots, (a_{\tau - 1}, r_{\tau - 1})$ .
We encode  $G_{\tau}$  to its representation  $g_{\tau}$ :
$$
g _ {\tau} = \text {M a x p o o l i n g} \left(z _ {0}, \dots , z _ {\tau}\right) \tag {11}
$$
$$
z _ {0}, \dots , z _ {\tau} = \operatorname {G C N} \left(z _ {0} ^ {(0)}, \dots , z _ {\tau} ^ {(0)}, G _ {\tau}\right) \tag {12}
$$
$$
z _ {j} ^ {(0)} = \left\{ \begin{array}{l l} \pi_ {t} \oplus y _ {i _ {f}}, & j = 0 \\ \pi_ {a _ {j}} \oplus y _ {i _ {r _ {j}}}, & j = 1, \dots , \tau \end{array} \right. \tag {13}
$$
where  $i_{f}$  and  $i_{r_j}$  denotes indices of  $f$  and  $r_j$  in FKG, and  $\pi_{a_j} = Q(i_j^s,i_j^e)$ . The GCN module is to encode partial FSG.
Based on the representation  $g_{\tau}$  of each snapshot  $G_{\tau}$ , KID predicts boundary positions of argument  $a_{\tau}$  and assign an FE  $r_{\tau}$  as its semantic role (section 4.3.2,4.3.3). The  $G_{\tau}$  will be updated to  $G_{\tau +1}$  with the new node  $(a_{\tau},r_{\tau})$  until the  $r_{\tau}$  is the special dummy node in FKG. Figure 4 shows how to find a new node and add it into the FSG.
5 Training and Inference
We train KID with all subtasks jointly by optimizing the loss function  $\mathcal{L}$  since representations of frames and FEs are learned in a unified vector space.
#exemplar	#train	#dev	#test
FN 1.5	153952	17143	2333	4458
FN 1.7	192461	19875	2309	6722
Table 1: Number of instances in two datasets.
$$
\mathcal {L} ^ {f} = - \log P (f = f ^ {\text {g o l d}} | S, t) \tag {14}
$$
$$
\mathcal {L} _ {s / e} ^ {a} = - \sum_ {\tau = 0} ^ {k - 1} \log P \left(i _ {\tau} ^ {s / e} = I _ {\tau} ^ {s / e} | S, G _ {\tau}\right) \tag {15}
$$
$$
\mathcal {L} ^ {r} = - \sum_ {\tau = 0} ^ {k} \log P \left(r _ {\tau} = r _ {\tau} ^ {\text {g o l d}} \mid S, G _ {\tau}, a _ {\tau}\right) \tag {16}
$$
$$
\mathcal {L} = \lambda_ {1} \mathcal {L} ^ {f} + \lambda_ {2} \left(\mathcal {L} _ {s} ^ {a} + \mathcal {L} _ {e} ^ {a}\right) + \lambda_ {3} \mathcal {L} ^ {r} \tag {17}
$$
where  $f^{\mathrm{gold}}$  is the gold frame and  $I_{\tau}^{s}, I_{\tau}^{e}, r_{\tau}^{\mathrm{gold}}$  are gold labels of argument  $a_{\tau}$ .  $r_{k}^{\mathrm{gold}}$  is "Dummy", indicating the end of the parsing. We force our model to identify arguments in a left-to-right order, i.e.  $a_0$  is the leftmost argument in  $S$ . We use gold frame in the initial node of FSG:  $G_0 = (t, f^{\mathrm{gold}})$  while other nodes are predicted autoregressively:  $G_{\tau + 1} = G_{\tau} + (\hat{a}_{\tau}, \hat{r}_{\tau}), \hat{r}_{\tau} \in \mathcal{R}_{f^{\mathrm{gold}}}$ .
KID predicts frame and all arguments with their roles in a sequential way. We use probabilities above with some constraints: 1. We use lexicon filtering strategy: for a target  $t$ , we can use the lemma  $\ell_t$  of it to find a subset of frames  $\mathcal{F}_{\ell_t} \subset \mathcal{F}$  so that we can reduce the searching space; 2. Similarly, we take  $R_{\hat{f}}$  instead of  $R$  as the set of candidate FEs; 3. In argument identification, we will mask spans that are already selected as arguments, and  $i_{\tau}^{e}$  should be no less than  $i_{\tau}^{s}$ .
We evaluate KID on two FrameNet datasets: FN 1.5 and FN 1.7. FN 1.7 is an extension version of FN 1.5, including more fine-grained frames and more instances. FN 1.5 defines 1019 frames and 9634 FEs while FN 1.7 defines 1221 frames and 11428 FEs. We use the same splits of datasets as Peng et al. (2018), and we also follow Kshirsagar et al. (2015); Yang and Mitchell (2017); Peng et al.
1https://framenet.icsi.berkeley.edu/ fndrupal/about
Model	Frame IdAccuracy	Arg Id (gold frame)	Full structure
Precision	Recall	F1-score	Precision	Recall	F1-score
SEMAFOR (2014)	83.6	65.6	53.8	59.1	-	-	-
Hermann et al. (2014)	88.4	-	-	-	74.3	66.0	69.9
SEMAFOR (HI) (2015)	-	67.2	54.8	60.4	-	-	-
Täckström et al. (2015)	-	-	-	-	75.4	65.8	70.3
FitzGerald et al. (2015)	-	-	-	-	74.8	65.5	69.9
open-SESAME (2017)	86.9	69.4	60.5	64.6	71.0	67.8	69.4
KID (GloVe)	89.5	64.6	68.2	66.4	73.8	76.8	75.3
SEMAFOR (HI + exemplar) (2015)	-	66.0	60.4	63.1	-	-	-
Yang and Mitchell (2017)	88.2	70.2	60.2	65.5	77.3	71.2	74.1
Swayamdipta et al. (2018)	-	67.8	66.2	67.0	-	-	-
Peng et al. (2018)	89.2	-	-	-	79.2	71.7	75.3
Marcheggiani and Titov (2020)	-	69.8	68.8	69.3	-	-	-
Chen et al. (2021)	89.4	-	-	-	75.1	76.9	76.0
KID (GloVe + exemplar)	90.0	66.8	73.7	70.1	75.5	80.1	77.7
Bastianelli et al. (2020) (JL)	90.1	74.6	74.4	74.5	-	-	-
Chen et al. (2021) (BERT)	90.5	-	-	-	78.2	82.4	80.2
Jiang and Riloff (2021)	91.3	-	-	-	-	-	-
Su et al. (2021)	92.1	-	-	-	-	-	-
KID (BERT)	91.7	71.7	79.0	75.2	79.3	84.2	81.7
Table 2: Empirical results on the test set of FN 1.5. All models are single-task, non-ensemble. The upper block lists models trained without exemplar instances, the lower block lists models with pretrained language models. KID outperforms other models under all conditions except Su et al. (2021). We also train our model with multiple runs and report the statistical analysis with a significance testing in appendix C.
(2018); Chen et al. (2021) to include exemplar instances as training instances. As Kshirsagar et al. (2015) states that there exists a domain gap between exemplar instances and original training instances, we follow Chen et al. (2021) to use exemplar instances as pre-training instances and further train our model in original training instances. Table 1 shows the numbers of instances in two datasets.
We compare KID with previous models (see appendix C) on FN 1.5 and FN 1.7. We focus on three metrics: frame acc, arg F1 and full structure F1. $^{2}$  Full structure F1 shows the performance of models on extracting full frame semantic structures from text, frame acc denotes accuracy of frame identification and arg F1 evaluates the results of argument identification and role classification with gold frames. All metrics are evaluated in test set.
Table 2 shows results on FN 1.5. For a fair comparison, we divide models into three parts: the first part of models do not use exemplar instances as training data; the second part of models use exemplar instances without any pretrained language models; the third part of models use pretrained language models. KID (GloVe) uses GloVe
(Pennington et al., 2014) as word embeddings and KID (BERT) fine-tunes pretrained language model BERT (Devlin et al., 2019) to encode word representations. KID achieves state-of-the-art of these metrics under almost all circumstances, and we also train our model with multiple runs, which shows KID (GloVe + exemplar) and KID (BERT) outperforms previous state-of-the-art models by 1.4 and 1.3 full structure F1-score averagely. There is an exception that our model with BERT does not outperform Su et al. (2021) on frame identification accuracy and we find that the number of train, validation and test instances reported by them are a little bit smaller than ours. Results on FN 1.7 and statistical analysis of our model with multiple runs are listed in appendix C.
It is worth noting that KID achieves much higher recall than other models. We attribute this to the incremental strategy of building FSG. By constructing FSG incrementally, KID can capture relations between arguments and identify arguments that are hard to find in other models.
To prove the effectiveness of double-graph architecture, we conduct further experiments with KID on FN 1.5. Table 3 shows ablation study on double-graph architecture. w/o FSG uses LSTM instead of our frame semantic graph decoder. It takes a