SlowGuess/ACL_2020_OCR huggingface dataset:
Dian  $\mathbf{Y}\mathbf{u}^{1\dagger}$  Kai Sun²² Claire Cardie² Dong  $\mathbf{Y}\mathbf{u}^{1}$
Tencent AI Lab, Bellevue, WA
$^{2}$ Cornell University, Ithaca, NY
{yudian, dyu}  $@$  Tencent.com, ks985@cornell.edu, cardie@cscornell.edu
We present the first human-annotated dialogue-based relation extraction (RE) dataset DialogRE, aiming to support the prediction of relation(s) between two arguments that appear in a dialogue. We further offer DialogRE as a platform for studying cross-sentence RE as most facts span multiple sentences. We argue that speaker-related information plays a critical role in the proposed task, based on an analysis of similarities and differences between dialogue-based and traditional RE tasks. Considering the timeliness of communication in a dialogue, we design a new metric to evaluate the performance of RE methods in a conversational setting and investigate the performance of several representative RE methods on DialogRE. Experimental results demonstrate that a speaker-aware extension on the best-performing model leads to gains in both the standard and conversational evaluation settings. DialogRE is available at https://dataset.org/dialogre/.
Cross-sentence relation extraction, which aims to identify relations between two arguments that are not mentioned in the same sentence or relations that cannot be supported by any single sentence, is an essential step in building knowledge bases from large-scale corpora automatically (Ji et al., 2010; Swampillai and Stevenson, 2010; Surdeanu, 2013). It has yet to receive extensive study in natural language processing, however. In particular, although dialogues readily exhibit cross-sentence relations, most existing relation extraction tasks focus on texts from formal genres such as professionally written and edited news reports or well-edited websites (Elsahar et al., 2018; Yao et al., 2019;
S1: Hey Pheebs.
S2: Hey!
S1: Any sign of your brother?
S2: No, but he's always late.
S1: I thought you only met him once?
S2: Yeah, I did. I think it sounds y'know big sistery, y'know, 'Frank's always late.'
S1: Well relax, he'll be here.
Argument pair	Trigger	Relation type
R1	(Frank, S2)	brother	per:siblings
R2	(S2, Frank)	brother	per:siblings
R3	(S2, Pheecs)	none	per:alternate_names
R4	(S1, Pheecs)	none	unanswerable
Table 1: A dialogue and its associated instances in DialogRE. S1, S2: anonymized speaker of each utterance.
Mesquita et al., 2019; Grishman, 2019), while dialogues have been under-studied.
In this paper, we take an initial step towards studying relation extraction in dialogues by constructing the first human-annotated dialogue-based relation extraction dataset, DialogRE. Specifically, we annotate all occurrences of 36 possible relation types that exist between pairs of arguments in the 1,788 dialogues originating from the complete transcripts of Friends, a corpus that has been widely employed in dialogue research in recent years (Catzzone et al., 2010; Chen and Choi, 2016; Chen et al., 2017; Zhou and Choi, 2018; Rashid and Blanco, 2018; Yang and Choi, 2019). Altogether, we annotate 10,168 relational triples. For each (subject, relation type, object) triple, we also annotate the minimal contiguous text span that most clearly expresses the relation; this may enable researchers to explore relation extraction methods that provide fine-grained explanations along with evidence sentences. For example, the bolded text span "brother" in Table 1 indicates the PER:SIBLINGS relation (R1 and R2) between speaker 2 (S2) and "Frank".
Our analysis of DialogRE indicates that the supporting text for most (approximately  $96.0\%$ ) an
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4927-4940 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
notated relational triples includes content from multiple sentences, making the dataset ideal for studying cross-sentence relation extraction. This is perhaps because of the higher person pronoun frequency (Biber, 1991) and lower information density (Wang and Liu, 2011) in conversational texts than those in formal written texts. In addition,  $65.9\%$  of relational triples involve arguments that never appear in the same turn, suggesting that multi-turn information may play an important role in dialogue-based relation extraction. For example, to justify that "Pheebs" is an alternate name of S2 in Table 1, the response of S2 in the second turn is required as well as the first turn.
We next conduct a thorough investigation of the similarities and differences between dialogue-based and traditional relation extraction tasks by comparing DialogRE and the Slot Filling dataset (McNamee and Dang, 2009; Ji et al., 2010, 2011; Surdeanu, 2013; Surdeanu and Ji, 2014), and we argue that a relation extraction system should be aware of speakers in dialogues. In particular, most relational triples in DialogRE  $(89.9\%)$  signify either an attribute of a speaker or a relation between two speakers. The same phenomenon occurs in an existing knowledge base constructed by encyclopedia collaborators, relevant to the same dialogue corpus we use for annotation (Section 3.2). Unfortunately, most previous work directly applies existing relation extraction systems to dialogues without explicitly considering the speakers involved (Yoshino et al., 2011; Wang and Cardie, 2012).
Moreover, traditional relation extraction methods typically output a set of relations only after they have read the entire document and are free to rely on the existence of multiple mentions of a relation throughout the text to confirm its existence. However, these methods may be insufficient for powering a number of practical real-time dialogue-based applications such as chatbots, which would likely require recognition of a relation at its first mention in an interactive conversation. To encourage automated methods to identify the relationship between two arguments in a dialogue as early as possible, we further design a new performance evaluation metric for the conversational setting, which can be used as a supplement to the standard F1 measure (Section 4.1).
In addition to dataset creation and metric design, we adapt a number of strong, representative learning-based relation extraction methods (Zeng
et al., 2014; Cai et al., 2016; Yao et al., 2019; Devlin et al., 2019) and evaluate them on DialogRE to establish baseline results on the dataset going forward. We also extend the best-performing method (Devlin et al., 2019) among them by letting the model be aware of the existence of arguments that are dialogue participants (Section 4.2). Experiments on DialogRE demonstrate that this simple extension nevertheless yields substantial gains on both standard and conversational RE evaluation metrics, supporting our assumption regarding the critical role of tracking speakers in dialogue-based relation extraction (Section 5).
The primary contributions of this work are as follows: (i) we construct the first human-annotated dialogue-based relation extraction dataset and thoroughly investigate the similarities and differences between dialogue-based and traditional relation extraction tasks, (ii) we design a new conversational evaluation metric that features the timeliness aspect of interactive communications in dialogue, and (iii) we establish a set of baseline relation extraction results on DialogRE using standard learning-based techniques and further demonstrate the importance of explicit recognition of speaker arguments in dialogue-based relation extraction.
2 Data Construction
We use the transcripts of all ten seasons (263 episodes in total) of an American television situation comedy *Friends*, covering a range of topics. We remove all content (usually in parentheses or square brackets) that describes non-verbal information such as behaviors and scene information.
2.1 Relation Schema
We follow the slot descriptions<sup>1</sup> of the Slot Filling (SF) task in the Text Analysis Conference Knowledge Base Population (TAC-KBP) (McNamee and Dang, 2009; Ji et al., 2010, 2011; Surdeanu, 2013; Surdeanu and Ji, 2014), which primarily focuses on biographical attributes of person (PER) entities and important attributes of organization (ORG) entities. As the range of topics in *Friends* is relatively restricted compared to large-scale news corpora such as Gigaword (Parker et al., 2011), some relation types (e.g., PER:CHARGES, and ORG:SUBSIDIARIES) seldom appear in the texts. Additionally, we consider new relation types such as PER:GIRL/BOYFRIEND and PER:NEIGHBOR that
<sup>1</sup> http://surdeanu.info/kbp2014/def.php.
ID	Subject	Relation Type	Object	Inverse Relation	TR (%)
1	PER	per:positive_impression	NAME	70.4
2	PER	per:negative_impression	NAME	60.9
3	PER	per:acquaintance	NAME	per:acquaintance	22.2
4	PER	per:alumni	NAME	per:alumni	72.5
5	PER	per:boss	NAME	per:subordinate	58.1
6	PER	per:subordinate	NAME	per:boss	58.1
7	PER	per:client	NAME	50.0
8	PER	per:dates	NAME	per:dates	72.5
9	PER	per:friends	NAME	per:friends	94.7
10	PER	per:girl/boyfriend	NAME	per:girl/boyfriend	86.1
11	PER	per:neighbor	NAME	per:neighbor	71.2
12	PER	per:roommate	NAME	per:roommate	89.9
13	PER	per:children*	NAME	per:parents	85.4
14	PER	per:other_family*	NAME	per:other_family	52.0
15	PER	per:parents*	NAME	per:children	85.4
16	PER	per:siblings*	NAME	per:siblings	80.5
17	PER	per:s spouse*	NAME	per:s spouse	86.7
18	PER	per:place_of_residence**	NAME	gpe:residents_of_place	42.9
19	PER	per:place_of_birth**	NAME	gpe:births_in_place	100.0
20	PER	per:visited_place	NAME	gpe:visitors_of_place	43.0
21	PER	per:origin*	NAME	3.8
22	PER	per:employee_or_member_of*	NAME	org:employees_or_membership	47.2
23	PER	per:schools_attended*	NAME	org:students	37.5
24	PER	per:works	NAME	27.0
25	PER	per:age*	VALUE	0.0
26	PER	per:date_of_birth*	VALUE	66.7
27	PER	per:major	STRING	50.0
28	PER	per:place_of_work	STRING	45.1
29	PER	per:title*	STRING	0.5
30	PER	per:alternate_names*	NAME/STRING	0.7
31	PER	per:pet	NAME/STRING	0.3
32	GPE	gpe:residents_of_place**	NAME	per:place_of_residence	42.9
33	GPE	gpe:births_in_place**	NAME	per:place_of_birth	100.0
34	GPE	gpe:visitors_of_place	NAME	per:visited_place	43.0
35	ORG	org:employees_or_members	NAME	per:employee_or_member_of	47.2
36	ORG	org:students*	NAME	per:schools_attended	37.5
37	NAME	unanswerable	NAME/STRING/VALUE	—
Table 2: Relation Types in DialogRE. Relation types with  $\star$  represent the existing relation types defined in the TAC-KBP SF task, and we combine three SF fine-grained relation types about cities, states, and countries in a single relation type with  $\star \star$ . TR: Trigger ratio, representing the percentage of relational triples of a certain relation type that are accompanied by triggers.
frequently appear in *Friends*. We list all 36 relation types that have at least one relational instance in the transcripts in Table 2 and provide definitions and examples of new relation types in Appendix A.1.
We focus on the annotation of relational triples (i.e., (subject, relation type, object)) in which at least one of the arguments is a named entity. We regard an uninterrupted stream of speech from one speaker and the name of this speaker as a turn.
As we follow the TAC-KBP guideline to annotate relation types and design new types, we use internal annotators (two authors of this paper) who are familiar with this task. For a pilot annotation, annotator A annotates relational triples in each scene in all transcripts and form a dialogue
by extracting the shortest snippet of contiguous turns that covers all annotated relational triples and sufficient supportive contexts in this scene. The guidelines are adjusted during the annotation. We prefer to use speaker name (i.e., the first word or phrase of a turn, followed by a colon) as one argument of a speaker-related triple if the corresponding full names or alternate names of the speaker name also appear in the same dialogue, except for relation PER:ALTERNATE_NAMES in which both mentions should be regarded as arguments. For an argument pair (i.e., (subject, object)), there may exist multiple relations between them, and we annotate all instances of all of them. For each
As the pilot annotation only involves one annotator, we admit there may exist a certain degree of bias in defining new relation types and labeling argument pairs.
triple, we also annotate its trigger: the smallest extent (i.e., span) of contiguous text in the dialogue that most clearly indicates the existence of the relation between two arguments. If there exist multiple spans that can serve as triggers, we only keep one for each triple. For relation types such as PER:TITLE and PER:ALTERNATE_NAMES, it is difficult to identify such supportive contexts, and therefore we leave their triggers empty. For each relational triple, we annotate its inverse triple if its corresponding inverse relation type exists in the schema (e.g., PER:CHILDREN and PER: PARENTS) while the trigger remains unchanged.
In the second process, annotator B annotates the possible relations between candidate pairs annotated by annotator A (previous relation labels are hidden). Cohen's kappa among annotators is around 0.87. We remove the cases when annotators cannot reach a consensus. On average, each dialogue in DialogRE contains 4.5 relational triples and 12.9 turns, as shown in Table 3. See Table 1 for relational triple examples (R1, R2, and R3).
DialogRE
Average dialogue length (in tokens)	225.8
Average # of turns	12.9
Average # of speakers	3.3
Average # of sentences	21.8
Average # of relational instances	4.5
Average # of no-relation instances	1.2
2.3 Negative Instance Generation, Data Split, and Speaker Name Anonymization
After our first round of annotation, we use any two annotated arguments associated with each dialogue to generate candidate relational triples, in which the relation between two arguments is unanswerable based on the given dialogue or beyond our relation schema. We manually filter out candidate triples for which there is "obviously" no relation between an argument pair in consideration of aspects such as argument type constraints (e.g., relation PER:SchoolS_ATTENDED can only exist between a PER name and an ORG name). After filtering, we keep 2,100 triples in total, whose two arguments are in "no relation", and we finally have 10,168 triples for 1,788 dialogues. We randomly split them at the dialogue level, with  $60\%$  for training,  $20\%$  for development, and  $20\%$  for testing.
The focus of the proposed task is to identify
relations between argument pairs based on a dialogue, rather than exploiting information in DialogRE beyond the given dialogue or leveraging external knowledge to predict the relations between arguments (e.g., characters) specific to a particular television show. Therefore, we anonymize all speaker names (Section 2.2) in each dialogue and annotated triples and rename them in chronological order within the given dialogue. For example, S1 and S2 in Table 1 represent the original speaker names "Rachel" and "Phoebe", respectively.
3 Data Comparisons and Discussions
3.1 Comparison Between DialogRE and SF
As a pilot study, we examine the similarities and differences between dialogue-based and traditional relation extraction datasets that are manually annotated. We compare DialogRE with the official SF (2013-2014) dataset (Surdeanu, 2013; Surdeanu and Ji, 2014) as  $47.2\%$  of relation types in DialogRE originate from the SF relation types (Section 2.1), and  $92.2\%$  of the source documents in it that contain ground truth relational triples are formally written newswire reports  $(72.8\%)$  or well-edited web documents  $(19.4\%)$  compared to the remaining documents from discussion fora. We show the relation distributions in DialogRE and SF in Figure 1 and Figure 2 (Appendix A.2), respectively. Half of the top ten relation types in DialogRE are newly defined (PER:GIRL/BOYFRIEND, PER:POSITIVE(NEGATIVE)_IMPRESSION, PER:FRIENDS, and PER:ROOMMATE), partially justifying the need for new relation types.
Argument Type: Based on the predefined SF and DialogRE relation types, a subject is expected to be an entity of type PER, ORG, or geo-political entity (GPE). Notably, subjects of most relational triples (96.8% vs. 69.7% in the SF dataset) in DialogRE are person names. The coarse-grained object type is entity, string, or value (i.e., a numerical value or a date). As shown in Table 4, we observe that a higher proportion (80.1%) of objects are entities in DialogRE compared to that in SF (65.3%).
Table 3: Statistics per dialogue of DialogRE.
DialogRE	SF
Entity	80.1 (6,460)	65.3 (2,167)
String	18.9 (1,524)	25.4 (843)
Value	1.0 (84)	9.2 (306)
Table 4: Coarse-grained object type distributions (%) of DialogRE and SF with frequencies in brackets.
In particular, the subjects of  $77.3\%$  of relational triples are speaker names, and more than  $90.0\%$  of relational triples contain at least one speaker argument. The high percentage of "speaker-centric" relational triples and the low percentage of ORG and GPE arguments in DialogRE is perhaps because the transcripts for annotation are from a single situation comedy that involves a small group of characters in a very limited number of scenes (see more discussions in Section 5.3).
Distance Between Argument Pairs: It has been shown that there is a longer distance between two arguments in the SF dataset (Surdeanu, 2013; Huang et al., 2017) compared to that in many widely used human-annotated relation extraction datasets such as ACE (Doddington et al., 2004) and SemEval (Hendrickx et al., 2010). However, it is not trivial to compute an accurate distance between two arguments in a dialogue, especially for cases containing arguments that are speaker names. We instead consider different types of distances (e.g., average and minimum) between two argument mentions in a dialogue. We argue that DialogRE exhibits a similar level of difficulty as SF from the perspective of the distance between two arguments.  $41.3\%$  of arguments are separated by at least seven words even considering the minimum distance, and the percentage can reach as high as  $96.5\%$  considering the average distance, contrast with  $46.0\%$  in SF (Huang et al., 2017) and  $59.8\%$  in a recently released cross-sentence relation extraction dataset DocRED, in which Wikipedia articles serve as documents (Yao et al., 2019). Note that the provenance/evidence sentences in SF and DocRED are provided by automated systems or annotators. Also,  $95.6\%$  of relational triples from an annotated subset of DialogRE (Section 5.2) require reasoning over multiple sentences in a dialogue, compared with  $40.7\%$  in DocRED (Table 7). See Figure 3 in Appendix A.3 for more details.
3.2 Comparison Between DialogRE and Existing Relational Triples
We also collect 2,341 relational triples related to Friends, which are summarized by a community of contributors, from a collaborative encyclopedia. We remove triples of content-independent relation types such as DIRECTED_BY, GUEST_STARs, and NUMBER_OF_EPISODES.
We find that  $93.8\%$  of all 224 relation types in these triples can be mapped to one of the 36 relation types in our relation schema (e.g., HUSBAND, EX-HUSBAND, and WIFE can be mapped to PER:SPOUSE) except for the remaining relatively rare or implicit relation types such as PROM_DATE and GENDER, and KISSED, demonstrating the relation schema we use for annotation is capable of covering most of the important relation types labeled by the encyclopedia community of contributors.
On the other hand, the relatively small number of the existing triples and the moderate size of our annotated triples in DialogRE may suggest the low information density (Wang and Liu, 2011) in conversational speech in terms of relation extraction. For example, the average annotated triple per sentence in DialogRE is merely 0.21, compared to other exhaustively annotated datasets ACE (0.73) and KnowledgeNet (Mesquita et al., 2019) (1.44), in which corpora are formal written news reports and Wikipedia articles, respectively.
3.3 Discussions on Triggers
As annotated triggers are rarely available in existing relation extraction datasets (Aguilar et al., 2014), the connections between different relation types and trigger existence are under-investigated.
Relation Type: In DialogRE,  $49.6\%$  of all relational triples are annotated with triggers. We find that argument pairs are frequently accompanied by triggers when (1) arguments have the same type such as PER:FRIENDS, (2) strong emotions are involved (e.g., PER:POSITIVE(NEGATIVE)_IMPRESSION), or (3) the relation type is related to death or birth (e.g., GPE:BIRTHS_INPLACE). In comparison, a relation between two arguments of different types (e.g., PER:ORIGIN and PER:AGE) is more likely to be implicitly expressed instead of relying on triggers. This is perhaps because there exist fewer possible relations between such an argument pair compared to arguments of the same type, and a relatively short distance between such an argument pair might be sufficient to help the listeners understand the message correctly. For each relation type, we report the percentage of relational triples with triggers in Table 2.
Argument Distance: We assume the existence of triggers may allow a longer distance between argument pairs in a text as they help to decrease ambiguity. This assumption may be empirically
<sup>3</sup>https://friends.fandom.com/wiki/Friends.
validated by the longer average distance (68.3 tokens) between argument pairs with triggers in a dialogue, compared to the distance (61.2 tokens) between argument pairs without any triggers.
4 Task Formulations and Methods
4.1 Dialogue-Based Relation Extraction
Given a dialogue  $D = s_{1}:t_{1},s_{2}:t_{2},\ldots ,s_{m}:t_{m}$  and an argument pair  $(a_{1},a_{2})$ , where  $s_i$  and  $t_i$  denote the speaker ID and text of the  $i^{\text{th}}$  turn, respectively, and  $m$  is the total number of turns, we evaluate the performance of approaches in extracting relations between  $a_1$  and  $a_2$  that appear in  $D$  in the following two settings.
Standard Setting: As the standard setting of relation extraction tasks, we regard dialogue  $D$  as document  $d$ . The input is  $a_1$ ,  $a_2$ , and  $d$ , and the expected output is the relation type(s) between  $a_1$  and  $a_2$  based on  $d$ . We adopt F1, which is the harmonic mean of precision (P) and recall (R), for evaluation.
Conversational Setting: Instead of only considering the entire dialogue, here we can regard the first  $i \leq m$  turns of the dialogue as  $d$ . Accordingly, we propose a new metric  $\mathrm{F1}_{\mathrm{c}}$ , the harmonic mean of conversational precision  $(\mathrm{P_c})$  and recall  $(\mathbb{R}_c)$  as a supplement to the standard F1. We start by introducing some notation that will be used in the definition of  $\mathrm{F1}_{\mathrm{c}}$ . Let  $O_{i}$  denote the set of predicted relation types when the input is  $a_1, a_2$ , and the first  $i$  turns (i.e.,  $d = s_1 : t_1, s_2 : t_2, \ldots, s_i : t_i$ ). For an argument pair  $(a_1, a_2)$ , let  $L$  denote its corresponding set of relation types that are manually annotated based on the full dialogue.  $R$  represents the set of 36 relation types. By definition,  $O_{i}, L \subseteq R$ . We define that auxiliary function  $\jmath(x)$  returns  $m$  if  $x$  does not appear in  $D$ . Otherwise, it returns the index of the turn where  $x$  first appears.
We define auxiliary function  $\iota(r)$  as: (i) For each relation type  $r \in L$ , if there exists an annotated trigger for  $r$ ,  $\iota(r) = \jmath(\lambda_r)$  where  $\lambda_r$  denotes the trigger. Otherwise,  $\iota(r) = m$ . (ii) For each  $r \in R \setminus L$ ,  $\iota(r) = 1$ . We define the set of relation types that are evaluable based on the first  $i$  turns by  $E_i$ :
$$
E _ {i} = \{r \mid i \geq \max  \{\jmath (a _ {1}), \jmath (a _ {2}), \iota (r) \} \} \quad (1)
$$
The interpretation of Equation 1 is that given  $d$  containing the first  $i$  turns in a dialogue, relation type  $r$  associated with  $a_1$  and  $a_2$  is evaluable if  $a_1$ ,  $a_2$ , and the trigger for  $r$  have all been mentioned in  $d$ . The definition is based on our assumption