SlowGuess/ACL_2020_OCR huggingface dataset:
that we can roughly estimate how many turns we require to predict the relations between two arguments based on the positions of the arguments and triggers, which most clearly express relations. See Section 5.2 for more discussions.
The conversational precision and recall for an input instance  $D$ ,  $a_1$ , and  $a_2$  are defined as:
$$
\mathrm {P} _ {\mathrm {c}} (D, a _ {1}, a _ {2}) = \frac {\sum_ {i = 1} ^ {m} \left| O _ {i} \cap L \cap E _ {i} \right|}{\sum_ {i = 1} ^ {m} \left| O _ {i} \cap E _ {i} \right|} \tag {2}
$$
$$
\mathrm {R} _ {\mathrm {c}} (D, a _ {1}, a _ {2}) = \frac {\sum_ {i = 1} ^ {m} \left| O _ {i} \cap L \cap E _ {i} \right|}{\sum_ {i = 1} ^ {m} \left| L \cap E _ {i} \right|} \tag {3}
$$
We average the conversational precision/recall scores of all instances to obtain the final conversational precision/recall.
$$
\mathrm {P} _ {\mathrm {c}} = \frac {\sum_ {D ^ {\prime} , a _ {1} ^ {\prime} , a _ {2} ^ {\prime}} \mathrm {P} _ {\mathrm {c}} \left(D ^ {\prime} , a _ {1} ^ {\prime} , a _ {2} ^ {\prime}\right)}{\sum_ {D ^ {\prime} , a _ {1} ^ {\prime} , a _ {2} ^ {\prime}} 1} \tag {4}
$$
$$
\mathrm {R} _ {\mathrm {c}} = \frac {\sum_ {D ^ {\prime} , a _ {1} ^ {\prime} , a _ {2} ^ {\prime}} \mathrm {R} _ {\mathrm {c}} \left(D ^ {\prime} , a _ {1} ^ {\prime} , a _ {2} ^ {\prime}\right)}{\sum_ {D ^ {\prime} , a _ {1} ^ {\prime} , a _ {2} ^ {\prime}} 1} \tag {5}
$$
and  $\mathrm{F1_c} = 2\cdot \mathrm{P_c}\cdot \mathrm{R_c} / (\mathrm{P_c} + \mathrm{R_c})$
Majority: If a given argument pair does not appear in the training set, output the majority relation type in the training set as the prediction. Otherwise, output the most frequent relation type associated with the two arguments in the training set.
CNN, LSTM, and BiLSTM: Following previous work (Yao et al., 2019), we adapt three baselines (Zeng et al., 2014; Cai et al., 2016) that use different document encoders. We refer readers to Yao et al. (2019) for more details.
BERT: We follow the framework of fine-tuning a pre-trained language model on a downstream task (Radford et al., 2018) and use BERT (Devlin et al., 2019) as the pre-trained model. We concatenate the given  $d$  and  $(a_{1},a_{2})$  with classification token [CLS] and separator token [SEP] in BERT as the input sequence [CLS]  $d[\mathrm{SEP}]a_1[\mathrm{SEP}]a_2[\mathrm{SEP}]$ . We denote the final hidden vector corresponding to [CLS] as  $C\in \mathbb{R}^H$ , where  $H$  is the hidden size. For each relation type  $i$ , we introduce a vector  $W_{i}\in \mathbb{R}^{H}$  and obtain the probability  $P_{i}$  of the existence of  $i$  between  $a_1$  and  $a_2$  based on  $d$  by  $P_{i} = \mathrm{sigmoid}(CW_{i}^{T})$ . The cross-entropy loss is used.
Method	Dev	Test
F1 (σ)	F1c (σ)	F1 (σ)	F1c (σ)
Majority	38.9 (0.0)	38.7 (0.0)	35.8 (0.0)	35.8 (0.0)
CNN	46.1 (0.7)	43.7 (0.5)	48.0 (1.5)	45.0 (1.4)
LSTM	46.7 (1.1)	44.2 (0.8)	47.4 (0.6)	44.9 (0.7)
BiLSTM	48.1 (1.0)	44.3 (1.3)	48.6 (1.0)	45.0 (1.3)
BERT	60.6 (1.2)	55.4 (0.9)	58.5 (2.0)	53.2 (1.6)
BERTs	63.0 (1.5)	57.3 (1.2)	61.2 (0.9)	55.4 (0.9)
Table 5: Performance of relation extraction methods on DialogRE in both the standard and conversational settings.
BERTs: We propose a modification to the input sequence of the above BERT baseline with two motivations: (1) help a model locate the start positions of relevant turns based on the arguments that are speaker names, and (2) prevent a model from overfitting to the training data. Formally, given an argument pair  $(a_{1},a_{2})$  and its associated document  $d = s_1:t_1,s_2:t_2,\ldots ,s_n:t_n$ , we construct  $\hat{d} = \hat{s}_1:t_1,\hat{s}_2:t_2,\ldots ,\hat{s}_n:t_n$ , where  $\hat{s}_i$  is:
$$
\hat {s} _ {i} = \left\{ \begin{array}{l l} {[ S _ {1} ]} & {\text {i f} s _ {i} = a _ {1}} \\ {[ S _ {2} ]} & {\text {i f} s _ {i} = a _ {2}} \\ {s _ {i}} & {\text {o t h e r w i s e}} \end{array} \right. \tag {6}
$$
where  $\left[\mathrm{S}_1\right]$  and  $\left[\mathrm{S}_2\right]$  are two newly-introduced special tokens. In addition, we define  $\hat{a}_k$  ( $k \in \{1, 2\}$ ) to be  $\left[\mathrm{S}_k\right]$  if  $\exists i(s_i = a_k)$ , and  $a_k$  otherwise. The modified input sequence to BERT is [CLS]  $\hat{d}[\mathrm{SEP}]\hat{a}_1[\mathrm{SEP}]\hat{a}_2[\mathrm{SEP}]$ . In Appendix A.4, we investigate in three alternative input sequences. It is worth mentioning that a modification that does not disambiguate speaker arguments from other arguments performs substantially worse than the above speaker-aware modification.
CNN, LSTM, and BiLSTM Baselines: The CNN/LSTM/BiLSTM encoder takes as features GloVe word embeddings (Pennington et al., 2014), mention embeddings, and type embeddings. We assign the same mention embedding to mentions of the same argument and obtain the type embeddings based on named entity types of the two arguments. We use  $\mathsf{spaCy}^4$  for entity typing.
Language Model Fine-Tuning: We use the uncased base model of BERT released by Devlin et al. (2019). We truncate a document when the input sequence length exceeds 512 and fine-tune BERT using a batch size of 24 and a learning rate of  $3 \times 10^{-5}$
for 20 epochs. Other parameters remain unchanged. The embeddings of newly-introduced special tokens (e.g.,  $[S_1]$ ) are initialized randomly.
5.2 Results and Discussions
We report the performance of all baselines in both the standard and conversational settings in Table 5. We run each experiment five times and report the average F1 and  $\mathrm{F1}_{\mathrm{c}}$  along with standard deviation  $(\sigma)$ . The fine-tuned BERT method already outperform other baselines (e.g., BiLSTM that achieves  $51.1\%$  in F1 on DocRED (Yao et al., 2019)), and our speaker-aware extension to the BERT baseline further leads to  $2.7\%$  and  $2.2\%$  improvements in F1 and  $\mathrm{F1}_{\mathrm{c}}$ , respectively, on the test set of DialogRE, demonstrating the importance of tracking speakers in dialogue-based relation extraction.
Conversational Metric: We randomly select 269 and 256 instances, which are associated with 50 dialogues from each of the dev and test sets, respectively. For each of relational instances (188 in total) that are previously labeled with triggers in the subsets, annotator A labels the smallest turn  $i^*$  such that the first  $i^*$  turns contain sufficient information to justify a relation. The average distance between  $i^*$  and our estimation  $\max \{j(a_1), j(a_2), \iota(r)\}$  in Equation (1) (Section 4.1) is only 0.9 turn, supporting our hypothesis that the positions of arguments and triggers may be good indicators for estimating the minimum turns for humans to make predictions.
For convenience, we use BERT for the following discussions and comparisons.
Ground Truth Argument Types: Methods in Table 5 are not provided with ground truth argument types considering the unavailability of this kind of annotation in practical use. To study the impacts of argument types on DialogRE, we report the performance of four methods, each of which additionally takes as input the ground truth argument types as previous work (Zhang et al., 2017; Yao et al., 2019). We adopt the same baseline for a direct comparison
4 https://spacy.io/.
except that the input sequence is changed.
In Method 1, we simply extend the original input sequence of BERT (Section 4.2) with newly-introduced special tokens that represent argument types. The input sequence is [CLS]  $d[\mathrm{SEP}]\tau_{1}a_{1}[\mathrm{SEP}]\tau_{2}a_{2}[\mathrm{SEP}]$ , where  $\tau_{i}$  is a special token representing the argument type of  $a_{i}$  ( $i\in \{1,2\}$ ). For example, given  $a_1$  of type PER and  $a_2$  of type STRING,  $\tau_{1}$  is [PER] and  $\tau_{2}$  is [STRING]. In Method 2, we extend the input sequence of BERTs with  $\tau_{i}$  defined in Method 1 (i.e., [CLS]  $\hat{d}[\mathrm{SEP}]\tau_{1}\hat{a}_{1}[\mathrm{SEP}]\tau_{1}\hat{a}_{2}[\mathrm{SEP}]$ ). We also follow the input sequence of previous single-sentence relation extraction methods (Shi and Lin, 2019; Joshi et al., 2020) and refer them as Method 3 and 4, respectively. We provide the implementation details in Appendix A.5. As shown in Table 6, the best performance achieved by Method 2 is not superior to that of BERTs, which does not leverage ground truth argument types. Therefore, we guess that ground truth argument types may only provide a limited, if at all positive, contribution to the performance on DialogRE.
Method 1	Method 2	Method 3	Method 4
Dev	60.6 (0.4)	62.9 (1.2)	55.6 (2.4)	61.9 (1.4)
Test	59.1 (0.7)	60.5 (1.9)	52.3 (3.2)	59.7 (0.6)
Table 6: Performance (F1  $(\sigma)$ ) comparison of methods with considering the ground truth argument types.
Ground Truth Triggers: We investigate what performance would be ideally attainable if the model could identify all triggers correctly. We append the ground truth triggers to the input sequence on the baseline, and the F1 of this model is  $74.9\%$ , a  $16.4\%$  absolute improvement compared to the BERT baseline. In particular, through the introduction of triggers, we observe a  $22.9\%$  absolute improvement in F1 on relation types whose inverse relation types are themselves (e.g., PER:ROOMMATE and PER:SPOUSE). These experimental results show the critical role of triggers in dialogue-based relation extraction. However, trigger identification is perhaps as difficult as relation extraction, and it is labor-intensive to annotate large-scale datasets with triggers. Future research may explore how to identify triggers based on a small amount of human-annotated triggers as seeds (Bronstein et al., 2015; Yu and Ji, 2016).
5.3 Error Analysis and Limitations
We analyze the outputs on the dev set and find that BERT tends to make more mistakes when there exists an asymmetric inverse relation of the relation to be predicted compared to those that have symmetric inverse relations. For example, the baseline mistakenly predicts S2 as the subordinate of S1 based on the following dialogue: "...S2: Oh. Well, I wish I could say no, but you can't stay my assistant forever. Neither can you Sophie, but for different reasons. S1: God, I am so glad you don't have a problem with this, because if you did, I wouldn't even consider applying..." Introducing triggers into the input sequence leads to a relatively small gain (11.0% in F1 on all types with an asymmetric inverse relation) perhaps because inverse relation types share the same triggers (e.g., "my assistant" serves as the trigger for both PER:BOSS and PER:SUBORDINATE). One possible solution may be the use of directed syntactic graphs constructed from the given dialogue, though the performance of coreference resolution and dependency parsing in dialogues may be relatively unsatisfying.
A major limitation in DialogRE is that all transcripts for annotation are from Friends, which may limit the diversity of scenarios and generality of the relation distributions. It may be useful to leverage existing triples in knowledge bases (e.g., Fandom) for thousands of movies or TV shows using distant supervision (Mintz et al., 2009), considering the time-consuming manual annotation process. In addition, dialogues in Friends presents less variation based on linguistic features (Biber, 1991) than natural conversations; nonetheless, compared to other registers such as personal letters and prepared speeches, there are noticeable linguistic similarities between natural conversations and television dialogues in Friends (Quaglio, 2009).
Cross-Sentence Relation Extraction Datasets Different from the sentence-level relation extraction (RE) datasets (Roth and Yih, 2004; Hendrickx et al., 2010; Riedel et al., 2010; Zhang and Wang, 2015; Zhang et al., 2017; Han et al., 2018), in which relations are between two arguments in the same sentence, we focus on cross-sentence RE tasks (Ji et al., 2011; Surdeanu, 2013; Surdeanu and Ji, 2014) and present the first dialogue-based RE dataset, in which dialogues serve as input contexts instead of formally written sentences or documents.
Task	style/source of doc	# rel	cross rate°	# doc	# triples*
— distant supervision —
Peng et al. (2017)	written/PubMed	4	75.2	960,000	140,661
DocRED (Yao et al., 2019)	written/Wikipedia	96	n/a	101,873	881,298
T-REx (Elsahar et al., 2018)	written/Wikipedia	353	n/a	3 million	11 million
— human annotation —
BC5CDR (Li et al., 2016)	written/PubMed	1	n/a	1,500	2,434
DocRED (Yao et al., 2019)	written/Wikipedia	96	40.7	5,053	56,354
KnowledgeNet (Mesquita et al., 2019)	written/Wikipedia and others	15	n/a	4,991	13,425
DialogRE (this work)	conversational/Friends	36	95.6	1,788	8,068
Table 7: Statistics of publicly available cross-sentence relation extraction datasets (o: the percentage (\%) of relational triples involving multiple sentences; ●: not include no-relation argument pairs).
We compare DialogRE and existing cross-sentence RE datasets (Li et al., 2016; Quirk and Poon, 2017; Yao et al., 2019; Mesquita et al., 2019) in Table 7. In this paper, we do not consider relations that take relations or events as arguments and are also likely to span multiple sentences (Pustejovsky and Verhagen, 2009; Do et al., 2012; Moschitti et al., 2013).
Relation Extraction Approaches Over the past few years, neural models have achieved remarkable success in RE (Nguyen and Grishman, 2015b,a; Adel et al., 2016; Yin et al., 2017; Levy et al., 2017; Su et al., 2018; Song et al., 2018; Luo et al., 2019), in which the input representation usually comes from shallow neural networks over pre-trained word and character embeddings (Xu et al., 2015; Zeng et al., 2015; Lin et al., 2016). Deep contextualized word representations such as the ELMo (Peters et al., 2018) are also applied as additional input features to boost the performance (Luan et al., 2018). A recent thread is to fine-tune pre-trained deep language models on downstream tasks (Radford et al., 2018; Devlin et al., 2019), leading to further performance gains on many RE tasks (Alt et al., 2019; Shi and Lin, 2019; Baldini Soares et al., 2019; Peters et al., 2019; Wadden et al., 2019). We propose an improved method that explicitly considers speaker arguments, which are seldom investigated in previous RE methods.
Dialogue-Based Natural Language Understanding To advance progress in spoken language understanding, researchers have studied dialogue-based tasks such as argument extraction (Swanson et al., 2015), named entity recognition (Chen and Choi, 2016; Choi and Chen, 2018; Bowden et al., 2018), coreference resolution (Chen et al., 2017; Zhou and Choi, 2018), emotion detection (Zahiri and Choi, 2018), and machine reading comprehension.
sion (Ma et al., 2018; Sun et al., 2019; Yang and Choi, 2019). Besides, some pioneer studies focus on participating in dialogues (Yoshino et al., 2011; Hixon et al., 2015) by asking users relation-related questions or using outputs of existing RE methods as inputs of other tasks (Klüwer et al., 2010; Wang and Cardie, 2012). In comparison, we focus on extracting relation triples from human-human dialogues, which is still under investigation.
We present the first human-annotated dialogue-based RE dataset DialogRE. We also design a new metric to evaluate the performance of RE methods in a conversational setting and argue that tracking speakers play a critical role in this task. We investigate the performance of several RE methods, and experimental results demonstrate that a speaker-aware extension on the best-performing model leads to substantial gains in both the standard and conversational settings.
In the future, we are interested in investigating the generality of our defined schema for other comedies and different conversational registers, identifying the temporal intervals when relations are valid (Surdeanu, 2013) in a dialogue, and joint dialogue-based information extraction as well as its potential combinations with multimodal signals from images, speech, and videos.
We would like to thank the anonymous reviewers for their constructive comments and suggestions.
Heike Adel, Benjamin Roth, and Hinrich Schütze. 2016. Comparing convolutional neural networks to
traditional models for slot filling. In Proceedings of NAACL-HLT, pages 828-838, San Diego, CA.
Jacqueline Aguilar, Charley Beller, Paul McNamee, Benjamin Van Durme, Stephanie Strassel, Zhiyi Song, and Joe Ellis. 2014. A comparison of the events and relations across ACE, ERE, TAC-KBP, and FrameNet annotation standards. In Proceedings of the Second Workshop on EVENTS: Definition, Detection, Coreference, and Representation, pages 45-53, Baltimore, MD.
Christoph Alt, Marc Hübner, and Leonhard Hennig. 2019. Improving relation extraction by pre-trained language representations. In Proceedings of AKBC, Amherst, MA.
Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019. Matching the blanks: Distributional similarity for relation learning. In Proceedings of ACL, pages 2895-2905, Florence, Italy.
Douglas Biber. 1991. Variation across speech and writing. Cambridge University Press.
Kevin Bowden, Jiaqi Wu, Shereen Oraby, Anita Misra, and Marilyn Walker. 2018. SlugNERDS: A named entity recognition tool for open domain dialogue systems. In Proceedings of LREC, pages 4462-4469, Miyazaki, Japan.
Ofer Bronstein, Ido Dagan, Qi Li, Heng Ji, and Anette Frank. 2015. Seed-based event trigger labeling: How far can event descriptions get us? In Proceedings of ACL-IJCNLP, pages 372-376, Beijing, China.
Rui Cai, Xiaodong Zhang, and Houfeng Wang. 2016. Bidirectional recurrent convolutional neural network for relation classification. In Proceedings of ACL, pages 756-765, Berlin, Germany.
Roberta Catizone, Alexiei Dingli, and Robert Gaizauskas. 2010. Using dialogue corpora to extend information extraction patterns for natural language understanding of dialogue. In Proceedings of LREC, pages 2136-2140, Valletta, Malta.
Henry Y Chen, Ethan Zhou, and Jinho D Choi. 2017. Robust coreference resolution and entity linking on dialogues: Character identification on tv show transcripts. In Proceedings of CoNLL, pages 216-225, Vancouver, Canada.
Yu-Hsin Chen and Jinho D. Choi. 2016. Character identification on multiparty conversation: Identifying mentions of characters in TV shows. In Proceedings of SIGDIAL, pages 90-100, Los Angeles, CA.
Jinho D. Choi and Henry Y. Chen. 2018. SemEval 2018 Task 4: Character identification on multiparty dialogues. In Proceedings of SemEval, pages 57-64, New Orleans, LA.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171-4186, Minneapolis, MN.
Quang Do, Wei Lu, and Dan Roth. 2012. Joint inference for event timeline construction. In Proceedings of EMNLP-CoNLL, pages 677-687, Jeju Island, Korea.
George Doddington, Alexis Mitchell, Mark Przybocki, Lance Ramshaw, Stephanie Strassel, and Ralph Weischedel. 2004. The automatic content extraction (ACE) program - tasks, data, and evaluation. In Proceedings of LREC, pages 837-840, Lisbon, Portugal.
Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frédérique Laforest, and Elena Simperl. 2018. T-REx: A large scale alignment of natural language with knowledge base triples. In Proceedings of LREC, pages 3448–3452, Miyazaki, Japan.
Ralph Grishman. 2019. Twenty-five years of information extraction. *Natural Language Engineering*, 25(6):677-692.
Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2018. FewRel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation. In Proceedings of EMNLP, pages 4803-4809, Brussels, Belgium.
Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid O Seaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2010. SemEval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals. In Proceedings of SemEval, pages 33-38, Uppsala, Sweden.
Ben Hixon, Peter Clark, and Hannaneh Hajishirzi. 2015. Learning knowledge graphs for question answering through conversational dialog. In Proceedings of NAACL-HLT, pages 851-861, Denver, CO.
Lifu Huang, Avirup Sil, Heng Ji, and Radu Florian. 2017. Improving slot filling performance with attentive neural networks on dependency structures. In Proceedings of EMNLP, pages 2588-2597, Copenhagen, Denmark.
Heng Ji, Ralph Grishman, and Hoa Trang Dang. 2011. Overview of the TAC2011 Knowledge Base Population Track. In Proceedings of TAC, Gaithersburg, MD.
Heng Ji, Ralph Grishman, Hoa Trang Dang, Kira Griffin, and Joe Ellis. 2010. Overview of the TAC 2010 knowledge base population track. In Proceedings of TAC, Gaithersburg, MD.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. 2020. Spanbert: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77.
Tina Kluwer, Hans Uszkoreit, and Feiyu Xu. 2010. Using syntactic and semantic based relations for dialogue act recognition. In Proceedings of COLING, pages 570-578, Beijing, China.
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017. Zero-shot relation extraction via reading comprehension. In Proceedings of CoNLL, pages 333-342, Vancouver, Canada.
Jiao Li, Yueping Sun, Robin J Johnson, Daniela Sciaky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J Mattingly, Thomas C Wiegers, and Zhiyong Lu. 2016. BioCreative V CDR task corpus: a resource for chemical disease relation extraction. Database, 2016.
Yankai Lin, Shiqi Shen, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. 2016. Neural relation extraction with selective attention over instances. In Proceedings of ACL, pages 2124-2133, Berlin, Germany.
Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In Proceedings of EMNLP, pages 3219-3232, Brussels, Belgium.
Fan Luo, Ajay Nagesh, Rebecca Sharp, and Mihai Surdeanu. 2019. Semi-supervised teacher-student architecture for relation extraction. In Proceedings of the Third Workshop on Structured Prediction for NLP, pages 29-37, Minneapolis, MN.
Kaixin Ma, Tomasz Jurczyk, and Jinho D. Choi. 2018. Challenging reading comprehension on daily conversation: Passage completion on multiparty dialog. In Proceedings of NAACL-HLT, pages 2039-2048, New Orleans, LA.
Paul McNamee and Hoa Trang Dang. 2009. Overview of the TAC 2009 knowledge base population track. In Proceedings of TAC, Gaithersburg, MD.
Filipe Mesquita, Matteo Cannaviccio, Jordan Schmidek, Paramita Mirza, and Denilson Barbosa. 2019. KnowledgeNet: A benchmark dataset for knowledge base population. In Proceedings of EMNLP-IJCNLP, pages 749-758, Hong Kong, China.
Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th ACL and the 4th IJCNLP of the AFNLP, pages 1003-1011, Suntec, Singapore.