SlowGuess/NAACL_2021_OCR huggingface dataset:
We also experimented with GPT-2 without finetuning (i.e., by directly using pre-trained weights). The non-finetuned GPT-2 fared poorly for both the tasks across all the metrics, getting a BLEU score of near 0 for Task 1. This dismal performance underscores the importance of fine-tuning on the end task for large-scale pre-trained language models.
Finally, we note that our method does not make any model-specific assumption, and can be used with any auto-regressive language model (i.e., a language model that generates a sequence of tokens from left-to-right). We use GPT-2 as a representative for large pre-trained language models.
Seq2Seq: We train a bi-directional LSTM (Hochreiter and Schmidhuber, 1997) based sequence-to-sequence model (Bahdanau et al., 2015) with global attention (Luong et al., 2015) and a hidden size of 500 as a baseline to contrast with GPT-2. The to
ken embeddings initialized using 300-dimensional pre-trained Glove (Pennington et al., 2014).
5.3 Task 1: Node Generation
Paragraph: Mr. Grier, a former defensive lineman for the New York Giants who was ordained as a minister in 1986, testified on Dec. 9 that he had visited Mr. Simpson a month earlier
Table 4: An example of GPT-2 fixing the label given by CAEVO. Given a query event after "Mr. Grier visited", CAEVO incorrectly extracts Mr. Grier ordained, whereas GPT-2 generates the correct event: Mr. Grier testified.
Metrics Given a query  $(C_r, e_q, r)$ , with  $C_r$  being the context (sentences containing events  $e_q, e_t$  and their neighboring sentences) and  $e_q$  as the source event, Task 1 is to generate a target event  $e_t$  such that  $r(e_q, e_t)$ . We format each query as "In the context of  $C$ , what happens  $r e_q$ ?". We found formatting the query in natural language to be empirically better. Let  $\hat{e}_t$  be the system generated event. We compare  $e_t$  vs.  $\hat{e}_t$  using BLEU (Papineni et al., 2002), METEOR (Denkowski and Lavie, 2011), and ROUGE (Lin, 2004)<sup>6</sup>, and measure the accuracy (ACC) as the fraction of examples where  $e_t = \hat{e}_t$ .
Results on TG-Gen The results are listed in Table 3. Unsurprisingly, GPT-2 achieves high scores across the metrics showing that it is highly effective in generating correct events. To test the generative capabilities of the models, we perform an ablation by removing the sentence containing the target event  $e_t$  from  $C_r$  (indicated with -C). Removal of context causes a drop in performance for both GPT-2 and Seq2Seq, showing that it is crucial for generating temporal events. However, GPT-2 obtains higher relative gains with context present, indicating that it uses its large architecture and pre-training to use the context more efficiently. GPT-2 also fares better as compared with Seq2Seq in terms of drop in performance for the out-of-domain TB-Dense dataset on metrics like accuracy  $(-21\%$  vs.  $-33\%)$  and BLEU  $(-16\%$  vs.  $-21\%)$ , indicating that pre-training makes helps GPT-2 in generalizing across the domains.
Human Evaluation To understand the nature of errors, we analyzed 100 randomly sampled incorrect generations. For  $53\%$  of the errors, GPT-2 generated a non-salient event which nevertheless
had the correct temporal relation with the query. Interestingly, for  $10\%$  of the events, we found that GPT-2 fixed the label assigned by CAEVO (Table 4), i.e.,  $e_t$  was incorrect but  $\hat{e}_t$  was correct.
5.4 Task 2: Graph Generation
Dataset	BLEU	MTR	RG	DOT%
Seq2Seq	TG-Gen	4.79	15.03	45.95	86.93
GPT-2	TG-Gen	37.77	37.22	64.24	94.47
Seq2Seq	TB-Dense	2.61	12.76	28.36	89.31
GPT-2	TB-Dense	26.61	29.49	49.26	92.37
Table 5: Graph string metrics.
Dataset	vP	vR	VF1	eP	eR	eF1
Seq2Seq	TG-Gen	36.84	24.89	28.11	9.65	4.29	4.70
GPT-2	TG-Gen	69.31	66.12	66.34	27.95	25.89	25.22
Seq2Seq	TB-Dense	24.86	15.25	17.99	4.7	0.14	0.24
CAEVO	TB-Dense	37.53	79.83	48.96	7.95	14.62	8.96
GPT-2	TB-Dense	45.96	48.44	44.97	8.74	8.89	7.96
Table 6: Graph semantic metrics.
Metrics Let  $\mathbf{G_i}(\mathbf{V_i},\mathbf{E_i})$  and  $\hat{\mathbf{G}}_{\mathrm{i}}(\hat{\mathbf{V}}_{\mathrm{i}},\hat{\mathbf{E}}_{\mathrm{i}})$  be the true and the generated graphs for an example  $i$  in the test corpus. Please recall that our proposed method generates a graph from a given document as a string in DOT. Let  $\pmb {y}_i$  and  $\hat{\pmb{y}}_i$  be the string representations of the true and generated graphs. We evaluate our generated graphs using three types of metrics:
1. Graph string metrics: To compare  $y_{i}$  vs.  $\hat{y}_i$ , we use BLEU, METEOR, and ROUGE, and also measure parse accuracy (DOT%) as the % of generated graphs  $\hat{y}_i$  which are valid DOT files.
2. Graph structure metrics To compare the structures of the graphs  $\mathbf{G}_{\mathrm{i}}$  vs.  $\hat{\mathbf{G}}_{\mathrm{i}}$ , we use i) Graph edit distance (GED) (Abu-Aisheh et al., 2015) - the minimum numbers of edits required to transform the predicted graph to the true graph by addition/removal of an edge/node; ii) Graph isomorphism (ISO) (Cordella et al., 2001) - a binary measure set to 1 if the graphs are isomorphic (without considering the node or edge attributes); iii) The average graph size  $(|\mathbf{V_i}|, |\mathbf{E_i}|, |\hat{\mathbf{V_i}}|, |\hat{\mathbf{E_i}}|)$  and the average degree  $(\mathrm{d}(\mathbf{V}))$ .
3. Graph semantic metrics: We evaluate the node sets  $(\mathbf{V}_{\mathbf{i}}$  vs.  $\hat{\mathbf{V}}_{\mathbf{i}})$  and the edge sets  $(\mathbf{E}_{\mathbf{i}}$  vs.  $\hat{\mathbf{E}}_{\mathbf{i}})$  to compare the semantics of the true and generated
$^{6}$ Sharma et al. (2017), https://github.com/Maluuba/nlg-eval
graphs. For every example  $i$ , we calculate node-set precision, recall, and  $F_{1}$  score, and average them over the test set to obtain node precision  $(v_{P})$ , recall  $(v_{R})$ , and  $F_{1}$ $(v_{F})$ . We evaluate the predicted edge set using temporal awareness (UzZaman and Allen, 2012; UzZaman et al., 2013). For an example  $i$ , we calculate  $e_{P}^{i} = \frac{|\hat{\mathbf{E}}_{\mathrm{i}}^{-}\cap\mathbf{E}_{\mathrm{i}}^{+}|}{|\hat{\mathbf{E}}_{\mathrm{i}}^{-}|}, e_{R}^{i} = \frac{|\hat{\mathbf{E}}_{\mathrm{i}}^{+}\cap\mathbf{E}_{\mathrm{i}}^{-}|}{|\mathbf{E}_{\mathrm{i}}^{-}|}$  where symbol  $+$  denotes the temporal transitive closure (Allen, 1983) of the edge set. Similarly,  $-$  indicates the reduced edge set, obtained by removing all the edges that can be inferred from other edges transitively. The  $F_{1}$  score  $e_{F_{1}}^{i}$  is the harmonic mean of  $e_{P}^{i}$  and  $e_{R}^{i}$ , and these metrics are averaged over the test set to obtain the temporal awareness precision  $(e_{P})$ , recall  $(e_{R})$ , and  $F_{1}$  score  $(e_{F_{1}})$ . Intuitively, the node metrics judge the quality of generated events in the graph, and the edge metrics evaluate the corresponding temporal relations.
Results Tables 5, 7, and 6 present results for graph generation, and we discuss them next.
Dataset	|V|	|E|	d(V)	GED ↓	ISO ↑
True	TG-Gen	4.15	5.47	1.54	0	100
Seq2Seq	TG-Gen	2.24	2.23	1.12	6.09	32.49
GPT-2	TG-Gen	3.81	4.60	1.40	2.62	41.66
True	TB-Dense	4.39	6.12	2.02	0	100
Seq2Seq	TB-Dense	2.21	2.20	1.11	6.22	23.08
CAEVO	TB-Dense	10.73	17.68	2.76	18.68	11.11
GPT-2	TB-Dense	3.72	4.65	1.75	4.05	24.00
Table 7: Graph structure metrics.
GPT-2 vs. Seq2Seq GPT-2 outperforms Seq2Seq on all the metrics by a large margin in both finetuned (TG-Gen) and zero-shot settings (TB-Dense). GPT-2 generated graphs are closer to the true graphs in size and topology, as shown by lower edit distance and a higher rate of isomorphism in Table 7. Both the systems achieve high parsing rates (DOT %), with GPT-2 generating valid DOT files  $94.6\%$  of the time. The high parsing rates are expected, as even simpler architectures like vanilla RNNs have been shown to generate syntactically valid complex structures like IATEXdocuments with ease (Karpathy, 2015).
GPT-2 vs. CAEVO We compare the graphs generated by GPT-2 with those extracted by CAEVO (Chambers et al., 2014)<sup>7</sup> from the TB-Dense documents. We remove all the vague
Top 10 Verbs: found, killed, began, called, want, took, came, used, trying, asked
Randomly Sampled Verbs: shooting, caused, accused, took, conceived, visit, vowing, play, withdraw, seems
Table 8: Verbs in GPT-2 generated graphs.
edges and the light verbs from the output of CAEVO for a fair comparison. Please recall that CAEVO is the tool we used for creating the training data for our method. Further, CAEVO was trained using TB-Dense, while GPT-2 was not. Thus, CAEVO forms an upper bound over the performance of GPT-2. The results in Tables 6 and 7 show that despite these challenges, GPT-2 performs strongly across a wide range of metrics, including GED, ISO, and temporal awareness. Comparing the node-set metrics, we see that GPT-2 leads CAEVO by over eight precision points  $(v_{P})$ , but loses on recall  $(v_{R})$  as CAEVO extracts nearly every verb in the document as a potential event. On temporal awareness (edge-metrics), GPT-2 outperforms both CAEVO and Seq2Seq in terms of average precision score  $e_{P}$  and achieves a competitive  $e_{F_{1}}$  score. These results have an important implication: they show that our method can best or match a pipeline of specialized systems given reasonable amounts of training data for temporal graph extraction. CAEVO involves several sub-modules to perform part-of-speech tagging, dependency parsing, event extraction, and several statistical and rule-based systems for temporal extraction. In contrast, our method involves no hand-curated features, is trained end-to-end (single GPT-2), and can be easily scaled to new datasets.
Node extraction and Edge Extraction The node-set metrics in Table 6 shows that GPT-2 avoids generating noisy events (high  $P$ ), and extracts salient events (high  $R$ ). This is confirmed by manual analysis, done by randomly sampling 100 graphs from the GPT-2 generated graphs and isolating the main verb in each node (Table 8). We provide several examples of generated graphs in the Appendix. We note from Table 6 that the relative difference between the  $e_{F_1}$  scores for GPT-2 and Seq2Seq (25.22 vs. 4.70) is larger than the relative difference between their  $v_{F_1}$  scores (66.34 vs. 28.11), showing that edge-extraction is the more challenging task which allows GPT-2 to take full advantage of its powerful architecture. We also observe that edge extraction  $(e_{F_1})$  is highly sensitive
<sup>7</sup>https://github.com/nchambers/caevo
Query (C, eq, r)	et	Explanation
The suspected car bombings...turning busy streets...Which event happened before the suspected car bombings?	many cars drove	Plausible:
The passage mentions busy streets and car bombing.
He...charged...killed one person. Which event happened after he was charged?	He was acquit-ted	Somewhat plausible: An acquittal is a possible outcome of a trial.
Table 9: Sample open-ended questions and the answers  $e_t$  generated by our system. Note that the answers generated by our system  $e_t$  are complete event phrases (not just verbs).
to node extraction  $(v_{F_1})$ ; for GPT-2, a  $27\%$  drop in  $v_{F_1}$  (66.34 on TG-Gen vs. 44.97 on TB-Dense) causes a  $68\%$  drop in  $e_{F_1}$  (25.22 on TG-Gen vs. 7.96 on TB-Dense). As each node is connected to multiple edges on average (Table 7), missing a node during the generation process might lead to multiple edges being omitted, thus affecting edge extraction metrics disproportionately.
5.5 Answering for Open-ended Questions
A benefit of our approach of using a pre-trained language model is that it can be used to generate an answer for open-ended temporal questions. Recently, Ning et al. (2020) introduced Torque, a temporal reading-comprehension dataset. Several questions in Torque have no answers, as they concern a time scope not covered by the passage (the question is about events not mentioned in the passage). We test the ability of our system for generating plausible answers for such questions out of the box (i.e., without training on Torque). Given a (passage, question) pair, we create a query  $(C, e_q, r)$ , where  $C$  is the passage, and  $e_q$  and  $r$  are the query event and temporal relation in the question. We then use our GPT-2 based model for node-generation trained without context and generate an answer  $e_t$  for the given query. A human-judge rated the answers generated for 100 such questions for plausibility, rating each answer as being plausible, somewhat plausible, or incorrect. For each answer rated as either plausible or somewhat plausible, the human-judge wrote a short explanation to provide a rationale for the plausibility of the generated event. Out of the 100 questions, the human-judge rated 22 of the gen
erated answers as plausible and ten as somewhat plausible, showing the promise of our method on this challenging task (Table 9).
Current methods for generating event-level temporal graphs are developed with relatively small amounts of hand-labeled data. On the other hand, the possibility of using pre-trained language models for this task has not received sufficient attention. This paper addresses this open challenge by first developing a data generation pipeline that uses existing IE/NLP/clustering techniques for automated acquisition of a large corpus of document-graph pairs, and by proposing a new formulation of the graph generation task as a sequence-to-sequence mapping task, allowing us to leverage and fine-tune pretrained language models. Our experiments strongly support the effectiveness of the proposed approach, which significantly outperforms strong baselines. We plan to explore techniques for adapting largescale language models on unseen domains and at multiple granularity levels in the future.
Thanks to Nathanael Chambers and Dheeraj Rajagopal for the helpful discussion, and to the anonymous reviewers for their constructive feedback. This material is based on research sponsored in part by the Air Force Research Laboratory under agreement number FA8750-19-2-0200. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the Air Force Research Laboratory or the U.S. Government.
Zeina Abu-Aisheh, Romain Raveaux, Jean-Yves Ramel, and Patrick Martineau. 2015. An exact graph edit distance algorithm for solving pattern recognition problems. In An exact graph edit distance algorithm for solving pattern recognition problems.
James F Allen. 1983. Maintaining knowledge about temporal intervals. Communications of the ACM, 26(11):832-843.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015.
Miguel Ballesteros, Rishita Anubhai, Shuai Wang, Nima Pourdamghani, Yogarshi Vyas, Jie Ma, Parminder Bhatia, Kathleen McKeown, and Yaser Al-Onaizan. 2020. Severing the edge between before and after: Neural architectures for temporal ordering of events. arXiv preprint arXiv:2004.04295.
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. Comet: Commonsense transformers for automatic knowledge graph construction. In ACL.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165.
Paweł Budzianowski and Ivan Vulić. 2019. Hello, it's gpt-2-how can i help you? towards the use of pretrained language models for task-oriented dialogue systems. arXiv preprint arXiv:1907.05774.
Taylor Cassidy, Bill McDowell, Nathaniel Chambers, and Steven Bethard. 2014. An annotation framework for dense event ordering. Technical report, CARNEGIE-MELLON UNIV PITTSBURGH PA.
Nathanael Chambers, Taylor Cassidy, Bill McDowell, and Steven Bethard. 2014. Dense event ordering with a multi-pass architecture. Transactions of the Association for Computational Linguistics, 2:273-284.
Nathanael Chambers and Dan Jurafsky. 2008. Unsupervised learning of narrative event chains. In Proceedings of ACL-08: HLT, pages 789-797.
Nathanael Chambers and Dan Jurafsky. 2009. Unsupervised learning of narrative schemas and their participants. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 602-610.
Fei Cheng and Yusuke Miyao. 2017. Classifying temporal relations by bidirectional LSTM over dependency paths. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1-6.
Aaron Clauset, Mark EJ Newman, and Christopher Moore. 2004. Finding community structure in very large networks. Physical review E, 70(6):066111.
Luigi Pietro Cordella, Pasquale Foggia, Carlo Sansone, and Mario Vento. 2001. An improved algorithm for matching large graphs. In 3rd IAPR-TC15 workshop on graph-based representations in pattern recognition, pages 149-159.
Michael Denkowski and Alon Lavie. 2011. Meteor 1.3: Automatic metric for reliable optimization and evaluation of machine translation systems. In Proceedings of the sixth workshop on statistical machine translation, pages 85-91. Association for Computational Linguistics.
Nicholas D Duran, Philip M McCarthy, Art C Graesser, and Danielle S McNamara. 2007. Using temporal cohesion to predict temporal coherence in narrative and expository texts. Behavior Research Methods, 39(2):212-223.
Jacqueline Evers-Vermeul, Jet Hoek, and Merel CJ Scholman. 2017. On temporality in discourse annotation: Theoretical and practical considerations. Dialogue & Discourse, 8(2):1-20.
Emden Gansner, Eleftherios Koutsofios, and Stephen North. 2006. Drawing graphs with dot.
Tanya Goyal and Greg Durrett. 2019. Embedding time expressions for deep temporal ordering models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4400-4406, Florence, Italy. Association for Computational Linguistics.
Aric A. Hagberg, Daniel A. Schult, and Pieter J. Swart. 2008. Exploring network structure, dynamics, and function using networkx. In Proceedings of the 7th Python in Science Conference, pages 11-15, Pasadena, CA USA.
Rujun Han, I-Hung Hsu, Mu Yang, Aram Galstyan, Ralph Weischedel, and Nanyun Peng. 2019. Deep structured neural network for event temporal relation extraction. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 666-106.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751.
Andrej Karpathy. 2015. The unreasonable effectiveness of recurrent neural networks. Andrej Karpathy blog, 21:23.
Chen Lin, Dmitriy Dligach, Timothy A Miller, Steven Bethard, and Guergana K Savova. 2016. Multilayered temporal modeling for the clinical domain. Journal of the American Medical Informatics Association, 23(2):387-395.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
Xiao Ling and Daniel S Weld. 2010. Temporal information extraction. In AAAI, volume 10, pages 1385-1390.
Zhengzhong Liu, Chenyan Xiong, Teruko Mitamura, and Eduard Hovy. 2018. Automatic event salience identification. arXiv preprint arXiv:1809.00647.
Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101.
Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412-1421.
Aman Madaan, Dheeraj Rajagopal, Yiming Yang, Abhilasha Ravichander, Eduard Hovy, and Shrimai Prabhumoye. 2020. Eigen: Event influence generation using pre-trained language models. arXiv preprint arXiv:2010.11764.
Juha Makkonen, Helena Ahonen-Myka, and Marko Salmenkivi. 2003. Topic detection and tracking with spatio-temporal evidence. In European Conference on Information Retrieval, pages 251-265. Springer.
Mark EJ Newman. 2004. Fast algorithm for detecting community structure in networks. Physical review E, 69(6):066133.
Mark EJ Newman and Michelle Girvan. 2004. Finding and evaluating community structure in networks. Physical review E, 69(2):026113.
Qiang Ning, Zhili Feng, and Dan Roth. 2019a. A structured learning approach to temporal relation extraction. arXiv preprint arXiv:1906.04943.
Qiang Ning, Sanjay Subramanian, and Dan Roth. 2019b. An improved neural baseline for temporal relation extraction. arXiv preprint arXiv:1909.00429.
Qiang Ning, Hao Wu, Rujun Han, Nanyun Peng, Matt Gardner, and Dan Roth. 2020. Torque: A reading comprehension dataset of temporal ordering questions. arXiv preprint arXiv:2005.00242.
Qiang Ning, Ben Zhou, Zhili Feng, Haoruo Peng, and Dan Roth. 2018. Cogcomptime: A tool for understanding time in natural language. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 72-77.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311-318. Association for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pretraining. URL https://s3-us-west-2. amazonaws.com/openai-assets/researchcovers/languageunsupervised/language understanding paper. pdf.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683.
Dheeraj Rajagopal, Aman Madaan, Niket Tandon, Yiming Yang, Shrimai Prabhumoye, Abhilasha Ravichander, Peter Clark, and Eduard Hovy. 2021. Curie: An iterative querying approach for reasoning about situations.
Marta Recasens, Eduard H Hovy, and Maria Antonia Martí. 2010. A typology of near-identity relations for coreference (nident). In LREC.
Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi. 2019. Atomic: An atlas of machine commonsense for if-then reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3027-3035.
Shikhar Sharma, Layla El Asri, Hannes Schulz, and Jeremie Zumer. 2017. Relevance of unsupervised metrics in task-oriented dialogue for evaluating natural language generation. CoRR, abs/1706.09799.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In Thirty-First AAAI Conference on Artificial Intelligence.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104-3112.
Naushad UzZaman and James F Allen. 2012. Interpreting the temporal aspects of language. CiteSeer.