mtybilly/Instruct2DS huggingface dataset:
Trainable Greedy Decoding for Neural Machine Translation
Authors: Kyunghyun Cho, Victor O.K. Li Jiatao Gu
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Recent research in neural machine translation has largely focused on two aspects; neural network architectures and end-to-end learning algorithms. The problem of decoding, however, has received relatively little attention from the research community. In this paper, we solely focus on the problem of decoding given a trained neural machine translation model. Instead of trying to build a new decoding algorithm for any specific decoding objective, we propose the idea of trainable decoding algorithm in which we train a decoding algorithm to find a translation that maximizes an arbitrary decoding objective. More specifically, we design an actor that observes and manipulates the hidden state of the neural machine translation decoder and propose to train it using a variant of deterministic policy gradient. We extensively evaluate the proposed algorithm using four language pairs and two decoding objectives and show that we can indeed train a trainable greedy decoder that generates a better translation (in terms of a target decoding objective) with minimal computational overhead.
Satirical News Detection and Analysis using Attention Mechanism and Linguistic Features
Authors: Arjun Mukherjee, Eduard Dragut Fan Yang
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Satirical news is considered to be entertainment, but it is potentially deceptive and harmful. Despite the embedded genre in the article, not everyone can recognize the satirical cues and therefore believe the news as true news. We observe that satirical cues are often reflected in certain paragraphs rather than the whole document. Existing works only consider document-level features to detect the satire, which could be limited. We consider paragraph-level linguistic features to unveil the satire by incorporating neural network and attention mechanism. We investigate the difference between paragraph-level features and document-level features, and analyze them on a large satirical news dataset. The evaluation shows that the proposed model detects satirical news effectively and reveals what features are important at which level.
Fine Grained Citation Span for References in Wikipedia
Authors: Katja Markert, Avishek Anand Besnik Fetahu
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Verifiability is one of the core editing principles in Wikipedia, where editors are encouraged to provide citations for the added content. For a Wikipedia article determining what content is covered by a citation or the citation span is not trivial, an important aspect for automated citation finding for uncovered content, or fact assessments. We address the problem of determining the citation span in Wikipedia articles. We approach this problem by classifying which textual fragments in an article are covered or hold true given a citation. We propose a sequence classification approach where for a paragraph and a citation, we determine the citation span at a fine-grained level. We provide a thorough experimental evaluation and compare our approach against baselines adopted from the scientific domain, where we show improvement for all evaluation metrics.
Identifying Semantic Edit Intentions from Revisions in Wikipedia
Authors: Aaron Halfaker, Robert Kraut, Eduard Hovy Diyi Yang
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Most studies on human editing focus merely on syntactic revision operations, failing to capture the intentions behind revision changes, which are essential for facilitating the single and collaborative writing process. In this work, we develop in collaboration with Wikipedia editors a 13-category taxonomy of the semantic intention behind edits in Wikipedia articles. Using labeled article edits, we build a computational classifier of intentions that achieved a micro-averaged F1 score of 0.621. We use this model to investigate edit intention effectiveness: how different types of edits predict the retention of newcomers and changes in the quality of articles, two key concerns for Wikipedia today. Our analysis shows that the types of edits that users make in their first session predict their subsequent survival as Wikipedia editors, and articles in different stages need different types of edits.
Accurate Supervised and Semi-Supervised Machine Reading for Long Documents
Authors: Llion Jones, Alexandre Lacoste, Izzeddin Gur Daniel Hewlett
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: We introduce a hierarchical architecture for machine reading capable of extracting precise information from long documents. The model divides the document into small, overlapping windows and encodes all windows in parallel with an RNN. It then attends over these window encodings, reducing them to a single encoding, which is decoded into an answer using a sequence decoder. This hierarchical approach allows the model to scale to longer documents without increasing the number of sequential steps. In a supervised setting, our model achieves state of the art accuracy of 76.8 on the WikiReading dataset. We also evaluate the model in a semi-supervised setting by downsampling the WikiReading training set to create increasingly smaller amounts of supervision, while leaving the full unlabeled document corpus to train a sequence autoencoder on document windows. We evaluate models that can reuse autoencoder states and outputs without fine-tuning their weights, allowing for more efficient training and inference.
Adversarial Examples for Evaluating Reading Comprehension Systems
Authors: Percy Liang Robin Jia
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Standard accuracy metrics indicate that reading comprehension systems are making rapid progress, but the extent to which these systems truly understand language remains unclear. To reward systems with real language understanding abilities, we propose an adversarial evaluation scheme for the Stanford Question Answering Dataset (SQuAD). Our method tests whether systems can answer questions about paragraphs that contain adversarially inserted sentences, which are automatically generated to distract computer systems without changing the correct answer or misleading humans. In this adversarial setting, the accuracy of sixteen published models drops from an average of 75% F1 score to 36%; when the adversary is allowed to add ungrammatical sequences of words, average accuracy on four models decreases further to 7%. We hope our insights will motivate the development of new models that understand language more precisely.
Reasoning with Heterogeneous Knowledge for Commonsense Machine Comprehension
Authors: Le Sun, Xianpei Han Hongyu Lin
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Reasoning with commonsense knowledge is critical for natural language understanding. Traditional methods for commonsense machine comprehension mostly only focus on one specific kind of knowledge, neglecting the fact that commonsense reasoning requires simultaneously considering different kinds of commonsense knowledge. In this paper, we propose a multi-knowledge reasoning method, which can exploit heterogeneous knowledge for commonsense machine comprehension. Specifically, we first mine different kinds of knowledge (including event narrative knowledge, entity semantic knowledge and sentiment coherent knowledge) and encode them as inference rules with costs. Then we propose a multi-knowledge reasoning model, which selects inference rules for a specific reasoning context using attention mechanism, and reasons by summarizing all valid inference rules. Experiments on RocStories show that our method outperforms traditional models significantly.
Document-Level Multi-Aspect Sentiment Classification as Machine Comprehension
Authors: Yangqiu Song, Ming Zhang Yichun Yin
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Document-level multi-aspect sentiment classification is an important task for customer relation management. In this paper, we model the task as a machine comprehension problem where pseudo question-answer pairs are constructed by a small number of aspect-related keywords and aspect ratings. A hierarchical iterative attention model is introduced to build aspectspecific representations by frequent and repeated interactions between documents and aspect questions. We adopt a hierarchical architecture to represent both word level and sentence level information, and use the attention operations for aspect questions and documents alternatively with the multiple hop mechanism. Experimental results on the TripAdvisor and BeerAdvocate datasets show that our model outperforms classical baselines. We will release our code and data for the method replicability.
What is the Essence of a Claim? Cross-Domain Claim Identification
Authors: Steffen Eger, Ivan Habernal, Christian Stab, Iryna Gurevych Johannes Daxenberger
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Argument mining has become a popular research area in NLP. It typically includes the identification of argumentative components, e.g. claims, as the central component of an argument. We perform a qualitative analysis across six different datasets and show that these appear to conceptualize claims quite differently. To learn about the consequences of such different conceptualizations of claim for practical applications, we carried out extensive experiments using state-of-the-art feature-rich and deep learning systems, to identify claims in a cross-domain fashion. While the divergent conceptualization of claims in different datasets is indeed harmful to cross-domain classification, we show that there are shared properties on the lexical level as well as system configurations that can help to overcome these gaps.
Identifying Where to Focus in Reading Comprehension for Neural Question Generation
Authors: Claire Cardie Xinya Du
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: A first step in the task of automatically generating questions for testing reading comprehension is to identify \textitquestion-worthy sentences, i.e. sentences in a text passage that humans find it worthwhile to ask questions about. We propose a hierarchical neural sentence-level sequence tagging model for this task, which existing approaches to question generation have ignored. The approach is fully data-driven --- with no sophisticated NLP pipelines or any hand-crafted rules/features --- and compares favorably to a number of baselines when evaluated on the SQuAD data set. When incorporated into an existing neural question generation system, the resulting end-to-end system achieves state-of-the-art performance for paragraph-level question generation for reading comprehension.
Break it Down for Me: A Study in Automated Lyric Annotation
Authors: Jason Naradowsky, Bill Byrne, Thomas Demeester, Chris Develder Lucas Sterckx
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Comprehending lyrics, as found in songs and poems, can pose a challenge to human and machine readers alike. This motivates the need for systems that can understand the ambiguity and jargon found in such creative texts, and provide commentary to aid readers in reaching the correct interpretation. We introduce the task of automated lyric annotation (ALA). Like text simplification, a goal of ALA is to rephrase the original text in a more easily understandable manner. However, in ALA the system must often include additional information to clarify niche terminology and abstract concepts. To stimulate research on this task, we release a large collection of crowdsourced annotations for song lyrics. We analyze the performance of translation and retrieval models on this task, measuring performance with both automated and human evaluation. We find that each model captures a unique type of information important to the task.
Cascaded Attention based Unsupervised Information Distillation for Compressive Summarization
Authors: Wai Lam, Lidong Bing, Weiwei Guo, Hang Li Piji Li
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: When people recall and digest what they have read for writing summaries, the important content is more likely to attract their attention. Inspired by this observation, we propose a cascaded attention based unsupervised model to estimate the salience information from the text for compressive multi-document summarization. The attention weights are learned automatically by an unsupervised data reconstruction framework which can capture the sentence salience. By adding sparsity constraints on the number of output vectors, we can generate condensed information which can be treated as word salience. Fine-grained and coarse-grained sentence compression strategies are incorporated to produce compressive summaries. Experiments on some benchmark data sets show that our framework achieves better results than the state-of-the-art methods.
Deep Recurrent Generative Decoder for Abstractive Text Summarization
Authors: Wai Lam, Lidong Bing, Zihao Wang Piji Li
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: We propose a new framework for abstractive text summarization based on a sequence-to-sequence oriented encoder-decoder model equipped with a deep recurrent generative decoder (DRGN). Latent structure information implied in the target summaries is learned based on a recurrent latent random model for improving the summarization quality. Neural variational inference is employed to address the intractable posterior inference for the recurrent latent variables. Abstractive summaries are generated based on both the generative latent variables and the discriminative deterministic states. Extensive experiments on some benchmark datasets in different languages show that DRGN achieves improvements over the state-of-the-art methods.
Extractive Summarization Using Multi-Task Learning with Document Classification
Authors: Toru Fujino, Junichiro Mori, Yutaka Matsuo, Ichiro Sakata Masaru Isonuma
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: The need for automatic document summarization that can be used for practical applications is increasing rapidly. In this paper, we propose a general framework for summarization that extracts sentences from a document using externally related information. Our work is aimed at single document summarization using small amounts of reference summaries. In particular, we address document summarization in the framework of multi-task learning using curriculum learning for sentence extraction and document classification. The proposed framework enables us to obtain better feature representations to extract sentences from documents. We evaluate our proposed summarization method on two datasets: financial report and news corpus. Experimental results demonstrate that our summarizers achieve performance that is comparable to state-of-the-art systems.
Towards Automatic Construction of News Overview Articles by News Synthesis
Authors: Xiaojun Wan Jianmin Zhang
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: In this paper we investigate a new task of automatically constructing an overview article from a given set of news articles about a news event. We propose a news synthesis approach to address this task based on passage segmentation, ranking, selection and merging. Our proposed approach is compared with several typical multi-document summarization methods on the Wikinews dataset, and achieves the best performance on both automatic evaluation and manual evaluation.
Joint Syntacto-Discourse Parsing and the Syntacto-Discourse Treebank
Authors: Liang Huang Kai Zhao
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Discourse parsing has long been treated as a stand-alone problem independent from constituency or dependency parsing. Most attempts at this problem rely on annotated text segmentations (Elementary Discourse Units, EDUs) and sophisticated sparse or continuous features to extract syntactic information. In this paper we propose the first end-to-end discourse parser that jointly parses in both syntax and discourse levels, as well as the first syntacto-discourse treebank by integrating the Penn Treebank and the RST Treebank. Built upon our recent span-based constituency parser, this joint syntacto-discourse parser requires no preprocessing efforts such as segmentation or feature extraction, making discourse parsing more convenient. Empirically, our parser achieves the state-of-the-art end-to-end discourse parsing accuracy.
Event Coreference Resolution by Iteratively Unfolding Inter-dependencies among Events
Authors: Ruihong Huang Prafulla Kumar Choubey
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: We introduce a novel iterative approach for event coreference resolution that gradually builds event clusters by exploiting inter-dependencies among event mentions within the same chain as well as across event chains. Among event mentions in the same chain, we distinguish within- and cross-document event coreference links by using two distinct pairwise classifiers, trained separately to capture differences in feature distributions of within- and cross-document event clusters. Our event coreference approach alternates between WD and CD clustering and combines arguments from both event clusters after every merge, continuing till no more merge can be made. And then it performs further merging between event chains that are both closely related to a set of other chains of events. Experiments on the ECB+ corpus show that our model outperforms state-of-the-art methods in joint task of WD and CD event coreference resolution.
When to Finish? Optimal Beam Search for Neural Text Generation (modulo beam size)
Authors: Kai Zhao, Mingbo Ma Liang Huang
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: In neural text generation such as neural machine translation, summarization, and image captioning, beam search is widely used to improve the output text quality. However, in the neural generation setting, hypotheses can finish in different steps, which makes it difficult to decide when to end beam search to ensure optimality. We propose a provably optimal beam search algorithm that will always return the optimal-score complete hypothesis (modulo beam size), and finish as soon as the optimality is established. To counter neural generation`s tendency for shorter hypotheses, we also introduce a bounded length reward mechanism which allows a modified version of our beam search algorithm to remain optimal. Experiments on neural machine translation demonstrate that our principled beam search algorithm leads to improvement in BLEU score over previously proposed alternatives.
Steering Output Style and Topic in Neural Response Generation
Authors: Nebojsa Jojic, Chris Brockett, Eric Nyberg Di Wang
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: We propose simple and flexible training and decoding methods for influencing output style and topic in neural encoder-decoder based language generation. This capability is desirable in a variety of applications, including conversational systems, where successful agents need to produce language in a specific style and generate responses steered by a human puppeteer or external knowledge. We decompose the neural generation process into empirically easier sub-problems: a faithfulness model and a decoding method based on selective-sampling. We also describe training and sampling algorithms that bias the generation process with a specific language style restriction, or a topic restriction. Human evaluation results show that our proposed methods are able to to restrict style and topic without degrading output quality in conversational tasks.
Preserving Distributional Information in Dialogue Act Classification
Authors: Ingrid Zukerman, Gholamreza Haffari Quan Hung Tran
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: This paper introduces a novel training/decoding strategy for sequence labeling. Instead of greedily choosing a label at each time step, and using it for the next prediction, we retain the probability distribution over the current label, and pass this distribution to the next prediction. This approach allows us to avoid the effect of label bias and error propagation in sequence learning/decoding. Our experiments on dialogue act classification demonstrate the effectiveness of this approach. Even though our underlying neural network model is relatively simple, it outperforms more complex neural models, achieving state-of-the-art results on the MapTask and Switchboard corpora.
Adversarial Learning for Neural Dialogue Generation
Authors: Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter, Dan Jurafsky Jiwei Li
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: We apply adversarial training to open-domain dialogue generation, training a system to produce sequences that are indistinguishable from human-generated dialogue utterances. We cast the task as a reinforcement learning problem where we jointly train two systems: a generative model to produce response sequences, and a discriminator---analagous to the human evaluator in the Turing test--- to distinguish between the human-generated dialogues and the machine-generated ones. In this generative adversarial network approach, the outputs from the discriminator are used to encourage the system towards more human-like dialogue. Further, we investigate models for adversarial evaluation that uses success in fooling an adversary as a dialogue evaluation metric, while avoiding a number of potential pitfalls. Experimental results on several metrics, including adversarial evaluation, demonstrate that the adversarially-trained system generates higher-quality responses than previous baselines
Using Context Information for Dialog Act Classification in DNN Framework
Authors: Kun Han, Zhao Tan, Yun Lei Yang Liu
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Previous work on dialog act (DA) classification has investigated different methods, such as hidden Markov models, maximum entropy, conditional random fields, graphical models, and support vector machines. A few recent studies explored using deep learning neural networks for DA classification, however, it is not clear yet what is the best method for using dialog context or DA sequential information, and how much gain it brings. This paper proposes several ways of using context information for DA classification, all in the deep learning framework. The baseline system classifies each utterance using the convolutional neural networks (CNN). Our proposed methods include using hierarchical models (recurrent neural networks (RNN) or CNN) for DA sequence tagging where the bottom layer takes the sentence CNN representation as input, concatenating predictions from the previous utterances with the CNN vector for classification, and performing sequence decoding based on the predictions from the sentence CNN model. We conduct thorough experiments and comparisons on the Switchboard corpus, demonstrate that incorporating context information significantly improves DA classification, and show that we achieve new state-of-the-art performance for this task.
Modeling Dialogue Acts with Content Word Filtering and Speaker Preferences
Authors: Michael Yoder, Hyeju Jang, Carolyn Rosé Yohan Jo
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: We present an unsupervised model of dialogue act sequences in conversation. By modeling topical themes as transitioning more slowly than dialogue acts in conversation, our model de-emphasizes content-related words in order to focus on conversational function words that signal dialogue acts. We also incorporate speaker tendencies to use some acts more than others as an additional predictor of dialogue act prevalence beyond temporal dependencies. According to the evaluation presented on two dissimilar corpora, the CNET forum and NPS Chat corpus, the effectiveness of each modeling assumption is found to vary depending on characteristics of the data. De-emphasizing content-related words yields improvement on the CNET corpus, while utilizing speaker tendencies is advantageous on the NPS corpus. The components of our model complement one another to achieve robust performance on both corpora and outperform state-of-the-art baseline models.
Towards Implicit Content-Introducing for Generative Short-Text Conversation Systems
Authors: Yaoyuan Zhang, Yansong Feng, Dongyan Zhao, Rui Yan Lili Yao
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: The study on human-computer conversation systems is a hot research topic nowadays. One of the prevailing methods to build the system is using the generative Sequence-to-Sequence (Seq2Seq) model through neural networks. However, the standard Seq2Seq model is prone to generate trivial responses. In this paper, we aim to generate a more meaningful and informative reply when answering a given question. We propose an implicit content-introducing method which incorporates additional information into the Seq2Seq model in a flexible way. Specifically, we fuse the general decoding and the auxiliary cue word information through our proposed hierarchical gated fusion unit. Experiments on real-life data demonstrate that our model consistently outperforms a set of competitive baselines in terms of BLEU scores and human evaluation.
Affordable On-line Dialogue Policy Learning
Authors: Runzhe Yang, Lu Chen, Xiang Zhou, Kai Yu Cheng Chang
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: The key to building an evolvable dialogue system in real-world scenarios is to ensure an affordable on-line dialogue policy learning, which requires the on-line learning process to be safe, efficient and economical. But in reality, due to the scarcity of real interaction data, the dialogue system usually grows slowly. Besides, the poor initial dialogue policy easily leads to bad user experience and incurs a failure of attracting users to contribute training data, so that the learning process is unsustainable. To accurately depict this, two quantitative metrics are proposed to assess safety and efficiency issues. For solving the unsustainable learning problem, we proposed a complete companion teaching framework incorporating the guidance from the human teacher. Since the human teaching is expensive, we compared various teaching schemes answering the question how and when to teach, to economically utilize teaching budget, so that make the online learning process affordable.
Generating High-Quality and Informative Conversation Responses with Sequence-to-Sequence Models
Authors: Stephan Gouws, Denny Britz, Anna Goldie, Brian Strope, Ray Kurzweil Yuanlong Shao
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Sequence-to-sequence models have been applied to the conversation response generation problem where the source sequence is the conversation history and the target sequence is the response. Unlike translation, conversation responding is inherently creative. The generation of long, informative, coherent, and diverse responses remains a hard task. In this work, we focus on the single turn setting. We add self-attention to the decoder to maintain coherence in longer responses, and we propose a practical approach, called the glimpse-model, for scaling to large datasets. We introduce a stochastic beam-search algorithm with segment-by-segment reranking which lets us inject diversity earlier in the generation process. We trained on a combined data set of over 2.3B conversation messages mined from the web. In human evaluation studies, our method produces longer responses overall, with a higher proportion rated as acceptable and excellent as length increases, compared to baseline sequence-to-sequence models with explicit length-promotion. A back-off strategy produces better responses overall, in the full spectrum of lengths.
Bootstrapping incremental dialogue systems from minimal data: the generalisation power of dialogue grammars
Authors: Igor Shalyminov, Oliver Lemon Arash Eshghi
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: We investigate an end-to-end method for automatically inducing task-based dialogue systems from small amounts of unannotated dialogue data. It combines an incremental semantic grammar - Dynamic Syntax and Type Theory with Records (DS-TTR) - with Reinforcement Learning (RL), where language generation and dialogue management are a joint decision problem. The systems thus produced are incremental: dialogues are processed word-by-word, shown previously to be essential in supporting natural, spontaneous dialogue. We hypothesised that the rich linguistic knowledge within the grammar should enable a combinatorially large number of dialogue variations to be processed, even when trained on very few dialogues. Our experiments show that our model can process 74% of the Facebook AI bAbI dataset even when trained on only 0.13% of the data (5 dialogues). It can in addition process 65% of bAbI+, a corpus we created by systematically adding incremental dialogue phenomena such as restarts and self-corrections to bAbI. We compare our model with a state-of-the-art retrieval model, MEMN2N. We find that, in terms of semantic accuracy, the MEMN2N model shows very poor robustness to the bAbI+ transformations even when trained on the full bAbI dataset.
Composite Task-Completion Dialogue Policy Learning via Hierarchical Deep Reinforcement Learning
Authors: Xiujun Li, Lihong Li, Jianfeng Gao, Asli Celikyilmaz, Sungjin Lee, Kam-Fai Wong Baolin Peng
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Building a dialogue agent to fulfill complex tasks, such as travel planning, is challenging because the agent has to learn to collectively complete multiple subtasks. For example, the agent needs to reserve a hotel and book a flight so that there leaves enough time for commute between arrival and hotel check-in. This paper addresses this challenge by formulating the task in the mathematical framework of options over Markov Decision Processes (MDPs), and proposing a hierarchical deep reinforcement learning approach to learning a dialogue manager that operates at different temporal scales. The dialogue manager consists of: (1) a top-level dialogue policy that selects among subtasks or options, (2) a low-level dialogue policy that selects primitive actions to complete the subtask given by the top-level policy, and (3) a global state tracker that helps ensure all cross-subtask constraints be satisfied. Experiments on a travel planning task with simulated and real users show that our approach leads to significant improvements over three baselines, two based on handcrafted rules and the other based on flat deep reinforcement learning.
Why We Need New Evaluation Metrics for NLG
Authors: Ondřej Dušek, Amanda Cercas Curry, Verena Rieser Jekaterina Novikova
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: The majority of NLG evaluation relies on automatic metrics, such as BLEU . In this paper, we motivate the need for novel, system- and data-independent automatic evaluation methods: We investigate a wide range of metrics, including state-of-the-art word-based and novel grammar-based ones, and demonstrate that they only weakly reflect human judgements of system outputs as generated by data-driven, end-to-end NLG. We also show that metric performance is data- and system-specific. Nevertheless, our results also suggest that automatic metrics perform reliably at system-level and can support system development by finding cases where a system performs poorly.
Challenges in Data-to-Document Generation
Authors: Stuart Shieber, Alexander Rush Sam Wiseman
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Recent neural models have shown significant progress on the problem of generating short descriptive texts conditioned on a small number of database records. In this work, we suggest a slightly more difficult data-to-text generation task, and investigate how effective current approaches are on this task. In particular, we introduce a new, large-scale corpus of data records paired with descriptive documents, propose a series of extractive evaluation methods for analyzing performance, and obtain baseline results using current neural generation methods. Experiments show that these models produce fluent text, but fail to convincingly approximate human-generated documents. Moreover, even templated baselines exceed the performance of these neural models on some metrics, though copy- and reconstruction-based extensions lead to noticeable improvements.
All that is English may be Hindi: Enhancing language identification through automatic ranking of the likeliness of word borrowing in social media
Authors: Bidisha Samanta, Saurabh Singh, Abhipsa Basu, Prithwish Mukherjee, Monojit Choudhury, Animesh Mukherjee Jasabanta Patro
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: n this paper, we present a set of computational methods to identify the likeliness of a word being borrowed, based on the signals from social media. In terms of Spearman`s correlation values, our methods perform more than two times better (\ensuremath∼ 0.62) in predicting the borrowing likeliness compared to the best performing baseline (\ensuremath∼ 0.26) reported in literature. Based on this likeliness estimate we asked annotators to re-annotate the language tags of foreign words in predominantly native contexts. In 88% of cases the annotators felt that the foreign language tag should be replaced by native language tag, thus indicating a huge scope for improvement of automatic language identification systems.
Multi-View Unsupervised User Feature Embedding for Social Media-based Substance Use Prediction
Authors: Warren K. Bickel, Shimei Pan Tao Ding
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: In this paper, we demonstrate how the state-of-the-art machine learning and text mining techniques can be used to build effective social media-based substance use detection systems. Since a substance use ground truth is difficult to obtain on a large scale, to maximize system performance, we explore different unsupervised feature learning methods to take advantage of a large amount of unsupervised social media data. We also demonstrate the benefit of using multi-view unsupervised feature learning to combine heterogeneous user information such as Facebook “likes” and “status updates” to enhance system performance. Based on our evaluation, our best models achieved 86% AUC for predicting tobacco use, 81% for alcohol use and 84% for illicit drug use, all of which significantly outperformed existing methods. Our investigation has also uncovered interesting relations between a user`s social media behavior (e.g., word usage) and substance use.
Demographic-aware word associations
Authors: Carmen Banea, Rada Mihalcea Aparna Garimella
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Variations of word associations across different groups of people can provide insights into people`s psychologies and their world views. To capture these variations, we introduce the task of demographic-aware word associations. We build a new gold standard dataset consisting of word association responses for approximately 300 stimulus words, collected from more than 800 respondents of different gender (male/female) and from different locations (India/United States), and show that there are significant variations in the word associations made by these groups. We also introduce a new demographic-aware word association model based on a neural net skip-gram architecture, and show how computational methods for measuring word associations that specifically account for writer demographics can outperform generic methods that are agnostic to such information.
A Factored Neural Network Model for Characterizing Online Discussions in Vector Space
Authors: Hao Fang, Mari Ostendorf Hao Cheng
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: We develop a novel factored neural model that learns comment embeddings in an unsupervised way leveraging the structure of distributional context in online discussion forums. The model links different context with related language factors in the embedding space, providing a way to interpret the factored embeddings. Evaluated on a community endorsement prediction task using a large collection of topic-varying Reddit discussions, the factored embeddings consistently achieve improvement over other text representations. Qualitative analysis shows that the model captures community style and topic, as well as response trigger patterns.
Dimensions of Interpersonal Relationships: Corpus and Experiments
Authors: Eduardo Blanco Farzana Rashid
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: This paper presents a corpus and experiments to determine dimensions of interpersonal relationships. We define a set of dimensions heavily inspired by work in social science. We create a corpus by retrieving pairs of people, and then annotating dimensions for their relationships. A corpus analysis shows that dimensions can be annotated reliably. Experimental results show that given a pair of people, values to dimensions can be assigned automatically.
Argument Mining on Twitter: Arguments, Facts and Sources
Authors: Elena Cabrio, Serena Villata Mihai Dusmanu
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Social media collect and spread on the Web personal opinions, facts, fake news and all kind of information users may be interested in. Applying argument mining methods to such heterogeneous data sources is a challenging open research issue, in particular considering the peculiarities of the language used to write textual messages on social media. In addition, new issues emerge when dealing with arguments posted on such platforms, such as the need to make a distinction between personal opinions and actual facts, and to detect the source disseminating information about such facts to allow for provenance verification. In this paper, we apply supervised classification to identify arguments on Twitter, and we present two new tasks for argument mining, namely facts recognition and source identification. We study the feasibility of the approaches proposed to address these tasks on a set of tweets related to the Grexit and Brexit news topics.
Distinguishing Japanese Non-standard Usages from Standard Ones
Authors: Ryohei Sasano, Hiroya Takamura, Manabu Okumura Tatsuya Aoki
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: We focus on non-standard usages of common words on social media. In the context of social media, words sometimes have other usages that are totally different from their original. In this study, we attempt to distinguish non-standard usages on social media from standard ones in an unsupervised manner. Our basic idea is that non-standardness can be measured by the inconsistency between the expected meaning of the target word and the given context. For this purpose, we use context embeddings derived from word embeddings. Our experimental results show that the model leveraging the context embedding outperforms other methods and provide us with findings, for example, on how to construct context embeddings and which corpus to use.
Connotation Frames of Power and Agency in Modern Films
Authors: Marcella Cindy Prasettio, Ari Holtzman, Hannah Rashkin, Yejin Choi Maarten Sap
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: The framing of an action influences how we perceive its actor. We introduce connotation frames of power and agency, a pragmatic formalism organized using frame semantic representations, to model how different levels of power and agency are implicitly projected on actors through their actions. We use the new power and agency frames to measure the subtle, but prevalent, gender bias in the portrayal of modern film characters and provide insights that deviate from the well-known Bechdel test. Our contributions include an extended lexicon of connotation frames along with a web interface that provides a comprehensive analysis through the lens of connotation frames.
Controlling Human Perception of Basic User Traits
Authors: Sharath Chandra Guntuku, Lyle Ungar Daniel Preoţiuc-Pietro
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Much of our online communication is text-mediated and, lately, more common with automated agents. Unlike interacting with humans, these agents currently do not tailor their language to the type of person they are communicating to. In this pilot study, we measure the extent to which human perception of basic user trait information -- gender and age -- is controllable through text. Using automatic models of gender and age prediction, we estimate which tweets posted by a user are more likely to mis-characterize his traits. We perform multiple controlled crowdsourcing experiments in which we show that we can reduce the human prediction accuracy of gender to almost random -- an over 20% drop in accuracy. Our experiments show that it is practically feasible for multiple applications such as text generation, text summarization or machine translation to be tailored to specific traits and perceived as such.
Topic Signatures in Political Campaign Speeches
Authors: Peggy Cellier, René Quiniou, Alexandre Termier Clément Gautrais
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Highlighting the recurrence of topics usage in candidates speeches is a key feature to identify the main ideas of each candidate during a political campaign. In this paper, we present a method combining standard topic modeling with signature mining for analyzing topic recurrence in speeches of Clinton and Trump during the 2016 American presidential campaign. The results show that the method extracts automatically the main ideas of each candidate and, in addition, provides information about the evolution of these topics during the campaign.
Assessing Objective Recommendation Quality through Political Forecasting
Authors: Masoud Rouhizadeh, Michael Bishop, Philip Tetlock, Barbara Mellers, Lyle Ungar H. Andrew Schwartz
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Recommendations are often rated for their subjective quality, but few researchers have studied comment quality in terms of objective utility. We explore recommendation quality assessment with respect to both subjective (i.e. users' ratings) and objective (i.e., did it influence? did it improve decisions?) metrics in a massive online geopolitical forecasting system, ultimately comparing linguistic characteristics of each quality metric. Using a variety of features, we predict all types of quality with better accuracy than the simple yet strong baseline of comment length. Looking at the most predictive content illustrates rater biases; for example, forecasters are subjectively biased in favor of comments mentioning business transactions or dealings as well as material things, even though such comments do not indeed prove any more useful objectively. Additionally, more complex sentence constructions, as evidenced by subordinate conjunctions, are characteristic of comments leading to objective improvements in forecasting.
Never Abandon Minorities: Exhaustive Extraction of Bursty Phrases on Microblogs Using Set Cover Problem
Authors: Takahiro Hara, Takuya Maekawa Masumi Shirakawa
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: We propose a language-independent data-driven method to exhaustively extract bursty phrases of arbitrary forms (e.g., phrases other than simple noun phrases) from microblogs. The burst (i.e., the rapid increase of the occurrence) of a phrase causes the burst of overlapping N-grams including incomplete ones. In other words, bursty incomplete N-grams inevitably overlap bursty phrases. Thus, the proposed method performs the extraction of bursty phrases as the set cover problem in which all bursty N-grams are covered by a minimum set of bursty phrases. Experimental results using Japanese Twitter data showed that the proposed method outperformed word-based, noun phrase-based, and segmentation-based methods both in terms of accuracy and coverage.
Maximum Margin Reward Networks for Learning from Explicit and Implicit Supervision
Authors: Ming-Wei Chang, Wen-tau Yih Haoruo Peng
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Neural networks have achieved state-of-the-art performance on several structured-output prediction tasks, trained in a fully supervised fashion. However, annotated examples in structured domains are often costly to obtain, which thus limits the applications of neural networks. In this work, we propose Maximum Margin Reward Networks, a neural network-based framework that aims to learn from both explicit (full structures) and implicit supervision signals (delayed feedback on the correctness of the predicted structure). On named entity recognition and semantic parsing, our model outperforms previous systems on the benchmark datasets, CoNLL-2003 and WebQuestionsSP.