mtybilly/Instruct2DS huggingface dataset:
Outta Control: Laws of Semantic Change and Inherent Biases in Word Representation Models
Authors: Daphna Weinshall, Eitan Grossman Haim Dubossarsky
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: This article evaluates three proposed laws of semantic change. Our claim is that in order to validate a putative law of semantic change, the effect should be observed in the genuine condition but absent or reduced in a suitably matched control condition, in which no change can possibly have taken place. Our analysis shows that the effects reported in recent literature must be substantially revised: (i) the proposed negative correlation between meaning change and word frequency is shown to be largely an artefact of the models of word representation used; (ii) the proposed negative correlation between meaning change and prototypicality is shown to be much weaker than what has been claimed in prior art; and (iii) the proposed positive correlation between meaning change and polysemy is largely an artefact of word frequency. These empirical observations are corroborated by analytical proofs that show that count representations introduce an inherent dependence on word frequency, and thus word frequency cannot be evaluated as an independent factor with these representations.
Human Centered NLP with User-Factor Adaptation
Authors: Youngseo Son, Vivek Kulkarni, Niranjan Balasubramanian, H. Andrew Schwartz Veronica Lynn
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: We pose the general task of user-factor adaptation -- adapting supervised learning models to real-valued user factors inferred from a background of their language, reflecting the idea that a piece of text should be understood within the context of the user that wrote it. We introduce a continuous adaptation technique, suited for real-valued user factors that are common in social science and bringing us closer to personalized NLP, adapting to each user uniquely. We apply this technique with known user factors including age, gender, and personality traits, as well as latent factors, evaluating over five tasks: POS tagging, PP-attachment, sentiment analysis, sarcasm detection, and stance detection. Adaptation provides statistically significant benefits for 3 of the 5 tasks: up to +1.2 points for PP-attachment, +3.4 points for sarcasm, and +3.0 points for stance.
Neural Sequence Learning Models for Word Sense Disambiguation
Authors: Claudio Delli Bovi, Roberto Navigli Alessandro Raganato
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Word Sense Disambiguation models exist in many flavors. Even though supervised ones tend to perform best in terms of accuracy, they often lose ground to more flexible knowledge-based solutions, which do not require training by a word expert for every disambiguation target. To bridge this gap we adopt a different perspective and rely on sequence learning to frame the disambiguation problem: we propose and study in depth a series of end-to-end neural architectures directly tailored to the task, from bidirectional Long Short-Term Memory to encoder-decoder models. Our extensive evaluation over standard benchmarks and in multiple languages shows that sequence learning enables more versatile all-words models that consistently lead to state-of-the-art results, even against word experts with engineered features.
Learning Word Relatedness over Time
Authors: Eytan Adar, Kira Radinsky Guy D. Rosin
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Search systems are often focused on providing relevant results for the “now”, assuming both corpora and user needs that focus on the present. However, many corpora today reflect significant longitudinal collections ranging from 20 years of the Web to hundreds of years of digitized newspapers and books. Understanding the temporal intent of the user and retrieving the most relevant historical content has become a significant challenge. Common search features, such as query expansion, leverage the relationship between terms but cannot function well across all times when relationships vary temporally. In this work, we introduce a temporal relationship model that is extracted from longitudinal data collections. The model supports the task of identifying, given two words, when they relate to each other. We present an algorithmic framework for this task and show its application for the task of query expansion, achieving high gain.
Inter-Weighted Alignment Network for Sentence Pair Modeling
Authors: Yunlun Yang, Zhi-Hong Deng Gehui Shen
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Sentence pair modeling is a crucial problem in the field of natural language processing. In this paper, we propose a model to measure the similarity of a sentence pair focusing on the interaction information. We utilize the word level similarity matrix to discover fine-grained alignment of two sentences. It should be emphasized that each word in a sentence has a different importance from the perspective of semantic composition, so we exploit two novel and efficient strategies to explicitly calculate a weight for each word. Although the proposed model only use a sequential LSTM for sentence modeling without any external resource such as syntactic parser tree and additional lexicon features, experimental results show that our model achieves state-of-the-art performance on three datasets of two tasks.
A Short Survey on Taxonomy Learning from Text Corpora: Issues, Resources and Recent Advances
Authors: Xiaofeng He, Aoying Zhou Chengyu Wang
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: A taxonomy is a semantic hierarchy, consisting of concepts linked by is-a relations. While a large number of taxonomies have been constructed from human-compiled resources (e.g., Wikipedia), learning taxonomies from text corpora has received a growing interest and is essential for long-tailed and domain-specific knowledge acquisition. In this paper, we overview recent advances on taxonomy construction from free texts, reorganizing relevant subtasks into a complete framework. We also overview resources for evaluation and discuss challenges for future research.
Idiom-Aware Compositional Distributed Semantics
Authors: Kaiyu Qian, Xipeng Qiu, Xuanjing Huang Pengfei Liu
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Idioms are peculiar linguistic constructions that impose great challenges for representing the semantics of language, especially in current prevailing end-to-end neural models, which assume that the semantics of a phrase or sentence can be literally composed from its constitutive words. In this paper, we propose an idiom-aware distributed semantic model to build representation of sentences on the basis of understanding their contained idioms. Our models are grounded in the literal-first psycholinguistic hypothesis, which can adaptively learn semantic compositionality of a phrase literally or idiomatically. To better evaluate our models, we also construct an idiom-enriched sentiment classification dataset with considerable scale and abundant peculiarities of idioms. The qualitative and quantitative experimental analyses demonstrate the efficacy of our models.
Macro Grammars and Holistic Triggering for Efficient Semantic Parsing
Authors: Panupong Pasupat, Percy Liang Yuchen Zhang
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: To learn a semantic parser from denotations, a learning algorithm must search over a combinatorially large space of logical forms for ones consistent with the annotated denotations. We propose a new online learning algorithm that searches faster as training progresses. The two key ideas are using macro grammars to cache the abstract patterns of useful logical forms found thus far, and holistic triggering to efficiently retrieve the most relevant patterns based on sentence similarity. On the WikiTableQuestions dataset, we first expand the search space of an existing model to improve the state-of-the-art accuracy from 38.7% to 42.7%, and then use macro grammars and holistic triggering to achieve an 11x speedup and an accuracy of 43.7%.
A Continuously Growing Dataset of Sentential Paraphrases
Authors: Siyu Qiu, Hua He, Wei Xu Wuwei Lan
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: A major challenge in paraphrase research is the lack of parallel corpora. In this paper, we present a new method to collect large-scale sentential paraphrases from Twitter by linking tweets through shared URLs. The main advantage of our method is its simplicity, as it gets rid of the classifier or human in the loop needed to select data before annotation and subsequent application of paraphrase identification algorithms in the previous work. We present the largest human-labeled paraphrase corpus to date of 51,524 sentence pairs and the first cross-domain benchmarking for automatic paraphrase identification. In addition, we show that more than 30,000 new sentential paraphrases can be easily and continuously captured every month at ~70% precision, and demonstrate their utility for downstream NLP tasks through phrasal paraphrase extraction. We make our code and data freely available.
Cross-domain Semantic Parsing via Paraphrasing
Authors: Xifeng Yan Yu Su
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Existing studies on semantic parsing mainly focus on the in-domain setting. We formulate cross-domain semantic parsing as a domain adaptation problem: train a semantic parser on some source domains and then adapt it to the target domain. Due to the diversity of logical forms in different domains, this problem presents unique and intriguing challenges. By converting logical forms into canonical utterances in natural language, we reduce semantic parsing to paraphrasing, and develop an attentive sequence-to-sequence paraphrase model that is general and flexible to adapt to different domains. We discover two problems, small micro variance and large macro variance, of pre-trained word embeddings that hinder their direct use in neural networks, and propose standardization techniques as a remedy. On the popular Overnight dataset, which contains eight domains, we show that both cross-domain training and standardized pre-trained word embeddings can bring significant improvement.
A Joint Sequential and Relational Model for Frame-Semantic Parsing
Authors: Tom Mitchell Bishan Yang
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: We introduce a new method for frame-semantic parsing that significantly improves the prior state of the art. Our model leverages the advantages of a deep bidirectional LSTM network which predicts semantic role labels word by word and a relational network which predicts semantic roles for individual text expressions in relation to a predicate. The two networks are integrated into a single model via knowledge distillation, and a unified graphical model is employed to jointly decode frames and semantic roles during inference. Experiments on the standard FrameNet data show that our model significantly outperforms existing neural and non-neural approaches, achieving a 5.7 F1 gain over the current state of the art, for full frame structure extraction.
Getting the Most out of AMR Parsing
Authors: Nianwen Xue Chuan Wang
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: This paper proposes to tackle the AMR parsing bottleneck by improving two components of an AMR parser: concept identification and alignment. We first build a Bidirectional LSTM based concept identifier that is able to incorporate richer contextual information to learn sparse AMR concept labels. We then extend an HMM-based word-to-concept alignment model with graph distance distortion and a rescoring method during decoding to incorporate the structural information in the AMR graph. We show integrating the two components into an existing AMR parser results in consistently better performance over the state of the art on various datasets.
AMR Parsing using Stack-LSTMs
Authors: Yaser Al-Onaizan Miguel Ballesteros
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: We present a transition-based AMR parser that directly generates AMR parses from plain text. We use Stack-LSTMs to represent our parser state and make decisions greedily. In our experiments, we show that our parser achieves very competitive scores on English using only AMR training data. Adding additional information, such as POS tags and dependency trees, improves the results further.
An End-to-End Deep Framework for Answer Triggering with a Novel Group-Level Objective
Authors: Yu Su, Ziyu Guan, Huan Sun Jie Zhao
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Given a question and a set of answer candidates, answer triggering determines whether the candidate set contains any correct answers. If yes, it then outputs a correct one. In contrast to existing pipeline methods which first consider individual candidate answers separately and then make a prediction based on a threshold, we propose an end-to-end deep neural network framework, which is trained by a novel group-level objective function that directly optimizes the answer triggering performance. Our objective function penalizes three potential types of error and allows training the framework in an end-to-end manner. Experimental results on the WikiQA benchmark show that our framework outperforms the state of the arts by a 6.6% absolute gain under F1 measure.
Predicting Word Association Strengths
Authors: Xiaojuan Ma Andrew Cattle
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: This paper looks at the task of predicting word association strengths across three datasets; WordNet Evocation (Boyd-Graber et al., 2006), University of Southern Florida Free Association norms (Nelson et al., 2004), and Edinburgh Associative Thesaurus (Kiss et al., 1973). We achieve results of r=0.357 and p=0.379, r=0.344 and p=0.300, and r=0.292 and p=0.363, respectively. We find Word2Vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014) cosine similarities, as well as vector offsets, to be the highest performing features. Furthermore, we examine the usefulness of Gaussian embeddings (Vilnis and McCallum, 2014) for predicting word association strength, the first work to do so.
Learning Contextually Informed Representations for Linear-Time Discourse Parsing
Authors: Mirella Lapata Yang Liu
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Recent advances in RST discourse parsing have focused on two modeling paradigms: (a) high order parsers which jointly predict the tree structure of the discourse and the relations it encodes; or (b) linear-time parsers which are efficient but mostly based on local features. In this work, we propose a linear-time parser with a novel way of representing discourse constituents based on neural networks which takes into account global contextual information and is able to capture long-distance dependencies. Experimental results show that our parser obtains state-of-the art performance on benchmark datasets, while being efficient (with time complexity linear in the number of sentences in the document) and requiring minimal feature engineering.
Multi-task Attention-based Neural Networks for Implicit Discourse Relationship Representation and Identification
Authors: Jianxiang Wang, Yuanbin Wu, Zheng-Yu Niu, Haifeng Wang Man Lan
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: We present a novel multi-task attention based neural network model to address implicit discourse relationship representation and identification through two types of representation learning, an attention based neural network for learning discourse relationship representation with two arguments and a multi-task framework for learning knowledge from annotated and unannotated corpora. The extensive experiments have been performed on two benchmark corpora (i.e., PDTB and CoNLL-2016 datasets). Experimental results show that our proposed model outperforms the state-of-the-art systems on benchmark corpora.
Chinese Zero Pronoun Resolution with Deep Memory Network
Authors: Yu Zhang, Weinan Zhang, Ting Liu Qingyu Yin
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Existing approaches for Chinese zero pronoun resolution typically utilize only syntactical and lexical features while ignoring semantic information. The fundamental reason is that zero pronouns have no descriptive information, which brings difficulty in explicitly capturing their semantic similarities with antecedents. Meanwhile, representing zero pronouns is challenging since they are merely gaps that convey no actual content. In this paper, we address this issue by building a deep memory network that is capable of encoding zero pronouns into vector representations with information obtained from their contexts and potential antecedents. Consequently, our resolver takes advantage of semantic information by using these continuous distributed representations. Experiments on the OntoNotes 5.0 dataset show that the proposed memory network could substantially outperform the state-of-the-art systems in various experimental settings.
How much progress have we made on RST discourse parsing? A replication study of recent results on the RST-DT
Authors: Philippe Muller, Nicholas Asher Mathieu Morey
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: This article evaluates purported progress over the past years in RST discourse parsing. Several studies report a relative error reduction of 24 to 51% on all metrics that authors attribute to the introduction of distributed representations of discourse units. We replicate the standard evaluation of 9 parsers, 5 of which use distributed representations, from 8 studies published between 2013 and 2017, using their predictions on the test set of the RST-DT. Our main finding is that most recently reported increases in RST discourse parser performance are an artefact of differences in implementations of the evaluation procedure. We evaluate all these parsers with the standard Parseval procedure to provide a more accurate picture of the actual RST discourse parsers performance in standard evaluation settings. Under this more stringent procedure, the gains attributable to distributed representations represent at most a 16% relative error reduction on fully-labelled structures.
What is it? Disambiguating the different readings of the pronoun \textquoteleftit'
Authors: Liane Guillou, Christian Hardmeier Sharid Loáiciga
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: In this paper, we address the problem of predicting one of three functions for the English pronoun \textquoteleftit': anaphoric, event reference or pleonastic. This disambiguation is valuable in the context of machine translation and coreference resolution. We present experiments using a MAXENT classifier trained on gold-standard data and self-training experiments of an RNN trained on silver-standard data, annotated using the MAXENT classifier. Lastly, we report on an analysis of the strengths of these two models.
Revisiting Selectional Preferences for Coreference Resolution
Authors: Nafise Sadat Moosavi, Michael Strube Benjamin Heinzerling
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Selectional preferences have long been claimed to be essential for coreference resolution. However, they are modeled only implicitly by current coreference resolvers. We propose a dependency-based embedding model of selectional preferences which allows fine-grained compatibility judgments with high coverage. Incorporating our model improves performance, matching state-of-the-art results of a more complex system. However, it comes with a cost that makes it debatable how worthwhile are such improvements.
Learning to Rank Semantic Coherence for Topic Segmentation
Authors: Sujian Li, Yajuan Lv, Houfeng Wang Liang Wang
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Topic segmentation plays an important role for discourse parsing and information retrieval. Due to the absence of training data, previous work mainly adopts unsupervised methods to rank semantic coherence between paragraphs for topic segmentation. In this paper, we present an intuitive and simple idea to automatically create a “quasi” training dataset, which includes a large amount of text pairs from the same or different documents with different semantic coherence. With the training corpus, we design a symmetric CNN neural network to model text pairs and rank the semantic coherence within the learning to rank framework. Experiments show that our algorithm is able to achieve competitive performance over strong baselines on several real-world datasets.
GRASP: Rich Patterns for Argumentation Mining
Authors: Ran Levy, Vikas Raykar, Noam Slonim Eyal Shnarch
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: GRASP (GReedy Augmented Sequential Patterns) is an algorithm for automatically extracting patterns that characterize subtle linguistic phenomena. To that end, GRASP augments each term of input text with multiple layers of linguistic information. These different facets of the text terms are systematically combined to reveal rich patterns. We report highly promising experimental results in several challenging text analysis tasks within the field of Argumentation Mining. We believe that GRASP is general enough to be useful for other domains too. For example, each of the following sentences includes a claim for a [topic]: 1. Opponents often argue that the open primary is unconstitutional. [Open Primaries] 2. Prof. Smith suggested that affirmative action devalues the accomplishments of the chosen. [Affirmative Action] 3. The majority stated that the First Amendment does not guarantee the right to offend others. [Freedom of Speech] These sentences share almost no words in common, however, they are similar at a more abstract level. A human observer may notice the following underlying common structure, or pattern: [someone][argue/suggest/state][that][topic term][sentiment term]. GRASP aims to automatically capture such underlying structures of the given data. For the above examples it finds the pattern [noun][express][that][noun,topic][sentiment], where [express] stands for all its (in)direct hyponyms, and [noun,topic] means a noun which is also related to the topic.
Patterns of Argumentation Strategies across Topics
Authors: Henning Wachsmuth, Matthias Hagen, Benno Stein Khalid Al-Khatib
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: This paper presents an analysis of argumentation strategies in news editorials within and across topics. Given nearly 29,000 argumentative editorials from the New York Times, we develop two machine learning models, one for determining an editorial`s topic, and one for identifying evidence types in the editorial. Based on the distribution and structure of the identified types, we analyze the usage patterns of argumentation strategies among 12 different topics. We detect several common patterns that provide insights into the manifestation of argumentation strategies. Also, our experiments reveal clear correlations between the topics and the detected patterns.
Using Argument-based Features to Predict and Analyse Review Helpfulness
Authors: Yang Gao, Pin Lv, Mengxue Li, Shiqiang Geng, Minglan Li, Hao Wang Haijing Liu
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: We study the helpful product reviews identification problem in this paper. We observe that the evidence-conclusion discourse relations, also known as arguments, often appear in product reviews, and we hypothesise that some argument-based features, e.g. the percentage of argumentative sentences, the evidences-conclusions ratios, are good indicators of helpful reviews. To validate this hypothesis, we manually annotate arguments in 110 hotel reviews, and investigate the effectiveness of several combinations of argument-based features. Experiments suggest that, when being used together with the argument-based features, the state-of-the-art baseline features can enjoy a performance boost (in terms of F1) of 11.01% in average.
Here`s My Point: Joint Pointer Architecture for Argument Mining
Authors: Alexey Romanov, Anna Rumshisky Peter Potash
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: In order to determine argument structure in text, one must understand how individual components of the overall argument are linked. This work presents the first neural network-based approach to link extraction in argument mining. Specifically, we propose a novel architecture that applies Pointer Network sequence-to-sequence attention modeling to structural prediction in discourse parsing tasks. We then develop a joint model that extends this architecture to simultaneously address the link extraction task and the classification of argument components. The proposed joint model achieves state-of-the-art results on two separate evaluation corpora, showing far superior performance than the previously proposed corpus-specific and heavily feature-engineered models. Furthermore, our results demonstrate that jointly optimizing for both tasks is crucial for high performance.
Identifying attack and support argumentative relations using deep learning
Authors: Francesca Toni Oana Cocarascu
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: We propose a deep learning architecture to capture argumentative relations of attack and support from one piece of text to another, of the kind that naturally occur in a debate. The architecture uses two (unidirectional or bidirectional) Long Short-Term Memory networks and (trained or non-trained) word embeddings, and allows to considerably improve upon existing techniques that use syntactic features and supervised classifiers for the same form of (relation-based) argument mining.
Neural Lattice-to-Sequence Models for Uncertain Inputs
Authors: Graham Neubig, Jan Niehues, Alex Waibel Matthias Sperber
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: The input to a neural sequence-to-sequence model is often determined by an up-stream system, e.g. a word segmenter, part of speech tagger, or speech recognizer. These up-stream models are potentially error-prone. Representing inputs through word lattices allows making this uncertainty explicit by capturing alternative sequences and their posterior probabilities in a compact form. In this work, we extend the TreeLSTM (Tai et al., 2015) into a LatticeLSTM that is able to consume word lattices, and can be used as encoder in an attentional encoder-decoder model. We integrate lattice posterior scores into this architecture by extending the TreeLSTM`s child-sum and forget gates and introducing a bias term into the attention mechanism. We experiment with speech translation lattices and report consistent improvements over baselines that translate either the 1-best hypothesis or the lattice without posterior scores.
Memory-augmented Neural Machine Translation
Authors: Shiyue Zhang, Andi Zhang, Dong Wang, Andrew Abel Yang Feng
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Neural machine translation (NMT) has achieved notable success in recent times, however it is also widely recognized that this approach has limitations with handling infrequent words and word pairs. This paper presents a novel memory-augmented NMT (M-NMT) architecture, which stores knowledge about how words (usually infrequently encountered ones) should be translated in a memory and then utilizes them to assist the neural model. We use this memory mechanism to combine the knowledge learned from a conventional statistical machine translation system and the rules learned by an NMT system, and also propose a solution for out-of-vocabulary (OOV) words based on this framework. Our experiments on two Chinese-English translation tasks demonstrated that the M-NMT architecture outperformed the NMT baseline by 9.0 and 2.7 BLEU points on the two tasks, respectively. Additionally, we found this architecture resulted in a much more effective OOV treatment compared to competitive methods.
Dynamic Data Selection for Neural Machine Translation
Authors: Arianna Bisazza, Christof Monz Marlies van der Wees
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Intelligent selection of training data has proven a successful technique to simultaneously increase training efficiency and translation performance for phrase-based machine translation (PBMT). With the recent increase in popularity of neural machine translation (NMT), we explore in this paper to what extent and how NMT can also benefit from data selection. While state-of-the-art data selection (Axelrod et al., 2011) consistently performs well for PBMT, we show that gains are substantially lower for NMT. Next, we introduce \textquoteleftdynamic data selection' for NMT, a method in which we vary the selected subset of training data between different training epochs. Our experiments show that the best results are achieved when applying a technique we call \textquoteleftgradual fine-tuning', with improvements up to +2.6 BLEU over the original data selection approach and up to +3.1 BLEU over a general baseline.
Neural Machine Translation Leveraging Phrase-based Models in a Hybrid Search
Authors: Evgeny Matusov, Pavel Petrushkov, Shahram Khadivi Leonard Dahlmann
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: In this paper, we introduce a hybrid search for attention-based neural machine translation (NMT). A target phrase learned with statistical MT models extends a hypothesis in the NMT beam search when the attention of the NMT model focuses on the source words translated by this phrase. Phrases added in this way are scored with the NMT model, but also with SMT features including phrase-level translation probabilities and a target language model. Experimental results on German-to-English news domain and English-to-Russian e-commerce domain translation tasks show that using phrase-based models in NMT search improves MT quality by up to 2.3% BLEU absolute as compared to a strong NMT baseline.
Translating Phrases in Neural Machine Translation
Authors: Zhaopeng Tu, Deyi Xiong, Min Zhang Xing Wang
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Phrases play an important role in natural language understanding and machine translation (Sag et al., 2002; Villavicencio et al., 2005). However, it is difficult to integrate them into current neural machine translation (NMT) which reads and generates sentences word by word. In this work, we propose a method to translate phrases in NMT by integrating a phrase memory storing target phrases from a phrase-based statistical machine translation (SMT) system into the encoder-decoder architecture of NMT. At each decoding step, the phrase memory is first re-written by the SMT model, which dynamically generates relevant target phrases with contextual information provided by the NMT model. Then the proposed model reads the phrase memory to make probability estimations for all phrases in the phrase memory. If phrase generation is carried on, the NMT decoder selects an appropriate phrase from the memory to perform phrase translation and updates its decoding state by consuming the words in the selected phrase. Otherwise, the NMT decoder generates a word from the vocabulary as the general NMT decoder does. Experiment results on the Chinese to English translation show that the proposed model achieves significant improvements over the baseline on various test sets.
Towards Bidirectional Hierarchical Representations for Attention-based Neural Machine Translation
Authors: Derek F. Wong, Tong Xiao, Lidia S. Chao, Jingbo Zhu Baosong Yang
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: This paper proposes a hierarchical attentional neural translation model which focuses on enhancing source-side hierarchical representations by covering both local and global semantic information using a bidirectional tree-based encoder. To maximize the predictive likelihood of target words, a weighted variant of an attention mechanism is used to balance the attentive information between lexical and phrase vectors. Using a tree-based rare word encoding, the proposed model is extended to sub-word level to alleviate the out-of-vocabulary (OOV) problem. Empirical results reveal that the proposed model significantly outperforms sequence-to-sequence attention-based and tree-based neural translation models in English-Chinese translation tasks.
Massive Exploration of Neural Machine Translation Architectures
Authors: Anna Goldie, Minh-Thang Luong, Quoc Le Denny Britz
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Neural Machine Translation (NMT) has shown remarkable progress over the past few years, with production systems now being deployed to end-users. As the field is moving rapidly, it has become unclear which elements of NMT architectures have a significant impact on translation quality. In this work, we present a large-scale analysis of the sensitivity of NMT architectures to common hyperparameters. We report empirical results and variance numbers for several hundred experimental runs, corresponding to over 250,000 GPU hours on a WMT English to German translation task. Our experiments provide practical insights into the relative importance of factors such as embedding size, network depth, RNN cell type, residual connections, attention mechanism, and decoding heuristics. As part of this contribution, we also release an open-source NMT framework in TensorFlow to make it easy for others to reproduce our results and perform their own experiments.
Learning Translations via Matrix Completion
Authors: Brendan Callahan, John Hewitt, Jie Gao, Xiao Ling, Marianna Apidianaki, Chris Callison-Burch Derry Tanti Wijaya
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Bilingual Lexicon Induction is the task of learning word translations without bilingual parallel corpora. We model this task as a matrix completion problem, and present an effective and extendable framework for completing the matrix. This method harnesses diverse bilingual and monolingual signals, each of which may be incomplete or noisy. Our model achieves state-of-the-art performance for both high and low resource languages.
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
Authors: Hal Daumé III, Jordan Boyd-Graber Khanh Nguyen
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Machine translation is a natural candidate problem for reinforcement learning from human feedback: users provide quick, dirty ratings on candidate translations to guide a system to improve. Yet, current neural machine translation training focuses on expensive human-generated reference translations. We describe a reinforcement learning algorithm that improves neural machine translation systems from simulated human feedback. Our algorithm combines the advantage actor-critic algorithm (Mnih et al., 2016) with the attention-based neural encoder-decoder architecture (Luong et al., 2015). This algorithm (a) is well-designed for problems with a large action space and delayed rewards, (b) effectively optimizes traditional corpus-level machine translation metrics, and (c) is robust to skewed, high-variance, granular feedback modeled after actual human behaviors.
Towards Compact and Fast Neural Machine Translation Using a Combined Method
Authors: Wei Chen, Feng Wang, Shuang Xu, Bo Xu Xiaowei Zhang
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Neural Machine Translation (NMT) lays intensive burden on computation and memory cost. It is a challenge to deploy NMT models on the devices with limited computation and memory budgets. This paper presents a four stage pipeline to compress model and speed up the decoding for NMT. Our method first introduces a compact architecture based on convolutional encoder and weight shared embeddings. Then weight pruning is applied to obtain a sparse model. Next, we propose a fast sequence interpolation approach which enables the greedy decoding to achieve performance on par with the beam search. Hence, the time-consuming beam search can be replaced by simple greedy decoding. Finally, vocabulary selection is used to reduce the computation of softmax layer. Our final model achieves 10 times speedup, 17 times parameters reduction, less than 35MB storage size and comparable performance compared to the baseline model.
Instance Weighting for Neural Machine Translation Domain Adaptation
Authors: Masao Utiyama, Lemao Liu, Kehai Chen, Eiichiro Sumita Rui Wang
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Instance weighting has been widely applied to phrase-based machine translation domain adaptation. However, it is challenging to be applied to Neural Machine Translation (NMT) directly, because NMT is not a linear model. In this paper, two instance weighting technologies, i.e., sentence weighting and domain weighting with a dynamic weight learning strategy, are proposed for NMT domain adaptation. Empirical results on the IWSLT English-German/French tasks show that the proposed methods can substantially improve NMT performance by up to 2.7-6.7 BLEU points, outperforming the existing baselines by up to 1.6-3.6 BLEU points.
Regularization techniques for fine-tuning in neural machine translation
Authors: Barry Haddow, Ulrich Germann, Rico Sennrich Antonio Valerio Miceli Barone
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: We investigate techniques for supervised domain adaptation for neural machine translation where an existing model trained on a large out-of-domain dataset is adapted to a small in-domain dataset. In this scenario, overfitting is a major challenge. We investigate a number of techniques to reduce overfitting and improve transfer learning, including regularization techniques such as dropout and L2-regularization towards an out-of-domain prior. In addition, we introduce tuneout, a novel regularization technique inspired by dropout. We apply these techniques, alone and in combination, to neural machine translation, obtaining improvements on IWSLT datasets for English\textrightarrowGerman and English\textrightarrowRussian. We also investigate the amounts of in-domain training data needed for domain adaptation in NMT, and find a logarithmic relationship between the amount of training data and gain in BLEU score.
Source-Side Left-to-Right or Target-Side Left-to-Right? An Empirical Comparison of Two Phrase-Based Decoding Algorithms
Authors: Michael Collins Yin-Wen Chang
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: This paper describes an empirical study of the phrase-based decoding algorithm proposed by Chang and Collins (2017). The algorithm produces a translation by processing the source-language sentence in strictly left-to-right order, differing from commonly used approaches that build the target-language sentence in left-to-right order. Our results show that the new algorithm is competitive with Moses (Koehn et al., 2007) in terms of both speed and BLEU scores.
Using Target-side Monolingual Data for Neural Machine Translation through Multi-task Learning
Authors: Felix Hieber Tobias Domhan
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: The performance of Neural Machine Translation (NMT) models relies heavily on the availability of sufficient amounts of parallel data, and an efficient and effective way of leveraging the vastly available amounts of monolingual data has yet to be found. We propose to modify the decoder in a neural sequence-to-sequence model to enable multi-task learning for two strongly related tasks: target-side language modeling and translation. The decoder predicts the next target word through two channels, a target-side language model on the lowest layer, and an attentional recurrent model which is conditioned on the source representation. This architecture allows joint training on both large amounts of monolingual and moderate amounts of bilingual data to improve NMT performance. Initial results in the news domain for three language pairs show moderate but consistent improvements over a baseline trained on bilingual data only.
Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling
Authors: Ivan Titov Diego Marcheggiani
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Semantic role labeling (SRL) is the task of identifying the predicate-argument structure of a sentence. It is typically regarded as an important step in the standard NLP pipeline. As the semantic representations are closely related to syntactic ones, we exploit syntactic information in our model. We propose a version of graph convolutional networks (GCNs), a recent class of neural networks operating on graphs, suited to model syntactic dependency graphs. GCNs over syntactic dependency trees are used as sentence encoders, producing latent feature representations of words in a sentence. We observe that GCN layers are complementary to LSTM ones: when we stack both GCN and LSTM layers, we obtain a substantial improvement over an already state-of-the-art LSTM SRL model, resulting in the best reported scores on the standard benchmark (CoNLL-2009) both for Chinese and English.
Neural Semantic Parsing with Type Constraints for Semi-Structured Tables
Authors: Pradeep Dasigi, Matt Gardner Jayant Krishnamurthy
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: We present a new semantic parsing model for answering compositional questions on semi-structured Wikipedia tables. Our parser is an encoder-decoder neural network with two key technical innovations: (1) a grammar for the decoder that only generates well-typed logical forms; and (2) an entity embedding and linking module that identifies entity mentions while generalizing across tables. We also introduce a novel method for training our neural model with question-answer supervision. On the WikiTableQuestions data set, our parser achieves a state-of-the-art accuracy of 43.3% for a single model and 45.9% for a 5-model ensemble, improving on the best prior score of 38.7% set by a 15-model ensemble. These results suggest that type constraints and entity linking are valuable components to incorporate in neural semantic parsers.
Joint Concept Learning and Semantic Parsing from Natural Language Explanations
Authors: Igor Labutov, Tom Mitchell Shashank Srivastava
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Natural language constitutes a predominant medium for much of human learning and pedagogy. We consider the problem of concept learning from natural language explanations, and a small number of labeled examples of the concept. For example, in learning the concept of a phishing email, one might say \textquoteleftthis is a phishing email because it asks for your bank account number'. Solving this problem involves both learning to interpret open ended natural language statements, and learning the concept itself. We present a joint model for (1) language interpretation (semantic parsing) and (2) concept learning (classification) that does not require labeling statements with logical forms. Instead, the model prefers discriminative interpretations of statements in context of observable features of the data as a weak signal for parsing. On a dataset of email-related concepts, our approach yields across-the-board improvements in classification performance, with a 30% relative improvement in F1 score over competitive methods in the low data regime.
Grasping the Finer Point: A Supervised Similarity Network for Metaphor Detection
Authors: Luana Bulat, Douwe Kiela, Ekaterina Shutova Marek Rei
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: The ubiquity of metaphor in our everyday communication makes it an important problem for natural language understanding. Yet, the majority of metaphor processing systems to date rely on hand-engineered features and there is still no consensus in the field as to which features are optimal for this task. In this paper, we present the first deep learning architecture designed to capture metaphorical composition. Our results demonstrate that it outperforms the existing approaches in the metaphor identification task.
Identifying civilians killed by police with distantly supervised entity-event extraction
Authors: Abram Handler, Michael Pinkham, Cara Magliozzi, Joshua McDuffie, Brendan O′Connor Katherine Keith
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: We propose a new, socially-impactful task for natural language processing: from a news corpus, extract names of persons who have been killed by police. We present a newly collected police fatality corpus, which we release publicly, and present a model to solve this problem that uses EM-based distant supervision with logistic regression and convolutional neural network classifiers. Our model outperforms two off-the-shelf event extractor systems, and it can suggest candidate victim names in some cases faster than one of the major manually-collected police fatality databases.