mtybilly/Instruct2DS huggingface dataset:
Neural Transition Based Parsing of Web Queries: An Entity Based Approach
Authors: Rivka Malca, Roi Reichart
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Web queries with question intent manifest a complex syntactic structure and the processing of this structure is important for their interpretation. Pinter et al. (2016) has formalized the grammar of these queries and proposed semi-supervised algorithms for the adaptation of parsers originally designed to parse according to the standard dependency grammar, so that they can account for the unique forest grammar of queries. However, their algorithms rely on resources typically not available outside of big web corporates. We propose a new BiLSTM query parser that: (1) Explicitly accounts for the unique grammar of web queries; and (2) Utilizes named entity (NE) information from a BiLSTM NE tagger, that can be jointly trained with the parser. In order to train our model we annotate the query treebank of Pinter et al. (2016) with NEs. When trained on 2500 annotated queries our parser achieves UAS of 83.5% and segmentation F1-score of 84.5, substantially outperforming existing state-of-the-art parsers.
An Investigation of the Interactions Between Pre-Trained Word Embeddings, Character Models and POS Tags in Dependency Parsing
Authors: Aaron Smith, Miryam de Lhoneux, Sara Stymne, Joakim Nivre
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: We provide a comprehensive analysis of the interactions between pre-trained word embeddings, character models and POS tags in a transition-based dependency parser. While previous studies have shown POS information to be less important in the presence of character models, we show that in fact there are complex interactions between all three techniques. In isolation each produces large improvements over a baseline system using randomly initialised word embeddings only, but combining them quickly leads to diminishing returns. We categorise words by frequency, POS tag and language in order to systematically investigate how each of the techniques affects parsing quality. For many word categories, applying any two of the three techniques is almost as good as the full combined system. Character models tend to be more important for low-frequency open-class words, especially in morphologically rich languages, while POS tags can help disambiguate high-frequency function words. We also show that large character embedding sizes help even for languages with small character sets, especially in morphologically rich languages.
Depth-bounding is effective: Improvements and evaluation of unsupervised PCFG induction
Authors: Lifeng Jin, Finale Doshi-Velez, Timothy Miller, William Schuler, Lane Schwartz
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: There have been several recent attempts to improve the accuracy of grammar induction systems by bounding the recursive complexity of the induction model. Modern depth-bounded grammar inducers have been shown to be more accurate than early unbounded PCFG inducers, but this technique has never been compared against unbounded induction within the same system, in part because most previous depth-bounding models are built around sequence models, the complexity of which grows exponentially with the maximum allowed depth. The present work instead applies depth bounds within a chart-based Bayesian PCFG inducer, where bounding can be switched on and off, and then samples trees with or without bounding. Results show that depth-bounding is indeed significantly effective in limiting the search space of the inducer and thereby increasing accuracy of resulting parsing model, independent of the contribution of modern Bayesian induction techniques. Moreover, parsing results on English, Chinese and German show that this bounded model is able to produce parse trees more accurately than or competitively with state-of-the-art constituency grammar induction models.
Incremental Computation of Infix Probabilities for Probabilistic Finite Automata
Authors: Marco Cognetta, Yo-Sub Han, Soon Chan Kwon
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: In natural language processing, a common task is to compute the probability of a phrase appearing in a document or to calculate the probability of all phrases matching a given pattern. For instance, one computes affix (prefix, suffix, infix, etc.) probabilities of a string or a set of strings with respect to a probability distribution of patterns. The problem of computing infix probabilities of strings when the pattern distribution is given by a probabilistic context-free grammar or by a probabilistic finite automaton is already solved, yet it was open to compute the infix probabilities in an incremental manner. The incremental computation is crucial when a new query is built from a previous query. We tackle this problem and suggest a method that computes infix probabilities incrementally for probabilistic finite automata by representing all the probabilities of matching strings as a series of transition matrix calculations. We show that the proposed approach is theoretically faster than the previous method and, using real world data, demonstrate that our approach has vastly better performance in practice.
Syntax Encoding with Application in Authorship Attribution
Authors: Richong Zhang, Zhiyuan Hu, Hongyu Guo, Yongyi Mao
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: We propose a novel strategy to encode the syntax parse tree of sentence into a learnable distributed representation. The proposed syntax encoding scheme is provably information-lossless. In specific, an embedding vector is constructed for each word in the sentence, encoding the path in the syntax tree corresponding to the word. The one-to-one correspondence between these “syntax-embedding” vectors and the words (hence their embedding vectors) in the sentence makes it easy to integrate such a representation with all word-level NLP models. We empirically show the benefits of the syntax embeddings on the Authorship Attribution domain, where our approach improves upon the prior art and achieves new performance records on five benchmarking data sets.
Sanskrit Word Segmentation Using Character-level Recurrent and Convolutional Neural Networks
Authors: Oliver Hellwig, Sebastian Nehrdich
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: The paper introduces end-to-end neural network models that tokenize Sanskrit by jointly splitting compounds and resolving phonetic merges (Sandhi). Tokenization of Sanskrit depends on local phonetic and distant semantic features that are incorporated using convolutional and recurrent elements. Contrary to most previous systems, our models do not require feature engineering or extern linguistic resources, but operate solely on parallel versions of raw and segmented text. The models discussed in this paper clearly improve over previous approaches to Sanskrit word segmentation. As they are language agnostic, we will demonstrate that they also outperform the state of the art for the related task of German compound splitting.
Session-level Language Modeling for Conversational Speech
Authors: Wayne Xiong, Lingfeng Wu, Jun Zhang, Andreas Stolcke
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: We propose to generalize language models for conversational speech recognition to allow them to operate across utterance boundaries and speaker changes, thereby capturing conversation-level phenomena such as adjacency pairs, lexical entrainment, and topical coherence. The model consists of a long-short-term memory (LSTM) recurrent network that reads the entire word-level history of a conversation, as well as information about turn taking and speaker overlap, in order to predict each next word. The model is applied in a rescoring framework, where the word history prior to the current utterance is approximated with preliminary recognition results. In experiments in the conversational telephone speech domain (Switchboard) we find that such a model gives substantial perplexity reductions over a standard LSTM-LM with utterance scope, as well as improvements in word error rate.
Towards Less Generic Responses in Neural Conversation Models: A Statistical Re-weighting Method
Authors: Yahui Liu, Wei Bi, Jun Gao, Xiaojiang Liu, Jian Yao, Shuming Shi
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Sequence-to-sequence neural generation models have achieved promising performance on short text conversation tasks. However, they tend to generate generic/dull responses, leading to unsatisfying dialogue experience. We observe that in the conversation tasks, each query could have multiple responses, which forms a 1-to-n or m-to-n relationship in the view of the total corpus. The objective function used in standard sequence-to-sequence models will be dominated by loss terms with generic patterns. Inspired by this observation, we introduce a statistical re-weighting method that assigns different weights for the multiple responses of the same query, and trains the common neural generation model with the weights. Experimental results on a large Chinese dialogue corpus show that our method improves the acceptance rate of generated responses compared with several baseline models and significantly reduces the number of generated generic responses.
Training Millions of Personalized Dialogue Agents
Authors: Pierre-Emmanuel Mazaré, Samuel Humeau, Martin Raison, Antoine Bordes
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Current dialogue systems fail at being engaging for users, especially when trained end-to-end without relying on proactive reengaging scripted strategies. Zhang et al. (2018) showed that the engagement level of end-to-end dialogue models increases when conditioning them on text personas providing some personalized back-story to the model. However, the dataset used in Zhang et al. (2018) is synthetic and only contains around 1k different personas. In this paper we introduce a new dataset providing 5 million personas and 700 million persona-based dialogues. Our experiments show that, at this scale, training using personas still improves the performance of end-to-end systems. In addition, we show that other tasks benefit from the wide coverage of our dataset by fine-tuning our model on the data from Zhang et al. (2018) and achieving state-of-the-art results.
Towards Universal Dialogue State Tracking
Authors: Liliang Ren, Kaige Xie, Lu Chen, Kai Yu
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Dialogue state tracker is the core part of a spoken dialogue system. It estimates the beliefs of possible user`s goals at every dialogue turn. However, for most current approaches, it`s difficult to scale to large dialogue domains. They have one or more of following limitations: (a) Some models don`t work in the situation where slot values in ontology changes dynamically; (b) The number of model parameters is proportional to the number of slots; (c) Some models extract features based on hand-crafted lexicons. To tackle these challenges, we propose StateNet, a universal dialogue state tracker. It is independent of the number of values, shares parameters across all slots, and uses pre-trained word vectors instead of explicit semantic dictionaries. Our experiments on two datasets show that our approach not only overcomes the limitations, but also significantly outperforms the performance of state-of-the-art approaches.
Semantic Parsing for Task Oriented Dialog using Hierarchical Representations
Authors: Sonal Gupta, Rushin Shah, Mrinal Mohit, Anuj Kumar, Mike Lewis
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Task oriented dialog systems typically first parse user utterances to semantic frames comprised of intents and slots. Previous work on task oriented intent and slot-filling work has been restricted to one intent per query and one slot label per token, and thus cannot model complex compositional requests. Alternative semantic parsing systems have represented queries as logical forms, but these are challenging to annotate and parse. We propose a hierarchical annotation scheme for semantic parsing that allows the representation of compositional queries, and can be efficiently and accurately parsed by standard constituency parsing models. We release a dataset of 44k annotated queries (r̆lhttp://fb.me/semanticparsingdialog), and show that parsing models outperform sequence-to-sequence approaches on this dataset.
Reducing Gender Bias in Abusive Language Detection
Authors: Ji Ho Park, Jamin Shin, Pascale Fung
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Abusive language detection models tend to have a problem of being biased toward identity words of a certain group of people because of imbalanced training datasets. For example, “You are a good woman” was considered “sexist” when trained on an existing dataset. Such model bias is an obstacle for models to be robust enough for practical use. In this work, we measure them on models trained with different datasets, while analyzing the effect of different pre-trained word embeddings and model architectures. We also experiment with three mitigation methods: (1) debiased word embeddings, (2) gender swap data augmentation, and (3) fine-tuning with a larger corpus. These methods can effectively reduce model bias by 90-98% and can be extended to correct model bias in other scenarios.
SafeCity: Understanding Diverse Forms of Sexual Harassment Personal Stories
Authors: Sweta Karlekar, Mohit Bansal
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: With the recent rise of #MeToo, an increasing number of personal stories about sexual harassment and sexual abuse have been shared online. In order to push forward the fight against such harassment and abuse, we present the task of automatically categorizing and analyzing various forms of sexual harassment, based on stories shared on the online forum SafeCity. For the labels of groping, ogling, and commenting, our single-label CNN-RNN model achieves an accuracy of 86.5%, and our multi-label model achieves a Hamming score of 82.5%. Furthermore, we present analysis using LIME, first-derivative saliency heatmaps, activation clustering, and embedding visualization to interpret neural model predictions and demonstrate how this helps extract features that can help automatically fill out incident reports, identify unsafe areas, avoid unsafe practices, and \textquoteleftpin the creeps'.
Learning multiview embeddings for assessing dementia
Authors: Chloé Pou-Prom, Frank Rudzicz
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: As the incidence of Alzheimer`s Disease (AD) increases, early detection becomes crucial. Unfortunately, datasets for AD assessment are often sparse and incomplete. In this work, we leverage the multiview nature of a small AD dataset, DementiaBank, to learn an embedding that captures different modes of cognitive impairment. We apply generalized canonical correlation analysis (GCCA) to our dataset and demonstrate the added benefit of using multiview embeddings in two downstream tasks: identifying AD and predicting clinical scores. By including multiview embeddings, we obtain an F1 score of 0.82 in the classification task and a mean absolute error of 3.42 in the regression task. Furthermore, we show that multiview embeddings can be obtained from other datasets as well.
WikiConv: A Corpus of the Complete Conversational History of a Large Online Collaborative Community
Authors: Yiqing Hua, Cristian Danescu-Niculescu-Mizil, Dario Taraborelli, Nithum Thain, Jeffery Sorensen, Lucas Dixon
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: We present a corpus that encompasses the complete history of conversations between contributors to Wikipedia, one of the largest online collaborative communities. By recording the intermediate states of conversations - including not only comments and replies, but also their modifications, deletions and restorations - this data offers an unprecedented view of online conversation. Our framework is designed to be language agnostic, and we show that it extracts high quality data in both Chinese and English. This level of detail supports new research questions pertaining to the process (and challenges) of large-scale online collaboration. We illustrate the corpus' potential with two case studies on English Wikipedia that highlight new perspectives on earlier work. First, we explore how a person`s conversational behavior depends on how they relate to the discussion`s venue. Second, we show that community moderation of toxic behavior happens at a higher rate than previously estimated.
Marginal Likelihood Training of BiLSTM-CRF for Biomedical Named Entity Recognition from Disjoint Label Sets
Authors: Nathan Greenberg, Trapit Bansal, Patrick Verga, Andrew McCallum
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Extracting typed entity mentions from text is a fundamental component to language understanding and reasoning. While there exist substantial labeled text datasets for multiple subsets of biomedical entity types---such as genes and proteins, or chemicals and diseases---it is rare to find large labeled datasets containing labels for all desired entity types together. This paper presents a method for training a single CRF extractor from multiple datasets with disjoint or partially overlapping sets of entity types. Our approach employs marginal likelihood training to insist on labels that are present in the data, while filling in “missing labels”. This allows us to leverage all the available data within a single model. In experimental results on the Biocreative V CDR (chemicals/diseases), Biocreative VI ChemProt (chemicals/proteins) and MedMentions (19 entity types) datasets, we show that joint training on multiple datasets improves NER F1 over training in isolation, and our methods achieve state-of-the-art results.
Adversarial training for multi-context joint entity and relation extraction
Authors: Giannis Bekoulis, Johannes Deleu, Thomas Demeester, Chris Develder
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Adversarial training (AT) is a regularization method that can be used to improve the robustness of neural network methods by adding small perturbations in the training data. We show how to use AT for the tasks of entity recognition and relation extraction. In particular, we demonstrate that applying AT to a general purpose baseline model for jointly extracting entities and relations, allows improving the state-of-the-art effectiveness on several datasets in different contexts (i.e., news, biomedical, and real estate data) and for different languages (English and Dutch).
Structured Multi-Label Biomedical Text Tagging via Attentive Neural Tree Decoding
Authors: Gaurav Singh, James Thomas, Iain Marshall, John Shawe-Taylor, Byron C. Wallace
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: We propose a model for tagging unstructured texts with an arbitrary number of terms drawn from a tree-structured vocabulary (i.e., an ontology). We treat this as a special case of sequence-to-sequence learning in which the decoder begins at the root node of an ontological tree and recursively elects to expand child nodes as a function of the input text, the current node, and the latent decoder state. We demonstrate that this method yields state-of-the-art results on the important task of assigning MeSH terms to biomedical abstracts.
Deep Exhaustive Model for Nested Named Entity Recognition
Authors: Mohammad Golam Sohrab, Makoto Miwa
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: We propose a simple deep neural model for nested named entity recognition (NER). Most NER models focused on flat entities and ignored nested entities, which failed to fully capture underlying semantic information in texts. The key idea of our model is to enumerate all possible regions or spans as potential entity mentions and classify them with deep neural networks. To reduce the computational costs and capture the information of the contexts around the regions, the model represents the regions using the outputs of shared underlying bidirectional long short-term memory. We evaluate our exhaustive model on the GENIA and JNLPBA corpora in biomedical domain, and the results show that our model outperforms state-of-the-art models on nested and flat NER, achieving 77.1% and 78.4% respectively in terms of F-score, without any external knowledge resources.
Evaluating the Utility of Hand-crafted Features in Sequence Labelling
Authors: Minghao Wu, Fei Liu, Trevor Cohn
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Conventional wisdom is that hand-crafted features are redundant for deep learning models, as they already learn adequate representations of text automatically from corpora. In this work, we test this claim by proposing a new method for exploiting handcrafted features as part of a novel hybrid learning approach, incorporating a feature auto-encoder loss component. We evaluate on the task of named entity recognition (NER), where we show that including manual features for part-of-speech, word shapes and gazetteers can improve the performance of a neural CRF model. We obtain a F 1 of 91.89 for the CoNLL-2003 English shared task, which significantly outperforms a collection of highly competitive baseline models. We also present an ablation study showing the importance of auto-encoding, over using features as either inputs or outputs alone, and moreover, show including the autoencoder components reduces training requirements to 60%, while retaining the same predictive accuracy.
Improved Dependency Parsing using Implicit Word Connections Learned from Unlabeled Data
Authors: Wenhui Wang, Baobao Chang, Mairgup Mansur
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Pre-trained word embeddings and language model have been shown useful in a lot of tasks. However, both of them cannot directly capture word connections in a sentence, which is important for dependency parsing given its goal is to establish dependency relations between words. In this paper, we propose to implicitly capture word connections from unlabeled data by a word ordering model with self-attention mechanism. Experiments show that these implicit word connections do improve our parsing model. Furthermore, by combining with a pre-trained language model, our model gets state-of-the-art performance on the English PTB dataset, achieving 96.35% UAS and 95.25% LAS.
A Framework for Understanding the Role of Morphology in Universal Dependency Parsing
Authors: Mathieu Dehouck, Pascal Denis
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: This paper presents a simple framework for characterizing morphological complexity and how it encodes syntactic information. In particular, we propose a new measure of morpho-syntactic complexity in terms of governor-dependent preferential attachment that explains parsing performance. Through experiments on dependency parsing with data from Universal Dependencies (UD), we show that representations derived from morphological attributes deliver important parsing performance improvements over standard word form embeddings when trained on the same datasets. We also show that the new morpho-syntactic complexity measure is predictive of the gains provided by using morphological attributes over plain forms on parsing scores, making it a tool to distinguish languages using morphology as a syntactic marker from others.
The Lazy Encoder: A Fine-Grained Analysis of the Role of Morphology in Neural Machine Translation
Authors: Arianna Bisazza, Clara Tump
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Neural sequence-to-sequence models have proven very effective for machine translation, but at the expense of model interpretability. To shed more light into the role played by linguistic structure in the process of neural machine translation, we perform a fine-grained analysis of how various source-side morphological features are captured at different levels of the NMT encoder while varying the target language. Differently from previous work, we find no correlation between the accuracy of source morphology encoding and translation quality. We do find that morphological features are only captured in context and only to the extent that they are directly transferable to the target words.
Imitation Learning for Neural Morphological String Transduction
Authors: Peter Makarov, Simon Clematide
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: We employ imitation learning to train a neural transition-based string transducer for morphological tasks such as inflection generation and lemmatization. Previous approaches to training this type of model either rely on an external character aligner for the production of gold action sequences, which results in a suboptimal model due to the unwarranted dependence on a single gold action sequence despite spurious ambiguity, or require warm starting with an MLE model. Our approach only requires a simple expert policy, eliminating the need for a character aligner or warm start. It also addresses familiar MLE training biases and leads to strong and state-of-the-art performance on several benchmarks.
An Encoder-Decoder Approach to the Paradigm Cell Filling Problem
Authors: Miikka Silfverberg, Mans Hulden
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: The Paradigm Cell Filling Problem in morphology asks to complete word inflection tables from partial ones. We implement novel neural models for this task, evaluating them on 18 data sets in 8 languages, showing performance that is comparable with previous work with far less training data. We also publish a new dataset for this task and code implementing the system described in this paper.
Generating Natural Language Adversarial Examples
Authors: Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, Kai-Wei Chang
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Deep neural networks (DNNs) are vulnerable to adversarial examples, perturbations to correctly classified examples which can cause the model to misclassify. In the image domain, these perturbations can often be made virtually indistinguishable to human perception, causing humans and state-of-the-art models to disagree. However, in the natural language domain, small perturbations are clearly perceptible, and the replacement of a single word can drastically alter the semantics of the document. Given these challenges, we use a black-box population-based optimization algorithm to generate semantically and syntactically similar adversarial examples that fool well-trained sentiment analysis and textual entailment models with success rates of 97% and 70%, respectively. We additionally demonstrate that 92.3% of the successful sentiment analysis adversarial examples are classified to their original label by 20 human annotators, and that the examples are perceptibly quite similar. Finally, we discuss an attempt to use adversarial training as a defense, but fail to yield improvement, demonstrating the strength and diversity of our adversarial examples. We hope our findings encourage researchers to pursue improving the robustness of DNNs in the natural language domain.
Multi-Head Attention with Disagreement Regularization
Authors: Jian Li, Zhaopeng Tu, Baosong Yang, Michael R. Lyu, Tong Zhang
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Multi-head attention is appealing for the ability to jointly attend to information from different representation subspaces at different positions. In this work, we introduce a disagreement regularization to explicitly encourage the diversity among multiple attention heads. Specifically, we propose three types of disagreement regularization, which respectively encourage the subspace, the attended positions, and the output representation associated with each attention head to be different from other heads. Experimental results on widely-used WMT14 English-German and WMT17 Chinese-English translation tasks demonstrate the effectiveness and universality of the proposed approach.
Deep Bayesian Active Learning for Natural Language Processing: Results of a Large-Scale Empirical Study
Authors: Aditya Siddhant, Zachary C. Lipton
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Several recent papers investigate Active Learning (AL) for mitigating the data dependence of deep learning for natural language processing. However, the applicability of AL to real-world problems remains an open question. While in supervised learning, practitioners can try many different methods, evaluating each against a validation set before selecting a model, AL affords no such luxury. Over the course of one AL run, an agent annotates its dataset exhausting its labeling budget. Thus, given a new task, we have no opportunity to compare models and acquisition functions. This paper provides a large-scale empirical study of deep active learning, addressing multiple tasks and, for each, multiple datasets, multiple models, and a full suite of acquisition functions. We find that across all settings, Bayesian active learning by disagreement, using uncertainty estimates provided either by Dropout or Bayes-by-Backprop significantly improves over i.i.d. baselines and usually outperforms classic uncertainty sampling.
Bayesian Compression for Natural Language Processing
Authors: Nadezhda Chirkova, Ekaterina Lobacheva, Dmitry Vetrov
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: In natural language processing, a lot of the tasks are successfully solved with recurrent neural networks, but such models have a huge number of parameters. The majority of these parameters are often concentrated in the embedding layer, which size grows proportionally to the vocabulary length. We propose a Bayesian sparsification technique for RNNs which allows compressing the RNN dozens or hundreds of times without time-consuming hyperparameters tuning. We also generalize the model for vocabulary sparsification to filter out unnecessary words and compress the RNN even further. We show that the choice of the kept words is interpretable.
Multimodal neural pronunciation modeling for spoken languages with logographic origin
Authors: Minh Nguyen, Gia H. Ngo, Nancy Chen
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Graphemes of most languages encode pronunciation, though some are more explicit than others. Languages like Spanish have a straightforward mapping between its graphemes and phonemes, while this mapping is more convoluted for languages like English. Spoken languages such as Cantonese present even more challenges in pronunciation modeling: (1) they do not have a standard written form, (2) the closest graphemic origins are logographic Han characters, of which only a subset of these logographic characters implicitly encodes pronunciation. In this work, we propose a multimodal approach to predict the pronunciation of Cantonese logographic characters, using neural networks with a geometric representation of logographs and pronunciation of cognates in historically related languages. The proposed framework improves performance by 18.1% and 25.0% respective to unimodal and multimodal baselines.
Chinese Pinyin Aided IME, Input What You Have Not Keystroked Yet
Authors: Yafang Huang, Hai Zhao
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Chinese pinyin input method engine (IME) converts pinyin into character so that Chinese characters can be conveniently inputted into computer through common keyboard. IMEs work relying on its core component, pinyin-to-character conversion (P2C). Usually Chinese IMEs simply predict a list of character sequences for user choice only according to user pinyin input at each turn. However, Chinese inputting is a multi-turn online procedure, which can be supposed to be exploited for further user experience promoting. This paper thus for the first time introduces a sequence-to-sequence model with gated-attention mechanism for the core task in IMEs. The proposed neural P2C model is learned by encoding previous input utterance as extra context to enable our IME capable of predicting character sequence with incomplete pinyin input. Our model is evaluated in different benchmark datasets showing great user experience improvement compared to traditional models, which demonstrates the first engineering practice of building Chinese aided IME.
Estimating Marginal Probabilities of n-grams for Recurrent Neural Language Models
Authors: Thanapon Noraset, Doug Downey, Lidong Bing
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Recurrent neural network language models (RNNLMs) are the current standard-bearer for statistical language modeling. However, RNNLMs only estimate probabilities for complete sequences of text, whereas some applications require context-independent phrase probabilities instead. In this paper, we study how to compute an RNNLM`s em marginal probability: the probability that the model assigns to a short sequence of text when the preceding context is not known. We introduce a simple method of altering the RNNLM training to make the model more accurate at marginal estimation. Our experiments demonstrate that the technique is effective compared to baselines including the traditional RNNLM probability and an importance sampling approach. Finally, we show how we can use the marginal estimation to improve an RNNLM by training the marginals to match n-gram probabilities from a larger corpus.
How to represent a word and predict it, too: Improving tied architectures for language modelling
Authors: Kristina Gulordava, Laura Aina, Gemma Boleda
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Recent state-of-the-art neural language models share the representations of words given by the input and output mappings. We propose a simple modification to these architectures that decouples the hidden state from the word embedding prediction. Our architecture leads to comparable or better results compared to previous tied models and models without tying, with a much smaller number of parameters. We also extend our proposal to word2vec models, showing that tying is appropriate for general word prediction tasks.
The Importance of Generation Order in Language Modeling
Authors: Nicolas Ford, Daniel Duckworth, Mohammad Norouzi, George Dahl
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Neural language models are a critical component of state-of-the-art systems for machine translation, summarization, audio transcription, and other tasks. These language models are almost universally autoregressive in nature, generating sentences one token at a time from left to right. This paper studies the influence of token generation order on model quality via a novel two-pass language model that produces partially-filled sentence “templates” and then fills in missing tokens. We compare various strategies for structuring these two passes and observe a surprisingly large variation in model quality. We find the most effective strategy generates function words in the first pass followed by content words in the second. We believe these experimental results justify a more extensive investigation of the generation order for neural language models.
Document-Level Neural Machine Translation with Hierarchical Attention Networks
Authors: Lesly Miculicich, Dhananjay Ram, Nikolaos Pappas, James Henderson
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Neural Machine Translation (NMT) can be improved by including document-level contextual information. For this purpose, we propose a hierarchical attention model to capture the context in a structured and dynamic manner. The model is integrated in the original NMT architecture as another level of abstraction, conditioning on the NMT model`s own previous hidden states. Experiments show that hierarchical attention significantly improves the BLEU score over a strong NMT baseline with the state-of-the-art in context-aware methods, and that both the encoder and decoder benefit from context in complementary ways.
Three Strategies to Improve One-to-Many Multilingual Translation
Authors: Yining Wang, Jiajun Zhang, Feifei Zhai, Jingfang Xu, Chengqing Zong
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Due to the benefits of model compactness, multilingual translation (including many-to-one, many-to-many and one-to-many) based on a universal encoder-decoder architecture attracts more and more attention. However, previous studies show that one-to-many translation based on this framework cannot perform on par with the individually trained models. In this work, we introduce three strategies to improve one-to-many multilingual translation by balancing the shared and unique features. Within the architecture of one decoder for all target languages, we first exploit the use of unique initial states for different target languages. Then, we employ language-dependent positional embeddings. Finally and especially, we propose to divide the hidden cells of the decoder into shared and language-dependent ones. The extensive experiments demonstrate that our proposed methods can obtain remarkable improvements over the strong baselines. Moreover, our strategies can achieve comparable or even better performance than the individually trained translation models.
Multi-Source Syntactic Neural Machine Translation
Authors: Anna Currey, Kenneth Heafield
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: We introduce a novel multi-source technique for incorporating source syntax into neural machine translation using linearized parses. This is achieved by employing separate encoders for the sequential and parsed versions of the same source sentence; the resulting representations are then combined using a hierarchical attention mechanism. The proposed model improves over both seq2seq and parsed baselines by over 1 BLEU on the WMT17 English-German task. Further analysis shows that our multi-source syntactic model is able to translate successfully without any parsed input, unlike standard parsed methods. In addition, performance does not deteriorate as much on long sentences as for the baselines.
Fixing Translation Divergences in Parallel Corpora for Neural MT
Authors: MinhQuang Pham, Josep Crego, Jean Senellart, François Yvon
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Corpus-based approaches to machine translation rely on the availability of clean parallel corpora. Such resources are scarce, and because of the automatic processes involved in their preparation, they are often noisy. This paper describes an unsupervised method for detecting translation divergences in parallel sentences. We rely on a neural network that computes cross-lingual sentence similarity scores, which are then used to effectively filter out divergent translations. Furthermore, similarity scores predicted by the network are used to identify and fix some partial divergences, yielding additional parallel segments. We evaluate these methods for English-French and English-German machine translation tasks, and show that using filtered/corrected corpora actually improves MT performance.
Loss in Translation: Learning Bilingual Word Mapping with a Retrieval Criterion
Authors: Armand Joulin, Piotr Bojanowski, Tomas Mikolov, Hervé Jégou, Edouard Grave
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Continuous word representations learned separately on distinct languages can be aligned so that their words become comparable in a common space. Existing works typically solve a quadratic problem to learn a orthogonal matrix aligning a bilingual lexicon, and use a retrieval criterion for inference. In this paper, we propose an unified formulation that directly optimizes a retrieval criterion in an end-to-end fashion. Our experiments on standard benchmarks show that our approach outperforms the state of the art on word translation, with the biggest improvements observed for distant language pairs such as English-Chinese.
Learning When to Concentrate or Divert Attention: Self-Adaptive Attention Temperature for Neural Machine Translation
Authors: Junyang Lin, Xu Sun, Xuancheng Ren, Muyu Li, Qi Su
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Most of the Neural Machine Translation (NMT) models are based on the sequence-to-sequence (Seq2Seq) model with an encoder-decoder framework equipped with the attention mechanism. However, the conventional attention mechanism treats the decoding at each time step equally with the same matrix, which is problematic since the softness of the attention for different types of words (e.g. content words and function words) should differ. Therefore, we propose a new model with a mechanism called Self-Adaptive Control of Temperature (SACT) to control the softness of attention by means of an attention temperature. Experimental results on the Chinese-English translation and English-Vietnamese translation demonstrate that our model outperforms the baseline models, and the analysis and the case study show that our model can attend to the most relevant elements in the source-side contexts and generate the translation of high quality.
Accelerating Asynchronous Stochastic Gradient Descent for Neural Machine Translation
Authors: Nikolay Bogoychev, Kenneth Heafield, Alham Fikri Aji, Marcin Junczys-Dowmunt
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: In order to extract the best possible performance from asynchronous stochastic gradient descent one must increase the mini-batch size and scale the learning rate accordingly. In order to achieve further speedup we introduce a technique that delays gradient updates effectively increasing the mini-batch size. Unfortunately with the increase of mini-batch size we worsen the stale gradient problem in asynchronous stochastic gradient descent (SGD) which makes the model convergence poor. We introduce local optimizers which mitigate the stale gradient problem and together with fine tuning our momentum we are able to train a shallow machine translation system 27% faster than an optimized baseline with negligible penalty in BLEU.
Learning to Jointly Translate and Predict Dropped Pronouns with a Shared Reconstruction Mechanism
Authors: Longyue Wang, Zhaopeng Tu, Andy Way, Qun Liu
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Pronouns are frequently omitted in pro-drop languages, such as Chinese, generally leading to significant challenges with respect to the production of complete translations. Recently, Wang et al. (2018) proposed a novel reconstruction-based approach to alleviating dropped pronoun (DP) translation problems for neural machine translation models. In this work, we improve the original model from two perspectives. First, we employ a shared reconstructor to better exploit encoder and decoder representations. Second, we jointly learn to translate and predict DPs in an end-to-end manner, to avoid the errors propagated from an external DP prediction model. Experimental results show that our approach significantly improves both translation performance and DP prediction accuracy.
Getting Gender Right in Neural Machine Translation
Authors: Eva Vanmassenhove, Christian Hardmeier, Andy Way
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Speakers of different languages must attend to and encode strikingly different aspects of the world in order to use their language correctly (Sapir, 1921; Slobin, 1996). One such difference is related to the way gender is expressed in a language. Saying “I am happy” in English, does not encode any additional knowledge of the speaker that uttered the sentence. However, many other languages do have grammatical gender systems and so such knowledge would be encoded. In order to correctly translate such a sentence into, say, French, the inherent gender information needs to be retained/recovered. The same sentence would become either “Je suis heureux”, for a male speaker or “Je suis heureuse” for a female one. Apart from morphological agreement, demographic factors (gender, age, etc.) also influence our use of language in terms of word choices or syntactic constructions (Tannen, 1991; Pennebaker et al., 2003). We integrate gender information into NMT systems. Our contribution is two-fold: (1) the compilation of large datasets with speaker information for 20 language pairs, and (2) a simple set of experiments that incorporate gender information into NMT for multiple language pairs. Our experiments show that adding a gender feature to an NMT system significantly improves the translation quality for some language pairs.