mtybilly/Instruct2DS huggingface dataset:
Multimodal neural pronunciation modeling for spoken languages with logographic origin
Authors: Gia H. Ngo, Nancy Chen Minh Nguyen
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Graphemes of most languages encode pronunciation, though some are more explicit than others. Languages like Spanish have a straightforward mapping between its graphemes and phonemes, while this mapping is more convoluted for languages like English. Spoken languages such as Cantonese present even more challenges in pronunciation modeling: (1) they do not have a standard written form, (2) the closest graphemic origins are logographic Han characters, of which only a subset of these logographic characters implicitly encodes pronunciation. In this work, we propose a multimodal approach to predict the pronunciation of Cantonese logographic characters, using neural networks with a geometric representation of logographs and pronunciation of cognates in historically related languages. The proposed framework improves performance by 18.1% and 25.0% respective to unimodal and multimodal baselines.
Chinese Pinyin Aided IME, Input What You Have Not Keystroked Yet
Authors: Hai Zhao Yafang Huang
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Chinese pinyin input method engine (IME) converts pinyin into character so that Chinese characters can be conveniently inputted into computer through common keyboard. IMEs work relying on its core component, pinyin-to-character conversion (P2C). Usually Chinese IMEs simply predict a list of character sequences for user choice only according to user pinyin input at each turn. However, Chinese inputting is a multi-turn online procedure, which can be supposed to be exploited for further user experience promoting. This paper thus for the first time introduces a sequence-to-sequence model with gated-attention mechanism for the core task in IMEs. The proposed neural P2C model is learned by encoding previous input utterance as extra context to enable our IME capable of predicting character sequence with incomplete pinyin input. Our model is evaluated in different benchmark datasets showing great user experience improvement compared to traditional models, which demonstrates the first engineering practice of building Chinese aided IME.
Estimating Marginal Probabilities of n-grams for Recurrent Neural Language Models
Authors: Doug Downey, Lidong Bing Thanapon Noraset
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Recurrent neural network language models (RNNLMs) are the current standard-bearer for statistical language modeling. However, RNNLMs only estimate probabilities for complete sequences of text, whereas some applications require context-independent phrase probabilities instead. In this paper, we study how to compute an RNNLM`s em marginal probability: the probability that the model assigns to a short sequence of text when the preceding context is not known. We introduce a simple method of altering the RNNLM training to make the model more accurate at marginal estimation. Our experiments demonstrate that the technique is effective compared to baselines including the traditional RNNLM probability and an importance sampling approach. Finally, we show how we can use the marginal estimation to improve an RNNLM by training the marginals to match n-gram probabilities from a larger corpus.
How to represent a word and predict it, too: Improving tied architectures for language modelling
Authors: Laura Aina, Gemma Boleda Kristina Gulordava
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Recent state-of-the-art neural language models share the representations of words given by the input and output mappings. We propose a simple modification to these architectures that decouples the hidden state from the word embedding prediction. Our architecture leads to comparable or better results compared to previous tied models and models without tying, with a much smaller number of parameters. We also extend our proposal to word2vec models, showing that tying is appropriate for general word prediction tasks.
The Importance of Generation Order in Language Modeling
Authors: Daniel Duckworth, Mohammad Norouzi, George Dahl Nicolas Ford
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Neural language models are a critical component of state-of-the-art systems for machine translation, summarization, audio transcription, and other tasks. These language models are almost universally autoregressive in nature, generating sentences one token at a time from left to right. This paper studies the influence of token generation order on model quality via a novel two-pass language model that produces partially-filled sentence “templates” and then fills in missing tokens. We compare various strategies for structuring these two passes and observe a surprisingly large variation in model quality. We find the most effective strategy generates function words in the first pass followed by content words in the second. We believe these experimental results justify a more extensive investigation of the generation order for neural language models.
Document-Level Neural Machine Translation with Hierarchical Attention Networks
Authors: Dhananjay Ram, Nikolaos Pappas, James Henderson Lesly Miculicich
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Neural Machine Translation (NMT) can be improved by including document-level contextual information. For this purpose, we propose a hierarchical attention model to capture the context in a structured and dynamic manner. The model is integrated in the original NMT architecture as another level of abstraction, conditioning on the NMT model`s own previous hidden states. Experiments show that hierarchical attention significantly improves the BLEU score over a strong NMT baseline with the state-of-the-art in context-aware methods, and that both the encoder and decoder benefit from context in complementary ways.
Three Strategies to Improve One-to-Many Multilingual Translation
Authors: Jiajun Zhang, Feifei Zhai, Jingfang Xu, Chengqing Zong Yining Wang
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Due to the benefits of model compactness, multilingual translation (including many-to-one, many-to-many and one-to-many) based on a universal encoder-decoder architecture attracts more and more attention. However, previous studies show that one-to-many translation based on this framework cannot perform on par with the individually trained models. In this work, we introduce three strategies to improve one-to-many multilingual translation by balancing the shared and unique features. Within the architecture of one decoder for all target languages, we first exploit the use of unique initial states for different target languages. Then, we employ language-dependent positional embeddings. Finally and especially, we propose to divide the hidden cells of the decoder into shared and language-dependent ones. The extensive experiments demonstrate that our proposed methods can obtain remarkable improvements over the strong baselines. Moreover, our strategies can achieve comparable or even better performance than the individually trained translation models.
Multi-Source Syntactic Neural Machine Translation
Authors: Kenneth Heafield Anna Currey
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: We introduce a novel multi-source technique for incorporating source syntax into neural machine translation using linearized parses. This is achieved by employing separate encoders for the sequential and parsed versions of the same source sentence; the resulting representations are then combined using a hierarchical attention mechanism. The proposed model improves over both seq2seq and parsed baselines by over 1 BLEU on the WMT17 English-German task. Further analysis shows that our multi-source syntactic model is able to translate successfully without any parsed input, unlike standard parsed methods. In addition, performance does not deteriorate as much on long sentences as for the baselines.
Fixing Translation Divergences in Parallel Corpora for Neural MT
Authors: Josep Crego, Jean Senellart, François Yvon MinhQuang Pham
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Corpus-based approaches to machine translation rely on the availability of clean parallel corpora. Such resources are scarce, and because of the automatic processes involved in their preparation, they are often noisy. This paper describes an unsupervised method for detecting translation divergences in parallel sentences. We rely on a neural network that computes cross-lingual sentence similarity scores, which are then used to effectively filter out divergent translations. Furthermore, similarity scores predicted by the network are used to identify and fix some partial divergences, yielding additional parallel segments. We evaluate these methods for English-French and English-German machine translation tasks, and show that using filtered/corrected corpora actually improves MT performance.
Adversarial Evaluation of Multimodal Machine Translation
Authors: Desmond Elliott
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: The promise of combining language and vision in multimodal machine translation is that systems will produce better translations by leveraging the image data. However, the evidence surrounding whether the images are useful is unconvincing due to inconsistencies between text-similarity metrics and human judgements. We present an adversarial evaluation to directly examine the utility of the image data in this task. Our evaluation tests whether systems perform better when paired with congruent images or incongruent images. This evaluation shows that only one out of three publicly available systems is sensitive to this perturbation of the data. We recommend that multimodal translation systems should be able to pass this sanity check in the future.
Loss in Translation: Learning Bilingual Word Mapping with a Retrieval Criterion
Authors: Piotr Bojanowski, Tomas Mikolov, Hervé Jégou, Edouard Grave Armand Joulin
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Continuous word representations learned separately on distinct languages can be aligned so that their words become comparable in a common space. Existing works typically solve a quadratic problem to learn a orthogonal matrix aligning a bilingual lexicon, and use a retrieval criterion for inference. In this paper, we propose an unified formulation that directly optimizes a retrieval criterion in an end-to-end fashion. Our experiments on standard benchmarks show that our approach outperforms the state of the art on word translation, with the biggest improvements observed for distant language pairs such as English-Chinese.
Learning When to Concentrate or Divert Attention: Self-Adaptive Attention Temperature for Neural Machine Translation
Authors: Xu Sun, Xuancheng Ren, Muyu Li, Qi Su Junyang Lin
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Most of the Neural Machine Translation (NMT) models are based on the sequence-to-sequence (Seq2Seq) model with an encoder-decoder framework equipped with the attention mechanism. However, the conventional attention mechanism treats the decoding at each time step equally with the same matrix, which is problematic since the softness of the attention for different types of words (e.g. content words and function words) should differ. Therefore, we propose a new model with a mechanism called Self-Adaptive Control of Temperature (SACT) to control the softness of attention by means of an attention temperature. Experimental results on the Chinese-English translation and English-Vietnamese translation demonstrate that our model outperforms the baseline models, and the analysis and the case study show that our model can attend to the most relevant elements in the source-side contexts and generate the translation of high quality.
Accelerating Asynchronous Stochastic Gradient Descent for Neural Machine Translation
Authors: Kenneth Heafield, Alham Fikri Aji, Marcin Junczys-Dowmunt Nikolay Bogoychev
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: In order to extract the best possible performance from asynchronous stochastic gradient descent one must increase the mini-batch size and scale the learning rate accordingly. In order to achieve further speedup we introduce a technique that delays gradient updates effectively increasing the mini-batch size. Unfortunately with the increase of mini-batch size we worsen the stale gradient problem in asynchronous stochastic gradient descent (SGD) which makes the model convergence poor. We introduce local optimizers which mitigate the stale gradient problem and together with fine tuning our momentum we are able to train a shallow machine translation system 27% faster than an optimized baseline with negligible penalty in BLEU.
Learning to Jointly Translate and Predict Dropped Pronouns with a Shared Reconstruction Mechanism
Authors: Zhaopeng Tu, Andy Way, Qun Liu Longyue Wang
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Pronouns are frequently omitted in pro-drop languages, such as Chinese, generally leading to significant challenges with respect to the production of complete translations. Recently, Wang et al. (2018) proposed a novel reconstruction-based approach to alleviating dropped pronoun (DP) translation problems for neural machine translation models. In this work, we improve the original model from two perspectives. First, we employ a shared reconstructor to better exploit encoder and decoder representations. Second, we jointly learn to translate and predict DPs in an end-to-end manner, to avoid the errors propagated from an external DP prediction model. Experimental results show that our approach significantly improves both translation performance and DP prediction accuracy.
Getting Gender Right in Neural Machine Translation
Authors: Christian Hardmeier, Andy Way Eva Vanmassenhove
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Speakers of different languages must attend to and encode strikingly different aspects of the world in order to use their language correctly (Sapir, 1921; Slobin, 1996). One such difference is related to the way gender is expressed in a language. Saying “I am happy” in English, does not encode any additional knowledge of the speaker that uttered the sentence. However, many other languages do have grammatical gender systems and so such knowledge would be encoded. In order to correctly translate such a sentence into, say, French, the inherent gender information needs to be retained/recovered. The same sentence would become either “Je suis heureux”, for a male speaker or “Je suis heureuse” for a female one. Apart from morphological agreement, demographic factors (gender, age, etc.) also influence our use of language in terms of word choices or syntactic constructions (Tannen, 1991; Pennebaker et al., 2003). We integrate gender information into NMT systems. Our contribution is two-fold: (1) the compilation of large datasets with speaker information for 20 language pairs, and (2) a simple set of experiments that incorporate gender information into NMT for multiple language pairs. Our experiments show that adding a gender feature to an NMT system significantly improves the translation quality for some language pairs.
Towards Two-Dimensional Sequence to Sequence Model in Neural Machine Translation
Authors: Christopher Brix, Hermann Ney Parnia Bahar
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: This work investigates an alternative model for neural machine translation (NMT) and proposes a novel architecture, where we employ a multi-dimensional long short-term memory (MDLSTM) for translation modelling. In the state-of-the-art methods, source and target sentences are treated as one-dimensional sequences over time, while we view translation as a two-dimensional (2D) mapping using an MDLSTM layer to define the correspondence between source and target words. We extend beyond the current sequence to sequence backbone NMT models to a 2D structure in which the source and target sentences are aligned with each other in a 2D grid. Our proposed topology shows consistent improvements over attention-based sequence to sequence model on two WMT 2017 tasks, German\ensuremath<-\ensuremath>English.
End-to-End Non-Autoregressive Neural Machine Translation with Connectionist Temporal Classification
Authors: Jindřich Helcl Jindřich Libovický
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Autoregressive decoding is the only part of sequence-to-sequence models that prevents them from massive parallelization at inference time. Non-autoregressive models enable the decoder to generate all output symbols independently in parallel. We present a novel non-autoregressive architecture based on connectionist temporal classification and evaluate it on the task of neural machine translation. Unlike other non-autoregressive methods which operate in several steps, our model can be trained end-to-end. We conduct experiments on the WMT English-Romanian and English-German datasets. Our models achieve a significant speedup over the autoregressive models, keeping the translation quality comparable to other non-autoregressive models.
Prediction Improves Simultaneous Neural Machine Translation
Authors: Maryam Siahbani, Anoop Sarkar Ashkan Alinejad
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Simultaneous speech translation aims to maintain translation quality while minimizing the delay between reading input and incrementally producing the output. We propose a new general-purpose prediction action which predicts future words in the input to improve quality and minimize delay in simultaneous translation. We train this agent using reinforcement learning with a novel reward function. Our agent with prediction has better translation quality and less delay compared to an agent-based simultaneous translation system without prediction.
Training Deeper Neural Machine Translation Models with Transparent Attention
Authors: Mia Chen, Orhan Firat, Yuan Cao, Yonghui Wu Ankur Bapna
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: While current state-of-the-art NMT models, such as RNN seq2seq and Transformers, possess a large number of parameters, they are still shallow in comparison to convolutional models used for both text and vision applications. In this work we attempt to train significantly (2-3x) deeper Transformer and Bi-RNN encoders for machine translation. We propose a simple modification to the attention mechanism that eases the optimization of deeper models, and results in consistent gains of 0.7-1.1 BLEU on the benchmark WMT`14 English-German and WMT`15 Czech-English tasks for both architectures.
Context and Copying in Neural Machine Translation
Authors: Philipp Koehn Rebecca Knowles
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Neural machine translation systems with subword vocabularies are capable of translating or copying unknown words. In this work, we show that they learn to copy words based on both the context in which the words appear as well as features of the words themselves. In contexts that are particularly copy-prone, they even copy words that they have already learned they should translate. We examine the influence of context and subword features on this and other types of copying behavior.
Encoding Gated Translation Memory into Neural Machine Translation
Authors: Deyi Xiong Qian Cao
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Translation memories (TM) facilitate human translators to reuse existing repetitive translation fragments. In this paper, we propose a novel method to combine the strengths of both TM and neural machine translation (NMT) for high-quality translation. We treat the target translation of a TM match as an additional reference input and encode it into NMT with an extra encoder. A gating mechanism is further used to balance the impact of the TM match on the NMT decoder. Experiment results on the UN corpus demonstrate that when fuzzy matches are higher than 50%, the quality of NMT translation can be significantly improved by over 10 BLEU points.
Automatic Post-Editing of Machine Translation: A Neural Programmer-Interpreter Approach
Authors: Gholamreza Haffari Thuy-Trang Vu
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Automated Post-Editing (PE) is the task of automatically correct common and repetitive errors found in machine translation (MT) output. In this paper, we present a neural programmer-interpreter approach to this task, resembling the way that human perform post-editing using discrete edit operations, wich we refer to as programs. Our model outperforms previous neural models for inducing PE programs on the WMT17 APE task for German-English up to +1 BLEU score and -0.7 TER scores.
Breaking the Beam Search Curse: A Study of (Re-)Scoring Methods and Stopping Criteria for Neural Machine Translation
Authors: Liang Huang, Mingbo Ma Yilin Yang
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Beam search is widely used in neural machine translation, and usually improves translation quality compared to greedy search. It has been widely observed that, however, beam sizes larger than 5 hurt translation quality. We explain why this happens, and propose several methods to address this problem. Furthermore, we discuss the optimal stopping criteria for these methods. Results show that our hyperparameter-free methods outperform the widely-used hyperparameter-free heuristic of length normalization by +2.0 BLEU, and achieve the best results among all methods on Chinese-to-English translation.
Multi-Multi-View Learning: Multilingual and Multi-Representation Entity Typing
Authors: Hinrich Schütze Yadollah Yaghoobzadeh
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Accurate and complete knowledge bases (KBs) are paramount in NLP. We employ mul-itiview learning for increasing the accuracy and coverage of entity type information in KBs. We rely on two metaviews: language and representation. For language, we consider high-resource and low-resource languages from Wikipedia. For representation, we consider representations based on the context distribution of the entity (i.e., on its embedding), on the entity`s name (i.e., on its surface form) and on its description in Wikipedia. The two metaviews language and representation can be freely combined: each pair of language and representation (e.g., German embedding, English description, Spanish name) is a distinct view. Our experiments on entity typing with fine-grained classes demonstrate the effectiveness of multiview learning. We release MVET, a large multiview --- and, in particular, multilingual --- entity typing dataset we created. Mono- and multilingual fine-grained entity typing systems can be evaluated on this dataset.
Word Embeddings for Code-Mixed Language Processing
Authors: Monojit Choudhury, Sunayana Sitaram Adithya Pratapa
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: We compare three existing bilingual word embedding approaches, and a novel approach of training skip-grams on synthetic code-mixed text generated through linguistic models of code-mixing, on two tasks - sentiment analysis and POS tagging for code-mixed text. Our results show that while CVM and CCA based embeddings perform as well as the proposed embedding technique on semantic and syntactic tasks respectively, the proposed approach provides the best performance for both tasks overall. Thus, this study demonstrates that existing bilingual embedding techniques are not ideal for code-mixed text processing and there is a need for learning multilingual word embedding from the code-mixed text.
On the Strength of Character Language Models for Multilingual Named Entity Recognition
Authors: Stephen Mayhew, Mark Sammons, Dan Roth Xiaodong Yu
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Character-level patterns have been widely used as features in English Named Entity Recognition (NER) systems. However, to date there has been no direct investigation of the inherent differences between name and nonname tokens in text, nor whether this property holds across multiple languages. This paper analyzes the capabilities of corpus-agnostic Character-level Language Models (CLMs) in the binary task of distinguishing name tokens from non-name tokens. We demonstrate that CLMs provide a simple and powerful model for capturing these differences, identifying named entity tokens in a diverse set of languages at close to the performance of full NER systems. Moreover, by adding very simple CLM-based features we can significantly improve the performance of an off-the-shelf NER system for multiple languages.
Code-switched Language Models Using Dual RNNs and Same-Source Pretraining
Authors: Tanmay Parekh, Preethi Jyothi Saurabh Garg
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: This work focuses on building language models (LMs) for code-switched text. We propose two techniques that significantly improve these LMs: 1) A novel recurrent neural network unit with dual components that focus on each language in the code-switched text separately 2) Pretraining the LM using synthetic text from a generative model estimated using the training data. We demonstrate the effectiveness of our proposed techniques by reporting perplexities on a Mandarin-English task and derive significant reductions in perplexity.
Part-of-Speech Tagging for Code-Switched, Transliterated Texts without Explicit Language Identification
Authors: Dan Garrette Kelsey Ball
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Code-switching, the use of more than one language within a single utterance, is ubiquitous in much of the world, but remains a challenge for NLP largely due to the lack of representative data for training models. In this paper, we present a novel model architecture that is trained exclusively on monolingual resources, but can be applied to unseen code-switched text at inference time. The model accomplishes this by jointly maintaining separate word representations for each of the possible languages, or scripts in the case of transliteration, allowing each to contribute to inferences without forcing the model to commit to a language. Experiments on Hindi-English part-of-speech tagging demonstrate that our approach outperforms standard models when training on monolingual text without transliteration, and testing on code-switched text with alternate scripts.
Zero-shot User Intent Detection via Capsule Neural Networks
Authors: Chenwei Zhang, Xiaohui Yan, Yi Chang, Philip Yu Congying Xia
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: User intent detection plays a critical role in question-answering and dialog systems. Most previous works treat intent detection as a classification problem where utterances are labeled with predefined intents. However, it is labor-intensive and time-consuming to label users' utterances as intents are diversely expressed and novel intents will continually be involved. Instead, we study the zero-shot intent detection problem, which aims to detect emerging user intents where no labeled utterances are currently available. We propose two capsule-based architectures: IntentCapsNet that extracts semantic features from utterances and aggregates them to discriminate existing intents, and IntentCapsNet-ZSL which gives IntentCapsNet the zero-shot learning ability to discriminate emerging intents via knowledge transfer from existing intents. Experiments on two real-world datasets show that our model not only can better discriminate diversely expressed existing intents, but is also able to discriminate emerging intents when no labeled utterances are available.
Hierarchical Neural Networks for Sequential Sentence Classification in Medical Scientific Abstracts
Authors: Peter Szolovits Di Jin
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Prevalent models based on artificial neural network (ANN) for sentence classification often classify sentences in isolation without considering the context in which sentences appear. This hampers the traditional sentence classification approaches to the problem of sequential sentence classification, where structured prediction is needed for better overall classification performance. In this work, we present a hierarchical sequential labeling network to make use of the contextual information within surrounding sentences to help classify the current sentence. Our model outperforms the state-of-the-art results by 2%-3% on two benchmarking datasets for sequential sentence classification in medical scientific abstracts.
Investigating Capsule Networks with Dynamic Routing for Text Classification
Authors: Jianbo Ye, Min Yang, Zeyang Lei, Suofei Zhang, Zhou Zhao Wei Zhao
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: In this study, we explore capsule networks with dynamic routing for text classification. We propose three strategies to stabilize the dynamic routing process to alleviate the disturbance of some noise capsules which may contain “background” information or have not been successfully trained. A series of experiments are conducted with capsule networks on six text classification benchmarks. Capsule networks achieve state of the art on 4 out of 6 datasets, which shows the effectiveness of capsule networks for text classification. We additionally show that capsule networks exhibit significant improvement when transfer single-label to multi-label text classification over strong baseline methods. To the best of our knowledge, this is the first work that capsule networks have been empirically investigated for text modeling.
Topic Memory Networks for Short Text Classification
Authors: Jing Li, Yan Song, Cuiyun Gao, Michael R. Lyu, Irwin King Jichuan Zeng
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Many classification models work poorly on short texts due to data sparsity. To address this issue, we propose topic memory networks for short text classification with a novel topic memory mechanism to encode latent topic representations indicative of class labels. Different from most prior work that focuses on extending features with external knowledge or pre-trained topics, our model jointly explores topic inference and text classification with memory networks in an end-to-end manner. Experimental results on four benchmark datasets show that our model outperforms state-of-the-art models on short text classification, meanwhile generates coherent topics.
Few-Shot and Zero-Shot Multi-Label Learning for Structured Label Spaces
Authors: Ramakanth Kavuluru Anthony Rios
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Large multi-label datasets contain labels that occur thousands of times (frequent group), those that occur only a few times (few-shot group), and labels that never appear in the training dataset (zero-shot group). Multi-label few- and zero-shot label prediction is mostly unexplored on datasets with large label spaces, especially for text classification. In this paper, we perform a fine-grained evaluation to understand how state-of-the-art methods perform on infrequent labels. Furthermore, we develop few- and zero-shot methods for multi-label text classification when there is a known structure over the label space, and evaluate them on two publicly available medical text datasets: MIMIC II and MIMIC III. For few-shot labels we achieve improvements of 6.2% and 4.8% in R@10 for MIMIC II and MIMIC III, respectively, over prior efforts; the corresponding R@10 improvements for zero-shot labels are 17.3% and 19%.
Automatic Poetry Generation with Mutual Reinforcement Learning
Authors: Maosong Sun, Ruoyu Li, Wenhao Li Xiaoyuan Yi
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Poetry is one of the most beautiful forms of human language art. As a crucial step towards computer creativity, automatic poetry generation has drawn researchers' attention for decades. In recent years, some neural models have made remarkable progress in this task. However, they are all based on maximum likelihood estimation, which only learns common patterns of the corpus and results in loss-evaluation mismatch. Human experts evaluate poetry in terms of some specific criteria, instead of word-level likelihood. To handle this problem, we directly model the criteria and use them as explicit rewards to guide gradient update by reinforcement learning, so as to motivate the model to pursue higher scores. Besides, inspired by writing theories, we propose a novel mutual reinforcement learning schema. We simultaneously train two learners (generators) which learn not only from the teacher (rewarder) but also from each other to further improve performance. We experiment on Chinese poetry. Based on a strong basic model, our method achieves better results and outperforms the current state-of-the-art method.
Variational Autoregressive Decoder for Neural Response Generation
Authors: Wenjie Li, Yulan He, Ruifeng Xu, Lidong Bing, Xuan Wang Jiachen Du
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Combining the virtues of probability graphic models and neural networks, Conditional Variational Auto-encoder (CVAE) has shown promising performance in applications such as response generation. However, existing CVAE-based models often generate responses from a single latent variable which may not be sufficient to model high variability in responses. To solve this problem, we propose a novel model that sequentially introduces a series of latent variables to condition the generation of each word in the response sequence. In addition, the approximate posteriors of these latent variables are augmented with a backward Recurrent Neural Network (RNN), which allows the latent variables to capture long-term dependencies of future tokens in generation. To facilitate training, we supplement our model with an auxiliary objective that predicts the subsequent bag of words. Empirical experiments conducted on Opensubtitle and Reddit datasets show that the proposed model leads to significant improvement on both relevance and diversity over state-of-the-art baselines.
Integrating Transformer and Paraphrase Rules for Sentence Simplification
Authors: Rui Meng, Daqing He, Andi Saptono, Bambang Parmanto Sanqiang Zhao
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Sentence simplification aims to reduce the complexity of a sentence while retaining its original meaning. Current models for sentence simplification adopted ideas from machine translation studies and implicitly learned simplification mapping rules from normal-simple sentence pairs. In this paper, we explore a novel model based on a multi-layer and multi-head attention architecture and we propose two innovative approaches to integrate the Simple PPDB (A Paraphrase Database for Simplification), an external paraphrase knowledge base for simplification that covers a wide range of real-world simplification rules. The experiments show that the integration provides two major benefits: (1) the integrated model outperforms multiple state-of-the-art baseline models for sentence simplification in the literature (2) through analysis of the rule utilization, the model seeks to select more accurate simplification rules. The code and models used in the paper are available at r̆lhttps://github.com/Sanqiang/text_simplification.
Learning Neural Templates for Text Generation
Authors: Stuart Shieber, Alexander Rush Sam Wiseman
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: While neural, encoder-decoder models have had significant empirical success in text generation, there remain several unaddressed problems with this style of generation. Encoder-decoder models are largely (a) uninterpretable, and (b) difficult to control in terms of their phrasing or content. This work proposes a neural generation system using a hidden semi-markov model (HSMM) decoder, which learns latent, discrete templates jointly with learning to generate. We show that this model learns useful templates, and that these templates make generation both more interpretable and controllable. Furthermore, we show that this approach scales to real data sets and achieves strong performance nearing that of encoder-decoder text generation models.
Multi-Reference Training with Pseudo-References for Neural Translation and Text Generation
Authors: Mingbo Ma, Liang Huang Renjie Zheng
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Neural text generation, including neural machine translation, image captioning, and summarization, has been quite successful recently. However, during training time, typically only one reference is considered for each example, even though there are often multiple references available, e.g., 4 references in NIST MT evaluations, and 5 references in image captioning data. We first investigate several different ways of utilizing multiple human references during training. But more importantly, we then propose an algorithm to generate exponentially many pseudo-references by first compressing existing human references into lattices and then traversing them to generate new pseudo-references. These approaches lead to substantial improvements over strong baselines in both machine translation (+1.5 BLEU) and image captioning (+3.1 BLEU / +11.7 CIDEr).
Knowledge Graph Embedding with Hierarchical Relation Structure
Authors: Fuzhen Zhuang, Meng Qu, Fen Lin, Qing He Zhao Zhang
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: The rapid development of knowledge graphs (KGs), such as Freebase and WordNet, has changed the paradigm for AI-related applications. However, even though these KGs are impressively large, most of them are suffering from incompleteness, which leads to performance degradation of AI applications. Most existing researches are focusing on knowledge graph embedding (KGE) models. Nevertheless, those models simply embed entities and relations into latent vectors without leveraging the rich information from the relation structure. Indeed, relations in KGs conform to a three-layer hierarchical relation structure (HRS), i.e., semantically similar relations can make up relation clusters and some relations can be further split into several fine-grained sub-relations. Relation clusters, relations and sub-relations can fit in the top, the middle and the bottom layer of three-layer HRS respectively. To this end, in this paper, we extend existing KGE models TransE, TransH and DistMult, to learn knowledge representations by leveraging the information from the HRS. Particularly, our approach is capable to extend other KGE models. Finally, the experiment results clearly validate the effectiveness of the proposed approach against baselines.
Embedding Multimodal Relational Data for Knowledge Base Completion
Authors: Liyan Chen, Sameer Singh Pouya Pezeshkpour
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Representing entities and relations in an embedding space is a well-studied approach for machine learning on relational data. Existing approaches, however, primarily focus on simple link structure between a finite set of entities, ignoring the variety of data types that are often used in knowledge bases, such as text, images, and numerical values. In this paper, we propose multimodal knowledge base embeddings (MKBE) that use different neural encoders for this variety of observed data, and combine them with existing relational models to learn embeddings of the entities and multimodal data. Further, using these learned embedings and different neural decoders, we introduce a novel multimodal imputation model to generate missing multimodal values, like text and images, from information in the knowledge base. We enrich existing relational datasets to create two novel benchmarks that contain additional information such as textual descriptions and images of the original entities. We demonstrate that our models utilize this additional information effectively to provide more accurate link prediction, achieving state-of-the-art results with a considerable gap of 5-7% over existing methods. Further, we evaluate the quality of our generated multimodal values via a user study.
Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction
Authors: Luheng He, Mari Ostendorf, Hannaneh Hajishirzi Yi Luan
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: We introduce a multi-task setup of identifying entities, relations, and coreference clusters in scientific articles. We create SciERC, a dataset that includes annotations for all three tasks and develop a unified framework called SciIE with shared span representations. The multi-task setup reduces cascading errors between tasks and leverages cross-sentence relations through coreference links. Experiments show that our multi-task model outperforms previous models in scientific information extraction without using any domain-specific features. We further show that the framework supports construction of a scientific knowledge graph, which we use to analyze information in scientific literature.
Playing 20 Question Game with Policy-Based Reinforcement Learning
Authors: Xianchao Wu, Bingfeng Luo, Chongyang Tao, Can Xu, Wei Wu, Zhan Chen Huang Hu
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: The 20 Questions (Q20) game is a well known game which encourages deductive reasoning and creativity. In the game, the answerer first thinks of an object such as a famous person or a kind of animal. Then the questioner tries to guess the object by asking 20 questions. In a Q20 game system, the user is considered as the answerer while the system itself acts as the questioner which requires a good strategy of question selection to figure out the correct object and win the game. However, the optimal policy of question selection is hard to be derived due to the complexity and volatility of the game environment. In this paper, we propose a novel policy-based Reinforcement Learning (RL) method, which enables the questioner agent to learn the optimal policy of question selection through continuous interactions with users. To facilitate training, we also propose to use a reward network to estimate the more informative reward. Compared to previous methods, our RL method is robust to noisy answers and does not rely on the Knowledge Base of objects. Experimental results show that our RL method clearly outperforms an entropy-based engineering system and has competitive performance in a noisy-free simulation environment.
Multi-Hop Knowledge Graph Reasoning with Reward Shaping
Authors: Richard Socher, Caiming Xiong Xi Victoria Lin
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Multi-hop reasoning is an effective approach for query answering (QA) over incomplete knowledge graphs (KGs). The problem can be formulated in a reinforcement learning (RL) setup, where a policy-based agent sequentially extends its inference path until it reaches a target. However, in an incomplete KG environment, the agent receives low-quality rewards corrupted by false negatives in the training data, which harms generalization at test time. Furthermore, since no golden action sequence is used for training, the agent can be misled by spurious search trajectories that incidentally lead to the correct answer. We propose two modeling advances to address both issues: (1) we reduce the impact of false negative supervision by adopting a pretrained one-hop embedding model to estimate the reward of unobserved facts; (2) we counter the sensitivity to spurious paths of on-policy RL by forcing the agent to explore a diverse set of paths using randomly generated edge masks. Our approach significantly improves over existing path-based KGQA models on several benchmark datasets and is comparable or better than embedding-based models.
Neural Transductive Learning and Beyond: Morphological Generation in the Minimal-Resource Setting
Authors: Hinrich Schütze Katharina Kann
Published in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2018

Abstract: Neural state-of-the-art sequence-to-sequence (seq2seq) models often do not perform well for small training sets. We address paradigm completion, the morphological task of, given a partial paradigm, generating all missing forms. We propose two new methods for the minimal-resource setting: (i) Paradigm transduction: Since we assume only few paradigms available for training, neural seq2seq models are able to capture relationships between paradigm cells, but are tied to the idiosyncracies of the training set. Paradigm transduction mitigates this problem by exploiting the input subset of inflected forms at test time. (ii) Source selection with high precision (SHIP): Multi-source models which learn to automatically select one or multiple sources to predict a target inflection do not perform well in the minimal-resource setting. SHIP is an alternative to identify a reliable source if training data is limited. On a 52-language benchmark dataset, we outperform the previous state of the art by up to 9.71% absolute accuracy.