mtybilly/Instruct2DS huggingface dataset:
A Hybrid Convolutional Variational Autoencoder for Text Generation
Authors: Stanislau Semeniuta, Aliaksei Severyn, Erhardt Barth
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: In this paper we explore the effect of architectural choices on learning a variational autoencoder (VAE) for text generation. In contrast to the previously introduced VAE model for text where both the encoder and decoder are RNNs, we propose a novel hybrid architecture that blends fully feed-forward convolutional and deconvolutional components with a recurrent language model. Our architecture exhibits several attractive properties such as faster run time and convergence, ability to better handle long sequences and, more importantly, it helps to avoid the issue of the VAE collapsing to a deterministic model.
Filling the Blanks (hint: plural noun) for Mad Libs Humor
Authors: Nabil Hossain, John Krumm, Lucy Vanderwende, Eric Horvitz, Henry Kautz
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Computerized generation of humor is a notoriously difficult AI problem. We develop an algorithm called Libitum that helps humans generate humor in a Mad Lib, which is a popular fill-in-the-blank game. The algorithm is based on a machine learned classifier that determines whether a potential fill-in word is funny in the context of the Mad Lib story. We use Amazon Mechanical Turk to create ground truth data and to judge humor for our classifier to mimic, and we make this data freely available. Our testing shows that Libitum successfully aids humans in filling in Mad Libs that are usually judged funnier than those filled in by humans with no computerized help. We go on to analyze why some words are better than others at making a Mad Lib funny.
Measuring Thematic Fit with Distributional Feature Overlap
Authors: Enrico Santus, Emmanuele Chersoni, Alessandro Lenci, Philippe Blache
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: In this paper, we introduce a new distributional method for modeling predicate-argument thematic fit judgments. We use a syntax-based DSM to build a prototypical representation of verb-specific roles: for every verb, we extract the most salient second order contexts for each of its roles (i.e. the most salient dimensions of typical role fillers), and then we compute thematic fit as a weighted overlap between the top features of candidate fillers and role prototypes. Our experiments show that our method consistently outperforms a baseline re-implementing a state-of-the-art system, and achieves better or comparable results to those reported in the literature for the other unsupervised systems. Moreover, it provides an explicit representation of the features characterizing verb-specific semantic roles.
SCDV : Sparse Composite Document Vectors using soft clustering over distributional representations
Authors: Dheeraj Mekala, Vivek Gupta, Bhargavi Paranjape, Harish Karnick
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: We present a feature vector formation technique for documents - Sparse Composite Document Vector (SCDV) - which overcomes several shortcomings of the current distributional paragraph vector representations that are widely used for text representation. In SCDV, word embeddings are clustered to capture multiple semantic contexts in which words occur. They are then chained together to form document topic-vectors that can express complex, multi-topic documents. Through extensive experiments on multi-class and multi-label classification tasks, we outperform the previous state-of-the-art method, NTSG. We also show that SCDV embeddings perform well on heterogeneous tasks like Topic Coherence, context-sensitive Learning and Information Retrieval. Moreover, we achieve a significant reduction in training and prediction times compared to other representation methods. SCDV achieves best of both worlds - better performance with lower time and space complexity.
Supervised Learning of Universal Sentence Representations from Natural Language Inference Data
Authors: Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, Antoine Bordes
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Many modern NLP systems rely on word embeddings, previously trained in an unsupervised manner on large corpora, as base features. Efforts to obtain embeddings for larger chunks of text, such as sentences, have however not been so successful. Several attempts at learning unsupervised representations of sentences have not reached satisfactory enough performance to be widely adopted. In this paper, we show how universal sentence representations trained using the supervised data of the Stanford Natural Language Inference datasets can consistently outperform unsupervised methods like SkipThought vectors on a wide range of transfer tasks. Much like how computer vision uses ImageNet to obtain features, which can then be transferred to other tasks, our work tends to indicate the suitability of natural language inference for transfer learning to other NLP tasks. Our encoder is publicly available.
Determining Semantic Textual Similarity using Natural Deduction Proofs
Authors: Hitomi Yanaka, Koji Mineshima, Pascual Martínez-Gómez, Daisuke Bekki
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Determining semantic textual similarity is a core research subject in natural language processing. Since vector-based models for sentence representation often use shallow information, capturing accurate semantics is difficult. By contrast, logical semantic representations capture deeper levels of sentence semantics, but their symbolic nature does not offer graded notions of textual similarity. We propose a method for determining semantic textual similarity by combining shallow features with features extracted from natural deduction proofs of bidirectional entailment relations between sentence pairs. For the natural deduction proofs, we use ccg2lambda, a higher-order automatic inference system, which converts Combinatory Categorial Grammar (CCG) derivation trees into semantic representations and conducts natural deduction proofs. Experiments show that our system was able to outperform other logic-based systems and that features derived from the proofs are effective for learning textual similarity.
Multi-Grained Chinese Word Segmentation
Authors: Chen Gong, Zhenghua Li, Min Zhang, Xinzhou Jiang
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Traditionally, word segmentation (WS) adopts the single-grained formalism, where a sentence corresponds to a single word sequence. However, Sproat et al. (1997) show that the inter-native-speaker consistency ratio over Chinese word boundaries is only 76%, indicating single-grained WS (SWS) imposes unnecessary challenges on both manual annotation and statistical modeling. Moreover, WS results of different granularities can be complementary and beneficial for high-level applications. This work proposes and addresses multi-grained WS (MWS). We build a large-scale pseudo MWS dataset for model training and tuning by leveraging the annotation heterogeneity of three SWS datasets. Then we manually annotate 1,500 test sentences with true MWS annotations. Finally, we propose three benchmark approaches by casting MWS as constituent parsing and sequence labeling. Experiments and analysis lead to many interesting findings.
Don`t Throw Those Morphological Analyzers Away Just Yet: Neural Morphological Disambiguation for Arabic
Authors: Nasser Zalmout, Nizar Habash
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: This paper presents a model for Arabic morphological disambiguation based on Recurrent Neural Networks (RNN). We train Long Short-Term Memory (LSTM) cells in several configurations and embedding levels to model the various morphological features. Our experiments show that these models outperform state-of-the-art systems without explicit use of feature engineering. However, adding learning features from a morphological analyzer to model the space of possible analyses provides additional improvement. We make use of the resulting morphological models for scoring and ranking the analyses of the morphological analyzer for morphological disambiguation. The results show significant gains in accuracy across several evaluation metrics. Our system results in 4.4% absolute increase over the state-of-the-art in full morphological analysis accuracy (30.6% relative error reduction), and 10.6% (31.5% relative error reduction) for out-of-vocabulary words.
Paradigm Completion for Derivational Morphology
Authors: Ryan Cotterell, Ekaterina Vylomova, Huda Khayrallah, Christo Kirov, David Yarowsky
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: The generation of complex derived word forms has been an overlooked problem in NLP; we fill this gap by applying neural sequence-to-sequence models to the task. We overview the theoretical motivation for a paradigmatic treatment of derivational morphology, and introduce the task of derivational paradigm completion as a parallel to inflectional paradigm completion. State-of-the-art neural models adapted from the inflection task are able to learn the range of derivation patterns, and outperform a non-neural baseline by 16.4%. However, due to semantic, historical, and lexical considerations involved in derivational morphology, future work will be needed to achieve performance parity with inflection-generating systems.
Do LSTMs really work so well for PoS tagging? -- A replication study
Authors: Tobias Horsmann, Torsten Zesch
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: A recent study by Plank et al. (2016) found that LSTM-based PoS taggers considerably improve over the current state-of-the-art when evaluated on the corpora of the Universal Dependencies project that use a coarse-grained tagset. We replicate this study using a fresh collection of 27 corpora of 21 languages that are annotated with fine-grained tagsets of varying size. Our replication confirms the result in general, and we additionally find that the advantage of LSTMs is even bigger for larger tagsets. However, we also find that for the very large tagsets of morphologically rich languages, hand-crafted morphological lexicons are still necessary to reach state-of-the-art performance.
The Labeled Segmentation of Printed Books
Authors: Lara McConnaughey, Jennifer Dai, David Bamman
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: We introduce the task of book structure labeling: segmenting and assigning a fixed category (such as Table of Contents, Preface, Index) to the document structure of printed books. We manually annotate the page-level structural categories for a large dataset totaling 294,816 pages in 1,055 books evenly sampled from 1750-1922, and present empirical results comparing the performance of several classes of models. The best-performing model, a bidirectional LSTM with rich features, achieves an overall accuracy of 95.8 and a class-balanced macro F-score of 71.4.
Cross-lingual Character-Level Neural Morphological Tagging
Authors: Ryan Cotterell, Georg Heigold
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Even for common NLP tasks, sufficient supervision is not available in many languages -- morphological tagging is no exception. In the work presented here, we explore a transfer learning scheme, whereby we train character-level recurrent neural taggers to predict morphological taggings for high-resource languages and low-resource languages together. Learning joint character representations among multiple related languages successfully enables knowledge transfer from the high-resource languages to the low-resource ones.
Word-Context Character Embeddings for Chinese Word Segmentation
Authors: Hao Zhou, Zhenting Yu, Yue Zhang, Shujian Huang, Xinyu Dai, Jiajun Chen
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Neural parsers have benefited from automatically labeled data via dependency-context word embeddings. We investigate training character embeddings on a word-based context in a similar way, showing that the simple method improves state-of-the-art neural word segmentation models significantly, beating tri-training baselines for leveraging auto-segmented data.
From Textbooks to Knowledge: A Case Study in Harvesting Axiomatic Knowledge from Textbooks to Solve Geometry Problems
Authors: Mrinmaya Sachan, Kumar Dubey, Eric Xing
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Textbooks are rich sources of information. Harvesting structured knowledge from textbooks is a key challenge in many educational applications. As a case study, we present an approach for harvesting structured axiomatic knowledge from math textbooks. Our approach uses rich contextual and typographical features extracted from raw textbooks. It leverages the redundancy and shared ordering across multiple textbooks to further refine the harvested axioms. These axioms are then parsed into rules that are used to improve the state-of-the-art in solving geometry problems.
RACE: Large-scale ReAding Comprehension Dataset From Examinations
Authors: Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, Eduard Hovy
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: We present RACE, a new dataset for benchmark evaluation of methods in the reading comprehension task. Collected from the English exams for middle and high school Chinese students in the age range between 12 to 18, RACE consists of near 28,000 passages and near 100,000 questions generated by human experts (English instructors), and covers a variety of topics which are carefully designed for evaluating the students' ability in understanding and reasoning. In particular, the proportion of questions that requires reasoning is much larger in RACE than that in other benchmark datasets for reading comprehension, and there is a significant gap between the performance of the state-of-the-art models (43%) and the ceiling human performance (95%). We hope this new dataset can serve as a valuable resource for research and evaluation in machine comprehension. The dataset is freely available at r̆lhttp://www.cs.cmu.edu/~glai1/data/race/and the code is available at \l̆https://github.com/qizhex/RACE_AR_baselines.
Beyond Sentential Semantic Parsing: Tackling the Math SAT with a Cascade of Tree Transducers
Authors: Mark Hopkins, Cristian Petrescu-Prahova, Roie Levin, Ronan Le Bras, Alvaro Herrasti, Vidur Joshi
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: We present an approach for answering questions that span multiple sentences and exhibit sophisticated cross-sentence anaphoric phenomena, evaluating on a rich source of such questions -- the math portion of the Scholastic Aptitude Test (SAT). By using a tree transducer cascade as its basic architecture, our system propagates uncertainty from multiple sources (e.g. coreference resolution or verb interpretation) until it can be confidently resolved. Experiments show the first-ever results 43% recall and 91% precision) on SAT algebra word problems. We also apply our system to the public Dolphin algebra question set, and improve the state-of-the-art F1-score from 73.9% to 77.0%.
Learning Fine-Grained Expressions to Solve Math Word Problems
Authors: Danqing Huang, Shuming Shi, Chin-Yew Lin, Jian Yin
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: This paper presents a novel template-based method to solve math word problems. This method learns the mappings between math concept phrases in math word problems and their math expressions from training data. For each equation template, we automatically construct a rich template sketch by aggregating information from various problems with the same template. Our approach is implemented in a two-stage system. It first retrieves a few relevant equation system templates and aligns numbers in math word problems to those templates for candidate equation generation. It then does a fine-grained inference to obtain the final answer. Experiment results show that our method achieves an accuracy of 28.4% on the linear Dolphin18K benchmark, which is 10% (54% relative) higher than previous state-of-the-art systems while achieving an accuracy increase of 12% (59% relative) on the TS6 benchmark subset.
Structural Embedding of Syntactic Trees for Machine Comprehension
Authors: Rui Liu, Junjie Hu, Wei Wei, Zi Yang, Eric Nyberg
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Deep neural networks for machine comprehension typically utilizes only word or character embeddings without explicitly taking advantage of structured linguistic information such as constituency trees and dependency trees. In this paper, we propose structural embedding of syntactic trees (SEST), an algorithm framework to utilize structured information and encode them into vector representations that can boost the performance of algorithms for the machine comprehension. We evaluate our approach using a state-of-the-art neural attention model on the SQuAD dataset. Experimental results demonstrate that our model can accurately identify the syntactic boundaries of the sentences and extract answers that are syntactically coherent over the baseline methods.
World Knowledge for Reading Comprehension: Rare Entity Prediction with Hierarchical LSTMs Using External Descriptions
Authors: Teng Long, Emmanuel Bengio, Ryan Lowe, Jackie Chi Kit Cheung, Doina Precup
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Humans interpret texts with respect to some background information, or world knowledge, and we would like to develop automatic reading comprehension systems that can do the same. In this paper, we introduce a task and several models to drive progress towards this goal. In particular, we propose the task of rare entity prediction: given a web document with several entities removed, models are tasked with predicting the correct missing entities conditioned on the document context and the lexical resources. This task is challenging due to the diversity of language styles and the extremely large number of rare entities. We propose two recurrent neural network architectures which make use of external knowledge in the form of entity descriptions. Our experiments show that our hierarchical LSTM model performs significantly better at the rare entity prediction task than those that do not make use of external resources.
Two-Stage Synthesis Networks for Transfer Learning in Machine Comprehension
Authors: David Golub, Po-Sen Huang, Xiaodong He, Li Deng
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: We develop a technique for transfer learning in machine comprehension (MC) using a novel two-stage synthesis network. Given a high performing MC model in one domain, our technique aims to answer questions about documents in another domain, where we use no labeled data of question-answer pairs. Using the proposed synthesis network with a pretrained model on the SQuAD dataset, we achieve an F1 measure of 46.6% on the challenging NewsQA dataset, approaching performance of in-domain models (F1 measure of 50.0%) and outperforming the out-of-domain baseline by 7.6%, without use of provided annotations.
Deep Neural Solver for Math Word Problems
Authors: Yan Wang, Xiaojiang Liu, Shuming Shi
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: This paper presents a deep neural solver to automatically solve math word problems. In contrast to previous statistical learning approaches, we directly translate math word problems to equation templates using a recurrent neural network (RNN) model, without sophisticated feature engineering. We further design a hybrid model that combines the RNN model and a similarity-based retrieval model to achieve additional performance improvement. Experiments conducted on a large dataset show that the RNN model and the hybrid model significantly outperform state-of-the-art statistical learning methods for math word problem solving.
Latent Space Embedding for Retrieval in Question-Answer Archives
Authors: Deepak P, Dinesh Garg, Shirish Shevade
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Community-driven Question Answering (CQA) systems such as Yahoo! Answers have become valuable sources of reusable information. CQA retrieval enables usage of historical CQA archives to solve new questions posed by users. This task has received much recent attention, with methods building upon literature from translation models, topic models, and deep learning. In this paper, we devise a CQA retrieval technique, LASER-QA, that embeds question-answer pairs within a unified latent space preserving the local neighborhood structure of question and answer spaces. The idea is that such a space mirrors semantic similarity among questions as well as answers, thereby enabling high quality retrieval. Through an empirical analysis on various real-world QA datasets, we illustrate the improved effectiveness of LASER-QA over state-of-the-art methods.
Question Generation for Question Answering
Authors: Nan Duan, Duyu Tang, Peng Chen, Ming Zhou
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: This paper presents how to generate questions from given passages using neural networks, where large scale QA pairs are automatically crawled and processed from Community-QA website, and used as training data. The contribution of the paper is 2-fold: First, two types of question generation approaches are proposed, one is a retrieval-based method using convolution neural network (CNN), the other is a generation-based method using recurrent neural network (RNN); Second, we show how to leverage the generated questions to improve existing question answering systems. We evaluate our question generation method for the answer sentence selection task on three benchmark datasets, including SQuAD, MS MARCO, and WikiQA. Experimental results show that, by using generated questions as an extra signal, significant QA improvement can be achieved.
Learning to Paraphrase for Question Answering
Authors: Li Dong, Jonathan Mallinson, Siva Reddy, Mirella Lapata
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Question answering (QA) systems are sensitive to the many different ways natural language expresses the same information need. In this paper we turn to paraphrases as a means of capturing this knowledge and present a general framework which learns felicitous paraphrases for various QA tasks. Our method is trained end-to-end using question-answer pairs as a supervision signal. A question and its paraphrases serve as input to a neural scoring model which assigns higher weights to linguistic expressions most likely to yield correct answers. We evaluate our approach on QA over Freebase and answer sentence selection. Experimental results on three datasets show that our framework consistently improves performance, achieving competitive results despite the use of simple QA models.
Temporal Information Extraction for Question Answering Using Syntactic Dependencies in an LSTM-based Architecture
Authors: Yuanliang Meng, Anna Rumshisky, Alexey Romanov
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: In this paper, we propose to use a set of simple, uniform in architecture LSTM-based models to recover different kinds of temporal relations from text. Using the shortest dependency path between entities as input, the same architecture is used to extract intra-sentence, cross-sentence, and document creation time relations. A “double-checking” technique reverses entity pairs in classification, boosting the recall of positive cases and reducing misclassifications between opposite classes. An efficient pruning algorithm resolves conflicts globally. Evaluated on QA-TempEval (SemEval2015 Task 5), our proposed technique outperforms state-of-the-art methods by a large margin. We also conduct intrinsic evaluation and post state-of-the-art results on Timebank-Dense.
Ranking Kernels for Structures and Embeddings: A Hybrid Preference and Classification Model
Authors: Kateryna Tymoshenko, Daniele Bonadiman, Alessandro Moschitti
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Recent work has shown that Tree Kernels (TKs) and Convolutional Neural Networks (CNNs) obtain the state of the art in answer sentence reranking. Additionally, their combination used in Support Vector Machines (SVMs) is promising as it can exploit both the syntactic patterns captured by TKs and the embeddings learned by CNNs. However, the embeddings are constructed according to a classification function, which is not directly exploitable in the preference ranking algorithm of SVMs. In this work, we propose a new hybrid approach combining preference ranking applied to TKs and pointwise ranking applied to CNNs. We show that our approach produces better results on two well-known and rather different datasets: WikiQA for answer sentence selection and SemEval cQA for comment selection in Community Question Answering.
Recovering Question Answering Errors via Query Revision
Authors: Semih Yavuz, Izzeddin Gur, Yu Su, Xifeng Yan
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: The existing factoid QA systems often lack a post-inspection component that can help models recover from their own mistakes. In this work, we propose to crosscheck the corresponding KB relations behind the predicted answers and identify potential inconsistencies. Instead of developing a new model that accepts evidences collected from these relations, we choose to plug them back to the original questions directly and check if the revised question makes sense or not. A bidirectional LSTM is applied to encode revised questions. We develop a scoring mechanism over the revised question encodings to refine the predictions of a base QA system. This approach can improve the F1 score of STAGG (Yih et al., 2015), one of the leading QA systems, from 52.5% to 53.9% on WEBQUESTIONS data.
An empirical study on the effectiveness of images in Multimodal Neural Machine Translation
Authors: Jean-Benoit Delbrouck, Stéphane Dupont
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: In state-of-the-art Neural Machine Translation (NMT), an attention mechanism is used during decoding to enhance the translation. At every step, the decoder uses this mechanism to focus on different parts of the source sentence to gather the most useful information before outputting its target word. Recently, the effectiveness of the attention mechanism has also been explored for multi-modal tasks, where it becomes possible to focus both on sentence parts and image regions that they describe. In this paper, we compare several attention mechanism on the multi-modal translation task (English, image \textrightarrow German) and evaluate the ability of the model to make use of images to improve translation. We surpass state-of-the-art scores on the Multi30k data set, we nevertheless identify and report different misbehavior of the machine while translating.
Sound-Word2Vec: Learning Word Representations Grounded in Sounds
Authors: Ashwin Vijayakumar, Ramakrishna Vedantam, Devi Parikh
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: To be able to interact better with humans, it is crucial for machines to understand sound -- a primary modality of human perception. Previous works have used sound to learn embeddings for improved generic semantic similarity assessment. In this work, we treat sound as a first-class citizen, studying downstream 6textual tasks which require aural grounding. To this end, we propose sound-word2vec -- a new embedding scheme that learns specialized word embeddings grounded in sounds. For example, we learn that two seemingly (semantically) unrelated concepts, like leaves and paper are similar due to the similar rustling sounds they make. Our embeddings prove useful in textual tasks requiring aural reasoning like text-based sound retrieval and discovering Foley sound effects (used in movies). Moreover, our embedding space captures interesting dependencies between words and onomatopoeia and outperforms prior work on aurally-relevant word relatedness datasets such as AMEN and ASLex.
The Promise of Premise: Harnessing Question Premises in Visual Question Answering
Authors: Aroma Mahendru, Viraj Prabhu, Akrit Mohapatra, Dhruv Batra, Stefan Lee
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: In this paper, we make a simple observation that questions about images often contain premises -- objects and relationships implied by the question -- and that reasoning about premises can help Visual Question Answering (VQA) models respond more intelligently to irrelevant or previously unseen questions. When presented with a question that is irrelevant to an image, state-of-the-art VQA models will still answer purely based on learned language biases, resulting in non-sensical or even misleading answers. We note that a visual question is irrelevant to an image if at least one of its premises is false (i.e. not depicted in the image). We leverage this observation to construct a dataset for Question Relevance Prediction and Explanation (QRPE) by searching for false premises. We train novel question relevance detection models and show that models that reason about premises consistently outperform models that do not. We also find that forcing standard VQA models to reason about premises during training can lead to improvements on tasks requiring compositional reasoning.
Guided Open Vocabulary Image Captioning with Constrained Beam Search
Authors: Peter Anderson, Basura Fernando, Mark Johnson, Stephen Gould
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Existing image captioning models do not generalize well to out-of-domain images containing novel scenes or objects. This limitation severely hinders the use of these models in real world applications dealing with images in the wild. We address this problem using a flexible approach that enables existing deep captioning architectures to take advantage of image taggers at test time, without re-training. Our method uses constrained beam search to force the inclusion of selected tag words in the output, and fixed, pretrained word embeddings to facilitate vocabulary expansion to previously unseen tag words. Using this approach we achieve state of the art results for out-of-domain captioning on MSCOCO (and improved results for in-domain captioning). Perhaps surprisingly, our results significantly outperform approaches that incorporate the same tag predictions into the learning algorithm. We also show that we can significantly improve the quality of generated ImageNet captions by leveraging ground-truth labels.
Zero-Shot Activity Recognition with Verb Attribute Induction
Authors: Rowan Zellers, Yejin Choi
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: In this paper, we investigate large-scale zero-shot activity recognition by modeling the visual and linguistic attributes of action verbs. For example, the verb “salute” has several properties, such as being a light movement, a social act, and short in duration. We use these attributes as the internal mapping between visual and textual representations to reason about a previously unseen action. In contrast to much prior work that assumes access to gold standard attributes for zero-shot classes and focuses primarily on object attributes, our model uniquely learns to infer action attributes from dictionary definitions and distributed word representations. Experimental results confirm that action attributes inferred from language can provide a predictive signal for zero-shot prediction of previously unseen activities.
Deriving continous grounded meaning representations from referentially structured multimodal contexts
Authors: Sina Zarrieß, David Schlangen
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Corpora of referring expressions paired with their visual referents are a good source for learning word meanings directly grounded in visual representations. Here, we explore additional ways of extracting from them word representations linked to multi-modal context: through expressions that refer to the same object, and through expressions that refer to different objects in the same scene. We show that continuous meaning representations derived from these contexts capture complementary aspects of similarity, , even if not outperforming textual embeddings trained on very large amounts of raw text when tested on standard similarity benchmarks. We propose a new task for evaluating grounded meaning representations---detection of potentially co-referential phrases---and show that it requires precise denotational representations of attribute meanings, which our method provides.
Hierarchically-Attentive RNN for Album Summarization and Storytelling
Authors: Licheng Yu, Mohit Bansal, Tamara Berg
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: We address the problem of end-to-end visual storytelling. Given a photo album, our model first selects the most representative (summary) photos, and then composes a natural language story for the album. For this task, we make use of the Visual Storytelling dataset and a model composed of three hierarchically-attentive Recurrent Neural Nets (RNNs) to: encode the album photos, select representative (summary) photos, and compose the story. Automatic and human evaluations show our model achieves better performance on selection, generation, and retrieval than baselines.
Video Highlight Prediction Using Audience Chat Reactions
Authors: Cheng-Yang Fu, Joon Lee, Mohit Bansal, Alexander Berg
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Sports channel video portals offer an exciting domain for research on multimodal, multilingual analysis. We present methods addressing the problem of automatic video highlight prediction based on joint visual features and textual analysis of the real-world audience discourse with complex slang, in both English and traditional Chinese. We present a novel dataset based on League of Legends championships recorded from North American and Taiwanese Twitch.tv channels (will be released for further research), and demonstrate strong results on these using multimodal, character-level CNN-RNN model architectures.
Reinforced Video Captioning with Entailment Rewards
Authors: Ramakanth Pasunuru, Mohit Bansal
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Sequence-to-sequence models have shown promising improvements on the temporal task of video captioning, but they optimize word-level cross-entropy loss during training. First, using policy gradient and mixed-loss methods for reinforcement learning, we directly optimize sentence-level task-based metrics (as rewards), achieving significant improvements over the baseline, based on both automatic metrics and human evaluation on multiple datasets. Next, we propose a novel entailment-enhanced reward (CIDEnt) that corrects phrase-matching based metrics (such as CIDEr) to only allow for logically-implied partial matches and avoid contradictions, achieving further significant improvements over the CIDEr-reward model. Overall, our CIDEnt-reward model achieves the new state-of-the-art on the MSR-VTT dataset.
Evaluating Hierarchies of Verb Argument Structure with Hierarchical Clustering
Authors: Jesse Mu, Joshua K. Hartshorne, Timothy O′Donnell
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Verbs can only be used with a few specific arrangements of their arguments (syntactic frames). Most theorists note that verbs can be organized into a hierarchy of verb classes based on the frames they admit. Here we show that such a hierarchy is objectively well-supported by the patterns of verbs and frames in English, since a systematic hierarchical clustering algorithm converges on the same structure as the handcrafted taxonomy of VerbNet, a broad-coverage verb lexicon. We also show that the hierarchies capture meaningful psychological dimensions of generalization by predicting novel verb coercions by human participants. We discuss limitations of a simple hierarchical representation and suggest similar approaches for identifying the representations underpinning verb argument structure.
Incorporating Global Visual Features into Attention-based Neural Machine Translation.
Authors: Iacer Calixto, Qun Liu
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: We introduce multi-modal, attention-based neural machine translation (NMT) models which incorporate visual features into different parts of both the encoder and the decoder. Global image features are extracted using a pre-trained convolutional neural network and are incorporated (i) as words in the source sentence, (ii) to initialise the encoder hidden state, and (iii) as additional data to initialise the decoder hidden state. In our experiments, we evaluate translations into English and German, how different strategies to incorporate global image features compare and which ones perform best. We also study the impact that adding synthetic multi-modal, multilingual data brings and find that the additional data have a positive impact on multi-modal NMT models. We report new state-of-the-art results and our best models also significantly improve on a comparable phrase-based Statistical MT (PBSMT) model trained on the Multi30k data set according to all metrics evaluated. To the best of our knowledge, it is the first time a purely neural model significantly improves over a PBSMT model on all metrics evaluated on this data set.
Mapping Instructions and Visual Observations to Actions with Reinforcement Learning
Authors: Dipendra Misra, John Langford, Yoav Artzi
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: We propose to directly map raw visual observations and text input to actions for instruction execution. While existing approaches assume access to structured environment representations or use a pipeline of separately trained models, we learn a single model to jointly reason about linguistic and visual input. We use reinforcement learning in a contextual bandit setting to train a neural network agent. To guide the agent`s exploration, we use reward shaping with different forms of supervision. Our approach does not require intermediate representations, planning procedures, or training different models. We evaluate in a simulated environment, and show significant improvements over supervised learning and common reinforcement learning variants.
An analysis of eye-movements during reading for the detection of mild cognitive impairment
Authors: Kathleen C. Fraser, Kristina Lundholm Fors, Dimitrios Kokkinakis, Arto Nordlund
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: We present a machine learning analysis of eye-tracking data for the detection of mild cognitive impairment, a decline in cognitive abilities that is associated with an increased risk of developing dementia. We compare two experimental configurations (reading aloud versus reading silently), as well as two methods of combining information from the two trials (concatenation and merging). Additionally, we annotate the words being read with information about their frequency and syntactic category, and use these annotations to generate new features. Ultimately, we are able to distinguish between participants with and without cognitive impairment with up to 86% accuracy.
A Structured Learning Approach to Temporal Relation Extraction
Authors: Qiang Ning, Zhili Feng, Dan Roth
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Identifying temporal relations between events is an essential step towards natural language understanding. However, the temporal relation between two events in a story depends on, and is often dictated by, relations among other events. Consequently, effectively identifying temporal relations between events is a challenging problem even for human annotators. This paper suggests that it is important to take these dependencies into account while learning to identify these relations and proposes a structured learning approach to address this challenge. As a byproduct, this provides a new perspective on handling missing relations, a known issue that hurts existing methods. As we show, the proposed approach results in significant improvements on the two commonly used data sets for this problem.
Importance sampling for unbiased on-demand evaluation of knowledge base population
Authors: Arun Chaganty, Ashwin Paranjape, Percy Liang, Christopher D. Manning
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Knowledge base population (KBP) systems take in a large document corpus and extract entities and their relations. Thus far, KBP evaluation has relied on judgements on the pooled predictions of existing systems. We show that this evaluation is problematic: when a new system predicts a previously unseen relation, it is penalized even if it is correct. This leads to significant bias against new systems, which counterproductively discourages innovation in the field. Our first contribution is a new importance-sampling based evaluation which corrects for this bias by annotating a new system`s predictions on-demand via crowdsourcing. We show this eliminates bias and reduces variance using data from the 2015 TAC KBP task. Our second contribution is an implementation of our method made publicly available as an online KBP evaluation service. We pilot the service by testing diverse state-of-the-art systems on the TAC KBP 2016 corpus and obtain accurate scores in a cost effective manner.
PACRR: A Position-Aware Neural IR Model for Relevance Matching
Authors: Kai Hui, Andrew Yates, Klaus Berberich, Gerard de Melo
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: In order to adopt deep learning for information retrieval, models are needed that can capture all relevant information required to assess the relevance of a document to a given user query. While previous works have successfully captured unigram term matches, how to fully employ position-dependent information such as proximity and term dependencies has been insufficiently explored. In this work, we propose a novel neural IR model named PACRR aiming at better modeling position-dependent interactions between a query and a document. Extensive experiments on six years' TREC Web Track data confirm that the proposed model yields better results under multiple benchmarks.
Globally Normalized Reader
Authors: Jonathan Raiman, John Miller
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Rapid progress has been made towards question answering (QA) systems that can extract answers from text. Existing neural approaches make use of expensive bi-directional attention mechanisms or score all possible answer spans, limiting scalability. We propose instead to cast extractive QA as an iterative search problem: select the answer`s sentence, start word, and end word. This representation reduces the space of each search step and allows computation to be conditionally allocated to promising search paths. We show that globally normalizing the decision process and back-propagating through beam search makes this representation viable and learning efficient. We empirically demonstrate the benefits of this approach using our model, Globally Normalized Reader (GNR), which achieves the second highest single model performance on the Stanford Question Answering Dataset (68.4 EM, 76.21 F1 dev) and is 24.7x faster than bi-attention-flow. We also introduce a data-augmentation method to produce semantically valid examples by aligning named entities to a knowledge base and swapping them with new entities of the same type. This method improves the performance of all models considered in this work and is of independent interest for a variety of NLP tasks.
Speech segmentation with a neural encoder model of working memory
Authors: Micha Elsner, Cory Shain
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: We present the first unsupervised LSTM speech segmenter as a cognitive model of the acquisition of words from unsegmented input. Cognitive biases toward phonological and syntactic predictability in speech are rooted in the limitations of human memory (Baddeley et al., 1998); compressed representations are easier to acquire and retain in memory. To model the biases introduced by these memory limitations, our system uses an LSTM-based encoder-decoder with a small number of hidden units, then searches for a segmentation that minimizes autoencoding loss. Linguistically meaningful segments (e.g. words) should share regular patterns of features that facilitate decoder performance in comparison to random segmentations, and we show that our learner discovers these patterns when trained on either phoneme sequences or raw acoustics. To our knowledge, ours is the first fully unsupervised system to be able to segment both symbolic and acoustic representations of speech.