mtybilly/Instruct2DS huggingface dataset:
Don`t Throw Those Morphological Analyzers Away Just Yet: Neural Morphological Disambiguation for Arabic
Authors: Nizar Habash Nasser Zalmout
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: This paper presents a model for Arabic morphological disambiguation based on Recurrent Neural Networks (RNN). We train Long Short-Term Memory (LSTM) cells in several configurations and embedding levels to model the various morphological features. Our experiments show that these models outperform state-of-the-art systems without explicit use of feature engineering. However, adding learning features from a morphological analyzer to model the space of possible analyses provides additional improvement. We make use of the resulting morphological models for scoring and ranking the analyses of the morphological analyzer for morphological disambiguation. The results show significant gains in accuracy across several evaluation metrics. Our system results in 4.4% absolute increase over the state-of-the-art in full morphological analysis accuracy (30.6% relative error reduction), and 10.6% (31.5% relative error reduction) for out-of-vocabulary words.
Paradigm Completion for Derivational Morphology
Authors: Ekaterina Vylomova, Huda Khayrallah, Christo Kirov, David Yarowsky Ryan Cotterell
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: The generation of complex derived word forms has been an overlooked problem in NLP; we fill this gap by applying neural sequence-to-sequence models to the task. We overview the theoretical motivation for a paradigmatic treatment of derivational morphology, and introduce the task of derivational paradigm completion as a parallel to inflectional paradigm completion. State-of-the-art neural models adapted from the inflection task are able to learn the range of derivation patterns, and outperform a non-neural baseline by 16.4%. However, due to semantic, historical, and lexical considerations involved in derivational morphology, future work will be needed to achieve performance parity with inflection-generating systems.
A Sub-Character Architecture for Korean Language Processing
Authors: Karl Stratos
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: We introduce a novel sub-character architecture that exploits a unique compositional structure of the Korean language. Our method decomposes each character into a small set of primitive phonetic units called jamo letters from which character- and word-level representations are induced. The jamo letters divulge syntactic and semantic information that is difficult to access with conventional character-level units. They greatly alleviate the data sparsity problem, reducing the observation space to 1.6% of the original while increasing accuracy in our experiments. We apply our architecture to dependency parsing and achieve dramatic improvement over strong lexical baselines.
Do LSTMs really work so well for PoS tagging? -- A replication study
Authors: Torsten Zesch Tobias Horsmann
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: A recent study by Plank et al. (2016) found that LSTM-based PoS taggers considerably improve over the current state-of-the-art when evaluated on the corpora of the Universal Dependencies project that use a coarse-grained tagset. We replicate this study using a fresh collection of 27 corpora of 21 languages that are annotated with fine-grained tagsets of varying size. Our replication confirms the result in general, and we additionally find that the advantage of LSTMs is even bigger for larger tagsets. However, we also find that for the very large tagsets of morphologically rich languages, hand-crafted morphological lexicons are still necessary to reach state-of-the-art performance.
The Labeled Segmentation of Printed Books
Authors: Jennifer Dai, David Bamman Lara McConnaughey
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: We introduce the task of book structure labeling: segmenting and assigning a fixed category (such as Table of Contents, Preface, Index) to the document structure of printed books. We manually annotate the page-level structural categories for a large dataset totaling 294,816 pages in 1,055 books evenly sampled from 1750-1922, and present empirical results comparing the performance of several classes of models. The best-performing model, a bidirectional LSTM with rich features, achieves an overall accuracy of 95.8 and a class-balanced macro F-score of 71.4.
Cross-lingual Character-Level Neural Morphological Tagging
Authors: Georg Heigold Ryan Cotterell
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Even for common NLP tasks, sufficient supervision is not available in many languages -- morphological tagging is no exception. In the work presented here, we explore a transfer learning scheme, whereby we train character-level recurrent neural taggers to predict morphological taggings for high-resource languages and low-resource languages together. Learning joint character representations among multiple related languages successfully enables knowledge transfer from the high-resource languages to the low-resource ones.
Word-Context Character Embeddings for Chinese Word Segmentation
Authors: Zhenting Yu, Yue Zhang, Shujian Huang, Xinyu Dai, Jiajun Chen Hao Zhou
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Neural parsers have benefited from automatically labeled data via dependency-context word embeddings. We investigate training character embeddings on a word-based context in a similar way, showing that the simple method improves state-of-the-art neural word segmentation models significantly, beating tri-training baselines for leveraging auto-segmented data.
Segmentation-Free Word Embedding for Unsegmented Languages
Authors: Takamasa Oshikiri
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: In this paper, we propose a new pipeline of word embedding for unsegmented languages, called segmentation-free word embedding, which does not require word segmentation as a preprocessing step. Unlike space-delimited languages, unsegmented languages, such as Chinese and Japanese, require word segmentation as a preprocessing step. However, word segmentation, that often requires manually annotated resources, is difficult and expensive, and unavoidable errors in word segmentation affect downstream tasks. To avoid these problems in learning word vectors of unsegmented languages, we consider word co-occurrence statistics over all possible candidates of segmentations based on frequent character n-grams instead of segmented sentences provided by conventional word segmenters. Our experiments of noun category prediction tasks on raw Twitter, Weibo, and Wikipedia corpora show that the proposed method outperforms the conventional approaches that require word segmenters.
From Textbooks to Knowledge: A Case Study in Harvesting Axiomatic Knowledge from Textbooks to Solve Geometry Problems
Authors: Kumar Dubey, Eric Xing Mrinmaya Sachan
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Textbooks are rich sources of information. Harvesting structured knowledge from textbooks is a key challenge in many educational applications. As a case study, we present an approach for harvesting structured axiomatic knowledge from math textbooks. Our approach uses rich contextual and typographical features extracted from raw textbooks. It leverages the redundancy and shared ordering across multiple textbooks to further refine the harvested axioms. These axioms are then parsed into rules that are used to improve the state-of-the-art in solving geometry problems.
RACE: Large-scale ReAding Comprehension Dataset From Examinations
Authors: Qizhe Xie, Hanxiao Liu, Yiming Yang, Eduard Hovy Guokun Lai
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: We present RACE, a new dataset for benchmark evaluation of methods in the reading comprehension task. Collected from the English exams for middle and high school Chinese students in the age range between 12 to 18, RACE consists of near 28,000 passages and near 100,000 questions generated by human experts (English instructors), and covers a variety of topics which are carefully designed for evaluating the students' ability in understanding and reasoning. In particular, the proportion of questions that requires reasoning is much larger in RACE than that in other benchmark datasets for reading comprehension, and there is a significant gap between the performance of the state-of-the-art models (43%) and the ceiling human performance (95%). We hope this new dataset can serve as a valuable resource for research and evaluation in machine comprehension. The dataset is freely available at r̆lhttp://www.cs.cmu.edu/~glai1/data/race/and the code is available at \l̆https://github.com/qizhex/RACE_AR_baselines.
Beyond Sentential Semantic Parsing: Tackling the Math SAT with a Cascade of Tree Transducers
Authors: Cristian Petrescu-Prahova, Roie Levin, Ronan Le Bras, Alvaro Herrasti, Vidur Joshi Mark Hopkins
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: We present an approach for answering questions that span multiple sentences and exhibit sophisticated cross-sentence anaphoric phenomena, evaluating on a rich source of such questions -- the math portion of the Scholastic Aptitude Test (SAT). By using a tree transducer cascade as its basic architecture, our system propagates uncertainty from multiple sources (e.g. coreference resolution or verb interpretation) until it can be confidently resolved. Experiments show the first-ever results 43% recall and 91% precision) on SAT algebra word problems. We also apply our system to the public Dolphin algebra question set, and improve the state-of-the-art F1-score from 73.9% to 77.0%.
Learning Fine-Grained Expressions to Solve Math Word Problems
Authors: Shuming Shi, Chin-Yew Lin, Jian Yin Danqing Huang
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: This paper presents a novel template-based method to solve math word problems. This method learns the mappings between math concept phrases in math word problems and their math expressions from training data. For each equation template, we automatically construct a rich template sketch by aggregating information from various problems with the same template. Our approach is implemented in a two-stage system. It first retrieves a few relevant equation system templates and aligns numbers in math word problems to those templates for candidate equation generation. It then does a fine-grained inference to obtain the final answer. Experiment results show that our method achieves an accuracy of 28.4% on the linear Dolphin18K benchmark, which is 10% (54% relative) higher than previous state-of-the-art systems while achieving an accuracy increase of 12% (59% relative) on the TS6 benchmark subset.
Structural Embedding of Syntactic Trees for Machine Comprehension
Authors: Junjie Hu, Wei Wei, Zi Yang, Eric Nyberg Rui Liu
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Deep neural networks for machine comprehension typically utilizes only word or character embeddings without explicitly taking advantage of structured linguistic information such as constituency trees and dependency trees. In this paper, we propose structural embedding of syntactic trees (SEST), an algorithm framework to utilize structured information and encode them into vector representations that can boost the performance of algorithms for the machine comprehension. We evaluate our approach using a state-of-the-art neural attention model on the SQuAD dataset. Experimental results demonstrate that our model can accurately identify the syntactic boundaries of the sentences and extract answers that are syntactically coherent over the baseline methods.
World Knowledge for Reading Comprehension: Rare Entity Prediction with Hierarchical LSTMs Using External Descriptions
Authors: Emmanuel Bengio, Ryan Lowe, Jackie Chi Kit Cheung, Doina Precup Teng Long
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Humans interpret texts with respect to some background information, or world knowledge, and we would like to develop automatic reading comprehension systems that can do the same. In this paper, we introduce a task and several models to drive progress towards this goal. In particular, we propose the task of rare entity prediction: given a web document with several entities removed, models are tasked with predicting the correct missing entities conditioned on the document context and the lexical resources. This task is challenging due to the diversity of language styles and the extremely large number of rare entities. We propose two recurrent neural network architectures which make use of external knowledge in the form of entity descriptions. Our experiments show that our hierarchical LSTM model performs significantly better at the rare entity prediction task than those that do not make use of external resources.
Two-Stage Synthesis Networks for Transfer Learning in Machine Comprehension
Authors: Po-Sen Huang, Xiaodong He, Li Deng David Golub
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: We develop a technique for transfer learning in machine comprehension (MC) using a novel two-stage synthesis network. Given a high performing MC model in one domain, our technique aims to answer questions about documents in another domain, where we use no labeled data of question-answer pairs. Using the proposed synthesis network with a pretrained model on the SQuAD dataset, we achieve an F1 measure of 46.6% on the challenging NewsQA dataset, approaching performance of in-domain models (F1 measure of 50.0%) and outperforming the out-of-domain baseline by 7.6%, without use of provided annotations.
Deep Neural Solver for Math Word Problems
Authors: Xiaojiang Liu, Shuming Shi Yan Wang
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: This paper presents a deep neural solver to automatically solve math word problems. In contrast to previous statistical learning approaches, we directly translate math word problems to equation templates using a recurrent neural network (RNN) model, without sophisticated feature engineering. We further design a hybrid model that combines the RNN model and a similarity-based retrieval model to achieve additional performance improvement. Experiments conducted on a large dataset show that the RNN model and the hybrid model significantly outperform state-of-the-art statistical learning methods for math word problem solving.
Latent Space Embedding for Retrieval in Question-Answer Archives
Authors: Dinesh Garg, Shirish Shevade Deepak P
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Community-driven Question Answering (CQA) systems such as Yahoo! Answers have become valuable sources of reusable information. CQA retrieval enables usage of historical CQA archives to solve new questions posed by users. This task has received much recent attention, with methods building upon literature from translation models, topic models, and deep learning. In this paper, we devise a CQA retrieval technique, LASER-QA, that embeds question-answer pairs within a unified latent space preserving the local neighborhood structure of question and answer spaces. The idea is that such a space mirrors semantic similarity among questions as well as answers, thereby enabling high quality retrieval. Through an empirical analysis on various real-world QA datasets, we illustrate the improved effectiveness of LASER-QA over state-of-the-art methods.
Question Generation for Question Answering
Authors: Duyu Tang, Peng Chen, Ming Zhou Nan Duan
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: This paper presents how to generate questions from given passages using neural networks, where large scale QA pairs are automatically crawled and processed from Community-QA website, and used as training data. The contribution of the paper is 2-fold: First, two types of question generation approaches are proposed, one is a retrieval-based method using convolution neural network (CNN), the other is a generation-based method using recurrent neural network (RNN); Second, we show how to leverage the generated questions to improve existing question answering systems. We evaluate our question generation method for the answer sentence selection task on three benchmark datasets, including SQuAD, MS MARCO, and WikiQA. Experimental results show that, by using generated questions as an extra signal, significant QA improvement can be achieved.
Learning to Paraphrase for Question Answering
Authors: Jonathan Mallinson, Siva Reddy, Mirella Lapata Li Dong
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Question answering (QA) systems are sensitive to the many different ways natural language expresses the same information need. In this paper we turn to paraphrases as a means of capturing this knowledge and present a general framework which learns felicitous paraphrases for various QA tasks. Our method is trained end-to-end using question-answer pairs as a supervision signal. A question and its paraphrases serve as input to a neural scoring model which assigns higher weights to linguistic expressions most likely to yield correct answers. We evaluate our approach on QA over Freebase and answer sentence selection. Experimental results on three datasets show that our framework consistently improves performance, achieving competitive results despite the use of simple QA models.
Temporal Information Extraction for Question Answering Using Syntactic Dependencies in an LSTM-based Architecture
Authors: Anna Rumshisky, Alexey Romanov Yuanliang Meng
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: In this paper, we propose to use a set of simple, uniform in architecture LSTM-based models to recover different kinds of temporal relations from text. Using the shortest dependency path between entities as input, the same architecture is used to extract intra-sentence, cross-sentence, and document creation time relations. A “double-checking” technique reverses entity pairs in classification, boosting the recall of positive cases and reducing misclassifications between opposite classes. An efficient pruning algorithm resolves conflicts globally. Evaluated on QA-TempEval (SemEval2015 Task 5), our proposed technique outperforms state-of-the-art methods by a large margin. We also conduct intrinsic evaluation and post state-of-the-art results on Timebank-Dense.
Ranking Kernels for Structures and Embeddings: A Hybrid Preference and Classification Model
Authors: Daniele Bonadiman, Alessandro Moschitti Kateryna Tymoshenko
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Recent work has shown that Tree Kernels (TKs) and Convolutional Neural Networks (CNNs) obtain the state of the art in answer sentence reranking. Additionally, their combination used in Support Vector Machines (SVMs) is promising as it can exploit both the syntactic patterns captured by TKs and the embeddings learned by CNNs. However, the embeddings are constructed according to a classification function, which is not directly exploitable in the preference ranking algorithm of SVMs. In this work, we propose a new hybrid approach combining preference ranking applied to TKs and pointwise ranking applied to CNNs. We show that our approach produces better results on two well-known and rather different datasets: WikiQA for answer sentence selection and SemEval cQA for comment selection in Community Question Answering.
Recovering Question Answering Errors via Query Revision
Authors: Izzeddin Gur, Yu Su, Xifeng Yan Semih Yavuz
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: The existing factoid QA systems often lack a post-inspection component that can help models recover from their own mistakes. In this work, we propose to crosscheck the corresponding KB relations behind the predicted answers and identify potential inconsistencies. Instead of developing a new model that accepts evidences collected from these relations, we choose to plug them back to the original questions directly and check if the revised question makes sense or not. A bidirectional LSTM is applied to encode revised questions. We develop a scoring mechanism over the revised question encodings to refine the predictions of a base QA system. This approach can improve the F1 score of STAGG (Yih et al., 2015), one of the leading QA systems, from 52.5% to 53.9% on WEBQUESTIONS data.
An empirical study on the effectiveness of images in Multimodal Neural Machine Translation
Authors: Stéphane Dupont Jean-Benoit Delbrouck
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: In state-of-the-art Neural Machine Translation (NMT), an attention mechanism is used during decoding to enhance the translation. At every step, the decoder uses this mechanism to focus on different parts of the source sentence to gather the most useful information before outputting its target word. Recently, the effectiveness of the attention mechanism has also been explored for multi-modal tasks, where it becomes possible to focus both on sentence parts and image regions that they describe. In this paper, we compare several attention mechanism on the multi-modal translation task (English, image \textrightarrow German) and evaluate the ability of the model to make use of images to improve translation. We surpass state-of-the-art scores on the Multi30k data set, we nevertheless identify and report different misbehavior of the machine while translating.
Sound-Word2Vec: Learning Word Representations Grounded in Sounds
Authors: Ramakrishna Vedantam, Devi Parikh Ashwin Vijayakumar
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: To be able to interact better with humans, it is crucial for machines to understand sound -- a primary modality of human perception. Previous works have used sound to learn embeddings for improved generic semantic similarity assessment. In this work, we treat sound as a first-class citizen, studying downstream 6textual tasks which require aural grounding. To this end, we propose sound-word2vec -- a new embedding scheme that learns specialized word embeddings grounded in sounds. For example, we learn that two seemingly (semantically) unrelated concepts, like leaves and paper are similar due to the similar rustling sounds they make. Our embeddings prove useful in textual tasks requiring aural reasoning like text-based sound retrieval and discovering Foley sound effects (used in movies). Moreover, our embedding space captures interesting dependencies between words and onomatopoeia and outperforms prior work on aurally-relevant word relatedness datasets such as AMEN and ASLex.
The Promise of Premise: Harnessing Question Premises in Visual Question Answering
Authors: Viraj Prabhu, Akrit Mohapatra, Dhruv Batra, Stefan Lee Aroma Mahendru
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: In this paper, we make a simple observation that questions about images often contain premises -- objects and relationships implied by the question -- and that reasoning about premises can help Visual Question Answering (VQA) models respond more intelligently to irrelevant or previously unseen questions. When presented with a question that is irrelevant to an image, state-of-the-art VQA models will still answer purely based on learned language biases, resulting in non-sensical or even misleading answers. We note that a visual question is irrelevant to an image if at least one of its premises is false (i.e. not depicted in the image). We leverage this observation to construct a dataset for Question Relevance Prediction and Explanation (QRPE) by searching for false premises. We train novel question relevance detection models and show that models that reason about premises consistently outperform models that do not. We also find that forcing standard VQA models to reason about premises during training can lead to improvements on tasks requiring compositional reasoning.
Guided Open Vocabulary Image Captioning with Constrained Beam Search
Authors: Basura Fernando, Mark Johnson, Stephen Gould Peter Anderson
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Existing image captioning models do not generalize well to out-of-domain images containing novel scenes or objects. This limitation severely hinders the use of these models in real world applications dealing with images in the wild. We address this problem using a flexible approach that enables existing deep captioning architectures to take advantage of image taggers at test time, without re-training. Our method uses constrained beam search to force the inclusion of selected tag words in the output, and fixed, pretrained word embeddings to facilitate vocabulary expansion to previously unseen tag words. Using this approach we achieve state of the art results for out-of-domain captioning on MSCOCO (and improved results for in-domain captioning). Perhaps surprisingly, our results significantly outperform approaches that incorporate the same tag predictions into the learning algorithm. We also show that we can significantly improve the quality of generated ImageNet captions by leveraging ground-truth labels.
Zero-Shot Activity Recognition with Verb Attribute Induction
Authors: Yejin Choi Rowan Zellers
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: In this paper, we investigate large-scale zero-shot activity recognition by modeling the visual and linguistic attributes of action verbs. For example, the verb “salute” has several properties, such as being a light movement, a social act, and short in duration. We use these attributes as the internal mapping between visual and textual representations to reason about a previously unseen action. In contrast to much prior work that assumes access to gold standard attributes for zero-shot classes and focuses primarily on object attributes, our model uniquely learns to infer action attributes from dictionary definitions and distributed word representations. Experimental results confirm that action attributes inferred from language can provide a predictive signal for zero-shot prediction of previously unseen activities.
Deriving continous grounded meaning representations from referentially structured multimodal contexts
Authors: David Schlangen Sina Zarrieß
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Corpora of referring expressions paired with their visual referents are a good source for learning word meanings directly grounded in visual representations. Here, we explore additional ways of extracting from them word representations linked to multi-modal context: through expressions that refer to the same object, and through expressions that refer to different objects in the same scene. We show that continuous meaning representations derived from these contexts capture complementary aspects of similarity, , even if not outperforming textual embeddings trained on very large amounts of raw text when tested on standard similarity benchmarks. We propose a new task for evaluating grounded meaning representations---detection of potentially co-referential phrases---and show that it requires precise denotational representations of attribute meanings, which our method provides.
Hierarchically-Attentive RNN for Album Summarization and Storytelling
Authors: Mohit Bansal, Tamara Berg Licheng Yu
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: We address the problem of end-to-end visual storytelling. Given a photo album, our model first selects the most representative (summary) photos, and then composes a natural language story for the album. For this task, we make use of the Visual Storytelling dataset and a model composed of three hierarchically-attentive Recurrent Neural Nets (RNNs) to: encode the album photos, select representative (summary) photos, and compose the story. Automatic and human evaluations show our model achieves better performance on selection, generation, and retrieval than baselines.
Video Highlight Prediction Using Audience Chat Reactions
Authors: Joon Lee, Mohit Bansal, Alexander Berg Cheng-Yang Fu
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Sports channel video portals offer an exciting domain for research on multimodal, multilingual analysis. We present methods addressing the problem of automatic video highlight prediction based on joint visual features and textual analysis of the real-world audience discourse with complex slang, in both English and traditional Chinese. We present a novel dataset based on League of Legends championships recorded from North American and Taiwanese Twitch.tv channels (will be released for further research), and demonstrate strong results on these using multimodal, character-level CNN-RNN model architectures.
Reinforced Video Captioning with Entailment Rewards
Authors: Mohit Bansal Ramakanth Pasunuru
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Sequence-to-sequence models have shown promising improvements on the temporal task of video captioning, but they optimize word-level cross-entropy loss during training. First, using policy gradient and mixed-loss methods for reinforcement learning, we directly optimize sentence-level task-based metrics (as rewards), achieving significant improvements over the baseline, based on both automatic metrics and human evaluation on multiple datasets. Next, we propose a novel entailment-enhanced reward (CIDEnt) that corrects phrase-matching based metrics (such as CIDEr) to only allow for logically-implied partial matches and avoid contradictions, achieving further significant improvements over the CIDEr-reward model. Overall, our CIDEnt-reward model achieves the new state-of-the-art on the MSR-VTT dataset.
Evaluating Hierarchies of Verb Argument Structure with Hierarchical Clustering
Authors: Joshua K. Hartshorne, Timothy O′Donnell Jesse Mu
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Verbs can only be used with a few specific arrangements of their arguments (syntactic frames). Most theorists note that verbs can be organized into a hierarchy of verb classes based on the frames they admit. Here we show that such a hierarchy is objectively well-supported by the patterns of verbs and frames in English, since a systematic hierarchical clustering algorithm converges on the same structure as the handcrafted taxonomy of VerbNet, a broad-coverage verb lexicon. We also show that the hierarchies capture meaningful psychological dimensions of generalization by predicting novel verb coercions by human participants. We discuss limitations of a simple hierarchical representation and suggest similar approaches for identifying the representations underpinning verb argument structure.
Incorporating Global Visual Features into Attention-based Neural Machine Translation.
Authors: Qun Liu Iacer Calixto
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: We introduce multi-modal, attention-based neural machine translation (NMT) models which incorporate visual features into different parts of both the encoder and the decoder. Global image features are extracted using a pre-trained convolutional neural network and are incorporated (i) as words in the source sentence, (ii) to initialise the encoder hidden state, and (iii) as additional data to initialise the decoder hidden state. In our experiments, we evaluate translations into English and German, how different strategies to incorporate global image features compare and which ones perform best. We also study the impact that adding synthetic multi-modal, multilingual data brings and find that the additional data have a positive impact on multi-modal NMT models. We report new state-of-the-art results and our best models also significantly improve on a comparable phrase-based Statistical MT (PBSMT) model trained on the Multi30k data set according to all metrics evaluated. To the best of our knowledge, it is the first time a purely neural model significantly improves over a PBSMT model on all metrics evaluated on this data set.
Mapping Instructions and Visual Observations to Actions with Reinforcement Learning
Authors: John Langford, Yoav Artzi Dipendra Misra
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: We propose to directly map raw visual observations and text input to actions for instruction execution. While existing approaches assume access to structured environment representations or use a pipeline of separately trained models, we learn a single model to jointly reason about linguistic and visual input. We use reinforcement learning in a contextual bandit setting to train a neural network agent. To guide the agent`s exploration, we use reward shaping with different forms of supervision. Our approach does not require intermediate representations, planning procedures, or training different models. We evaluate in a simulated environment, and show significant improvements over supervised learning and common reinforcement learning variants.
An analysis of eye-movements during reading for the detection of mild cognitive impairment
Authors: Kristina Lundholm Fors, Dimitrios Kokkinakis, Arto Nordlund Kathleen C. Fraser
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: We present a machine learning analysis of eye-tracking data for the detection of mild cognitive impairment, a decline in cognitive abilities that is associated with an increased risk of developing dementia. We compare two experimental configurations (reading aloud versus reading silently), as well as two methods of combining information from the two trials (concatenation and merging). Additionally, we annotate the words being read with information about their frequency and syntactic category, and use these annotations to generate new features. Ultimately, we are able to distinguish between participants with and without cognitive impairment with up to 86% accuracy.
A Structured Learning Approach to Temporal Relation Extraction
Authors: Zhili Feng, Dan Roth Qiang Ning
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Identifying temporal relations between events is an essential step towards natural language understanding. However, the temporal relation between two events in a story depends on, and is often dictated by, relations among other events. Consequently, effectively identifying temporal relations between events is a challenging problem even for human annotators. This paper suggests that it is important to take these dependencies into account while learning to identify these relations and proposes a structured learning approach to address this challenge. As a byproduct, this provides a new perspective on handling missing relations, a known issue that hurts existing methods. As we show, the proposed approach results in significant improvements on the two commonly used data sets for this problem.
Importance sampling for unbiased on-demand evaluation of knowledge base population
Authors: Ashwin Paranjape, Percy Liang, Christopher D. Manning Arun Chaganty
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Knowledge base population (KBP) systems take in a large document corpus and extract entities and their relations. Thus far, KBP evaluation has relied on judgements on the pooled predictions of existing systems. We show that this evaluation is problematic: when a new system predicts a previously unseen relation, it is penalized even if it is correct. This leads to significant bias against new systems, which counterproductively discourages innovation in the field. Our first contribution is a new importance-sampling based evaluation which corrects for this bias by annotating a new system`s predictions on-demand via crowdsourcing. We show this eliminates bias and reduces variance using data from the 2015 TAC KBP task. Our second contribution is an implementation of our method made publicly available as an online KBP evaluation service. We pilot the service by testing diverse state-of-the-art systems on the TAC KBP 2016 corpus and obtain accurate scores in a cost effective manner.
PACRR: A Position-Aware Neural IR Model for Relevance Matching
Authors: Andrew Yates, Klaus Berberich, Gerard de Melo Kai Hui
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: In order to adopt deep learning for information retrieval, models are needed that can capture all relevant information required to assess the relevance of a document to a given user query. While previous works have successfully captured unigram term matches, how to fully employ position-dependent information such as proximity and term dependencies has been insufficiently explored. In this work, we propose a novel neural IR model named PACRR aiming at better modeling position-dependent interactions between a query and a document. Extensive experiments on six years' TREC Web Track data confirm that the proposed model yields better results under multiple benchmarks.
Globally Normalized Reader
Authors: John Miller Jonathan Raiman
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Rapid progress has been made towards question answering (QA) systems that can extract answers from text. Existing neural approaches make use of expensive bi-directional attention mechanisms or score all possible answer spans, limiting scalability. We propose instead to cast extractive QA as an iterative search problem: select the answer`s sentence, start word, and end word. This representation reduces the space of each search step and allows computation to be conditionally allocated to promising search paths. We show that globally normalizing the decision process and back-propagating through beam search makes this representation viable and learning efficient. We empirically demonstrate the benefits of this approach using our model, Globally Normalized Reader (GNR), which achieves the second highest single model performance on the Stanford Question Answering Dataset (68.4 EM, 76.21 F1 dev) and is 24.7x faster than bi-attention-flow. We also introduce a data-augmentation method to produce semantically valid examples by aligning named entities to a knowledge base and swapping them with new entities of the same type. This method improves the performance of all models considered in this work and is of independent interest for a variety of NLP tasks.
Speech segmentation with a neural encoder model of working memory
Authors: Cory Shain Micha Elsner
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: We present the first unsupervised LSTM speech segmenter as a cognitive model of the acquisition of words from unsegmented input. Cognitive biases toward phonological and syntactic predictability in speech are rooted in the limitations of human memory (Baddeley et al., 1998); compressed representations are easier to acquire and retain in memory. To model the biases introduced by these memory limitations, our system uses an LSTM-based encoder-decoder with a small number of hidden units, then searches for a segmentation that minimizes autoencoding loss. Linguistically meaningful segments (e.g. words) should share regular patterns of features that facilitate decoder performance in comparison to random segmentations, and we show that our learner discovers these patterns when trained on either phoneme sequences or raw acoustics. To our knowledge, ours is the first fully unsupervised system to be able to segment both symbolic and acoustic representations of speech.
Speaking, Seeing, Understanding: Correlating semantic models with conceptual representation in the brain
Authors: Stephen Clark, Ekaterina Shutova Luana Bulat
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Research in computational semantics is increasingly guided by our understanding of human semantic processing. However, semantic models are typically studied in the context of natural language processing system performance. In this paper, we present a systematic evaluation and comparison of a range of widely-used, state-of-the-art semantic models in their ability to predict patterns of conceptual representation in the human brain. Our results provide new insights both for the design of computational semantic models and for further research in cognitive neuroscience.
Multi-modal Summarization for Asynchronous Collection of Text, Image, Audio and Video
Authors: Junnan Zhu, Cong Ma, Jiajun Zhang, Chengqing Zong Haoran Li
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: The rapid increase of the multimedia data over the Internet necessitates multi-modal summarization from collections of text, image, audio and video. In this work, we propose an extractive Multi-modal Summarization (MMS) method which can automatically generate a textual summary given a set of documents, images, audios and videos related to a specific topic. The key idea is to bridge the semantic gaps between multi-modal contents. For audio information, we design an approach to selectively use its transcription. For vision information, we learn joint representations of texts and images using a neural network. Finally, all the multi-modal aspects are considered to generate the textural summary by maximizing the salience, non-redundancy, readability and coverage through budgeted optimization of submodular functions. We further introduce an MMS corpus in English and Chinese. The experimental results on this dataset demonstrate that our method outperforms other competitive baseline methods.
Tensor Fusion Network for Multimodal Sentiment Analysis
Authors: Minghai Chen, Soujanya Poria, Erik Cambria, Louis-Philippe Morency Amir Zadeh
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Multimodal sentiment analysis is an increasingly popular research area, which extends the conventional language-based definition of sentiment analysis to a multimodal setup where other relevant modalities accompany language. In this paper, we pose the problem of multimodal sentiment analysis as modeling intra-modality and inter-modality dynamics. We introduce a novel model, termed Tensor Fusion Networks, which learns both such dynamics end-to-end. The proposed approach is tailored for the volatile nature of spoken language in online videos as well as accompanying gestures and voice. In the experiments, our model outperforms state-of-the-art approaches for both multimodal and unimodal sentiment analysis.
ConStance: Modeling Annotation Contexts to Improve Stance Classification
Authors: Lisa Friedland, William Hobbs, David Lazer, Oren Tsur Kenneth Joseph
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Manual annotations are a prerequisite for many applications of machine learning. However, weaknesses in the annotation process itself are easy to overlook. In particular, scholars often choose what information to give to annotators without examining these decisions empirically. For subjective tasks such as sentiment analysis, sarcasm, and stance detection, such choices can impact results. Here, for the task of political stance detection on Twitter, we show that providing too little context can result in noisy and uncertain annotations, whereas providing too strong a context may cause it to outweigh other signals. To characterize and reduce these biases, we develop ConStance, a general model for reasoning about annotations across information conditions. Given conflicting labels produced by multiple annotators seeing the same instances with different contexts, ConStance simultaneously estimates gold standard labels and also learns a classifier for new instances. We show that the classifier learned by ConStance outperforms a variety of baselines at predicting political stance, while the model`s interpretable parameters shed light on the effects of each context.
Deeper Attention to Abusive User Content Moderation
Authors: Prodromos Malakasiotis, Ion Androutsopoulos John Pavlopoulos
Published in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Main Conference Track), 2017

Abstract: Experimenting with a new dataset of 1.6M user comments from a news portal and an existing dataset of 115K Wikipedia talk page comments, we show that an RNN operating on word embeddings outpeforms the previous state of the art in moderation, which used logistic regression or an MLP classifier with character or word n-grams. We also compare against a CNN operating on word embeddings, and a word-list baseline. A novel, deep, classificationspecific attention mechanism improves the performance of the RNN further, and can also highlight suspicious words for free, without including highlighted words in the training data. We consider both fully automatic and semi-automatic moderation.