mtybilly/Instruct2DS huggingface dataset:
Hierarchical Transfer Learning for Multi-label Text Classification
Authors: Cem Akkaya, Francisco Perez-Sorrosal, Kostas Tsioutsiouliklis Siddhartha Banerjee
Published in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2019

Abstract: Multi-Label Hierarchical Text Classification (MLHTC) is the task of categorizing documents into one or more topics organized in an hierarchical taxonomy. MLHTC can be formulated by combining multiple binary classification problems with an independent classifier for each category. We propose a novel transfer learning based strategy, HTrans, where binary classifiers at lower levels in the hierarchy are initialized using parameters of the parent classifier and fine-tuned on the child category classification task. In HTrans, we use a Gated Recurrent Unit (GRU)-based deep learning architecture coupled with attention. Compared to binary classifiers trained from scratch, our HTrans approach results in significant improvements of 1% on micro-F1 and 3% on macro-F1 on the RCV1 dataset. Our experiments also show that binary classifiers trained from scratch are significantly better than single multi-label models.
Bias Analysis and Mitigation in the Evaluation of Authorship Verification
Authors: Matthias Hagen, Benno Stein, Martin Potthast Janek Bevendorff
Published in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2019

Abstract: The PAN series of shared tasks is well known for its continuous and high quality research in the field of digital text forensics. Among others, PAN contributions include original corpora, tailored benchmarks, and standardized experimentation platforms. In this paper we review, theoretically and practically, the authorship verification task and conclude that the underlying experiment design cannot guarantee pushing forward the state of the art---in fact, it allows for top benchmarking with a surprisingly straightforward approach. In this regard, we present a “Basic and Fairly Flawed” (BAFF) authorship verifier that is on a par with the best approaches submitted so far, and that illustrates sources of bias that should be eliminated. We pinpoint these sources in the evaluation chain and present a refined authorship corpus as effective countermeasure.
Numeracy-600K: Learning Numeracy for Detecting Exaggerated Information in Market Comments
Authors: Hen-Hsen Huang, Hiroya Takamura, Hsin-Hsi Chen Chung-Chi Chen
Published in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2019

Abstract: In this paper, we attempt to answer the question of whether neural network models can learn numeracy, which is the ability to predict the magnitude of a numeral at some specific position in a text description. A large benchmark dataset, called Numeracy-600K, is provided for the novel task. We explore several neural network models including CNN, GRU, BiGRU, CRNN, CNN-capsule, GRU-capsule, and BiGRU-capsule in the experiments. The results show that the BiGRU model gets the best micro-averaged F1 score of 80.16%, and the GRU-capsule model gets the best macro-averaged F1 score of 64.71%. Besides discussing the challenges through comprehensive experiments, we also present an important application scenario, i.e., detecting exaggerated information, for the task.
Large-Scale Multi-Label Text Classification on EU Legislation
Authors: Emmanouil Fergadiotis, Prodromos Malakasiotis, Ion Androutsopoulos Ilias Chalkidis
Published in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2019

Abstract: We consider Large-Scale Multi-Label Text Classification (LMTC) in the legal domain. We release a new dataset of 57k legislative documents from EUR-LEX, annotated with \ensuremath∼4.3k EUROVOC labels, which is suitable for LMTC, few- and zero-shot learning. Experimenting with several neural classifiers, we show that BIGRUs with label-wise attention perform better than other current state of the art methods. Domain-specific WORD2VEC and context-sensitive ELMO embeddings further improve performance. We also find that considering only particular zones of the documents is sufficient. This allows us to bypass BERT`s maximum text length limit and fine-tune BERT, obtaining the best results in all but zero-shot learning cases.
Why Didn`t You Listen to Me? Comparing User Control of Human-in-the-Loop Topic Models
Authors: Alison Smith-Renner, Leah Findlater, Kevin Seppi, Jordan Boyd-Graber Varun Kumar
Published in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2019

Abstract: To address the lack of comparative evaluation of Human-in-the-Loop Topic Modeling (HLTM) systems, we implement and evaluate three contrasting HLTM modeling approaches using simulation experiments. These approaches extend previously proposed frameworks, including constraints and informed prior-based methods. Users should have a sense of control in HLTM systems, so we propose a control metric to measure whether refinement operations' results match users' expectations. Informed prior-based methods provide better control than constraints, but constraints yield higher quality topics.
Encouraging Paragraph Embeddings to Remember Sentence Identity Improves Classification
Authors: Mohit Iyyer Tu Vu
Published in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2019

Abstract: While paragraph embedding models are remarkably effective for downstream classification tasks, what they learn and encode into a single vector remains opaque. In this paper, we investigate a state-of-the-art paragraph embedding method proposed by Zhang et al. (2017) and discover that it cannot reliably tell whether a given sentence occurs in the input paragraph or not. We formulate a sentence content task to probe for this basic linguistic property and find that even a much simpler bag-of-words method has no trouble solving it. This result motivates us to replace the reconstruction-based objective of Zhang et al. (2017) with our sentence content probe objective in a semi-supervised setting. Despite its simplicity, our objective improves over paragraph reconstruction in terms of (1) downstream classification accuracies on benchmark datasets, (2) faster training, and (3) better generalization ability.
A Multi-Task Architecture on Relevance-based Neural Query Translation
Authors: Hamed Bonab, James Allan Sheikh Muhammad Sarwar
Published in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2019

Abstract: We describe a multi-task learning approach to train a Neural Machine Translation (NMT) model with a Relevance-based Auxiliary Task (RAT) for search query translation. The translation process for Cross-lingual Information Retrieval (CLIR) task is usually treated as a black box and it is performed as an independent step. However, an NMT model trained on sentence-level parallel data is not aware of the vocabulary distribution of the retrieval corpus. We address this problem and propose a multi-task learning architecture that achieves 16% improvement over a strong baseline on Italian-English query-document dataset. We show using both quantitative and qualitative analysis that our model generates balanced and precise translations with the regularization effect it achieves from multi-task learning paradigm.
Topic Modeling with Wasserstein Autoencoders
Authors: Ran Ding, Ramesh Nallapati, Bing Xiang Feng Nan
Published in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2019

Abstract: We propose a novel neural topic model in the Wasserstein autoencoders (WAE) framework. Unlike existing variational autoencoder based models, we directly enforce Dirichlet prior on the latent document-topic vectors. We exploit the structure of the latent space and apply a suitable kernel in minimizing the Maximum Mean Discrepancy (MMD) to perform distribution matching. We discover that MMD performs much better than the Generative Adversarial Network (GAN) in matching high dimensional Dirichlet distribution. We further discover that incorporating randomness in the encoder output during training leads to significantly more coherent topics. To measure the diversity of the produced topics, we propose a simple topic uniqueness metric. Together with the widely used coherence measure NPMI, we offer a more wholistic evaluation of topic quality. Experiments on several real datasets show that our model produces significantly better topics than existing topic models.
Dense Procedure Captioning in Narrated Instructional Videos
Authors: Lei Ji, Yaobo Liang, Nan Duan, Peng Chen, Zhendong Niu, Ming Zhou Botian Shi
Published in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2019

Abstract: Understanding narrated instructional videos is important for both research and real-world web applications. Motivated by video dense captioning, we propose a model to generate procedure captions from narrated instructional videos which are a sequence of step-wise clips with description. Previous works on video dense captioning learn video segments and generate captions without considering transcripts. We argue that transcripts in narrated instructional videos can enhance video representation by providing fine-grained complimentary and semantic textual information. In this paper, we introduce a framework to (1) extract procedures by a cross-modality module, which fuses video content with the entire transcript; and (2) generate captions by encoding video frames as well as a snippet of transcripts within each extracted procedure. Experiments show that our model can achieve state-of-the-art performance in procedure extraction and captioning, and the ablation studies demonstrate that both the video frames and the transcripts are important for the task.
Latent Variable Model for Multi-modal Translation
Authors: Miguel Rios, Wilker Aziz Iacer Calixto
Published in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2019

Abstract: In this work, we propose to model the interaction between visual and textual features for multi-modal neural machine translation (MMT) through a latent variable model. This latent variable can be seen as a multi-modal stochastic embedding of an image and its description in a foreign language. It is used in a target-language decoder and also to predict image features. Importantly, our model formulation utilises visual and textual inputs during training but does not require that images be available at test time. We show that our latent variable MMT formulation improves considerably over strong baselines, including a multi-task learning approach (Elliott and Kadar, 2017) and a conditional variational auto-encoder approach (Toyama et al., 2016). Finally, we show improvements due to (i) predicting image features in addition to only conditioning on them, (ii) imposing a constraint on the KL term to promote models with non-negligible mutual information between inputs and latent variable, and (iii) by training on additional target-language image descriptions (i.e. synthetic data).
Identifying Visible Actions in Lifestyle Vlogs
Authors: Laura Burdick, Jia Deng, Rada Mihalcea Oana Ignat
Published in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2019

Abstract: We consider the task of identifying human actions visible in online videos. We focus on the widely spread genre of lifestyle vlogs, which consist of videos of people performing actions while verbally describing them. Our goal is to identify if actions mentioned in the speech description of a video are visually present. We construct a dataset with crowdsourced manual annotations of visible actions, and introduce a multimodal algorithm that leverages information derived from visual and linguistic clues to automatically infer which actions are visible in a video.
A Corpus for Reasoning about Natural Language Grounded in Photographs
Authors: Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, Yoav Artzi Alane Suhr
Published in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2019

Abstract: We introduce a new dataset for joint reasoning about natural language and images, with a focus on semantic diversity, compositionality, and visual reasoning challenges. The data contains 107,292 examples of English sentences paired with web photographs. The task is to determine whether a natural language caption is true about a pair of photographs. We crowdsource the data using sets of visually rich images and a compare-and-contrast task to elicit linguistically diverse language. Qualitative analysis shows the data requires compositional joint reasoning, including about quantities, comparisons, and relations. Evaluation using state-of-the-art visual reasoning methods shows the data presents a strong challenge.
Learning to Discover, Ground and Use Words with Segmental Neural Language Models
Authors: Chris Dyer, Phil Blunsom Kazuya Kawakami
Published in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2019

Abstract: We propose a segmental neural language model that combines the generalization power of neural networks with the ability to discover word-like units that are latent in unsegmented character sequences. In contrast to previous segmentation models that treat word segmentation as an isolated task, our model unifies word discovery, learning how words fit together to form sentences, and, by conditioning the model on visual context, how words' meanings ground in representations of nonlinguistic modalities. Experiments show that the unconditional model learns predictive distributions better than character LSTM models, discovers words competitively with nonparametric Bayesian word segmentation models, and that modeling language conditional on visual context improves performance on both.
What Should I Ask? Using Conversationally Informative Rewards for Goal-oriented Visual Dialog.
Authors: Carlos Elmadjian, Richika Sharan, Vivek Kulkarni, Matthew Turk, William Yang Wang Pushkar Shukla
Published in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2019

Abstract: The ability to engage in goal-oriented conversations has allowed humans to gain knowledge, reduce uncertainty, and perform tasks more efficiently. Artificial agents, however, are still far behind humans in having goal-driven conversations. In this work, we focus on the task of goal-oriented visual dialogue, aiming to automatically generate a series of questions about an image with a single objective. This task is challenging since these questions must not only be consistent with a strategy to achieve a goal, but also consider the contextual information in the image. We propose an end-to-end goal-oriented visual dialogue system, that combines reinforcement learning with regularized information gain. Unlike previous approaches that have been proposed for the task, our work is motivated by the Rational Speech Act framework, which models the process of human inquiry to reach a goal. We test the two versions of our model on the GuessWhat?! dataset, obtaining significant results that outperform the current state-of-the-art models in the task of generating questions to find an undisclosed object in an image.
Symbolic Inductive Bias for Visually Grounded Learning of Spoken Language
Authors: Grzegorz Chrupała
Published in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2019

Abstract: A widespread approach to processing spoken language is to first automatically transcribe it into text. An alternative is to use an end-to-end approach: recent works have proposed to learn semantic embeddings of spoken language from images with spoken captions, without an intermediate transcription step. We propose to use multitask learning to exploit existing transcribed speech within the end-to-end setting. We describe a three-task architecture which combines the objectives of matching spoken captions with corresponding images, speech with text, and text with images. We show that the addition of the speech/text task leads to substantial performance improvements on image retrieval when compared to training the speech/image task in isolation. We conjecture that this is due to a strong inductive bias transcribed speech provides to the model, and offer supporting evidence for this.
Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog
Authors: Yu Cheng, Ahmed Kholy, Linjie Li, Jingjing Liu, Jianfeng Gao Zhe Gan
Published in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2019

Abstract: This paper presents a new model for visual dialog, Recurrent Dual Attention Network (ReDAN), using multi-step reasoning to answer a series of questions about an image. In each question-answering turn of a dialog, ReDAN infers the answer progressively through multiple reasoning steps. In each step of the reasoning process, the semantic representation of the question is updated based on the image and the previous dialog history, and the recurrently-refined representation is used for further reasoning in the subsequent step. On the VisDial v1.0 dataset, the proposed ReDAN model achieves a new state-of-the-art of 64.47% NDCG score. Visualization on the reasoning process further demonstrates that ReDAN can locate context-relevant visual and textual clues via iterative refinement, which can lead to the correct answer step-by-step.
Lattice Transformer for Speech Translation
Authors: Niyu Ge, Boxing Chen, Kai Fan Pei Zhang
Published in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2019

Abstract: Recent advances in sequence modeling have highlighted the strengths of the transformer architecture, especially in achieving state-of-the-art machine translation results. However, depending on the up-stream systems, e.g., speech recognition, or word segmentation, the input to translation system can vary greatly. The goal of this work is to extend the attention mechanism of the transformer to naturally consume the lattice in addition to the traditional sequential input. We first propose a general lattice transformer for speech translation where the input is the output of the automatic speech recognition (ASR) which contains multiple paths and posterior scores. To leverage the extra information from the lattice structure, we develop a novel controllable lattice attention mechanism to obtain latent representations. On the LDC Spanish-English speech translation corpus, our experiments show that lattice transformer generalizes significantly better and outperforms both a transformer baseline and a lattice LSTM. Additionally, we validate our approach on the WMT 2017 Chinese-English translation task with lattice inputs from different BPE segmentations. In this task, we also observe the improvements over strong baselines.
Informative Image Captioning with External Sources of Information
Authors: Piyush Sharma, Tomer Levinboim, Radu Soricut Sanqiang Zhao
Published in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2019

Abstract: An image caption should fluently present the essential information in a given image, including informative, fine-grained entity mentions and the manner in which these entities interact. However, current captioning models are usually trained to generate captions that only contain common object names, thus falling short on an important “informativeness” dimension. We present a mechanism for integrating image information together with fine-grained labels (assumed to be generated by some upstream models) into a caption that describes the image in a fluent and informative manner. We introduce a multimodal, multi-encoder model based on Transformer that ingests both image features and multiple sources of entity labels. We demonstrate that we can learn to control the appearance of these entity labels in the output, resulting in captions that are both fluent and informative.
CoDraw: Collaborative Drawing as a Testbed for Grounded Goal-driven Communication
Authors: Nikita Kitaev, Xinlei Chen, Marcus Rohrbach, Byoung-Tak Zhang, Yuandong Tian, Dhruv Batra, Devi Parikh Jin-Hwa Kim
Published in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2019

Abstract: In this work, we propose a goal-driven collaborative task that combines language, perception, and action. Specifically, we develop a Collaborative image-Drawing game between two agents, called CoDraw. Our game is grounded in a virtual world that contains movable clip art objects. The game involves two players: a Teller and a Drawer. The Teller sees an abstract scene containing multiple clip art pieces in a semantically meaningful configuration, while the Drawer tries to reconstruct the scene on an empty canvas using available clip art pieces. The two players communicate with each other using natural language. We collect the CoDraw dataset of ~10K dialogs consisting of ~138K messages exchanged between human players. We define protocols and metrics to evaluate learned agents in this testbed, highlighting the need for a novel “crosstalk” evaluation condition which pairs agents trained independently on disjoint subsets of the training data. We present models for our task and benchmark them using both fully automated evaluation and by having them play the game live with humans.
Bridging by Word: Image Grounded Vocabulary Construction for Visual Captioning
Authors: Zhongyu Wei, Siyuan Wang, Xuanjing Huang Zhihao Fan
Published in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2019

Abstract: Image Captioning aims at generating a short description for an image. Existing research usually employs the architecture of CNN-RNN that views the generation as a sequential decision-making process and the entire dataset vocabulary is used as decoding space. They suffer from generating high frequent n-gram with irrelevant words. To tackle this problem, we propose to construct an image-grounded vocabulary, based on which, captions are generated with limitation and guidance. In specific, a novel hierarchical structure is proposed to construct the vocabulary incorporating both visual information and relations among words. For generation, we propose a word-aware RNN cell incorporating vocabulary information into the decoding process directly. Reinforce algorithm is employed to train the generator using constraint vocabulary as action space. Experimental results on MS COCO and Flickr30k show the effectiveness of our framework compared to some state-of-the-art models.
Distilling Translations with Visual Awareness
Authors: Pranava Madhyastha, Lucia Specia Julia Ive
Published in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2019

Abstract: Previous work on multimodal machine translation has shown that visual information is only needed in very specific cases, for example in the presence of ambiguous words where the textual context is not sufficient. As a consequence, models tend to learn to ignore this information. We propose a translate-and-refine approach to this problem where images are only used by a second stage decoder. This approach is trained jointly to generate a good first draft translation and to improve over this draft by (i) making better use of the target language textual context (both left and right-side contexts) and (ii) making use of visual context. This approach leads to the state of the art results. Additionally, we show that it has the ability to recover from erroneous or missing words in the source language.
VIFIDEL: Evaluating the Visual Fidelity of Image Descriptions
Authors: Josiah Wang, Lucia Specia Pranava Madhyastha
Published in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2019

Abstract: We address the task of evaluating image description generation systems. We propose a novel image-aware metric for this task: VIFIDEL. It estimates the faithfulness of a generated caption with respect to the content of the actual image, based on the semantic similarity between labels of objects depicted in images and words in the description. The metric is also able to take into account the relative importance of objects mentioned in human reference descriptions during evaluation. Even if these human reference descriptions are not available, VIFIDEL can still reliably evaluate system descriptions. The metric achieves high correlation with human judgments on two well-known datasets and is competitive with metrics that depend on and rely exclusively on human references.
Are You Looking? Grounding to Multiple Modalities in Vision-and-Language Navigation
Authors: Daniel Fried, Anna Rohrbach, Dan Klein, Trevor Darrell, Kate Saenko Ronghang Hu
Published in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2019

Abstract: Vision-and-Language Navigation (VLN) requires grounding instructions, such as “turn right and stop at the door”, to routes in a visual environment. The actual grounding can connect language to the environment through multiple modalities, e.g. “stop at the door” might ground into visual objects, while “turn right” might rely only on the geometric structure of a route. We investigate where the natural language empirically grounds under two recent state-of-the-art VLN models. Surprisingly, we discover that visual features may actually hurt these models: models which only use route structure, ablating visual features, outperform their visual counterparts in unseen new environments on the benchmark Room-to-Room dataset. To better use all the available modalities, we propose to decompose the grounding procedure into a set of expert models with access to different modalities (including object detections) and ensemble them at prediction time, improving the performance of state-of-the-art models on the VLN task.
Multimodal Transformer for Unaligned Multimodal Language Sequences
Authors: Shaojie Bai, Paul Pu Liang, J. Zico Kolter, Louis-Philippe Morency, Ruslan Salakhutdinov Yao-Hung Hubert Tsai
Published in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2019

Abstract: Human language is often multimodal, which comprehends a mixture of natural language, facial gestures, and acoustic behaviors. However, two major challenges in modeling such multimodal human language time-series data exist: 1) inherent data non-alignment due to variable sampling rates for the sequences from each modality; and 2) long-range dependencies between elements across modalities. In this paper, we introduce the Multimodal Transformer (MulT) to generically address the above issues in an end-to-end manner without explicitly aligning the data. At the heart of our model is the directional pairwise crossmodal attention, which attends to interactions between multimodal sequences across distinct time steps and latently adapt streams from one modality to another. Comprehensive experiments on both aligned and non-aligned multimodal time-series show that our model outperforms state-of-the-art methods by a large margin. In addition, empirical analysis suggests that correlated crossmodal signals are able to be captured by the proposed crossmodal attention mechanism in MulT.
Show, Describe and Conclude: On Exploiting the Structure Information of Chest X-ray Reports
Authors: Zeya Wang, Eric Xing Baoyu Jing
Published in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2019

Abstract: Chest X-Ray (CXR) images are commonly used for clinical screening and diagnosis. Automatically writing reports for these images can considerably lighten the workload of radiologists for summarizing descriptive findings and conclusive impressions. The complex structures between and within sections of the reports pose a great challenge to the automatic report generation. Specifically, the section Impression is a diagnostic summarization over the section Findings; and the appearance of normality dominates each section over that of abnormality. Existing studies rarely explore and consider this fundamental structure information. In this work, we propose a novel framework which exploits the structure information between and within report sections for generating CXR imaging reports. First, we propose a two-stage strategy that explicitly models the relationship between Findings and Impression. Second, we design a novel co-operative multi-agent system that implicitly captures the imbalanced distribution between abnormality and normality. Experiments on two CXR report datasets show that our method achieves state-of-the-art performance in terms of various evaluation metrics. Our results expose that the proposed approach is able to generate high-quality medical reports through integrating the structure information.
Visual Story Post-Editing
Authors: Chieh-Yang Huang, Yen-Chia Hsu, Ting-Hao Huang Ting-Yao Hsu
Published in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2019

Abstract: We introduce the first dataset for human edits of machine-generated visual stories and explore how these collected edits may be used for the visual story post-editing task. The dataset ,VIST-Edit, includes 14,905 human-edited versions of 2,981 machine-generated visual stories. The stories were generated by two state-of-the-art visual storytelling models, each aligned to 5 human-edited versions. We establish baselines for the task, showing how a relatively small set of human edits can be leveraged to boost the performance of large visual storytelling models. We also discuss the weak correlation between automatic evaluation scores and human ratings, motivating the need for new automatic metrics.
Multimodal Abstractive Summarization for How2 Videos
Authors: Jindřich Libovický, Spandana Gella, Florian Metze Shruti Palaskar
Published in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2019

Abstract: In this paper, we study abstractive summarization for open-domain videos. Unlike the traditional text news summarization, the goal is less to “compress” text information but rather to provide a fluent textual summary of information that has been collected and fused from different source modalities, in our case video and audio transcripts (or text). We show how a multi-source sequence-to-sequence model with hierarchical attention can integrate information from different modalities into a coherent output, compare various models trained with different modalities and present pilot experiments on the \textitHow2 corpus of instructional videos. We also propose a new evaluation metric (Content F1) for abstractive summarization task that measures semantic adequacy rather than fluency of the summaries, which is covered by metrics like ROUGE and BLEU.
Learning to Relate from Captions and Bounding Boxes
Authors: Joel Ruben Antony Moniz, Anshu Aviral, Priyatham Bollimpalli Sarthak Garg
Published in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2019

Abstract: In this work, we propose a novel approach that predicts the relationships between various entities in an image in a weakly supervised manner by relying on image captions and object bounding box annotations as the sole source of supervision. Our proposed approach uses a top-down attention mechanism to align entities in captions to objects in the image, and then leverage the syntactic structure of the captions to align the relations. We use these alignments to train a relation classification network, thereby obtaining both grounded captions and dense relationships. We demonstrate the effectiveness of our model on the Visual Genome dataset by achieving a recall@50 of 15% and recall@100 of 25% on the relationships present in the image. We also show that the model successfully predicts relations that are not present in the corresponding captions.
Toward Future Scenario Generation: Extracting Event Causality Exploiting Semantic Relation, Context, and Association Features
Authors: Kentaro Torisawa, Julien Kloetzer, Motoki Sano, István Varga, Jong-Hoon Oh, Yutaka Kidawara Chikara Hashimoto
Published in: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (Long Paper Track), 2014
A Generalized Language Model as the Combination of Skipped n-grams and Modified Kneser Ney Smoothing
Authors: Thomas Gottron, Martin Körner, Paul Georg Wagner, Till Speicher, Steffen Staab Rene Pickhardt
Published in: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (Long Paper Track), 2014
Be Appropriate and Funny: Automatic Entity Morph Encoding
Authors: Hongzhao Huang, Xiaoman Pan, Heng Ji, Kevin Knight, Zhen Wen, Yizhou Sun, Jiawei Han, Bulent Yener Boliang Zhang
Published in: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) (Short Paper Track), 2014
Parallel Multiple Context-Free Grammars, Finite-State Translation Systems, and Polynomial-Time Recognizable Subclasses of Lexical-Functional Grammars
Authors: Ryuichi Nakanishi, Yuichi Kaji, Sachiko Ando, Tadao Kasami Hiroyuki Seki
Published in: 31st Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 1993
An Integrated Archictecture for Shallow and Deep Processing
Authors: Anette Frank, Bernd Kiefer, Stefan Mueller, Guenter Neumann, Jakub Piskorski, Ulrich Schaefer, Melanie Siegel, Hans Uszkoreit, Feiyu Xu, Markus Becker, Hans-Ulrich Krieger Berthold Crysmann
Published in: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2002
Encoding Source Language with Convolutional Neural Network for Machine Translation
Authors: Zhengdong Lu, Mingxuan Wang, Hang Li, Wenbin Jiang, Qun Liu Fandong Meng
Published in: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (Long Paper Track), 2015
Statistical Machine Translation Features with Multitask Tensor Networks
Authors: Zhongqiang Huang, Jacob Devlin, Thomas Lamar, Rabih Zbib, Richard Schwartz, John Makhoul Hendra Setiawan
Published in: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (Long Paper Track), 2015
Stacked Ensembles of Information Extractors for Knowledge-Base Population
Authors: Nazneen Fatema Rajani, Yinon Bentor, Raymond Mooney Vidhoon Viswanathan
Published in: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (Long Paper Track), 2015
Syntax-based Simultaneous Translation through Prediction of Unseen Syntactic Constituents
Authors: Graham Neubig, Sakriani Sakti, Tomoki Toda, Satoshi Nakamura Yusuke Oda
Published in: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (Long Paper Track), 2015
Generating High Quality Proposition Banks for Multilingual Semantic Role Labeling
Authors: Laura Chiticariu, Marina Danilevsky, Yunyao Li, Shivakumar Vaithyanathan, Huaiyu Zhu Alan Akbik
Published in: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (Long Paper Track), 2015
Aligning Opinions: Cross-Lingual Opinion Mining with Dependencies
Authors: Cláudia Pinto, Helena Figueira, Pedro Mendes, André F. T. Martins Mariana S. C. Almeida
Published in: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (Long Paper Track), 2015
Encoding Distributional Semantics into Triple-Based Knowledge Ranking for Document Enrichment
Authors: Bing Qin, Mao Zheng, Graeme Hirst, Ting Liu Muyu Zhang
Published in: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (Long Paper Track), 2015
Bring you to the past: Automatic Generation of Topically Relevant Event Chronicles
Authors: Wenzhe Pei, Heng Ji, Sujian Li, Baobao Chang, Zhifang Sui Tao Ge
Published in: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (Long Paper Track), 2015
Context-aware Entity Morph Decoding
Authors: Hongzhao Huang, Xiaoman Pan, Sujian Li, Chin-Yew Lin, Heng Ji, Kevin Knight, Zhen Wen, Yizhou Sun, Jiawei Han, Bulent Yener Boliang Zhang
Published in: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (Long Paper Track), 2015
Omnia Mutantur, Nihil Interit: Connecting Past with Present by Finding Corresponding Terms across Time
Authors: Adam Jatowt, Sourav Bhowmick, Katsumi Tanaka Yating Zhang
Published in: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (Long Paper Track), 2015
Learning Word Representations from Scarce and Noisy Data with Embedding Subspaces
Authors: Silvio Amir, Wang Ling, Mário Silva, Isabel Trancoso Ramon F. Astudillo
Published in: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (Long Paper Track), 2015
Predicting Polarities of Tweets by Composing Word Embeddings with Long Short-Term Memory
Authors: Yuanchao Liu, Chengjie Sun, Baoxun Wang, Xiaolong Wang Xin Wang
Published in: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (Long Paper Track), 2015
Learning Tag Embeddings and Tag-specific Composition Functions in Recursive Neural Network
Authors: Bo Tian, Minlie Huang, Yang Liu, Xuan Zhu, Xiaoyan Zhu Qiao Qian
Published in: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (Long Paper Track), 2015
Tea Party in the House: A Hierarchical Ideal Point Topic Model and Its Application to Republican Legislators in the 112th Congress
Authors: Jordan Boyd-Graber, Philip Resnik, Kristina Miler Viet-An Nguyen
Published in: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (Long Paper Track), 2015
A Computationally Efficient Algorithm for Learning Topical Collocation Models
Authors: Lan Du, Benjamin Börschinger, John K Pate, Massimiliano Ciaramita, Mark Steedman, Mark Johnson Zhendong Zhao
Published in: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (Long Paper Track), 2015
Adding Semantics to Data-Driven Paraphrasing
Authors: Johan Bos, Malvina Nissim, Charley Beller, Benjamin Van Durme, Chris Callison-Burch Ellie Pavlick
Published in: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (Long Paper Track), 2015
Abstractive Multi-Document Summarization via Phrase Selection and Merging
Authors: Piji Li, Yi Liao, Wai Lam, Weiwei Guo, Rebecca Passonneau Lidong Bing
Published in: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (Long Paper Track), 2015
Linguistic Harbingers of Betrayal: A Case Study on an Online Strategy Game
Authors: Srijan Kumar, Jordan Boyd-Graber, Cristian Danescu-Niculescu-Mizil Vlad Niculae
Published in: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (Long Paper Track), 2015
Who caught a cold ? - Identifying the subject of a symptom
Authors: Mamoru Komachi, Naoaki Okazaki, Eiji Aramaki, Hiroshi Ishikawa Shin Kanouchi
Published in: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (Long Paper Track), 2015
Inverted indexing for cross-lingual NLP
Authors: Željko Agić, Héctor Mart\ńez Alonso, Barbara Plank, Bernd Bohnet, Anders Johannsen Anders Søgaard
Published in: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (Long Paper Track), 2015
Inducing Word and Part-of-Speech with Pitman-Yor Hidden Semi-Markov Models
Authors: Hiroshi Tsukahara, Daichi Mochihashi Kei Uchiumi
Published in: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (Long Paper Track), 2015
Domain-Specific Paraphrase Extraction
Authors: Juri Ganitkevitch, Tsz Ping Chan, Xuchen Yao, Benjamin Van Durme, Chris Callison-Burch Ellie Pavlick
Published in: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) (Short Paper Track), 2015
Language Models for Image Captioning: The Quirks and What Works
Authors: Hao Cheng, Hao Fang, Saurabh Gupta, Li Deng, Xiaodong He, Geoffrey Zweig, Margaret Mitchell Jacob Devlin
Published in: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) (Short Paper Track), 2015
Discriminative Preordering Meets Kendall`s $τ$ Maximization
Authors: Yusuke Miyao, Katsuhito Sudoh, Katsuhiko Hayashi, Masaaki Nagata Sho Hoshino
Published in: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) (Short Paper Track), 2015
Exploring the Planet of the APEs: a Comparative Study of State-of-the-art Methods for MT Automatic Post-Editing
Authors: Marion Weller, Matteo Negri, Marco Turchi Rajen Chatterjee
Published in: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) (Short Paper Track), 2015
Frame-Semantic Role Labeling with Heterogeneous Annotations
Authors: Sam Thomson, Nathan Schneider, Jaime Carbonell, Noah A. Smith, Chris Dyer Meghana Kshirsagar
Published in: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) (Short Paper Track), 2015
Semantic Interpretation of Superlative Expressions via Structured Knowledge Bases
Authors: Yansong Feng, Songfang Huang, Kun Xu, Zhe Han, Dongyan Zhao Sheng Zhang
Published in: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) (Short Paper Track), 2015
A Lexicalized Tree Kernel for Open Information Extraction
Authors: Christoph Ringlstetter, Mi-Young Kim, Grzegorz Kondrak, Randy Goebel, Yusuke Miyao Ying Xu
Published in: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) (Short Paper Track), 2015
Learning Topic Hierarchies for Wikipedia Categories
Authors: Xuzhong Wang, Mengdi Zhang, Juanzi Li, Xiaoli Li, Chao Shao, Jie Tang, Yongbin Liu Linmei Hu
Published in: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) (Short Paper Track), 2015
Semantic Clustering and Convolutional Neural Network for Short Text Categorization
Authors: Jiaming Xu, Bo Xu, Chenglin Liu, Heng Zhang, Fangyuan Wang, Hongwei Hao Peng Wang
Published in: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) (Short Paper Track), 2015
FrameNet+: Fast Paraphrastic Tripling of FrameNet
Authors: Travis Wolfe, Pushpendre Rastogi, Chris Callison-Burch, Mark Dredze, Benjamin Van Durme Ellie Pavlick
Published in: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) (Short Paper Track), 2015
PPDB 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classification
Authors: Pushpendre Rastogi, Juri Ganitkevitch, Benjamin Van Durme, Chris Callison-Burch Ellie Pavlick
Published in: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) (Short Paper Track), 2015
deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse Targets
Authors: Chris Brockett, Alessandro Sordoni, Yangfeng Ji, Michael Auli, Chris Quirk, Margaret Mitchell, Jianfeng Gao, Bill Dolan Michel Galley
Published in: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) (Short Paper Track), 2015
The Fixed-Size Ordinally-Forgetting Encoding Method for Neural Network Language Models
Authors: Hui Jiang, MingBin Xu, JunFeng Hou, LiRong Dai ShiLiang Zhang
Published in: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) (Short Paper Track), 2015
A Computational Approach to Automatic Prediction of Drunk-Texting
Authors: Abhijit Mishra, Balamurali AR, Pushpak Bhattacharyya, Mark J. Carman Aditya Joshi
Published in: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) (Short Paper Track), 2015
Reducing infrequent-token perplexity via variational corpora
Authors: Pranjal Daga, Yu Cheng, Kunpeng Zhang, Ankit Agrawal, Alok Choudhary Yusheng Xie
Published in: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) (Short Paper Track), 2015