Reviewerly/exHarmony huggingface dataset:
AssistSR: Task-oriented Video Segment Retrieval for Personal AI Assistant

It is still a pipe dream that personal AI assistants on the phone and AR glasses can assist our daily life in addressing our questions like "how to adjust the date for this watch?"and "how to set its heating duration?(while pointing at an oven)".The queries used in conventional tasks (i.e.Video Question Answering, Video Retrieval, Moment Localization) are often factoid and based on pure text.In contrast, we present a new task called Task-oriented Question-driven Video Segment Retrieval (TQVSR).Each of our questions is an image-box-text query that focuses on affordance of items in our daily life and expects relevant answer segments to be retrieved from a corpus of instructional videotranscript segments.To support the study of this TQVSR task, we construct a new dataset called AssistSR.We design novel guidelines to create high-quality samples.This dataset contains 3.2k multimodal questions on 1.6k video segments from instructional videos on diverse daily-used items.To address TQVSR, we develop a simple yet effective model called Dual Multimodal Encoders (DME) that significantly outperforms several baseline methods while still having large room for improvement in the future.Moreover, we present detailed ablation analyses.Code and data are available at https://github.com/StanLei52/TQVSR.
Language Prior Is Not the Only Shortcut: A Benchmark for Shortcut Learning in VQA

Visual Question Answering (VQA) models are prone to learn the shortcut solution formed by dataset biases rather than the intended solution.To evaluate the VQA models' reasoning ability beyond shortcut learning, the VQA-CP v2 dataset introduces an answer distribution shift between the training and test set given a question type.In this way, the model cannot use the training set shortcut to perform well on the test set.However, VQA-CP v2 only considers one type of shortcut (from question type to answer) and thus still cannot guarantee that the model relies on the intended solution rather than a solution specific to this shortcut.To overcome this limitation, we propose a new dataset that considers varying types of shortcuts by constructing different distribution shifts in multiple OOD test sets.In addition, we overcome three troubling practices in the use of VQA-CP v2, e.g., selecting models using OOD test sets, and further standardize OOD evaluation procedure.Our benchmark provides a more rigorous and comprehensive testbed for shortcut learning in VQA.We benchmark recent methods and find that methods specifically designed for particular shortcuts fail to simultaneously generalize to our varying OOD test sets.We also systematically study the varying shortcuts and provide several valuable findings, which may promote the exploration of shortcut learning in VQA.
Towards Table-to-Text Generation with Pretrained Language Model: A Table Structure Understanding and Text Deliberating Approach

Although remarkable progress on the neural table-to-text methods has been made, the generalization issues hinder the applicability of these models due to the limited source tables.Large-scale pretrained language models sound like a promising solution to tackle such issues.However, how to effectively bridge the gap between the structured table and the text input by fully leveraging table information to fuel the pretrained model is still not well explored.Besides, another challenge of integrating the deliberation mechanism into the textto-text pretrained model for solving the tableto-text task remains seldom studied.In this paper, to implement the table-to-text generation with pretrained language model, we propose a table structure understanding and text deliberating approach, namely TASD.To be specific, we devise a three-layered multi-head attention network to realize the table-structureaware text generation model with the help of the pretrained language model.Furthermore, a multi-pass decoder framework is adopted to enhance the capability of polishing generated text for table descriptions.The empirical studies, as well as human evaluation, on two public datasets, validate that our approach can generate faithful and fluent descriptive texts for different types of tables.
Bootstrapping meaning through listening: Unsupervised learning of spoken sentence embeddings

Inducing semantic representations directly from speech signals is a highly challenging task but has many useful applications in speech mining and spoken language understanding.This study tackles the unsupervised learning of semantic representations for spoken utterances.Through converting speech signals into hidden units generated from acoustic unit discovery, we propose WavEmbed, a multimodal sequential autoencoder that predicts hidden units from a dense representation of speech.Secondly, we also propose S-HuBERT to induce meaning through knowledge distillation, in which a sentence embedding model is first trained on hidden units and passes its knowledge to a speech encoder through contrastive learning.The best performing model achieves a moderate correlation (0.5∼0.6) with human judgments, without relying on any labels or transcriptions.Furthermore, these models can also be easily extended to leverage textual transcriptions of speech to learn much better speech embeddings that are strongly correlated with human annotations.Our proposed methods are applicable to the development of purely data-driven systems for speech mining, indexing and search.
Transformers on Multilingual Clause-Level Morphology

This paper describes our winning systems in MRL: The 1st Shared Task on Multilingual Clause-level Morphology (EMNLP 2022 Workshop) designed by KUIS AI NLP team.We present our work for all three parts of the shared task: inflection, reinflection, and analysis.We mainly explore transformers with two approaches: (i) training models from scratch in combination with data augmentation, and (ii) transfer learning with prefix-tuning at multilingual morphological tasks.Data augmentation significantly improves performance for most languages in the inflection and reinflection tasks.On the other hand, Prefix-tuning on a pre-trained mGPT model helps us to adapt analysis tasks in low-data and multilingual settings.While transformer architectures with data augmentation achieved the most promising results for inflection and reinflection tasks, prefix-tuning on mGPT received the highest results for the analysis task.Our systems received 1st place in all three tasks in MRL 2022. 1
Discord Questions: A Computational Approach To Diversity Analysis in News Coverage

There are many potential benefits to news readers accessing diverse sources.Modern news aggregators do the hard work of organizing the news, offering readers a plethora of source options, but choosing which source to read remains challenging.We propose a new framework to assist readers in identifying source differences and gaining an understanding of news coverage diversity.The framework is based on the generation of Discord Questions: questions with a diverse answer pool, explicitly illustrating source differences.To assemble a prototype of the framework, we focus on two components: (1) discord question generation, the task of generating questions answered differently by sources, for which we propose an automatic scoring method, and create a model that improves performance from current question generation (QG) methods by 5%, (2) answer consolidation, the task of grouping answers to a question that are semantically similar, for which we collect data and repurpose a method that achieves 81% balanced accuracy on our realistic test set.We illustrate the framework's feasibility through a prototype interface.Even though model performance at discord QG still lags human performance by more than 15%, generated questions are judged to be more interesting than factoid questions and can reveal differences in the level of detail, sentiment, and reasoning of sources in news coverage. 1.
Intriguing Properties of Compression on Multilingual Models

Multilingual models are often particularly dependent on scaling to generalize to a growing number of languages.Compression techniques are widely relied upon to reconcile the growth in model size with real world resource constraints, but compression can have a disparate effect on model performance for lowresource languages.It is thus crucial to understand the trade-offs between scale, multilingualism, and compression.In this work, we propose an experimental framework to characterize the impact of sparsifying multilingual pre-trained language models during finetuning.Applying this framework to mBERT named entity recognition models across 40 languages, we find that compression confers several intriguing and previously unknown generalization properties.In contrast to prior findings, we find that compression may improve model robustness over dense models.We additionally observe that under certain sparsification regimes compression may aid, rather than disproportionately impact the performance of low-resource languages.
Masked Language Models Know Which are Popular: A Simple Ranking Strategy for Commonsense Question Answering

We propose a simple ranking strategy to solve a generative commonsense question answering (QA) problem.Compared with multiple-choice QA, it is challenging because the answers to a question are not unique and they are supposed to be popular and diverse.Our strategy exploits the dataset itself and negative samples that we collect from WordNet to train a ranker that picks out the most popular answers for commonsense questions.The effectiveness of our strategy is verified on different pre-trained masked language models (MLMs) in a pipeline framework, where an MLM reranks the generated answers.Further, we explore an endto-end framework where MLMs are utilized to guide the generation of generative language models (GLMs).Taking advantage of reinforcement learning, we apply policy gradient to train a GLM with the rewards fed back by an MLM.Empirical results on ProtoQA dataset demonstrate that MLMs can acquire the ability to distinguish the popular answers and improve the typical answer generation of GLMs as well.
R2D2: Robust Data-to-Text with Replacement Detection

Unfaithful text generation is a common problem for text generation systems.In the case of Data-to-Text (D2T) systems, the factuality of the generated text is particularly crucial for any real-world applications.We introduce R2D2, a training framework that addresses unfaithful Data-to-Text generation by training a system both as a generator and a faithfulness discriminator with additional replacement detection and unlikelihood learning tasks.To facilitate such training, we propose two methods for sampling unfaithful sentences.We argue that the poor entity retrieval capability of D2T systems is one of the primary sources of unfaithfulness, so in addition to the existing metrics, we further propose named entity based metrics to evaluate the fidelity of D2T generations.Our experimental results show that R2D2 systems could effectively mitigate the unfaithful text generation, and they achieve new state-of-the-art results on FeTaQA, LogicNLG, and ToTTo, all with significant improvements.
Does Self-Rationalization Improve Robustness to Spurious Correlations?

Rationalization is fundamental to human reasoning and learning.NLP models trained to produce rationales along with predictions, called self-rationalization models, have been investigated for their interpretability and utility to end-users.However, the extent to which training with human-written rationales facilitates learning remains an under-explored question.We ask whether training models to selfrationalize can aid in their learning to solve tasks for the right reasons.Specifically, we evaluate how training self-rationalization models with free-text rationales affects robustness to spurious correlations in fine-tuned encoderdecoder and decoder-only models of six different sizes.We evaluate robustness to spurious correlations by measuring performance on 1) manually annotated challenge datasets and 2) subsets of original test sets where reliance on spurious correlations would fail to produce correct answers.We find that while self-rationalization can improve robustness to spurious correlations in low-resource settings, it tends to hurt robustness in higher-resource settings.Furthermore, these effects depend on model family and size, as well as on rationale content.Together, our results suggest that explainability can come at the cost of robustness; thus, appropriate care should be taken when training self-rationalizing models with the goal of creating more trustworthy models.2 Such approaches have also been referred to as explainthen-predict (Camburu et al., 2018) and rationalize-thenpredict (Chen et al., 2022) models.3 See Wiegreffe et al. ( 2021) for a detailed discussion of pipeline and self-rationalization approaches to rationalization.
Machine Translation for Multilingual Intent Detection and Slots Filling

We expect to interact with home assistants irrespective of our language.However, scaling the Natural Language Understanding pipeline to multiple languages while keeping the same level of accuracy remains a challenge.In this work, we leverage the inherent multilingual aspect of translation models for the task of multilingual intent classification and slot filling.Our experiments reveal that they work equally well with general-purpose multilingual text-to-text models.Furthermore, their accuracy can be further improved by artificially increasing the size of the training set.Unfortunately, increasing the training set also increases the overlap with the test set, leading to overestimating their true capabilities.As a result, we propose two new evaluation methods capable of accounting for an overlap between the training and test set.
Toward Privacy-preserving Text Embedding Similarity with Homomorphic Encryption

Text embedding is an essential component to build efficient natural language applications based on text similarities such as search engines and chatbots.Certain industries like finance and healthcare demand strict privacy-preserving conditions that user's data should not be exposed to any potential malicious users even including service providers.From a privacy standpoint, text embeddings seem impossible to be interpreted but there is still a privacy risk that they can be recovered to original texts through inversion attacks.To satisfy such privacy requirements, in this paper, we study a Homomorphic Encryption (HE) based text similarity inference.To validate our method, we perform extensive experiments on two vital text similarity tasks.Through text embedding inversion tests, we prove that the benchmark datasets are vulnerable to inversion attacks and another privacy preserving approach, dχ-privacy, a relaxed version of Local Differential Privacy method fails to prevent them.We show that our approach preserves the performance of models compared to that the baseline has degradation up to 10% of scores for the minimum security.
Finding Memo: Extractive Memorization in Constrained Sequence Generation Tasks

Memorization presents a challenge for several constrained Natural Language Generation (NLG) tasks such as Neural Machine Translation (NMT), wherein the proclivity of neural models to memorize noisy and atypical samples reacts adversely with the noisy (web crawled) datasets.However, previous studies of memorization in constrained NLG tasks have only focused on counterfactual memorization, linking it to the problem of hallucinations.In this work, we propose a new, inexpensive algorithm for extractive memorization (exact training data generation under insufficient context) in constrained sequence generation tasks and use it to study extractive memorization and its effects in NMT.We demonstrate that extractive memorization poses a serious threat to NMT reliability by qualitatively and quantitatively characterizing the memorized samples as well as the model behavior in their vicinity.Based on empirical observations, we develop a simple algorithm which elicits non-memorized translations of memorized samples from the same model, for a large fraction of such samples.Finally, we show that the proposed algorithm could also be leveraged to mitigate memorization in the model through finetuning.We have released the code to reproduce our results at https://github.com/vyraun/Finding-Memo.
Textual Backdoor Attacks Can Be More Harmful via Two Simple Tricks

Backdoor attacks are a kind of emergent security threat in deep learning.After being injected with a backdoor, a deep neural model will behave normally on standard inputs but give adversary-specified predictions once the input contains specific backdoor triggers.In this paper, we find two simple tricks that can make existing textual backdoor attacks much more harmful.The first trick is to add an extra training task to distinguish poisoned and clean data during the training of the victim model, and the second one is to use all the clean training data rather than remove the original clean data corresponding to the poisoned data.These two tricks are universally applicable to different attack models.We conduct experiments in three tough situations including clean data fine-tuning, low-poisoningrate, and label-consistent attacks.Experimental results show that the two tricks can significantly improve attack performance.This paper exhibits the great potential harmfulness of backdoor attacks.All the code and data can be obtained at https://github.com/thunlp/StyleAttack.
Active Learning for Abstractive Text Summarization

Construction of human-curated annotated datasets for abstractive text summarization (ATS) is very time-consuming and expensive because creating each instance requires a human annotator to read a long document and compose a shorter summary that would preserve the key information relayed by the original document.Active Learning (AL) is a technique developed to reduce the amount of annotation required to achieve a certain level of machine learning model performance.In information extraction and text classification, AL can reduce the amount of labor up to multiple times.Despite its potential for aiding expensive annotation, as far as we know, there were no effective AL query strategies for ATS.This stems from the fact that many AL strategies rely on uncertainty estimation, while as we show in our work, uncertain instances are usually noisy, and selecting them can degrade the model performance compared to passive annotation.We address this problem by proposing the first effective query strategy for AL in ATS based on diversity principles.We show that given a certain annotation budget, using our strategy in AL annotation helps to improve the model performance in terms of ROUGE and consistency scores.Additionally, we analyze the effect of self-learning and show that it can further increase the performance of the model.
Learning From the Source Document: Unsupervised Abstractive Summarization

Most of the state-of-the-art methods for abstractive text summarization are under supervised learning settings, while heavily relying on highquality and large-scale parallel corpora.In this paper, we remove the need for reference summaries and present an unsupervised learning method SCR (Summarize, Contrast and Review) for abstractive summarization, which leverages contrastive learning and is the first work to apply contrastive learning for unsupervised abstractive summarization.Particularly, we use the true source documents as positive source document examples, and strategically generated fake source documents as negative source document examples to train the model to generate good summaries.Furthermore, we consider and improve the writing quality of the generated summaries by guiding them to be similar to human-written texts.The promising results on extensive experiments show that SCR outperforms other unsupervised abstractive summarization baselines, which demonstrates its effectiveness.
Towards Teachable Reasoning Systems: Using a Dynamic Memory of User Feedback for Continual System Improvement

Our goal is a teachable reasoning system for question-answering (QA), where a user can interact with faithful answer explanations, and correct its errors so that the system improves over time.Our approach is to augment a QA model with a dynamic memory of user feedback, containing user-supplied corrections to erroneous model beliefs that users identify during interaction.Retrievals from memory are used as additional context for QA, to help avoid previous mistakes in similar new situationsa novel application of memory-based continuous learning.With simulated feedback, we find that our system (called TeachMe 1 ) continually improves with time, and without model retraining, requiring feedback on only 25% of training examples to reach within 1% of the upper-bound (feedback on all examples).Similarly, in experiments with real users, we observe a similar trend, with performance improving by over 15% on a hidden test set after teaching.This suggests new opportunities for using frozen language models in an interactive setting where users can inspect, debug, and correct the model's beliefs, leading to improved system's performance over time.
Prompt-based Connective Prediction Method for Fine-grained Implicit Discourse Relation Recognition

Due to the absence of connectives, implicit discourse relation recognition (IDRR) is still a challenging and crucial task in discourse analysis.Most of the current work adopted multitask learning to aid IDRR through explicit discourse relation recognition (EDRR) or utilized dependencies between discourse relation labels to constrain model predictions.But these methods still performed poorly on fine-grained IDRR and even utterly misidentified on most of the few-shot discourse relation classes.To address these problems, we propose a novel Prompt-based Connective Prediction (PCP) method for IDRR.Our method instructs largescale pre-trained models to use knowledge relevant to discourse relation and utilizes the strong correlation between connectives and discourse relation to help the model recognize implicit discourse relations.Experimental results show that our method surpasses the current state-ofthe-art model and achieves significant improvements on those fine-grained few-shot discourse relation.Moreover, our approach is able to be transferred to EDRR and obtain acceptable results.Our code is released in https:// github.com/zh-i9/PCP-for-IDRR.
Adaptive Token-level Cross-lingual Feature Mixing for Multilingual Neural Machine Translation

Multilingual neural machine translation aims to translate multiple language pairs in a single model and has shown great success thanks to the knowledge transfer across languages with the shared parameters.Despite promising, this share-all paradigm suffers from insufficient ability to capture language-specific features.Currently, the common practice is to insert or search language-specific networks to balance the shared and specific features.However, those two types of features are not sufficient enough to model the complex commonality and divergence across languages, such as the locally shared features among similar languages, which leads to sub-optimal transfer, especially in massively multilingual translation.In this paper, we propose a novel token-level feature mixing method that enables the model to capture different features and dynamically determine the feature sharing across languages.Based on the observation that the tokens in the multilingual model are usually shared by different languages, we insert a feature mixing layer into each Transformer sublayer and model each token representation as a mix of different features, with a proportion indicating its feature preference.In this way, we can perform fine-grained feature sharing and achieve better multilingual transfer.Experimental results on multilingual datasets show that our method outperforms various strong baselines and can be extended to zero-shot translation.Further analyses reveal that our method can capture different linguistic features and bridge the representation gap across languages.