Reviewerly/exHarmony huggingface dataset:
Multimodal Relation Extraction with Cross-Modal Retrieval and Synthesis

Multimodal relation extraction (MRE) is the task of identifying the semantic relationships between two entities based on the context of the sentence image pair.Existing retrievalaugmented approaches mainly focused on modeling the retrieved textual knowledge, but this may not be able to accurately identify complex relations.To improve the prediction, this research proposes to retrieve textual and visual evidence based on the object, sentence, and whole image.We further develop a novel approach to synthesize the object-level, imagelevel, and sentence-level information for better reasoning between the same and different modalities.Extensive experiments and analyses show that the proposed method is able to effectively select and compare evidence across modalities and significantly outperforms stateof-the-art models.Code and data are available 1 .
Easy Guided Decoding in Providing Suggestions for Interactive Machine Translation

Machine translation technology has made great progress in recent years, but it cannot guarantee error-free results.Human translators perform post-editing on machine translations to correct errors in the scene of computer-aided translation.In favor of expediting the post-editing process, many works have investigated machine translation in interactive modes, in which machines can automatically refine the rest of translations constrained by human's edits.Translation Suggestion (TS), as an interactive mode to assist human translators, requires machines to generate alternatives for specific incorrect words or phrases selected by human translators.In this paper, we utilize the parameterized objective function of neural machine translation (NMT) and propose a novel constrained decoding algorithm, namely Prefix-Suffix Guided Decoding (PSGD), to deal with the TS problem without additional training.Compared to the state-of-the-art lexically constrained decoding method, PSGD improves translation quality by an average of 10.87 BLEU and 8.62 BLEU on the WeTS 1 and the WMT 2022 Translation Suggestion datasets 2 , respectively, and reduces decoding time overhead by an average of 63.4% tested on the WMT translation datasets.Furthermore, on both of the TS benchmark datasets, it is superior to other supervised learning systems trained with TS annotated data.
Clustering-Aware Negative Sampling for Unsupervised Sentence Representation

Contrastive learning has been widely studied in sentence representation learning.However, earlier works mainly focus on the construction of positive examples, while in-batch samples are often simply treated as negative examples.This approach overlooks the importance of selecting appropriate negative examples, potentially leading to a scarcity of hard negatives and the inclusion of false negatives.To address these issues, we propose ClusterNS (Clusteringaware Negative Sampling), a novel method that incorporates cluster information into contrastive learning for unsupervised sentence representation learning.We apply a modified Kmeans clustering algorithm to supply hard negatives and recognize in-batch false negatives during training, aiming to solve the two issues in one unified framework.Experiments on semantic textual similarity (STS) tasks demonstrate that our proposed ClusterNS compares favorably with baselines in unsupervised sentence representation learning.Our code has been made publicly available.
A Unified Generative Approach to Product Attribute-Value Identification

Product attribute-value identification (PAVI) has been studied to link products on e-commerce sites with their attribute values (e.g., ⟨Material, Cotton⟩) using product text as clues.Technical demands from real-world e-commerce platforms require PAVI methods to handle unseen values, multi-attribute values, and canonicalized values, which are only partly addressed in existing extraction-and classification-based approaches.Motivated by this, we explore a generative approach to the PAVI task.We finetune a pre-trained generative model, T5, to decode a set of attribute-value pairs as a target sequence from the given product text.Since the attributevalue pairs are unordered set elements, how to linearize them will matter; we, thus, explore methods of composing an attribute-value pair and ordering the pairs for the task.Experimental results confirm that our generation-based approach outperforms the existing extractionand classification-based methods on large-scale real-world datasets meant for those methods.
Towards Understanding Omission in Dialogue Summarization

Dialogue summarization aims to condense the lengthy dialogue into a concise summary, and has recently achieved significant progress.However, the result of existing methods is still far from satisfactory.Previous works indicated that omission is a major factor in affecting the quality of summarization, but few of them have further explored the omission problem, such as how omission affects summarization results and how to detect omission, which is critical for reducing omission and improving summarization quality.Moreover, analyzing and detecting omission relies on summarization datasets with omission labels (i.e., which dialogue utterances are omitted in the summarization), which are not available in the current literature.In this paper, we propose the OLDS dataset, which provides high-quality Omission Labels for Dialogue Summarization.By analyzing this dataset, we find that a large improvement in summarization quality can be achieved by providing ground-truth omission labels for the summarization model to recover omission information, which demonstrates the importance of omission detection for omission mitigation in dialogue summarization.Therefore, we formulate an omission detection task and demonstrate our proposed dataset can support the training and evaluation of this task well.We also call for research action on omission detection based on our proposed datasets.Our dataset and codes are publicly available 1 .
Improved Visual Story Generation with Adaptive Context Modeling

Diffusion models developed on top of powerful text-to-image generation models like Stable Diffusion achieve remarkable success in visual story generation.However, the best-performing approach considers historically generated results as flattened memory cells, ignoring the fact that not all preceding images contribute equally to the generation of the characters and scenes at the current stage.To address this, we present a simple method that improves the leading system with adaptive context modeling, which is not only incorporated in the encoder but also adopted as additional guidance in the sampling stage to boost the global consistency of the generated story.We evaluate our model on PororoSV and FlintstonesSV datasets and show that our approach achieves state-of-theart FID scores on both story visualization and continuation scenarios.We conduct detailed model analysis and show that our model excels at generating semantically consistent images for stories.
Characterizing the Impacts of Instances on Robustness

Building robust deep neural networks (DNNs) against adversarial attacks is an important but challenging task.Previous defense approaches mainly focus on developing new model structures or training algorithms, but they do little to tap the potential of training instances, especially instances with robust patterns carring innate robustness.In this paper, we show that robust and non-robust instances in the training dataset, though are both important for test performance, have contrary impacts on robustness, which makes it possible to build a highly robust model by leveraging the training dataset in a more effective way.We propose a new method that can distinguish robust instances from nonrobust ones according to the model's sensitivity to perturbations on individual instances during training.Surprisingly, we find that the model under standard training easily overfits the robust instances by relying on their simple patterns before the model completely learns their robust features.Finally, we propose a new mitigation algorithm to further release the potential of robust instances.Experimental results show that proper use of robust instances in the original dataset is a new line to achieve highly robust models.Our codes are publicly available at https://github.com/ruizheng20/robust_data.
Learning “O” Helps for Learning More: Handling the Unlabeled Entity Problem for Class-incremental NER

As the categories of named entities rapidly increase, the deployed NER models are required to keep updating toward recognizing more entity types, creating a demand for classincremental learning for NER.Considering the privacy concerns and storage constraints, the standard paradigm for class-incremental NER updates the models with training data only annotated with the new classes, yet the entities from other entity classes are unlabeled, regarded as "Non-entity" (or "O").In this work, we conduct an empirical study on the "Unlabeled Entity Problem" and find that it leads to severe confusion between "O" and entities, decreasing class discrimination of old classes and declining the model's ability to learn new classes.To solve the Unlabeled Entity Problem, we propose a novel representation learning method to learn discriminative representations for the entity classes and "O".Specifically, we propose an entity-aware contrastive learning method that adaptively detects entity clusters in "O".Furthermore, we propose two effective distance-based relabeling strategies for better learning the old classes.We introduce a more realistic and challenging benchmark for classincremental NER, and the proposed method achieves up to 10.62% improvement over the baseline methods.
BITE: Textual Backdoor Attacks with Iterative Trigger Injection

Backdoor attacks have become an emerging threat to NLP systems.By providing poisoned training data, the adversary can embed a "backdoor" into the victim model, which allows input instances satisfying certain textual patterns (e.g., containing a keyword) to be predicted as a target label of the adversary's choice.In this paper, we demonstrate that it is possible to design a backdoor attack that is both stealthy (i.e., hard to notice) and effective (i.e., has a high attack success rate).We propose BITE, a backdoor attack that poisons the training data to establish strong correlations between the target label and a set of "trigger words".These trigger words are iteratively identified and injected into the target-label instances through natural word-level perturbations.The poisoned training data instruct the victim model to predict the target label on inputs containing trigger words, forming the backdoor.Experiments on four text classification datasets show that our proposed attack is significantly more effective than baseline methods while maintaining decent stealthiness, raising alarm on the usage of untrusted training data.We further propose a defense method named DeBITE based on potential trigger word removal, which outperforms existing methods in defending against BITE and generalizes well to handling other backdoor attacks. 1
K-UniMorph: Korean Universal Morphology and its Feature Schema

We present in this work a new Universal Morphology dataset for Korean.Previously, the Korean language has been underrepresented in the field of morphological paradigms amongst hundreds of diverse world languages.Hence, we propose this Universal Morphological paradigms for the Korean language that preserve its distinct characteristics.For our K-UniMorph dataset, we outline each grammatical criterion in detail for the verbal endings, clarify how to extract inflected forms, and demonstrate how we generate the morphological schemata.This dataset adopts morphological feature schema from Sylak-Glassman et al. ( 2015) and Sylak-Glassman (2016) for the Korean language as we extract inflected verb forms from the Sejong morphologically analyzed corpus that is one of the largest annotated corpora for Korean.During the data creation, our methodology also includes investigating the correctness of the conversion from the Sejong corpus.Furthermore, we carry out the inflection task using three different Korean word forms: letters, syllables and morphemes.Finally, we discuss and describe future perspectives on Korean morphological paradigms and the dataset.
Augmentation-Adapted Retriever Improves Generalization of Language Models as Generic Plug-In

Retrieval augmentation can aid language models (LMs) in knowledge-intensive tasks by supplying them with external information.Prior works on retrieval augmentation usually jointly fine-tune the retriever and the LM, making them closely coupled.In this paper, we explore the scheme of generic retrieval plug-in: the retriever is to assist target LMs that may not be known beforehand or are unable to be fine-tuned together.To retrieve useful documents for unseen target LMs, we propose augmentation-adapted retriever (AAR), which learns LM's preferences obtained from a known source LM.Experiments on the MMLU and PopQA datasets demonstrate that our AAR trained with a small source LM is able to significantly improve the zero-shot generalization of larger target LMs ranging from 250M Flan-T5 to 175B InstructGPT.Further analysis indicates that the preferences of different LMs overlap, enabling AAR trained with a single source LM to serve as a generic plug-in for various target LMs.Our code is open-sourced at https://github.com/OpenMatch/Augmentation-Adapted-Retriever.
SINAI at RadSum23: Radiology Report Summarization Based on Domain-Specific Sequence-To-Sequence Transformer Model

This paper covers participation of the SINAI team in the shared task 1B: Radiology Report Summarization at the BioNLP workshop held on ACL 2023.Our proposal follows a "sequence-to-sequence" approach which leverages pre-trained multilingual general domain and monolingual biomedical domain pretrained language models.The best performing system based on domain-specific model reached 33.96 F1RadGraph score which is the fourth best result among the challenge participants.This model was made publicly available on HuggingFace.We also describe an attempt of Proximal Policy Optimization Reinforcement Learning that was made in order to improve the factual correctness measured with F1RadGraph but did not lead to satisfactory results.
How Many Answers Should I Give? An Empirical Study of Multi-Answer Reading Comprehension

The multi-answer phenomenon, where a question may have multiple answers scattered in the document, can be well handled by humans but is challenging enough for machine reading comprehension (MRC) systems.Despite recent progress in multi-answer MRC, there lacks a systematic analysis of how this phenomenon arises and how to better address it.In this work, we design a taxonomy to categorize commonlyseen multi-answer MRC instances, with which we inspect three multi-answer datasets and analyze where the multi-answer challenge comes from.We further analyze how well different paradigms of current multi-answer MRC models deal with different types of multi-answer instances.We find that some paradigms capture well the key information in the questions while others better model the relationship between questions and contexts.We thus explore strategies to make the best of the strengths of different paradigms.Experiments show that generation models can be a promising platform to incorporate different paradigms.Our annotations and code are released for further research 1 .
KILM: Knowledge Injection into Encoder-Decoder Language Models

Large pre-trained language models (PLMs) have been shown to retain implicit knowledge within their parameters.To enhance this implicit knowledge, we propose Knowledge Injection into Language Models (KILM), a novel approach that injects entity-related knowledge into encoder-decoder PLMs, via a generative knowledge infilling objective through continued pre-training.This is done without architectural modifications to the PLMs or adding additional parameters.Experimental results over a suite of knowledge-intensive tasks spanning numerous datasets show that KILM enables models to retain more knowledge and hallucinate less while preserving their original performance on general NLU and NLG tasks.KILM also demonstrates improved zero-shot performances on tasks such as entity disambiguation, outperforming state-of-the-art models having 30x more parameters.
A Textual Dataset for Situated Proactive Response Selection

Recent data-driven conversational models are able to return fluent, consistent, and informative responses to many kinds of requests and utterances in task-oriented scenarios.However, these responses are typically limited to just the immediate local topic instead of being widerranging and proactively taking the conversation further, for example making suggestions to help customers achieve their goals.This inadequacy reflects a lack of understanding of the interlocutor's situation and implicit goal.To address the problem, we introduce a task of proactive response selection based on situational information.We present a manuallycurated dataset of 1.7k English conversation examples that include situational background information plus for each conversation a set of responses, only some of which are acceptable in the situation.A responsive and informed conversation system should select the appropriate responses and avoid inappropriate ones; doing so demonstrates the ability to adequately understand the initiating request and situation.Our benchmark experiments show that this is not an easy task even for strong neural models, offering opportunities for future research.
Aerial Vision-and-Dialog Navigation

The ability to converse with humans and follow natural language commands is crucial for intelligent unmanned aerial vehicles (a.k.a.drones).It can relieve people's burden of holding a controller all the time, allow multitasking, and make drone control more accessible for people with disabilities or with their hands occupied.To this end, we introduce Aerial Vision-and-Dialog Navigation (AVDN), to navigate a drone via natural language conversation.We build a drone simulator with a continuous photorealistic environment and collect a new AVDN dataset of over 3k recorded navigation trajectories with asynchronous human-human dialogs between commanders and followers.The commander provides initial navigation instruction and further guidance by request, while the follower navigates the drone in the simulator and asks questions when needed.During data collection, followers' attention on the drone's visual observation is also recorded.Based on the AVDN dataset, we study the tasks of aerial navigation from (full) dialog history and propose an effective Human Attention Aided Transformer model (HAA-Transformer), which learns to predict both navigation waypoints and human attention.Dataset and code are released:
Modeling Adversarial Attack on Pre-trained Language Models as Sequential Decision Making

Pre-trained language models (PLMs) have been widely used to underpin various downstream tasks.However, the adversarial attack task has found that PLMs are vulnerable to small perturbations.Mainstream methods adopt a detached two-stage framework to attack without considering the subsequent influence of substitution at each step.In this paper, we formally model the adversarial attack task on PLMs as a sequential decision-making problem, where the whole attack process is sequential with two decision-making problems, i.e., word finder and word substitution.Considering the attack process can only receive the final state without any direct intermediate signals, we propose to use reinforcement learning to find an appropriate sequential attack path to generate adversaries, named SDM-ATTACK.Extensive experimental results show that SDM-ATTACK achieves the highest attack success rate with a comparable modification rate and semantic similarity to attack fine-tuned BERT.Furthermore, our analyses demonstrate the generalization and transferability of SDM-ATTACK.
Character-Aware Models Improve Visual Text Rendering

Current image generation models struggle to reliably produce well-formed visual text.In this paper, we investigate a key contributing factor: popular text-to-image models lack character-level input features, making it much harder to predict a word's visual makeup as a series of glyphs.To quantify this effect, we conduct a series of experiments comparing character-aware vs. character-blind text encoders.In the text-only domain, we find that character-aware models provide large gains on a novel spelling task (WikiSpell).Applying our learnings to the visual domain, we train a suite of image generation models, and show that character-aware variants outperform their character-blind counterparts across a range of novel text rendering tasks (our DrawText benchmark).Our models set a much higher state-of-the-art on visual spelling, with 30+ point accuracy gains over competitors on rare words, despite training on far fewer examples.* Equal contribution. 1 https://huggingface.co/stabilityai/stable-diffusion-2-1
Improving Knowledge Graph Completion with Generative Hard Negative Mining

Contrastive learning has recently shown great potential to improve text-based knowledge graph completion (KGC).In this paper, we propose to learn a more semantically structured entity representation space in text-based KGC via hard negatives mining.Specifically, we novelly leverage a sequence-to-sequence architecture to generate high-quality hard negatives.These negatives are sampled from the same decoding distributions as the anchor (or correct entity), inherently being semantically close to the anchor and thus enjoying good hardness.A self-information-enhanced contrasting strategy is further incorporated into the Seq2Seq generator to systematically diversify the produced negatives.Extensive experiments on three KGC benchmarks demonstrate the sound hardness and diversity of our generated negatives and the resulting performance superiority on KGC.
On the Blind Spots of Model-Based Evaluation Metrics for Text Generation

In this work, we explore a useful but often neglected methodology for robustness analysis of text generation evaluation metrics: stress tests with synthetic data.Basically, we design and synthesize a wide range of potential errors and check whether they result in a commensurate drop in the metric scores.We examine a range of recently proposed evaluation metrics based on pretrained language models, for the tasks of open-ended generation, translation, and summarization.Our experiments reveal interesting insensitivities, biases, or even loopholes in existing metrics.For example, we find that BERTScore is confused by truncation errors in summarization, and MAUVE (built on top of GPT-2) is insensitive to errors at the beginning or middle of generations.Further, we investigate the reasons behind these blind spots and suggest practical workarounds for a more reliable evaluation of text generation.We have released our code and data at https://github.com/cloudygoose/blindspot_nlg.