Reviewerly/exHarmony huggingface dataset:
Robust Multi-bit Natural Language Watermarking through Invariant Features

Recent years have witnessed a proliferation of valuable original natural language contents found in subscription-based media outlets, web novel platforms, and outputs of large language models.However, these contents are susceptible to illegal piracy and potential misuse without proper security measures.This calls for a secure watermarking system to guarantee copyright protection through leakage tracing or ownership identification.To effectively combat piracy and protect copyrights, a multi-bit watermarking framework should be able to embed adequate bits of information and extract the watermarks in a robust manner despite possible corruption.In this work, we explore ways to advance both payload and robustness by following a well-known proposition from image watermarking and identify features in natural language that are invariant to minor corruption.Through a systematic analysis of the possible sources of errors, we further propose a corruption-resistant infill model.Our full method improves upon the previous work on robustness by +16.8% point on average on four datasets, three corruption types, and two corruption ratios. 1
HyperPELT: Unified Parameter-Efficient Language Model Tuning for Both Language and Vision-and-Language Tasks

With the scale and capacity of pretrained models growing rapidly, parameter-efficient language model tuning has emerged as a popular paradigm for solving various NLP and Visionand-Language (V&L) tasks.In this paper, we design a unified parameter-efficient multitask learning framework that works effectively on both NLP and V&L tasks.In particular, we use a shared hypernetwork that takes trainable hyper-embeddings and visual modality as input, and outputs weights for different modules in a pretrained language model, such as the parameters inserted into multi-head attention blocks (i.e., prefix-tuning) and feed-forward blocks (i.e., adapter-tuning.).Our proposed framework adds fewer trainable parameters in multi-task learning while achieving superior performances and transfer ability compared to state-of-the-art methods.Empirical results on the GLUE benchmark and multiple V&L tasks confirm the effectiveness of our framework.
COST-EFF: Collaborative Optimization of Spatial and Temporal Efficiency with Slenderized Multi-exit Language Models

Transformer-based pre-trained language models (PLMs) mostly suffer from excessive overhead despite their advanced capacity.For resource-constrained devices, there is an urgent need for a spatially and temporally efficient model which retains the major capacity of PLMs.However, existing statically compressed models are unaware of the diverse complexities between input instances, potentially resulting in redundancy and inadequacy for simple and complex inputs.Also, miniature models with early exiting encounter challenges in the trade-off between making predictions and serving the deeper layers.Motivated by such considerations, we propose a collaborative optimization for PLMs that integrates static model compression and dynamic inference acceleration.Specifically, the PLM is slenderized in width while the depth remains intact, complementing layer-wise early exiting to speed up inference dynamically.To address the trade-off of early exiting, we propose a joint training approach that calibrates slenderization and preserves contributive structures to each exit instead of only the final layer.Experiments are conducted on GLUE benchmark and the results verify the Pareto optimality of our approach at high compression and acceleration rate with 1/8 parameters and 1/19 FLOPs of BERT.
Understanding Jargon: Combining Extraction and Generation for Definition Modeling

Can machines know what twin prime is?From the composition of this phrase, machines may guess twin prime is a certain kind of prime, but it is still difficult to deduce exactly what twin stands for without additional knowledge.Here, twin prime is a jargon-a specialized term used by experts in a particular field.Explaining jargon is challenging since it usually requires domain knowledge to understand.Recently, there is an increasing interest in extracting and generating definitions of words automatically.However, existing approaches, either extraction or generation, perform poorly on jargon.In this paper, we propose to combine extraction and generation for jargon definition modeling: first extract self-and correlative definitional information of target jargon from the Web and then generate the final definitions by incorporating the extracted definitional information.Our framework is remarkably simple but effective: experiments demonstrate our method can generate high-quality definitions for jargon and outperform state-of-the-art models significantly, e.g., BLEU score from 8.76 to 22.66 and human-annotated score from 2.34 to 4.04. 1
Context-Situated Pun Generation

Previous work on pun generation commonly begins with a given pun word (a pair of homophones for heterographic pun generation and a polyseme for homographic pun generation) and seeks to generate an appropriate pun.While this may enable efficient pun generation, we believe that a pun is most entertaining if it fits appropriately within a given context, e.g., a given situation or dialogue.In this work, we propose a new task, context-situated pun generation, where a specific context represented by a set of keywords is provided, and the task is to first identify suitable pun words that are appropriate for the context, then generate puns based on the context keywords and the identified pun words.We collect CUP (Context-sitUated Pun), containing 4.5k tuples of context words and pun pairs.Based on the new data and setup, we propose a pipeline system for context-situated pun generation, including a pun word retrieval module that identifies suitable pun words for a given context, and a generation module that generates puns from context keywords and pun words.Human evaluation shows that 69% of our top retrieved pun words can be used to generate context-situated puns, and our generation module yields successful puns 31% of the time given a plausible tuple of context words and pun pair, almost tripling the yield of a state-of-the-art pun generation model.With an end-to-end evaluation, our pipeline system with the top-1 retrieved pun pair for a given context can generate successful puns 40% of the time, better than all other modeling variations but 32% lower than the human success rate.This highlights the difficulty of the task, and encourages more research in this direction.
TRIPS: Efficient Vision-and-Language Pre-training with Text-Relevant Image Patch Selection

Vision Transformers (ViTs) have been widely used in large-scale Vision and Language Pretraining (VLP) models.Though previous VLP works have proved the effectiveness of ViTs, they still suffer from computational efficiency brought by the long visual sequence.To tackle this problem, in this paper, we propose an efficient vision-and-language pre-training model with Text-Relevant Image Patch Selection, namely TRIPS, which reduces the visual sequence progressively with a text-guided patchselection layer in the visual backbone for efficient training and inference.The patchselection layer can dynamically compute textdependent visual attention to identify the attentive image tokens with text guidance and fuse inattentive ones in an end-to-end manner.Meanwhile, TRIPS does not introduce extra parameters to ViTs.Experimental results on a variety of popular benchmark datasets demonstrate that TRIPS gain a speedup of 40% over previous similar VLP models, yet with competitive or better downstream task performance.
Generating Information-Seeking Conversations from Unlabeled Documents

Synthesizing datasets for conversational question answering (CQA) from unlabeled documents remains challenging due to its interactive nature.Moreover, while modeling information needs is an essential key, only few studies have discussed it.In this paper, we introduce a novel framework, SIMSEEK, (Simulating information-Seeking conversation from unlabeled documents), and compare its two variants.In our baseline SIMSEEK-SYM, a questioner generates follow-up questions upon the predetermined answer by an answerer.On the contrary, SIMSEEK-ASYM first generates the question and then finds its corresponding answer under the conversational context.Our experiments show that they can synthesize effective training resources for CQA and conversational search tasks.As a result, conversations from SIMSEEK-ASYM not only make more improvements in our experiments but also are favorably reviewed in a human evaluation.We finally release a large-scale resource of synthetic conversations, WIKI-SIMSEEK, containing 2 million CQA pairs built upon Wikipedia documents.With the dataset, our CQA model achieves the state-of-the-art performance on a recent CQA benchmark, QuAC (Choi et al., 2018) 1 .Prior-grounded CQG !"#$% & | ℬ , '& Conversational Answer Finder " "#$% & | & , , '& (b) SimSeek-asym (a) SimSeek-sym Conversational
MedConQA: Medical Conversational Question Answering System based on Knowledge Graphs

The medical conversational system can relieve doctors’ burden and improve healthcare efficiency, especially during the COVID-19 pandemic. However, the existing medical dialogue systems have the problems of weak scalability, insufficient knowledge, and poor controllability. Thus, we propose a medical conversational question-answering (CQA) system based on the knowledge graph, namely MedConQA, which is designed as a pipeline framework to maintain high flexibility. Our system utilizes automated medical procedures, including medical triage, consultation, image-text drug recommendation, and record. Each module has been open-sourced as a tool, which can be used alone or in combination, with robust scalability. Besides, to conduct knowledge-grounded dialogues with users, we first construct a Chinese Medical Knowledge Graph (CMKG) and collect a large-scale Chinese Medical CQA (CMCQA) dataset, and we design a series of methods for reasoning more intellectually. Finally, we use several state-of-the-art (SOTA) techniques to keep the final generated response more controllable, which is further assured by hospital and professional evaluations. We have open-sourced related code, datasets, web pages, and tools, hoping to advance future research.
CogKTR: A Knowledge-Enhanced Text Representation Toolkit for Natural Language Understanding

As the first step of modern natural language processing, text representation encodes discrete texts as continuous embeddings.Pre-trained language models (PLMs) have demonstrated strong ability in text representation and significantly promoted the development of natural language understanding (NLU).However, existing PLMs represent a text solely by its context, which is not enough to support knowledgeintensive NLU tasks.Knowledge is power, and fusing external knowledge explicitly into PLMs can provide knowledgeable text representations.Since previous knowledge-enhanced methods differ in many aspects, making it difficult for us to reproduce previous methods, implement new methods, and transfer between different methods.It is highly desirable to have a unified paradigm to encompass all kinds of methods in one framework.In this paper, we propose , a knowledgeenhanced text representation toolkit for natural language understanding.According to our proposed Unified Knowledge-Enhanced Paradigm (UniKEP), CogKTR consists of four key stages, including knowledge acquisition, knowledge representation, knowledge injection, and knowledge application.CogKTR currently supports easy-to-use knowledge acquisition interfaces, multi-source knowledge embeddings, diverse knowledge-enhanced models, and various knowledge-intensive NLU tasks.Our unified, knowledgeable and modular toolkit is publicly available at GitHub 1 , with an online system 2 and a short instruction video 3 .
ExPUNations: Augmenting Puns with Keywords and Explanations

The tasks of humor understanding and generation are challenging and subjective even for humans, requiring commonsense and realworld knowledge to master.Puns, in particular, add the challenge of fusing that knowledge with the ability to interpret lexical-semantic ambiguity.In this paper, we present the Ex-PUNations (ExPUN) dataset, in which we augment an existing dataset of puns with detailed crowdsourced annotations of keywords denoting the most distinctive words that make the text funny, pun explanations describing why the text is funny, and fine-grained funniness ratings.This is the first humor dataset with such extensive and fine-grained annotations specifically for puns.Based on these annotations, we propose two tasks: explanation generation to aid with pun classification and keywordconditioned pun generation, to challenge the current state-of-the-art natural language understanding and generation models' ability to understand and generate humor.We showcase that the annotated keywords we collect are helpful for generating better novel humorous texts in human evaluation, and that our natural language explanations can be leveraged to improve both the accuracy and robustness of humor classifiers.
R2F: A General Retrieval, Reading and Fusion Framework for Document-level Natural Language Inference

Document-level natural language inference (DOCNLI) is a new challenging task in natural language processing, aiming at judging the entailment relationship between a pair of hypothesis and premise documents.Current datasets and baselines largely follow sentence-level settings, but fail to address the issues raised by longer documents.In this paper, we establish a general solution, named Retrieval, Reading and Fusion (R 2 F) framework, and a new setting, by analyzing the main challenges of DOCNLI: interpretability, long-range dependency, and cross-sentence inference.The basic idea of the framework is to simplify documentlevel task into a set of sentence-level tasks, and improve both performance and interpretability with the power of evidence.For each hypothesis sentence, the framework retrieves evidence sentences from the premise, and reads to estimate its credibility.Then the sentencelevel results are fused to judge the relationship between the documents.For the setting, we contribute complementary evidence and entailment label annotation on hypothesis sentences, for interpretability study.Our experimental results show that R 2 F framework can obtain state-of-the-art performance and is robust for diverse evidence retrieval methods.Moreover, it can give more interpretable prediction results.Our model and code are released at https://github.com/phoenixsecularbird/R2F.
Do Text-to-Text Multi-Task Learners Suffer from Task Conflict?

Traditional multi-task learning architectures learn a single model across multiple tasks through a shared encoder followed by taskspecific decoders.Learning these models often requires specialized training algorithms that address task-conflict in the shared parameter updates, which otherwise can lead to negative transfer.A new type of multi-task learning within NLP homogenizes multi-task architectures as a shared encoder and language model decoder, which does surprisingly well across a range of diverse tasks (Raffel et al., 2020).Does this new architecture suffer from taskconflicts that require specialized training algorithms?We study how certain factors in the shift towards text-to-text models affects multitask conflict and negative transfer, finding that both directional conflict and transfer are surprisingly constant across architectures.
Non-Autoregressive Models for Fast Sequence Generation

Autoregressive (AR) models have achieved great success in various sequence generation tasks. However, AR models can only generate target sequence word-by-word due to the AR mechanism and hence suffer from slow inference. Recently, non-autoregressive (NAR) models, which generate all the tokens in parallel by removing the sequential dependencies within the target sequence, have received increasing attention in sequence generation tasks such as neural machine translation (NMT), automatic speech recognition (ASR), and text to speech (TTS). In this tutorial, we will provide a comprehensive introduction to non-autoregressive sequence generation.
Exploring Compositional Image Retrieval with Hybrid Compositional Learning and Heuristic Negative Mining

Compositional image retrieval (CIR) is a challenging retrieval task, where the query is composed of a reference image and a modification text, and the target is another image reflecting the modification to the reference image.Due to the great success of the pre-trained vision-andlanguage model CLIP and its favorable applicability to large-scale retrieval tasks, we propose a CIR model HyCoLe-HNM with CLIP as the backbone.In HyCoLe-HNM, we follow the contrastive pre-training method of CLIP to perform cross-modal representation learning.On this basis, we propose a hybrid compositional learning mechanism, which includes both image compositional learning and text compositional learning.In hybrid compositional learning, we borrow a gated fusion mechanism from a question answering model to perform compositional fusion, and propose a heuristic negative mining method to filter negative samples.Privileged information in the form of image-related texts is utilized in cross-modal representation learning and hybrid compositional learning.Experimental results show that HyCoLe-HNM achieves state-of-the-art performance on three CIR datasets, namely FashionIQ, Fashion200K, and MIT-States.
Knowledge-grounded Dialog State Tracking

Knowledge (including structured knowledge such as schema and ontology, and unstructured knowledge such as web corpus) is a critical part of dialog understanding, especially for unseen tasks and domains.Traditionally, such domain-specific knowledge is encoded implicitly into model parameters for the execution of downstream tasks, which makes training inefficient.In addition, such models are not easily transferable to new tasks with different schemas.In this work, we propose to perform dialog state tracking grounded on knowledge encoded externally.We query relevant knowledge of various forms based on the dialog context where such information can ground the prediction of dialog states.We demonstrate superior performance of our proposed method over strong baselines, especially in the fewshot learning setting.
Improving English-Arabic Transliteration with Phonemic Memories

Transliteration is an important task in natural language processing (NLP) which aims to convert a name in the source language to the target language without changing its pronunciation.Particularly, transliteration from English to Arabic is highly needed in many applications, especially in countries (e.g., United Arab Emirates (UAE)) whose most citizens are foreigners but the official language is Arabic.In such a task-oriented scenario, namely transliterating the English names to the corresponding Arabic ones, the performance of the transliteration model is highly important.However, most existing neural approaches mainly apply a universal transliteration model with advanced encoders and decoders to the task, where limited attention is paid to leveraging the phonemic association between English and Arabic to further improve model performance.In this paper, we focus on transliteration of people's names from English to Arabic for the general public.In doing so, we collect a corpus named EANames by extracting high quality name pairs from online resources which better represent the names in the general public than linked Wikipedia entries that are always names of famous people).We propose a model for English-Arabic transliteration, where a memory module modeling the phonemic association between English and Arabic is used to guide the transliteration process.We run experiments on the collected data and the results demonstrate the effectiveness of our approach for English-Arabic transliteration. 1
MCP: Self-supervised Pre-training for Personalized Chatbots with Multi-level Contrastive Sampling

Personalized chatbots focus on endowing the chatbots with a consistent personality to behave like real users and further act as personal assistants.Previous studies have explored generating implicit user profiles from the user's dialogue history for building personalized chatbots.However, these studies only use the response generation loss to train the entire model, thus it is prone to suffer from the problem of data sparsity.Besides, they overemphasize the final generated response's quality while ignoring the correlations and fusions between the user's dialogue history, leading to rough data representations and performance degradation.To tackle these problems, we propose a selfsupervised learning framework MCP for capturing better representations from users' dialogue history for personalized chatbots.Specifically, we apply contrastive sampling methods to leverage the supervised signals hidden in user dialog history, and generate the pretraining samples for enhancing the model.We design three pre-training tasks based on three types of contrastive pairs from user dialogue history, namely response pairs, sequence augmentation pairs, and user pairs.We pre-train the utterance encoder and the history encoder towards the contrastive objectives and use these pre-trained encoders for generating user profiles while personalized response generation.Experimental results on two real-world datasets show a significant improvement in our proposed model MCP compared with the existing methods.
MovieUN: A Dataset for Movie Understanding and Narrating

Automatic movie narration generation and narration grounding are very important to provide a true movie experience for the blind and visually impaired.To tell the movie story well, it is necessary to mention plot-related details (such as character names) and keep the narrations in a plot coherent.Taking these two points into consideration, we construct a Chinese large-scale video benchmark from 101 movies for Movie Understanding and Narrating (MovieUN) to support the Movie Clip Narrating (MCN) task and Temporal Narration Grounding (TNG) task.We split movies in MovieUN into movie clips according to plots, and pair them with corresponding narrations provided by the movie narrators.Ultimately, the TNG task involves 3,253 long video clips totaling 179 hours.The MCN task contains 33,060 video clips totaling 105 hours.We benchmark state-of-the-art video captioning models and temporal grounding models in MCN and TNG tasks, respectively.Furthermore, to accurately comprehend plots of different characters, we propose methods to incorporate portraits of actors as external knowledge in both tasks.The experiment results demonstrate the effectiveness of our proposed methods.The dataset and codes are released at https:// github.com/yuezih/MovieUN.
Snapshot-Guided Domain Adaptation for ELECTRA

Discriminative pre-trained language models, such as ELECTRA, have achieved promising performances in a variety of general tasks.However, these generic pre-trained models struggle to capture domain-specific knowledge of domain-related tasks.In this work, we propose a novel domain-adaptation method for ELECTRA, which can dynamically select domain-specific tokens and guide the discriminator to emphasize them, without introducing new training parameters.We show that by reweighting the losses of domain-specific tokens, ELECTRA can be effectively adapted to different domains.The experimental results in both computer science and biomedical domains show that the proposed method can achieve state-of-the-art results on the domain-related tasks.