Reviewerly/exHarmony huggingface dataset:
Getting the Most out of Simile Recognition

Simile recognition involves two subtasks: simile sentence classification that discriminates whether a sentence contains simile, and simile component extraction that locates the corresponding objects (i.e., tenors and vehicles).Recent work ignores features other than surface strings.In this paper, we explore expressive features for this task to achieve more effective data utilization.Particularly, we study two types of features: 1) input-side features that include POS tags, dependency trees and word definitions, and 2) decoding features that capture the interdependence among various decoding decisions.We further construct a model named HGSR, which merges the input-side features as a heterogeneous graph and leverages decoding features via distillation.Experiments show that HGSR significantly outperforms the current state-of-the-art systems and carefully designed baselines, verifying the effectiveness of introduced features.Our code is available at https://github.com/DeepLearnXMU/HGSR.
Controllable Text Generation for All Ages: Evaluating a Plug-and-Play Approach to Age-Adapted Dialogue

To be trusted and perceived as natural and coherent, conversational systems must adapt to the language of their users.While personalized dialogue is a promising direction, controlling generation for fine-grained language features remains a challenge in this approach.A recent line of research showed the effectiveness of leveraging pre-trained language models toward adapting to a text's topic or sentiment.In this study, we build on these approaches and focus on a higher-level dimension of language variation: speakers' age.We frame the task as a dialogue response generation, and test methods based on bag-of-words (BoW) and neural discriminators (Disc) to condition the output of GPT-2 and DialoGPT without altering the parameters of the language models.We show that Disc models achieve a higher degree of detectable control than BoW models based on automatic evaluation.In contrast, humans can partially detect age differences in BoW but not Disc responses.Since BoW responses are deemed better than Disc ones by humans, simple controllable methods thus appear to be a better tradeoff between adaptation and language quality.Our work confirms the challenges of adapting to higherlevel dimensions of language variation.Moreover, it highlights the need to evaluate natural language generation thoroughly.
A Federated Approach to Predicting Emojis in Hindi Tweets

The use of emojis affords a visual modality to, often private, textual communication.The task of predicting emojis however provides a challenge for machine learning as emoji use tends to cluster into the frequently used and the rarely used emojis.Much of the machine learning research on emoji use has focused on high resource languages and has conceptualised the task of predicting emojis around traditional server-side machine learning approaches.However, traditional machine learning approaches for private communication can introduce privacy concerns, as these approaches require all data to be transmitted to a central storage.In this paper, we seek to address the dual concerns of emphasising high resource languages for emoji prediction and risking the privacy of people's data.We introduce a new dataset of 118k tweets (augmented from 25k unique tweets) for emoji prediction in Hindi, 1 and propose a modification to the federated learning algorithm, CausalFedGSD, which aims to strike a balance between model performance and user privacy.We show that our approach obtains comparative scores with more complex centralised models while reducing the amount of data required to optimise the models and minimising risks to user privacy.
Pseudo-Relevance for Enhancing Document Representation

This paper studies how to enhance the document representation for the bi-encoder approach in dense document retrieval.The biencoder, separately encoding a query and a document as a single vector, is favored for high efficiency in large-scale information retrieval, compared to more effective but complex architectures.To combine the strength of the two, the multi-vector representation of documents for bi-encoder, such as ColBERT preserving all token embeddings, has been widely adopted.Our contribution is to reduce the size of the multi-vector representation, without compromising the effectiveness, supervised by query logs.Our proposed solution decreases the latency and the memory footprint, up to 8-and 3fold, validated on MSMARCO and real-world search query logs 1 .
Eliciting and Understanding Cross-task Skills with Task-level Mixture-of-Experts

Recent works suggest that transformer models are capable of multi-tasking on diverse NLP tasks and adapting to new tasks efficiently.However, the potential of these multi-task models may be limited as they use the same set of parameters for all tasks.In contrast, humans tackle tasks in a more flexible way, by making proper presumptions on what skills and knowledge are relevant and executing only the necessary computations.Inspired by this, we propose to use task-level mixture-of-expert models, which has a collection of transformer layers (i.e., experts) and a router component that chooses from these experts dynamically and flexibly.We find that these models help improve the average performance gain (ARG) metric by 2.6% when adapting to unseen tasks in the few-shot setting and by 5.6% in the zeroshot generalization setting.Further, we show that the learned routing decisions partly rediscover human categorization of NLP tasks -certain experts are strongly associated with extractive tasks, some with classification tasks, and some with tasks requiring world knowledge.
Tackling Temporal Questions in Natural Language Interface to Databases

Temporal aspect is one of the most challenging areas in Natural Language Interface to Databases (NLIDB).This paper addresses and examines how temporal questions being studied and supported by the research community at both levels: popular annotated dataset (e.g.Spider) and recent advanced models.We present a new dataset with accompanied databases supporting temporal questions in NLIDB.We experiment with two SOTA models (Picard and ValueNet) to investigate how our new dataset helps these models learn and improve performance in temporal aspect.
BELA: Bot for English Language Acquisition

In this paper, we introduce a conversational agent (chatbot) for Hindi-speaking youth called BELA-Bot for English Language Acquisition.Developed for the young underprivileged students at an Indian non-profit 1 , the agent supports both Hindi and Hinglish (code-switched Hindi and English, written primarily with English orthography) utterances.BELA has two interaction modes: a question-answering mode for classic English language learning tasks like word meanings, translations, reading passage comprehensions, etc., and an open-domain dialogue system mode to allow users to practice their language skills.We present a high-level overview of the design of BELA, including the implementation details and the preliminary results of our early prototype.We also report the challenges in creating an English-language learning chatbot for a largely Hindi-speaking population.
SEMGraph: Incorporating Sentiment Knowledge and Eye Movement into Graph Model for Sentiment Analysis

This paper investigates the sentiment analysis task from a novel perspective by incorporating sentiment knowledge and eye movement into a graph architecture, aiming to draw the eye movement-based sentiment relationships for learning the sentiment expression of the context.To be specific, we first explore a linguistic probing eye movement paradigm to extract eye movement features based on the close relationship between linguistic features and the early and late processes of human reading behavior.Furthermore, to derive eye movement features with sentiment concepts, we devise a novel weighting strategy to integrate sentiment scores extracted from affective commonsense knowledge into eye movement features, called sentiment-eye movement weights.Then, the sentiment-eye movement weights are exploited to build the sentiment-eye movement guided graph (SEMGraph) model, so as to model the intricate sentiment relationships in the context.Experimental results on two sentiment analysis datasets with eye movement signals and three sentiment analysis datasets without eye movement signals show that the proposed SEM-Graph achieves state-of-the-art performance, and can also be directly generalized to those sentiment analysis datasets without eye movement signals.
CodeExp: Explanatory Code Document Generation

Developing models that can automatically generate detailed code explanation can greatly benefit software maintenance and programming education.However, existing code-totext generation models often produce only high-level summaries of code that do not capture implementation-level choices essential for these scenarios.To fill in this gap, we propose the code explanation generation task.We first conducted a human study to identify the criteria for high-quality explanatory docstring for code.Based on that, we collected and refined a largescale code docstring corpus and formulated automatic evaluation metrics that best match human assessments.Finally, we present a multistage fine-tuning strategy and baseline models for the task.Our experiments show that (1) our refined training dataset lets models achieve better performance in the explanation generation tasks compared to larger unrefined data (15× larger), and (2) fine-tuned models can generate well-structured long docstrings comparable to human-written ones.We envision our training dataset, human-evaluation protocol, recommended metrics, and fine-tuning strategy can boost future code explanation research.
Unsupervised Non-transferable Text Classification

Training a good deep learning model requires substantial data and computing resources, which makes the resulting neural model a valuable intellectual property.To prevent the neural network from being undesirably exploited, non-transferable learning has been proposed to reduce the model generalization ability in specific target domains.However, existing approaches require labeled data for the target domain which can be difficult to obtain.Furthermore, they do not have the mechanism to still recover the model's ability to access the target domain.In this paper, we propose a novel unsupervised non-transferable learning method for the text classification task that does not require annotated target domain data.We further introduce a secret key component in our approach for recovering the access to the target domain, where we design both an explicit and an implicit method for doing so.Extensive experiments demonstrate the effectiveness of our approach.
Task-driven augmented data evaluation

The main focus of data augmentation research has been on the enhancement of generation models, leaving the examination and improvements of synthetic data evaluation methods less explored.In our work, we explore a number of sentence similarity measures in the context of data generation filtering, and evaluate their impact on the performance of the targeted Natural Language Understanding problem for the example of intent classification and named entity recognition tasks.Our experiments on ATIS dataset show that the right choice of filtering technique can bring up to 33% in sentence accuracy improvement for targeted underrepresented intents.
Exploring Robustness of Prefix Tuning in Noisy Data: A Case Study in Financial Sentiment Analysis

The invention of transformer-based models such as BERT, GPT, and RoBERTa has enabled researchers and financial companies to finetune these powerful models and use them in different downstream tasks to achieve state-of-the-art performance.Recently, a lightweight alternative (approximately 0.1% -3% of the original model parameters) to fine-tuning, known as prefix tuning has been introduced.This method freezes the model parameters and only updates the prefix to achieve performance comparable to full fine-tuning.Prefix tuning enables researchers and financial practitioners to achieve similar results with much fewer parameters.In this paper, we explore the robustness of prefix tuning when facing noisy data.Our experiments demonstrate that fine-tuning is more robust to noise than prefix tuning-the latter method faces a significant decrease in performance on most corrupted data sets with increasing noise levels.Furthermore, prefix tuning has high variances on the F1 scores compared to fine-tuning in many corruption methods.We strongly advocate that caution should be carefully taken when applying the state-of-the-art prefix tuning method to noisy data.
Probing Script Knowledge from Pre-Trained Models

Script knowledge is critical for humans to understand the broad daily tasks and routine activities in the world.Recently researchers have explored the large-scale pre-trained language models (PLMs) to perform various script related tasks, such as story generation, temporal ordering of event, future event prediction and so on.However, it's still not well studied in terms of how well the PLMs capture the script knowledge.To answer this question, we design three probing tasks: inclusive sub-event selection, starting sub-event selection and temporal ordering to investigate the capabilities of PLMs with and without fine-tuning.The three probing tasks can be further used to automatically induce a script for each main event given all the possible sub-events.Taking BERT as a case study, by analyzing its performance on script induction as well as each individual probing task, we conclude that the stereotypical temporal knowledge among the sub-events is well captured in BERT, however the inclusive or starting sub-event knowledge is barely encoded.
Reflect, Not Reflex: Inference-Based Common Ground Improves Dialogue Response Quality

Human communication relies on common ground (CG), the mutual knowledge and beliefs shared by participants, to produce coherent and interesting conversations.In this paper, we demonstrate that current response generation (RG) models produce generic and dull responses in dialogues because they act reflexively, failing to explicitly model CG, both due to the lack of CG in training data and the standard RG training procedure.We introduce Reflect, a dataset that annotates dialogues with explicit CG (materialized as inferences approximating shared knowledge and beliefs) and solicits 9k diverse human-generated responses each following one common ground.Using Reflect, we showcase the limitations of current dialogue data and RG models: less than half of the responses in current data is rated as high quality (sensible, specific, and interesting) and models trained using this data have even lower quality, while most Reflect responses are judged high quality.Next, we analyze whether CG can help models produce better quality responses by using Reflect CG to guide RG models.Surprisingly, we find that simply prompting GPT3 to "think" about CG generates 30% more quality responses, showing promising benefits to integrating CG into the RG process. 1
Pre-training Transformer Models with Sentence-Level Objectives for Answer Sentence Selection

An important task for designing QA systems is answer sentence selection (AS2): selecting the sentence containing (or constituting) the answer to a question from a set of retrieved relevant documents.In this paper, we propose three novel sentence-level transformer pre-training objectives that incorporate paragraph-level semantics within and across documents, to improve the performance of transformers for AS2, and mitigate the requirement of large labeled datasets.Specifically, the model is tasked to predict whether: (i) two sentences are extracted from the same paragraph, (ii) a given sentence is extracted from a given paragraph, and (iii) two paragraphs are extracted from the same document.Our experiments on three public and one industrial AS2 datasets demonstrate the empirical superiority of our pre-trained transformers over baseline models such as RoBERTa and ELECTRA for AS2.
Robust Question Answering against Distribution Shifts with Test-Time Adaption: An Empirical Study

A deployed question answering (QA) model can easily fail when the test data has a distribution shift compared to the training data.Robustness tuning (RT) methods have been widely studied to enhance model robustness against distribution shifts before model deployment.However, can we improve a model after deployment?To answer this question, we evaluate test-time adaptation (TTA) to improve a model after deployment.We first introduce COLDQA, a unified evaluation benchmark for robust QA against text corruption and changes in language and domain.We then evaluate previous TTA methods on COLDQA and compare them to RT methods.We also propose a novel TTA method called online imitation learning (OIL).Through extensive experiments, we find that TTA is comparable to RT methods, and applying TTA after RT can significantly boost the performance on COLDQA.Our proposed OIL improves TTA to be more robust to variation in hyper-parameters and test distributions over time 1 .
Differentiable Data Augmentation for Contrastive Sentence Representation Learning

Fine-tuning a pre-trained language model via the contrastive learning framework with a large amount of unlabeled sentences or labeled sentence pairs is a common way to obtain highquality sentence representations.Although the contrastive learning framework has shown its superiority on sentence representation learning over previous methods, the potential of such a framework is under-explored so far due to the simple method it used to construct positive pairs.Motivated by this, we propose a method that makes hard positives from the original training examples.A pivotal ingredient of our approach is the use of prefix that is attached to a pre-trained language model, which allows for differentiable data augmentation during contrastive learning.Our method can be summarized in two steps: supervised prefixtuning followed by joint contrastive fine-tuning with unlabeled or labeled examples.Our experiments confirm the effectiveness of our data augmentation approach.The proposed method yields significant improvements over existing methods under both semi-supervised and supervised settings.Our experiments under a low labeled data setting also show that our method is more label-efficient than the state-of-the-art contrastive learning methods. 1
WSpeller: Robust Word Segmentation for Enhancing Chinese Spelling Check

Chinese spelling check (CSC) detects and corrects spelling errors in Chinese texts.Previous approaches have combined character-level phonetic and graphic information, ignoring the importance of segment-level information.According to our pilot study, spelling errors are always associated with incorrect word segmentation.When appropriate word boundaries are provided, CSC performance is greatly enhanced.Based on these findings, we present WSpeller, a CSC model that takes into account word segmentation.A fundamental component of WSpeller is a W-MLM, which is trained by predicting visually and phonetically similar words.Through modification of the embedding layer's input, word segmentation information can be incorporated.Additionally, a robust module is trained to assist the W-MLM-based correction module by predicting the correct word segmentations from sentences containing spelling errors.We evaluate WSpeller on the widely used benchmark datasets SIGHAN13, SIGHAN14, and SIGHAN15.Our model is superior to state-of-the-art baselines on SIGHAN13 and SIGHAN15 and maintains equal performance on SIGHAN14.
ELEVANT: A Fully Automatic Fine-Grained Entity Linking Evaluation and Analysis Tool

We present Elevant, a tool for the fully automatic fine-grained evaluation of a set of entity linkers on a set of benchmarks.Elevant provides an automatic breakdown of the performance by various error categories and by entity type.Elevant also provides a rich and compact, yet very intuitive and self-explanatory visualization of the results of a linker on a benchmark in comparison to the ground truth.A live demo, the link to the complete code base on GitHub and a link to a demo video are provided under https://elevant.cs.uni-freiburg.de .
Unobserved Local Structures Make Compositional Generalization Hard

While recent work has shown that sequence-tosequence models struggle to generalize to new compositions (termed compositional generalization), little is known on what makes compositional generalization hard on a particular test instance.In this work, we investigate the factors that make generalization to certain test instances challenging.We first substantiate that some examples are more difficult than others by showing that different models consistently fail or succeed on the same test instances.Then, we propose a criterion for the difficulty of an example: a test instance is hard if it contains a local structure that was not observed at training time.We formulate a simple decision rule based on this criterion and empirically show it predicts instance-level generalization well across 5 different semantic parsing datasets, substantially better than alternative decision rules.Last, we show local structures can be leveraged for creating difficult adversarial compositional splits and also to improve compositional generalization under limited training budgets by strategically selecting examples for the training set.
Low-resource Interactive Active Labeling for Fine-tuning Language Models

Recently, active learning (AL) methods have been used to effectively fine-tune pre-trained language models for various NLP tasks such as sentiment analysis and document classification.However, given the task of fine-tuning language models, understanding the impact of different aspects on AL methods such as labeling cost, sample acquisition latency, and the diversity of the datasets necessitates a deeper investigation.This paper examines the performance of existing AL methods within a lowresource, interactive labeling setting.We observe that existing methods often underperform in such a setting while exhibiting higher latency and a lack of generalizability.To overcome these challenges, we propose a novel active learning method TYROGUE that employs a hybrid sampling strategy to minimize labeling cost and acquisition latency while providing a framework for adapting to dataset diversity via user guidance.Through our experiments, we observe that compared to SOTA methods, TY-ROGUE reduces the labeling cost by up to 43% and the acquisition latency by as much as 11X, while achieving comparable accuracy.Finally, we discuss the strengths and weaknesses of TYROGUE by exploring the impact of dataset characteristics.
Unsupervised Syntactically Controlled Paraphrase Generation with Abstract Meaning Representations

Syntactically controlled paraphrase generation has become an emerging research direction in recent years.Most existing approaches require annotated paraphrase pairs for training and are thus costly to extend to new domains.Unsupervised approaches, on the other hand, do not need paraphrase pairs but suffer from relatively poor performance in terms of syntactic control and quality of generated paraphrases.In this paper, we demonstrate that leveraging Abstract Meaning Representations (AMR) can greatly improve the performance of unsupervised syntactically controlled paraphrase generation.Our proposed model, AMR-enhanced Paraphrase Generator (AMRPG), separately encodes the AMR graph and the constituency parse of the input sentence into two disentangled semantic and syntactic embeddings.A decoder is then learned to reconstruct the input sentence from the semantic and syntactic embeddings.Our experiments show that AMRPG generates more accurate syntactically controlled paraphrases, both quantitatively and qualitatively, compared to the existing unsupervised approaches.We also demonstrate that the paraphrases generated by AMRPG can be used for data augmentation to improve the robustness of NLP models.