Reviewerly/exHarmony huggingface dataset:
StoryTrans: Non-Parallel Story Author-Style Transfer with Discourse Representations and Content Enhancing

Non-parallel text style transfer is an important task in natural language generation.However, previous studies concentrate on the token or sentence level, such as sentence sentiment and formality transfer, but neglect long style transfer at the discourse level.Long texts usually involve more complicated author linguistic preferences such as discourse structures than sentences.In this paper, we formulate the task of non-parallel story author-style transfer, which requires transferring an input story into a specified author style while maintaining source semantics.To tackle this problem, we propose a generation model, named StoryTrans, which leverages discourse representations to capture source content information and transfer them to target styles with learnable style embeddings.We use an additional training objective to disentangle stylistic features from the learned discourse representation to prevent the model from degenerating to an auto-encoder.Moreover, to enhance content preservation, we design a mask-and-fill framework to explicitly fuse style-specific keywords of source texts into generation.Furthermore, we constructed new datasets for this task in Chinese and English, respectively.Extensive experiments show that our model outperforms strong baselines in overall performance of style transfer and content preservation.
ContraCLM: Contrastive Learning For Causal Language Model

Despite exciting progress in causal language models, the expressiveness of their representations is largely limited due to poor discrimination ability.To remedy this issue, we present CONTRACLM, a novel contrastive learning framework at both the token-level and the sequence-level.We assess CONTRACLM on a variety of downstream tasks.We show that CONTRACLM enhances the discrimination of representations and bridges the gap with encoder-only models, which makes causal language models better suited for tasks beyond language generation.Specifically, we attain 44% relative improvement on the Semantic Textual Similarity tasks and 34% on Code-to-Code Search tasks.Furthermore, by improving the expressiveness of representations, CONTRA-CLM also boosts the source code generation capability with 9% relative improvement on execution accuracy on the HumanEval benchmark.
Efficient Shapley Values Estimation by Amortization for Text Classification

Despite the popularity of Shapley Values in explaining neural text classification models, computing them is prohibitive for large pretrained models due to a large number of model evaluations.In practice, Shapley Values are often estimated with a small number of stochastic model evaluations.However, we show that the estimated Shapley Values are sensitive to random seed choices -the top-ranked features often have little overlap across different seeds, especially on examples with longer input texts.This can only be mitigated by aggregating thousands of model evaluations, which on the other hand, induces substantial computational overheads.To mitigate the trade-off between stability and efficiency, we develop an amortized model that directly predicts each input feature's Shapley Value without additional model evaluations.It is trained on a set of examples whose Shapley Values are estimated from a large number of model evaluations to ensure stability.Experimental results on two text classification datasets demonstrate that our amortized model estimates Shapley Values accurately with up to 60 times speedup compared to traditional methods.Furthermore, the estimated values are stable as the inference is deterministic.We release our code at https:
MTR: A Dataset Fusing Inductive, Deductive, and Defeasible Reasoning

A long-standing difficulty in AI is the introduction of human-like reasoning in machine reading comprehension.Since algorithmic models can already perform as well as humans on simple quality assurance tasks thanks to the development of deep learning techniques, more difficult reasoning datasets have been presented.However, these datasets mainly focus on a single type of reasoning.There are still significant gaps in the studies when compared to the complex reasoning used in daily life because we can mix and match different types of reasoning unconsciously.In this work, we introduce a brandnew dataset, named MT R.There are two subsets of it: (1)the first is mainly used to explore mixed reasoning abilities and combines deductive and inductive reasoning; (2)the second integrates inductive and defeasible reasoning for detecting non-monotonic reasoning ability.It consists of more than 30k instances, requiring models to infer relations between characters in short stories.Compared with the corresponding single reasoning datasets, MT R serves as a more challenging one, highlighting the gap in language models' ability to handle sophisticated inference.
CASE: Aligning Coarse-to-Fine Cognition and Affection for Empathetic Response Generation

Empathetic conversation is psychologically supposed to be the result of conscious alignment and interaction between the cognition and affection of empathy.However, existing empathetic dialogue models usually consider only the affective aspect or treat cognition and affection in isolation, which limits the capability of empathetic response generation.In this work, we propose the CASE model for empathetic dialogue generation.It first builds upon a commonsense cognition graph and an emotional concept graph and then aligns the user's cognition and affection at both the coarse-grained and fine-grained levels.Through automatic and manual evaluation, we demonstrate that CASE outperforms state-of-the-art baselines of empathetic dialogues and can generate more empathetic and informative responses.
Exploring Continual Learning for Code Generation Models

Large-scale code generation models such as Codex and CodeT5 have achieved impressive performance.However, libraries are upgraded or deprecated very frequently and re-training large-scale language models is computationally expensive.Therefore, Continual Learning (CL) is an important aspect that remains underexplored in the code domain.In this paper, we introduce a benchmark called CODETASK-CL that covers a wide range of tasks, including code generation, translation, summarization, and refinement, with different input and output programming languages.Next, on our CODETASK-CL benchmark, we compare popular CL techniques from NLP and Vision domains.We find that effective methods like Prompt Pooling (PP) suffer from catastrophic forgetting due to the unstable training of the prompt selection mechanism caused by stark distribution shifts in coding tasks.We address this issue with our proposed method, Prompt Pooling with Teacher Forcing (PP-TF), that stabilizes training by enforcing constraints on the prompt selection mechanism and leads to a 21.54% improvement over Prompt Pooling.Along with the benchmark, we establish a training pipeline that can be used for CL on code models, which we believe can motivate further development of CL methods for code models.
RobustQA: Benchmarking the Robustness of Domain Adaptation for Open-Domain Question Answering

Open-domain question answering (ODQA) is a crucial task in natural language processing.A typical ODQA system relies on a retriever module to select relevant contexts from a large corpus for a downstream reading comprehension model.Existing ODQA datasets consist mainly of Wikipedia corpus, and are insufficient to study models' generalizability across diverse domains, as models are trained and evaluated on the same genre of data.We propose RobustQA 1 , a novel benchmark consisting of datasets from 8 different domains, which facilitates the evaluation of ODQA's domain robustness.To build RobustQA, we annotate QA pairs in retrieval datasets with rigorous quality control.We further examine improving QA performances by incorporating unsupervised learning methods with target-domain corpus and adopting large generative language models.These methods can effectively improve model performances on RobustQA.However, experimental results demonstrate a significant gap from in-domain training, suggesting that Ro-bustQA is a challenging benchmark to evaluate ODQA domain robustness.
We Understand Elliptical Sentences, and Language Models should Too: A New Dataset for Studying Ellipsis and its Interaction with Thematic Fit

Ellipsis is a linguistic phenomenon characterized by the omission of one or more sentence elements.Solving such a linguistic construction is not a trivial issue in natural language processing since it involves the retrieval of non-overtly expressed verbal material, which might in turn require the model to integrate human-like syntactic and semantic knowledge.In this paper, we explored the issue of how the prototypicality of event participants affects the ability of Language Models (LMs) to handle elliptical sentences, and to identify the omitted arguments at different degrees of thematic fit, ranging from highly typical participants to semantically anomalous ones.With this purpose in mind, we built ELLie, the first dataset composed entirely of utterances containing different types of elliptical constructions, and structurally suited for evaluating the effect of argument thematic fit in solving ellipsis and reconstructing the missing element.Our tests demonstrated that the probability scores assigned by the models are higher for typical events than for atypical and impossible ones in different elliptical contexts, confirming the influence of prototypicality of the event participants in interpreting such linguistic structures.Finally, we conducted a retrieval task of the elided verb in the sentence in which the low performance of LMs highlighted a considerable difficulty in reconstructing the correct event.
VCSUM: A Versatile Chinese Meeting Summarization Dataset

Compared to news and chat summarization, the development of meeting summarization is hugely decelerated by the limited data.To this end, we introduce a versatile Chinese meeting summarization dataset, dubbed VCSUM, consisting of 239 real-life meetings, with a total duration of over 230 hours.We claim our dataset is versatile because we provide the annotations of topic segmentation, headlines, segmentation summaries, overall meeting summaries, and salient sentences for each meeting transcript.As such, the dataset can adapt to various summarization tasks or methods, including segmentation-based summarization, multi-granularity summarization and retrievalthen-generate summarization.Our analysis confirms the effectiveness and robustness of VCSUM.We also provide a set of benchmark models regarding different downstream summarization tasks on VCSUM to facilitate further research.The dataset and code will be released at https://github.com/hahahawu/VCSum.
Improving Reading Comprehension Question Generation with Data Augmentation and Overgenerate-and-rank

Reading comprehension is a crucial skill in many aspects of education, including language learning, cognitive development, and fostering early literacy skills in children.Automated answer-aware reading comprehension question generation has significant potential to scale up learner support in educational activities.One key technical challenge in this setting is that there can be multiple questions, sometimes very different from each other, with the same answer; a trained question generation method may not necessarily know which question human educators would prefer.To address this challenge, we propose 1) a data augmentation method that enriches the training dataset with diverse questions given the same context and answer and 2) an overgenerate-and-rank method to select the best question from a pool of candidates.We evaluate our method on the FairytaleQA dataset, showing a 5% absolute improvement in ROUGE-L over the best existing method.We also demonstrate the effectiveness of our method in generating harder, "implicit" questions, where the answers are not contained in the context as text spans.
Modeling What-to-ask and How-to-ask for Answer-unaware Conversational Question Generation

Conversational Question Generation (CQG) is a critical task for machines to assist humans in fulfilling their information needs through conversations.The task is generally cast into two different settings: answer-aware and answerunaware.While the former facilitates the models by exposing the expected answer, the latter is more realistic and receiving growing attentions recently.What-to-ask and how-to-ask are the two main challenges in the answer-unaware setting.To address the first challenge, existing methods mainly select sequential sentences in context as the rationales.We argue that the conversation generated using such naive heuristics may not be natural enough as in reality, the interlocutors often talk about the relevant contents that are not necessarily sequential in context.Additionally, previous methods decide the type of question (boolean/span-based) to be generated implicitly.Modeling the question type explicitly is crucial in this (answerunaware) setting, as the answer which hints the models to generate a boolean or span-based question, is unavailable.To this end, we present SG-CQG, a two-stage CQG framework.For the what-to-ask stage, a sentence is selected as the rationale from a semantic graph that we construct, and extract the answer span from it.For the how-to-ask stage, a classifier determines the target answer type of the question via two explicit control signals before generating and filtering.In addition, we propose Conv-Distinct, a novel evaluation metric for CQG, to evaluate the diversity of the generated conversation from a context.Compared with the existing answerunaware CQG models, the proposed SG-CQG achieves state-of-the-art performance.
CAME: Confidence-guided Adaptive Memory Efficient Optimization

Adaptive gradient methods, such as Adam and LAMB, have demonstrated excellent performance in the training of large language models.Nevertheless, the need for adaptivity requires maintaining second-moment estimates of the per-parameter gradients, which entails a high cost of extra memory overheads.To solve this problem, several memory-efficient optimizers (e.g., Adafactor) have been proposed to obtain a drastic reduction in auxiliary memory usage, but with a performance penalty.In this paper, we first study a confidence-guided strategy to reduce the instability of existing memory efficient optimizers.Based on this strategy, we propose CAME to simultaneously achieve two goals: fast convergence as in traditional adaptive methods, and low memory usage as in memory-efficient methods.Extensive experiments demonstrate the training stability and superior performance of CAME across various NLP tasks such as BERT and GPT-2 training.Notably, for BERT pre-training on the large batch size of 32,768, our proposed optimizer attains faster convergence and higher accuracy compared with the Adam optimizer.The implementation of CAME is publicly available 1 .
Cognitive Reframing of Negative Thoughts through Human-Language Model Interaction

A proven therapeutic technique to overcome negative thoughts is to replace them with a more hopeful "reframed thought."Although therapy can help people practice and learn this Cognitive Reframing of Negative Thoughts, clinician shortages and mental health stigma commonly limit people's access to therapy.In this paper, we conduct a human-centered study of how language models may assist people in reframing negative thoughts.Based on psychology literature, we define a framework of seven linguistic attributes that can be used to reframe a thought.We develop automated metrics to measure these attributes and validate them with expert judgements from mental health practitioners.We collect a dataset of 600 situations, thoughts and reframes from practitioners and use it to train a retrieval-enhanced in-context learning model that effectively generates reframed thoughts and controls their linguistic attributes.To investigate what constitutes a "high-quality" reframe, we conduct an IRBapproved randomized field study on a large mental health website with over 2,000 participants.Amongst other findings, we show that people prefer highly empathic or specific reframes, as opposed to reframes that are overly positive.Our findings provide key implications for the use of LMs to assist people in overcoming negative thoughts.
A Hierarchical Explanation Generation Method Based on Feature Interaction Detection

The opaqueness of deep NLP models has motivated efforts to explain how deep models predict.Recently, work has introduced hierarchical attribution explanations, which calculate attribution scores for compositional text hierarchically to capture compositional semantics.Existing work on hierarchical attributions tends to limit the text groups to a continuous text span, which we call the connecting rule.While easy for humans to read, limiting the attribution unit to a continuous span might lose important long-distance feature interactions for reflecting model predictions.In this work, we introduce a novel strategy for capturing feature interactions and employ it to build hierarchical explanations without the connecting rule.The proposed method can convert ubiquitous non-hierarchical explanations (e.g., LIME) into their corresponding hierarchical versions.Experimental results show the effectiveness of our approach in building high-quality hierarchical explanations.
Hypothetical Training for Robust Machine Reading Comprehension of Tabular Context

Machine Reading Comprehension (MRC) models easily learn spurious correlations from complex contexts such as tabular data. Counterfactual training—using the factual and counterfactual data by augmentation—has become a promising solution. However, it is costly to construct faithful counterfactual examples because it is tricky to maintain the consistency and dependency of the tabular data. In this paper, we take a more efficient fashion to ask hypothetical questions like “in which year would the net profit be larger if the revenue in 2019 were $38,298?”, whose effects on the answers are equivalent to those expensive counterfactual tables. We propose a hypothetical training framework that uses paired examples with different hypothetical questions to supervise the direction of model gradient towards the counterfactual answer change. The superior generalization results on tabular MRC datasets, including a newly constructed stress test and MultiHiertt, validate our effectiveness.
Few-Shot Data-to-Text Generation via Unified Representation and Multi-Source Learning

We present a novel approach for structured datato-text generation that addresses the limitations of existing methods that primarily focus on specific types of structured data.Our proposed method aims to improve performance in multitask training, zero-shot and few-shot scenarios by providing a unified representation that can handle various forms of structured data such as tables, knowledge graph triples, and meaning representations.We demonstrate that our proposed approach can effectively adapt to new structured forms, and can improve performance in comparison to current methods.For example, our method resulted in a 66% improvement in zero-shot BLEU scores when transferring models trained on table inputs to a knowledge graph dataset.Our proposed method is an important step towards a more general data-to-text generation framework. 1
A Side-by-side Comparison of Transformers for Implicit Discourse Relation Classification

Though discourse parsing can help multiple NLP fields, there has been no wide language model search done on implicit discourse relation classification.This hinders researchers from fully utilizing public-available models in discourse analysis.This work is a straightforward, fine-tuned discourse performance comparison of seven pre-trained language models.We use PDTB-3, a popular discourse relation annotated dataset.Through our model search, we raise SOTA to 0.671 ACC and obtain novel observations.Some are contrary to what has been reported before (Shi and Demberg, 2019b), that sentence-level pre-training objectives (NSP, SBO, SOP) generally fail to produce the best performing model for implicit discourse relation classification.Counterintuitively, similar-sized PLMs with MLM and full attention led to better performance.
Small Character Models Match Large Word Models for Autocomplete Under Memory Constraints

Autocomplete is a task where the user inputs a piece of text, termed prompt, which is conditioned by the model to generate semantically coherent continuation.Existing works for this task have primarily focused on datasets (e.g., email, chat) with high frequency user prompt patterns (or focused prompts) where word-based language models have been quite effective.In this work, we study the more challenging setting consisting of low frequency user prompt patterns (or broad prompts, e.g., prompt about 93 rd academy awards) and demonstrate the effectiveness of character-based language models.We study this problem under memoryconstrained settings (e.g., edge devices and smartphones), where character-based representation is effective in reducing the overall model size (in terms of parameters).We use WikiText-103 benchmark to simulate broad prompts and demonstrate that character models rival word models in exact match accuracy for the autocomplete task, when controlled for the model size.For instance, we show that a 20M parameter character model performs similar to an 80M parameter word model in the vanilla setting.We further propose novel methods to improve character models by incorporating inductive bias in the form of compositional information and representation transfer from large word models.Datasets and code used in this work are available at https://github.com/UBC-NLP/char_autocomplete.
Exploring the Impact of Transliteration on NLP Performance: Treating Maltese as an Arabic Dialect

Multilingual models such as mBERT have been demonstrated to exhibit impressive crosslingual transfer for a number of languages.Despite this, the performance drops for lowerresourced languages, especially when they are not part of the pre-training setup and when there are script differences.In this work we consider Maltese, a low-resource language of Arabic and Romance origins written in Latin script.Specifically, we investigate the impact of transliterating Maltese into Arabic scipt on a number of downstream tasks: Part-of-Speech Tagging, Dependency Parsing, and Sentiment Analysis.We compare multiple transliteration pipelines ranging from deterministic character maps to more sophisticated alternatives, including manually annotated word mappings and non-deterministic character mappings.For the latter, we show that selection techniques using n-gram language models of Tunisian Arabic, the dialect with the highest degree of mutual intelligibility to Maltese, yield better results on downstream tasks.Moreover, our experiments highlight that the use of an Arabic pre-trained model paired with transliteration outperforms mBERT.Overall, our results show that transliterating Maltese can be considered an option to improve the cross-lingual transfer capabilities.