Reviewerly/exHarmony huggingface dataset:
How “Multi” is Multi-Document Summarization?

The task of multi-document summarization (MDS) aims at models that, given multiple documents as input, are able to generate a summary that combines disperse information, originally spread across these documents.Accordingly, it is expected that both reference summaries in MDS datasets, as well as system summaries, would indeed be based on such dispersed information.In this paper, we argue for quantifying and assessing this expectation.To that end, we propose an automated measure for evaluating the degree to which a summary is "disperse", in the sense of the number of source documents needed to cover its content.We apply our measure to empirically analyze several popular MDS datasets, with respect to their reference summaries, as well as the output of state-of-the-art systems.Our results show that certain MDS datasets barely require combining information from multiple documents, where a single document often covers the full summary content.Overall, we advocate using our metric for assessing and improving the degree to which summarization datasets require combining multi-document information, and similarly how summarization models actually meet this challenge.
Entropy- and Distance-Based Predictors From GPT-2 Attention Patterns Predict Reading Times Over and Above GPT-2 Surprisal

Transformer-based large language models are trained to make predictions about the next word by aggregating representations of previous tokens through their self-attention mechanism.In the field of cognitive modeling, such attention patterns have recently been interpreted as embodying the process of cue-based retrieval, in which attention over multiple targets is taken to generate interference and latency during retrieval.Under this framework, this work first defines an entropy-based predictor that quantifies the diffuseness of self-attention, as well as distance-based predictors that capture the incremental change in attention patterns across timesteps.Moreover, following recent studies that question the informativeness of attention weights, we also experiment with alternative methods for incorporating vector norms into attention weights.Regression experiments using predictors calculated from the GPT-2 language model show that these predictors deliver a substantially better fit to held-out self-paced reading and eye-tracking data over a rigorous baseline including GPT-2 surprisal.Additionally, the distance-based predictors generally demonstrated higher predictive power, with effect sizes of up to 6.59 ms per standard deviation on self-paced reading times (compared to 2.82 ms for surprisal) and 1.05 ms per standard deviation on eye-gaze durations (compared to 3.81 ms for surprisal).
Prompt-and-Rerank: A Method for Zero-Shot and Few-Shot Arbitrary Textual Style Transfer with Small Language Models

We propose a method for arbitrary textual style transfer (TST)-the task of transforming a text into any given style-utilizing general-purpose pre-trained language models.Our method, Prompt-and-Rerank, is based on a mathematical formulation of the TST task, decomposing it into three constituent components: textual similarity, target style strength, and fluency.Our method uses zero-shot or few-shot prompting to obtain a set of candidate generations in the target style, and then re-ranks them according to the three components.Our method enables small pre-trained language models to perform on par with state-of-the-art large-scale models while using two orders of magnitude less compute and memory.We also investigate the effect of model size and prompt design (e.g., prompt paraphrasing and delimiter-pair choice) on style transfer quality across seven diverse textual style transfer datasets, finding, among other things, that delimiter-pair choice has a large impact on performance, and that models have biases on the direction of style transfer. 1
Unifying the Convergences in Multilingual Neural Machine Translation

Although all-in-one-model multilingual neural machine translation (MNMT) has achieved remarkable progress, the convergence inconsistency in the joint training is ignored, i.e.,different language pairs reaching convergence in different epochs.This leads to the trained MNMT model over-fitting low-resource language translations while under-fitting highresource ones.In this paper, we propose a novel training strategy named LSSD (Language-Specific Self-Distillation), which can alleviate the convergence inconsistency and help MNMT models achieve the best performance on each language pair simultaneously.Specifically, LSSD picks up language-specific best checkpoints for each language pair to teach the current model on the fly.Furthermore, we systematically explore three sample-level manipulations of knowledge transferring.Experimental results on three datasets show that LSSD obtains consistent improvements towards all language pairs and achieves the state-of-the-art 1 .
EtriCA: Event-Triggered Context-Aware Story Generation Augmented by Cross Attention

One of the key challenges of automatic story generation is how to generate a long narrative that can maintain fluency, relevance, and coherence.Despite recent progress, current story generation systems still face the challenge of how to effectively capture contextual and event features, which has a profound impact on a model's generation performance.To address these challenges, we present EtriCA, a novel neural generation model, which improves the relevance and coherence of the generated stories through residually mapping context features to event sequences with a crossattention mechanism.Such a feature capturing mechanism allows our model to better exploit the logical relatedness between events when generating stories.Extensive experiments based on both automatic and human evaluations show that our model significantly outperforms state-of-the-art baselines, demonstrating the effectiveness of our model in leveraging context and event features.
Human perceiving behavior modeling in evaluation of code generation models

Within this study, we evaluated a series of code generation models based on CodeGen and GPTNeo to compare the metric-based performance and human evaluation. For a deeper analysis of human perceiving within the evaluation procedure we’ve implemented a 5-level Likert scale assessment of the model output using a perceiving model based on the Theory of Planned Behavior (TPB). Through such analysis, we showed an extension of model assessment as well as a deeper understanding of the quality and applicability of generated code for practical question answering. The approach was evaluated with several model settings in order to assess diversity in quality and style of answer. With the TPB-based model, we showed a different level of perceiving the model result, namely personal understanding, agreement level, and readiness to use the particular code. With such analysis, we investigate a series of issues in code generation as natural language generation (NLG) problems observed in a practical context of programming question-answering with code.
Language as a fingerprint: Self-supervised learning of user encodings using transformers

The way we talk carries information about who we are.Demographics, personality, clinical conditions, political preferences influence what we speak about and how, suggesting that many individual attributes could be inferred from adequate encodings of linguistic behavior.Conversely, conditioning text representations on author attributes has been shown to improve model performance in many NLP tasks.Previous research on individual differences and language representations has mainly focused on predicting selected attributes from text, or on conditioning text representations on such attributes for author-based contextualization.Here, we present a self-supervised approach to learning language-based user encodings using transformers.Using a large corpus of Reddit submissions, we fine-tune DistilBERT on user-based triplet loss.We show that fine-tuned models can pick up on complex linguistic signatures of users, and that they are able to infer rich information about their profiles.Through a series of intrinsic analyses and probing tasks, we provide evidence that fine-tuning enhances models' ability to abstract generalizable user information, which yields performance advantages for user-based downstream tasks.We discuss applications in language-based assessment and contextualized and personalized NLP.
Pre-training Language Models with Deterministic Factual Knowledge

Previous works show that Pre-trained Language Models (PLMs) can capture factual knowledge.However, some analyses reveal that PLMs fail to perform it robustly, e.g., being sensitive to the changes of prompts when extracting factual knowledge.To mitigate this issue, we propose to let PLMs learn the deterministic relationship between the remaining context and the masked content.The deterministic relationship ensures that the masked factual content can be deterministically inferable based on the existing clues in the context.That would provide more stable patterns for PLMs to capture factual knowledge than randomly masking.Two pre-training tasks are further introduced to motivate PLMs to rely on the deterministic relationship when filling masks.Specifically, we use an external Knowledge Base (KB) to identify deterministic relationships and continuously pre-train PLMs with the proposed methods.The factual knowledge probing experiments indicate that the continuously pre-trained PLMs achieve better robustness in factual knowledge capturing.Further experiments on question-answering datasets show that trying to learn a deterministic relationship with the proposed methods can also help other knowledge-intensive tasks.
End-to-End Unsupervised Vision-and-Language Pre-training with Referring Expression Matching

Recently there has been an emerging interest in unsupervised vision-and-language pre-training (VLP) that learns multimodal representations without parallel image-caption data.These pioneering works significantly reduce the cost of VLP on data collection and achieve promising results compared to supervised VLP.However, existing unsupervised VLP methods take as input pre-extracted region-based visual features from external object detectors, which both limits flexibility and reduces computational efficiency.In this paper, we explore end-to-end unsupervised VLP with a vision encoder to directly encode images.The vision encoder is pre-trained on image-only data and jointly optimized during multimodal pre-training.To further enhance the learned cross-modal features, we propose a novel pre-training task that predicts which patches contain an object referred to in natural language from the encoded visual features.Extensive experiments on four visionand-language tasks show that our approach outperforms previous unsupervised VLP methods and obtains new state-of-the-art results 1 .
SentBS: Sentence-level Beam Search for Controllable Summarization

A wide range of control perspectives have been explored in controllable text generation.Structure-controlled summarization is recently proposed as a useful and interesting research direction.However, current structure-controlling methods have limited effectiveness in enforcing the desired structure.To address this limitation, we propose a sentence-level beam search generation method (SentBS), where evaluation is conducted throughout the generation process to select suitable sentences for subsequent generations.We experiment with different combinations of decoding methods to be used as subcomponents by SentBS and evaluate results on the structure-controlled dataset MReD.Experiments show that all explored combinations for SentBS can improve the agreement between the generated text and the desired structure, with the best method significantly reducing the structural discrepancies suffered by the existing model, by approximately 68%.
Measuring Harmful Representations in Scandinavian Language Models

Scandinavian countries are perceived as rolemodels when it comes to gender equality.With the advent of pre-trained language models and their widespread usage, we investigate to what extent gender-based harmful and toxic content exist in selected Scandinavian language models.We examine nine models, covering Danish, Swedish, and Norwegian, by manually creating template-based sentences and probing the models for completion.We evaluate the completions using two methods for measuring harmful and toxic completions and provide a thorough analysis of the results.We show that Scandinavian pre-trained language models contain harmful and gender-based stereotypes with similar values across all languages.This finding goes against the general expectations related to gender equality in Scandinavian countries and shows the possible problematic outcomes of using such models in real-world settings.Warning: this paper contains unobfuscated examples some readers may find upsetting and offensive.
Truncation Sampling as Language Model Desmoothing

Long samples of text from neural language models can be of poor quality.Truncation sampling algorithms-like top-p or top-k-address this by setting some words' probabilities to zero at each step.This work provides framing for the aim of truncation, and an improved algorithm for that aim.We propose thinking of a neural language model as a mixture of a true distribution and a smoothing distribution that avoids infinite perplexity.In this light, truncation algorithms aim to perform desmoothing, estimating a subset of the support of the true distribution.Finding a good subset is crucial: we show that top-p unnecessarily truncates high-probability words, for example causing it to truncate all words but Trump for a document that starts with Donald.We introduce η-sampling, which truncates words below an entropy-dependent probability threshold.Compared to previous algorithms, η-sampling generates more plausible long English documents according to humans, is better at breaking out of repetition, and behaves more reasonably on a battery of test distributions.
Eliciting Knowledge from Large Pre-Trained Models for Unsupervised Knowledge-Grounded Conversation

Recent advances in large-scale pre-training provide large models with the potential to learn knowledge from the raw text.It is thus natural to ask whether it is possible to leverage these large models as knowledge bases for downstream tasks.In this work, we answer the aforementioned question in unsupervised knowledge-grounded conversation.We explore various methods that best elicit knowledge from large models.Our human study indicates that, though hallucinations exist, large models post the unique advantage of being able to output common sense and summarize facts that cannot be directly retrieved from the search engine.To better exploit such generated knowledge in dialogue generation, we treat the generated knowledge as a noisy knowledge source and propose the posterior-based reweighing as well as the noisy training strategy.Empirical results on two benchmarks show advantages over the state-of-the-art methods.
Learning to Revise References for Faithful Summarization

In real-world scenarios with naturally occurring datasets, reference summaries are noisy and may contain information that cannot be inferred from the source text.On large news corpora, removing low quality samples has been shown to reduce model hallucinations.Yet, for smaller, and/or noisier corpora, filtering is detrimental to performance.To improve reference quality while retaining all data, we propose a new approach: to selectively rewrite unsupported reference sentences to better reflect source data.We automatically generate a synthetic dataset of positive and negative revisions by corrupting supported sentences and learn to revise reference sentences with contrastive learning.The intensity of revisions is treated as a controllable attribute so that, at inference, diverse candidates can be over-generated-then-rescored to balance faithfulness and abstraction.To test our methods, we extract noisy references from publicly available MIMIC-III discharge summaries for the task of hospital-course summarization, and vary the data on which models are trained.According to metrics and human evaluation, models trained on revised clinical references are much more faithful, informative, and fluent than models trained on original or filtered data.
Sharpness-Aware Minimization with Dynamic Reweighting

Deep neural networks are often overparameterized and may not easily achieve model generalization.Adversarial training has shown effectiveness in improving generalization by regularizing the change of loss on top of adversarially chosen perturbations.The recently proposed sharpness-aware minimization (SAM) algorithm conducts adversarial weight perturbation, encouraging the model to converge to a flat minima.SAM finds a common adversarial weight perturbation per-batch.Although perinstance adversarial weight perturbations are stronger adversaries and can potentially lead to better generalization performance, their computational cost is very high and thus it is impossible to use per-instance perturbations efficiently in SAM.In this paper, we tackle this efficiency bottleneck and propose sharpness-aware minimization with dynamic reweighting (δ-SAM).Our theoretical analysis motivates that it is possible to approach the stronger, per-instance adversarial weight perturbations using reweighted per-batch weight perturbations.δ-SAM dynamically reweights perturbation within each batch according to the theoretically principled weighting factors, serving as a good approximation to per-instance perturbation.Experiments on various natural language understanding tasks demonstrate the effectiveness of δ-SAM.
Balancing out Bias: Achieving Fairness Through Balanced Training

Group bias in natural language processing tasks manifests as disparities in system error rates across texts authorized by different demographic groups, typically disadvantaging minority groups.Dataset balancing has been shown to be effective at mitigating bias, however existing approaches do not directly account for correlations between author demographics and linguistic variables, limiting their effectiveness.To achieve Equal Opportunity fairness, such as equal job opportunity without regard to demographics, this paper introduces a simple, but highly effective, objective for countering bias using balanced training.We extend the method in the form of a gated model, which incorporates protected attributes as input, and show that it is effective at reducing bias in predictions through demographic input perturbation, outperforming all other bias mitigation techniques when combined with balanced training. 1
Simplified Graph Learning for Inductive Short Text Classification

Short text classification (STC) is hard as short texts lack context information and labeled data is not enough.Graph neural networks obtain the state-of-the-art on STC since they can merge various auxiliary information via the message passing framework.However, existing works conduct transductive learning, which requires retraining to accommodate new samples and takes large memory.In this paper, we present SimpleSTC which handles inductive STC problem but only leverages words.We construct word graph from an external large corpus to compensate for the lack of semantic information, and learn text graph to handle the lack of labeled data.Results show that Sim-pleSTC obtains state-of-the-art performance with lower memory consumption and faster inference speed.
Speaker Overlap-aware Neural Diarization for Multi-party Meeting Analysis

Recently, hybrid systems of clustering and neural diarization models have been successfully applied in multi-party meeting analysis.However, current models always treat overlapped speaker diarization as a multi-label classification problem, where speaker dependency and overlaps are not well considered.To overcome the disadvantages, we reformulate overlapped speaker diarization task as a single-label prediction problem via the proposed power set encoding (PSE).Through this formulation, speaker dependency and overlaps can be explicitly modeled.To fully leverage this formulation, we further propose the speaker overlap-aware neural diarization (SOND) model, which consists of a contextindependent (CI) scorer to model global speaker discriminability, a context-dependent scorer (CD) to model local discriminability, and a speaker combining network (SCN) to combine and reassign speaker activities.Experimental results show that using the proposed formulation can outperform the state-ofthe-art methods based on target speaker voice activity detection, and the performance can be further improved with SOND, resulting in a 6.30% relative diarization error reduction.
LPC: A Logits and Parameter Calibration Framework for Continual Learning

When we execute the typical fine-tuning paradigm on continuously sequential tasks, the model will suffer from the catastrophic forgetting problem (i.e., the model tends to adjust old parameters according to the new knowledge, which leads to the loss of previously acquired concepts).People proposed replay-based methods by accessing old data from extra storage and maintaining the parameters of old concepts, which actually raise the privacy issue and larger memory requirements.In this work, we aim to achieve the sequential/continual learning of knowledge without accessing the old data.The core idea is to calibrate the parameters and logits (output) so that preserving old parameters and generalized learning on new concepts can be solved simultaneously.Our proposed framework includes two major components, Logits Calibration (LC) and Parameter Calibration (PC).The LC focuses on calibrating the learning of novel models with old models, and PC aims to preserve the parameters of old models.These two operations can maintain the old knowledge while learning new tasks without storing previous data.We conduct experiments on various scenarios of the GLUE (the General Language Understanding Evaluation) benchmark.The experimental results show that our model achieves state-of-the-art performance in all scenarios.
EdiT5: Semi-Autoregressive Text Editing with T5 Warm-Start

We present EDIT5 1 -a novel semiautoregressive text-editing model designed to combine the strengths of non-autoregressive text-editing and autoregressive decoding.EDIT5 is faster during inference than conventional sequence-to-sequence (seq2seq) models, while being capable of modeling flexible input-output transformations.This is achieved by decomposing the generation process into three sub-tasks: (1) tagging to decide on the subset of input tokens to be preserved in the output, (2) re-ordering to define their order in the output text, and (3) insertion to infill the missing tokens that are not present in the input.The tagging and reordering steps, which are responsible for generating the largest portion of the output, are non-autoregressive, while the insertion step uses an autoregressive decoder.Depending on the task, EDIT5 on average requires significantly fewer autoregressive steps, demonstrating speedups of up to 25x when compared to seq2seq models.Quality-wise, EDIT5 is initialized with a pre-trained T5 checkpoint yielding comparable performance to T5 in high-resource settings when evaluated on three NLG tasks: Sentence Fusion, Grammatical Error Correction, and Decontextualization while clearly outperforming T5 in lowresource settings.