Reviewerly/exHarmony huggingface dataset:
Alleviating Exposure Bias via Multi-level Contrastive Learning and Deviation Simulation in Abstractive Summarization

Most Transformer based abstractive summarization systems have a severe mismatch between training and inference, i.e., exposure bias.From diverse perspectives, we introduce a simple multi-level contrastive learning framework for abstractive summarization (SimMCS) and a tailored sparse decoder self-attention pattern (SDSA) to bridge the gap between training and inference to improve model performance.Compared with previous contrastive objectives focusing only on the relative order of probability mass assigned to non-gold summaries, SimMCS additionally takes their absolute positions into account, which guarantees that the relatively high-quality (positive) summaries among them can be properly assigned high probability mass, and further enhances the capability of discriminating summary quality beyond exploiting potential artifacts of specific metrics.SDSA simulates the possible inference scenarios of deviation in the training phase to get closer to the ideal paradigm.Our approaches outperform the previous state-ofthe-art results on two summarization datasets while just adding fairly low overhead.Further empirical analysis shows our model preserves the advantages of prior contrastive methods and possesses strong few-shot learning ability.Our code is available at https://github.com/xjwnlp/SimMCS.Summarizer Generation + + … + High-quality Summaries (Positive) Teacher Forcing Encoder Input
Pre-training Language Model as a Multi-perspective Course Learner

ELECTRA (Clark et al., 2020), the generatordiscriminator pre-training framework, has achieved impressive semantic construction capability among various downstream tasks.Despite the convincing performance, ELECTRA still faces the challenges of monotonous training and deficient interaction.Generator with only masked language modeling (MLM) leads to biased learning and label imbalance for discriminator, decreasing learning efficiency; no explicit feedback loop from discriminator to generator results in the chasm between these two components, underutilizing the course learning.In this study, a multi-perspective course learning (MCL) method is proposed to fetch a many degrees and visual angles for sample-efficient pre-training, and to fully leverage the relationship between generator and discriminator.Concretely, three self-supervision courses are designed to alleviate inherent flaws of MLM and balance the label in a multiperspective way.Besides, two self-correction courses are proposed to bridge the chasm between the two encoders by creating a "correction notebook" for secondary-supervision.Moreover, a course soups trial is conducted to solve the "tug-of-war" dynamics problem of MCL, evolving a stronger pre-trained model.Experimental results show that our method significantly improves ELECTRA's average performance by 2.8% and 3.2% absolute points respectively on GLUE and SQuAD 2.0 benchmarks, and overshadows recent advanced ELECTRA-style models under the same settings.The pre-trained MCL model is available at https://huggingface.co/McmanusChen/MCLbase.
Actively Supervised Clustering for Open Relation Extraction

Current clustering-based Open Relation Extraction (OpenRE) methods usually adopt a two-stage pipeline.The first stage simultaneously learns relation representations and assignments.The second stage manually labels several instances and thus names the relation for each cluster.However, unsupervised objectives struggle to optimize the model to derive accurate clustering assignments, and the number of clusters has to be supplied in advance.In this paper, we present a novel setting, named actively supervised clustering for OpenRE.Our insight lies in that clustering learning and relation labeling can be alternately performed, providing the necessary guidance for clustering without a significant increase in human effort.The key to the setting is selecting which instances to label.Instead of using classical active labeling strategies designed for fixed known classes, we propose a new strategy, which is applicable to dynamically discover clusters of unknown relations.Experimental results show that our method is able to discover almost all relational clusters in the data and improve the SOTA methods by 10.3% and 5.2%, on two datasets respectively.
DeakinNLP at ProbSum 2023: Clinical Progress Note Summarization with Rules and Language ModelsClinical Progress Note Summarization with Rules and Languague Models

This paper summarizes two approaches developed for BioNLP2023 workshop task 1A 1 : clinical progress note summarization.We develop two types of methods with either rules or pre-trained language models.In the rule-based summarization model, we leverage UMLS (Unified Medical Language System) and a negation detector to extract text spans to represent the summary.We also fine tune three pre-trained language models (BART, T5 and GPT2) to generate the summaries.Experiment results show the rule based system returns extractive summaries but lower ROUGE-L score (0.043), while the fine tuned T5 returns a higher ROUGE-L score (0.208).
Python Code Generation by Asking Clarification Questions

Code generation from text requires understanding the user's intent from a natural language description and generating an executable code snippet that satisfies this intent.While recent pretrained language models demonstrate remarkable performance for this task, these models fail when the given natural language description is under-specified.In this work, we introduce a novel and more realistic setup for this task.We hypothesize that the underspecification of a natural language description can be resolved by asking clarification questions.Therefore, we collect and introduce a new dataset named CodeClarQA containing pairs of natural language descriptions and code with created synthetic clarification questions and answers.The empirical results of our evaluation of pretrained language model performance on code generation show that clarifications result in more precisely generated code, as shown by the substantial improvement of model performance in all evaluation metrics.Alongside this, our task and dataset introduce new challenges to the community, including when and what clarification questions should be asked.Our code and dataset are available on GitHub.
An Efficient Conversational Smart Compose System

Online conversation is a ubiquitous way to share information and connect everyone but repetitive idiomatic text typing takes users a lot of time.This paper demonstrates a simple yet effective cloud based smart compose system to improve human-to-human conversation efficiency.Heuristics from different perspectives are designed to achieve the best trade-off between quality and latency.From the modeling side, the decoder-only model exploited the previous turns of conversational history in a computation lightweight manner.Besides, a novel phrase tokenizer is proposed to reduce latency without losing the composing quality further.Additionally, the caching mechanism is applied to the serving framework.The demo video of the system is available at https://youtu.
MGR: Multi-generator Based Rationalization

Rationalization is to employ a generator and a predictor to construct a self-explaining NLP model in which the generator selects a subset of human-intelligible pieces of the input text to the following predictor.However, rationalization suffers from two key challenges, i.e., spurious correlation and degeneration, where the predictor overfits the spurious or meaningless pieces solely selected by the not-yet well-trained generator and in turn deteriorates the generator.Although many studies have been proposed to address the two challenges, they are usually designed separately and do not take both of them into account.In this paper, we propose a simple yet effective method named MGR to simultaneously solve the two problems.The key idea of MGR is to employ multiple generators such that the occurrence stability of real pieces is improved and more meaningful pieces are delivered to the predictor.Empirically 1 , we show that MGR improves the F1 score by up to 20.9% as compared to state-of-the-art methods.
Rehearsal-free Continual Language Learning via Efficient Parameter Isolation

We study the problem of defying catastrophic forgetting when learning a series of language processing tasks.Compared with previous methods, we emphasize the importance of not caching history tasks' data, which makes the problem more challenging.Our proposed method applies the parameter isolation strategy.For each task, it allocates a small portion of private parameters and learns them with a shared pre-trained model.To load correct parameters at testing time, we introduce a simple yet effective non-parametric method.Experiments on continual language learning benchmarks show that our method is significantly better than all existing no-data-cache methods, and is comparable (or even better) than those using historical data 1 .
Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformers

This paper explores the effectiveness of modelgenerated signals in improving zero-shot generalization of text-to-text Transformers such as T5.We study various designs to pretrain T5 using an auxiliary model to construct more challenging token replacements for the main model to denoise.Key aspects under study include the decoding target, the location of the RTD head, and the masking pattern.Based on these studies, we develop a new model, METRO-T0, which is pretrained using the redesigned ELECTRA-Style pretraining strategies and then prompt-finetuned on a mixture of NLP tasks.METRO-T0 outperforms all similarsized baselines on prompted NLP benchmarks, such as T0 Eval and MMLU, and rivals the state-of-the-art T0 11B model with only 8% of its parameters.Our analysis on model's neural activation and parameter sensitivity reveals that the effectiveness of METRO-T0 stems from more balanced contribution of parameters and better utilization of their capacity.The code and model checkpoints are available at https: //github.com/gonglinyuan/metro_t0.
Adversarial Knowledge Stimulated Contrastive Prompting for Few-shot Language Learners

Prompt-based fine-tuning has boosted the performance of Pre-trained Language Models (PLMs) on few-shot Natural Language Understanding (NLU) tasks by employing taskspecific prompts.Yet, PLMs are unfamiliar with prompt-style expressions during pretraining, which limits the few-shot learning performance on downstream tasks.It would be desirable if the models can stimulate prompting knowledge while adaptation to specific NLU tasks.We present the Adversarial Knowledge Stimulated Contrastive Prompting (AKSCP) framework, leading to better few-shot NLU tasks for language models by implicitly stimulate knowledge from pretrained language model.In AKSCP, a novel paradigm Clozedriven prompt is proposed for joint prompt tuning across word cloze task and prompt-based learning, forcing PLMs to stimulate prompting knowledge.We further design an Adversarial Contrastive learning method to improve the generalization ability of PLM for different downstream tasks.Experiments over a variety of NLU tasks show that AKSCP consistently outperforms state-of-the-arts for prompt-based fine-tuning.
An efficient method for Natural Language Querying on Structured Data

We present an efficient and reliable approach to Natural Language Querying (NLQ) on databases (DB) which is not based on text-to-SQL type semantic parsing.Our approach simplifies the NLQ on structured data problem to the following "bread and butter" NLP tasks: (a) Domain classification, for choosing which DB table to query, whether the question is out-ofscope (b) Multi-head slot/entity extraction (SE) to extract the field criteria and other attributes such as its role (filter, sort etc) from the raw text and (c) Slot value disambiguation (SVD) to resolve/normalize raw spans from SE to format suitable to query a DB.This is a general purpose, DB language agnostic approach and the output can be used to query any DB and return results to the user.Also each of these tasks is extremely well studied, mature, easier to collect data for and enables better error analysis by tracing problems to specific components when something goes wrong.
Open Set Relation Extraction via Unknown-Aware Training

The existing supervised relation extraction methods have achieved impressive performance in a closed-set setting, where the relations during both training and testing remain the same.In a more realistic open-set setting, unknown relations may appear in the test set.Due to the lack of supervision signals from unknown relations, a well-performing closedset relation extractor can still confidently misclassify them into known relations.In this paper, we propose an unknown-aware training method, regularizing the model by dynamically synthesizing negative instances.To facilitate a compact decision boundary, "difficult" negative instances are necessary.Inspired by text adversarial attacks, we adaptively apply small but critical perturbations to original training instances and thus synthesizing negative instances that are more likely to be mistaken by the model as known relations.Experimental results show that this method achieves SOTA unknown relation detection without compromising the classification of known relations.
Arthur Caplan at SemEval-2023 Task 4: Enhancing Human Value Detection through Fine-tuned Pre-trained Models

The computational identification of human values is a novel and challenging research that holds the potential to offer valuable insights into the nature of human behavior and cognition.This paper presents the methodology adopted by the Arthur-Caplan research team for the SemEval-2023 Task 4, which entailed the detection of human values behind arguments.The proposed system integrates BERT, ERNIE2.0,RoBERTA and XLNet models with fine tuning.Experimental results show that the macro F1 score of our system achieved 0.512, which overperformed baseline methods by 9.2% on the test set.
I2D2: Inductive Knowledge Distillation with NeuroLogic and Self-Imitation

Commonsense capabilities of pre-trained language models dramatically improve with scale, leading many to believe that scale is the only winning recipe.But is it?Here, we investigate an alternative that a priori seems impossible: can smaller language models (e.g., GPT-2) win over models that are orders of magnitude larger and better (e.g., GPT-3), if powered with novel commonsense distillation algorithms?The key intellectual challenge is to design a learning algorithm that achieves a competitive level of commonsense acquisition, without relying on the benefits of scale.In particular, we study generative models of commonsense knowledge, focusing on the task of generating generics, statements of commonsense facts about everyday concepts, e.g., birds can fly.We introduce I2D2, a novel commonsense distillation framework that loosely follows West et al. ( 2022)'s Symbolic Knowledge Distillation but breaks the dependence on the extremescale teacher model with two innovations: (1) the novel adaptation of NeuroLogic Decoding (Lu et al., 2021) to enhance the generation quality of the weak, off-the-shelf language models, and (2) self-imitation learning to iteratively learn from the model's own enhanced commonsense acquisition capabilities.Empirical results suggest that scale is not the only way, as novel algorithms can be a promising alternative.Moreover, our study leads to a new corpus of generics, Gen-A-tomic, that is the largest and highest-quality available to date.
DSP: Discriminative Soft Prompts for Zero-Shot Entity and Relation Extraction

Prompt-based methods have shown their efficacy in transferring general knowledge within pre-trained language models (PLMs) for lowresource scenarios.Typically, prompt-based methods convert downstream tasks to clozestyle problems and map all labels to verbalizers.However, when applied to zero-shot entity and relation extraction, vanilla promptbased methods may struggle with the limited coverage of verbalizers to labels and the slow inference speed.In this work, we propose a novel Discriminative Soft Prompts (DSP) approach to take advantage of the prompt-based methods to strengthen the transmission of general knowledge.Specifically, we develop a discriminative prompt method, which reformulates zero-shot tasks into token discrimination tasks without having to construct verbalizers.Furthermore, to improve the inference speed of the prompt-based methods, we design a soft prompt co-reference strategy, which leverages soft prompts to approximately refer to the vector representation of text tokens.The experimental results demonstrate that, our model outperforms baselines on two zero-shot entity recognition datasets with higher inference speed, and obtains a 7.5% average relation F1score improvement over previous state-of-theart models on Wiki-ZSL and FewRel.
DT-Solver: Automated Theorem Proving with Dynamic-Tree Sampling Guided by Proof-level Value Function

Recent advances in neural theorem-proving resort to large language models and tree searches.When proving a theorem, a language model advises single-step actions based on the current proving state and the tree search finds a sequence of correct steps using actions given by the language model.However, prior works often conduct constant computation efforts for each proving state while ignoring that the hard states often need more exploration than easy states.Moreover, they evaluate and guide the proof search solely depending on the current proof state instead of considering the whole proof trajectory as human reasoning does.Here, to accommodate general theorems, we propose a novel Dynamic-Tree Driven Theorem Solver (DT-Solver) by guiding the search procedure with state confidence and proof-level values.Specifically, DT-Solver introduces a dynamic-tree Monte-Carlo search algorithm, which dynamically allocates computing budgets for different state confidences, guided by a new proof-level value function to discover proof states that require substantial exploration.Experiments on two popular theorem-proving datasets, PISA and Mathlib, show significant performance gains by our DT-Solver over the state-of-the-art approaches, with a 6.65% improvement on average in terms of success rate.And especially under low computing resource settings (11.03% improvement on average).
Introducing Semantics into Speech Encoders

Recent studies find existing self-supervised speech encoders contain primarily acoustic rather than semantic information.As a result, pipelined supervised automatic speech recognition (ASR) to large language model (LLM) systems achieve state-of-the-art results on semantic spoken language tasks by utilizing rich semantic representations from the LLM.These systems come at the cost of labeled audio transcriptions, which is expensive and time-consuming to obtain.We propose a taskagnostic unsupervised way of incorporating semantic information from LLMs into selfsupervised speech encoders without labeled audio transcriptions.By introducing semantics, we improve existing speech encoder spoken language understanding (SLU) performance by over 5% on intent classification (IC), with modest gains in named entity resolution (NER) and slot filling (SF), and spoken question answering (SQA) FF1 score by over 2%.Our approach, which uses no ASR data, achieves similar performance as methods trained on over 100 hours of labeled audio transcripts, demonstrating the feasibility of unsupervised semantic augmentations to existing speech encoders.
Attend, Select and Eliminate: Accelerating Multi-turn Response Selection with Dual-attention-based Content Elimination

Although the incorporation of pre-trained language models (PLMs) significantly pushes the research frontier of multi-turn response selection, it brings a new issue of heavy computation costs.To alleviate this problem and make the PLM-based response selection model both effective and efficient, we propose an inference framework together with a post-training strategy that builds upon any pre-trained transformer-based response selection models to accelerate inference by progressively selecting and eliminating unimportant content under the guidance of context-response dual-attention.Specifically, at each transformer layer, we first identify the importance of each word based on context-to-response and response-to-context attention, then select a number of unimportant words to be eliminated following a retention configuration derived from evolutionary search while passing the rest of the representations into deeper layers.To mitigate the training-inference gap posed by content elimination, we introduce a posttraining strategy where we use knowledge distillation to force the model with progressively eliminated content to mimic the predictions of the original model with no content elimination.Experiments on three benchmarks indicate that our method can effectively speeds-up SOTA models without much performance degradation and shows a better trade-off between speed and performance than previous methods.
JECC: Commonsense Reasoning Tasks Derived from Interactive Fictions

Commonsense reasoning simulates the human ability to make presumptions about our physical world, and it is an essential cornerstone in building general AI systems.We propose a new commonsense reasoning dataset based on human's Interactive Fiction (IF) gameplay walkthroughs as human players demonstrate plentiful and diverse commonsense reasoning.The new dataset provides a natural mixture of various reasoning types and requires multi-hop reasoning.Moreover, the IF game-based construction procedure requires much less human interventions than previous ones.Different from existing benchmarks, our dataset focuses on the assessment of functional commonsense knowledge rules rather than factual knowledge.Hence, in order to achieve higher performance on our tasks, models need to effectively utilize such functional knowledge to infer the outcomes of actions, rather than relying solely on memorizing facts.Experiments show that the introduced dataset is challenging to previous machine reading models as well as the new large language models with a significant 20% performance gap compared to human experts.
Question-Interlocutor Scope Realized Graph Modeling over Key Utterances for Dialogue Reading Comprehension

We focus on dialogue reading comprehension (DRC) that extracts answers from dialogues.Compared to standard RC tasks, DRC has raised challenges because of the complex speaker information and noisy dialogue context.Essentially, the challenges come from the speaker-centric nature of dialogue utterances -an utterance is usually insufficient in its surface form, but requires to incorporate the role of its speaker and the dialogue context to fill the latent pragmatic and intention information.We propose to deal with these problems in two folds.First, we propose a new keyutterances-extracting method, which can realize more answer-contained utterances.Second, based on the extracted utterances, we then propose a Question-Interlocutor Scope Realized Graph (QuISG).QuISG involves the question and question-mentioning speaker as nodes.To realize interlocutor scopes, utterances are connected with corresponding speakers in the dialogue.Experiments on the benchmarks show that our method achieves state-of-the-art performance against previous works. 1