Reviewerly/exHarmony huggingface dataset:
GrounDialog: A Dataset for Repair and Grounding in Task-oriented Spoken Dialogues for Language Learning

Improving conversational proficiency is a key target for students learning a new language.While acquiring conversational proficiency, students must learn the linguistic mechanisms of Repair and Grounding (R&G) to negotiate meaning and find common ground with their interlocutor so conversational breakdowns can be resolved.Task-oriented Spoken Dialogue Systems (SDS) have long been sought as a tool to hone conversational proficiency.However, the R&G patterns for language learners interacting with a task-oriented spoken dialogue system are not reflected explicitly in any existing datasets.Therefore, to move the needle in Spoken Dialogue Systems for language learning we present GrounDialog: an annotated dataset of spoken conversations where we elicit a rich set of R&G patterns.
Follow the Wisdom of the Crowd: Effective Text Generation via Minimum Bayes Risk Decoding

In open-ended natural-language generation, existing text decoding methods typically struggle to produce text which is both diverse and highquality.Greedy and beam search are known to suffer from text degeneration and linguistic diversity issues, while temperature, top-k, and nucleus sampling yield diverse but often lowerquality outputs.In this work, we build upon Minimum Bayes Risk Decoding (MBRD), a family of decoding methods based on Bayesian risk minimization, to address this diversityquality trade-off.Inspired by the principle of the wisdom of the crowd, MBRD seeks to select a candidate from a pool of candidates that has the least expected risk under a generative model according to a given utility function.The crowd of candidates serves as an approximation for the distribution over human-generated references.We show that MBRD generalizes numerous decoding methods, including majority voting, and can be used as a drop-in replacement for existing sampling methods.Across a wide range of tasks-such as summarization, datato-text, translation, and textual style transfer-MBRD yields 3-7 ROUGE and BLEU point improvements, including state-of-the-art results on WebNLG and WMT'16. 1
Better Zero-Shot Reasoning with Self-Adaptive Prompting

Modern large language models (LLMs) have demonstrated impressive capabilities at sophisticated tasks, often through step-by-step reasoning similar to humans.This is made possible by their strong few and zero-shot abilities -they can effectively learn from a handful of handcrafted, completed responses ("incontext examples"), or are prompted to reason spontaneously through specially designed triggers.Nonetheless, some limitations have been observed.First, performance in the few-shot setting is sensitive to the choice of examples, whose design requires significant human effort.Moreover, given the diverse downstream tasks of LLMs, it may be difficult or laborious to handcraft per-task labels.Second, while the zero-shot setting does not require handcrafting, its performance is limited due to the lack of guidance to the LLMs.To address these limitations, we propose Consistency-based Selfadaptive Prompting (COSP), a novel prompt design method for LLMs.Requiring neither handcrafted responses nor ground-truth labels, COSP selects and builds the set of examples from the LLM zero-shot outputs via carefully designed criteria that combine consistency, diversity and repetition.In the zero-shot setting for three different LLMs, we show that using only LLM predictions, COSP improves performance up to 15% compared to zero-shot baselines and matches or exceeds few-shot baselines for a range of reasoning tasks.
Improving Self-training for Cross-lingual Named Entity Recognition with Contrastive and Prototype Learning

In cross-lingual named entity recognition (NER), self-training is commonly used to bridge the linguistic gap by training on pseudolabeled target-language data.However, due to sub-optimal performance on target languages, the pseudo labels are often noisy and limit the overall performance.In this work, we aim to improve self-training for cross-lingual NER by combining representation learning and pseudo label refinement in one coherent framework.Our proposed method, namely ContProto mainly comprises two components: (1) contrastive self-training and (2) prototype-based pseudo-labeling.Our contrastive self-training facilitates span classification by separating clusters of different classes, and enhances crosslingual transferability by producing closelyaligned representations between the source and target language.Meanwhile, prototype-based pseudo-labeling effectively improves the accuracy of pseudo labels during training.We evaluate ContProto on multiple transfer pairs, and experimental results show our method brings in substantial improvements over current stateof-the-art methods. 1
Exploring Lottery Prompts for Pre-trained Language Models

Consistently scaling pre-trained language models (PLMs) imposes substantial burdens on model adaptation, necessitating more efficient alternatives to conventional fine-tuning.Given the advantage of prompting in the zero-shot setting and the observed performance fluctuation among different prompts, we explore the instance-level prompt and their generalizability.By searching through the prompt space, we first validate the assumption that for every instance, there is almost always a lottery prompt that induces the correct prediction from the PLM, and such prompt can be obtained at a low cost thanks to the inherent ability of PLMs.Meanwhile, we find that some strong lottery prompts have high performance over the whole training set, and they are equipped with distinguishable linguistic features.Lastly, we attempt to generalize the searched strong lottery prompts to unseen data with prompt ensembling method without any parameter tuning.Experiments are conducted on various types of NLP classification tasks and demonstrate that the proposed method can achieve comparable results with other gradient-free and optimization-free baselines.
AI Coach Assist: An Automated Approach for Call Recommendation in Contact Centers for Agent Coaching

In recent years, the utilization of Artificial Intelligence (AI) in the contact center industry is on the rise.One area where AI can have a significant impact is in the coaching of contact center agents.By analyzing call transcripts using Natural Language Processing (NLP) techniques, it would be possible to quickly determine which calls are most relevant for coaching purposes.In this paper, we present "AI Coach Assist", which leverages the pre-trained transformerbased language models to determine whether a given call is coachable or not based on the quality assurance (QA) questions asked by the contact center managers or supervisors.The system was trained and evaluated on a large dataset collected from real-world contact centers and provides an effective way to recommend calls to the contact center managers that are more likely to contain coachable moments.Our experimental findings demonstrate the potential of AI Coach Assist to improve the coaching process, resulting in enhancing the performance of contact center agents.
A Study on Knowledge Distillation from Weak Teacher for Scaling Up Pre-trained Language Models

Distillation from Weak Teacher (DWT) is a method of transferring knowledge from a smaller, weaker teacher model to a larger student model to improve its performance.Previous studies have shown that DWT can be effective in the vision domain and natural language processing (NLP) pre-training stage.Specifically, DWT shows promise in practical scenarios, such as enhancing new generation or larger models using pre-trained yet older or smaller models and lacking a resource budget.However, the optimal conditions for using DWT have yet to be fully investigated in NLP pretraining.Therefore, this study examines three key factors to optimize DWT, distinct from those used in the vision domain or traditional knowledge distillation.These factors are: (i) the impact of teacher model quality on DWT effectiveness, (ii) guidelines for adjusting the weighting value for DWT loss, and (iii) the impact of parameter remapping as a student model initialization technique for DWT.
Query Encoder Distillation via Embedding Alignment is a Strong Baseline Method to Boost Dense Retriever Online Efficiency

The information retrieval community has made significant progress in improving the efficiency of Dual Encoder (DE) dense passage retrieval systems, making them suitable for latencysensitive settings.However, many proposed procedures are often too complex or resourceintensive, which makes it difficult for practitioners to adopt them or identify sources of empirical gains.Therefore, in this work, we propose a trivially simple recipe to serve as a baseline method for boosting the efficiency of DE retrievers leveraging an asymmetric architecture.Our results demonstrate that even a 2-layer, BERT-based query encoder can still retain 92.5% of the full DE performance on the BEIR benchmark via unsupervised distillation and proper student initialization.We hope that our findings will encourage the community to re-evaluate the trade-offs between method complexity and performance improvements.
ALEXSIS+: Improving Substitute Generation and Selection for Lexical Simplification with Information Retrieval

Lexical simplification (LS) automatically replaces words that are deemed difficult to understand for a given target population with simpler alternatives, whilst preserving the meaning of the original sentence.The TSAR-2022 shared task on LS provided participants with a multilingual lexical simplification test set.It contained nearly 1,200 complex words in English, Portuguese, and Spanish and presented multiple candidate substitutions for each complex word.The competition did not make training data available; therefore, teams had to use either off-the-shelf pre-trained large language models (LLMs) or out-domain data to develop their LS systems.As such, participants were unable to fully explore the capabilities of LLMs by re-training and/or fine-tuning them on in-domain data.To address this important limitation, we present ALEXSIS+, a multilingual dataset in the aforementioned three languages, and ALEXSIS++, an English monolingual dataset that together contains more than 50,000 unique sentences retrieved from news corpora and annotated with cosine similarities to the original complex word and sentence.Using these additional contexts, we are able to generate new high-quality candidate substitutions that improve LS performance on the TSAR-2022 test set regardless of the language or model.
shs-nlp at RadSum23: Domain-Adaptive Pre-training of Instruction-tuned LLMs for Radiology Report Impression Generation

Instruction-tuned generative large language models (LLMs), such as ChatGPT and Bloomz, possess excellent generalization abilities.However, they face limitations in understanding radiology reports, particularly when generating the IMPRESSIONS section from the FIND-INGS section.These models tend to produce either verbose or incomplete IMPRESSIONS, mainly due to insufficient exposure to medical text data during training.We present a system that leverages large-scale medical text data for domain-adaptive pre-training of instructiontuned LLMs, enhancing their medical knowledge and performance on specific medical tasks.We demonstrate that this system performs better in a zero-shot setting compared to several pretrain-and-finetune adaptation methods on the IMPRESSIONS generation task.Furthermore, it ranks 1st among participating systems in Task 1B: Radiology Report Summarization at the BioNLP 2023 workshop.
Complex Reasoning in Natural Languag

Wenting Zhao, Mor Geva, Bill Yuchen Lin, Michihiro Yasunaga, Aman Madaan, Tao Yu. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 6: Tutorial Abstracts). 2023.
Varta: A Large-Scale Headline-Generation Dataset for Indic Languages

We present Vārta, a large-scale multilingual dataset for headline generation in Indic languages.This dataset includes 41.8 million news articles in 14 different Indic languages (and English), which come from a variety of high-quality sources.To the best of our knowledge, this is the largest collection of curated articles for Indic languages currently available.We use the data collected in a series of experiments to answer important questions related to Indic NLP and multilinguality research in general.We show that the dataset is challenging even for state-of-the-art abstractive models and that they perform only slightly better than extractive baselines.Owing to its size, we also show that the dataset can be used to pretrain strong language models that outperform competitive baselines in both NLU and NLG benchmarks.The data and models are available at https://github.com/rahular/varta.
XDailyDialog: A Multilingual Parallel Dialogue Corpus

High-quality corpora are significant to the development of dialogue models.However, most existing corpora for open-domain dialogue modeling are limited to a single language.The absence of multilingual open-domain dialog corpora not only limits the research on multilingual or cross-lingual transfer learning but also hinders the development of robust opendomain dialogue systems that can be deployed in other parts of the world.In this paper, we provide a multilingual parallel open-domain dialog dataset, XDailyDialog, 1 to enable researchers to explore the challenging task of multilingual and cross-lingual open-domain dialogue.XDailyDialog includes 13K dialogues aligned across 4 languages (52K dialogues and 410K utterances in total).We then propose a dialogue generation model, kNN-Chat, which has a novel kNN-search mechanism to support unified response retrieval for monolingual, multilingual, and cross-lingual dialogue.Experiment results show the effectiveness of this framework.
RECAP: Retrieval-Enhanced Context-Aware Prefix Encoder for Personalized Dialogue Response Generation

Endowing chatbots with a consistent persona is essential to an engaging conversation, yet it remains an unresolved challenge.In this work, we propose a new retrieval-enhanced approach for personalized response generation.Specifically, we design a hierarchical transformer retriever trained on dialogue domain data to perform personalized retrieval and a context-aware prefix encoder that fuses the retrieved information to the decoder more effectively.Extensive experiments on a real-world dataset demonstrate the effectiveness of our model at generating more fluent and personalized responses.We quantitatively evaluate our model's performance under a suite of human and automatic metrics and find it to be superior compared to state-of-the-art baselines on English Reddit conversations. 1
Distinguish Before Answer: Generating Contrastive Explanation as Knowledge for Commonsense Question Answering

Existing knowledge-enhanced methods have achieved remarkable results in certain Q&A tasks via obtaining diverse knowledge from different knowledge bases.However, limited by the properties of retrieved knowledge, they still have trouble benefiting from both the knowledge relevance and distinguishment simultaneously.To address the challenge, we propose CPACE, a Concept-centric Prompt-bAsed Contrastive Explanation Generation model, which aims to convert obtained symbolic knowledge into the contrastive explanation for better distinguishing the differences among given candidates.Firstly, following previous works, we retrieve different types of symbolic knowledge with a concept-centric knowledge extraction module.After that, we generate corresponding contrastive explanation using acquired symbolic knowledge and explanation prompt as guidance for better modeling the knowledge distinguishment and interpretability.Finally, we regard the generated contrastive explanation as external knowledge for downstream task enhancement.We conduct a series of experiments on three widely-used questionanswering datasets: CSQA, QASC, and OBQA.Experimental results demonstrate that with the help of generated contrastive explanation, our CPACE model achieves new SOTA on CSQA (89.8% on the testing set, 0.9% higher than human performance), and gains impressive improvement on QASC and OBQA (4.2% and 3.5%, respectively).
Fusion or Defusion? Flexible Vision-and-Language Pre-Training

Existing approaches in the vision-and-language pre-training (VLP) paradigm mainly deploy either fusion-based encoders or dual-encoders, failing to achieve both effectiveness and efficiency in downstream multimodal tasks.In this paper, we build a flexible VLP model by incorporating cross-modal fusions into a dualencoder architecture, where the introduced fusion modules can be easily decoupled from the dual encoder so as to switch the model to a fusion-free one.To better absorb cross-modal features from the fusion modules, we design a cross-modal knowledge transfer strategy along with other comprehensive pre-training tasks to guide the training process, which can further strengthen both the fusion-based and fusionfree representation learning.Extensive experiments conducted on various downstream visionlanguage tasks show that our proposed model is well-equipped with effectiveness as well as efficiency, demonstrating a superior performance compared with other strong VLP models.
The Devil is in the Details: On the Pitfalls of Event Extraction Evaluation

Event extraction (EE) is a crucial task aiming at extracting events from texts, which includes two subtasks: event detection (ED) and event argument extraction (EAE).In this paper, we check the reliability of EE evaluations and identify three major pitfalls: (1) The data preprocessing discrepancy makes the evaluation results on the same dataset not directly comparable, but the data preprocessing details are not widely noted and specified in papers.(2) The output space discrepancy of different model paradigms makes differentparadigm EE models lack grounds for comparison and also leads to unclear mapping issues between predictions and annotations.(3) The absence of pipeline evaluation of many EAE-only works makes them hard to be directly compared with EE works and may not well reflect the model performance in realworld pipeline scenarios.We demonstrate the significant influence of these pitfalls through comprehensive meta-analyses of recent papers and empirical experiments.To avoid these pitfalls, we suggest a series of remedies, including specifying data preprocessing, standardizing outputs, and providing pipeline evaluation results.To help implement these remedies, we develop a consistent evaluation framework OMNIEVENT, which can be obtained from https://github.com/THU-KEG/OmniEvent.
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues

User Satisfaction Modeling (USM) is one of the popular choices for task-oriented dialogue systems evaluation, where user satisfaction typically depends on whether the user's task goals were fulfilled by the system.Task-oriented dialogue systems use task schema, which is a set of task attributes, to encode the user's task goals.Existing studies on USM neglect explicitly modeling the user's task goals fulfillment using the task schema.In this paper, we propose SG-USM, a novel schema-guided user satisfaction modeling framework.It explicitly models the degree to which the user's preferences regarding the task attributes are fulfilled by the system for predicting the user's satisfaction level.SG-USM employs a pre-trained language model for encoding dialogue context and task attributes.Further, it employs a fulfillment representation layer for learning how many task attributes have been fulfilled in the dialogue, an importance predictor component for calculating the importance of task attributes.Finally, it predicts the user satisfaction based on task attribute fulfillment and task attribute importance.Experimental results on benchmark datasets (i.e.MWOZ, SGD, ReDial, and JDDC) show that SG-USM consistently outperforms competitive existing methods.Our extensive analysis demonstrates that SG-USM can improve the interpretability of user satisfaction modeling, has good scalability as it can effectively deal with unseen tasks and can also effectively work in low-resource settings by leveraging unlabeled data. 1 * Work done while Yue Feng was an intern at Amazon, Alexa Shopping.
Can Diffusion Model Achieve Better Performance in Text Generation ? Bridging the Gap between Training and Inference !

Diffusion models have been successfully adapted to text generation tasks by mapping the discrete text into the continuous space.However, there exist nonnegligible gaps between training and inference, owing to the absence of the forward process during inference.Thus, the model only predicts based on the previously generated reverse noise rather than the noise computed by the forward process.Besides, the widely-used downsampling strategy in speeding up the inference will cause the mismatch of diffusion trajectories between training and inference.To understand and mitigate the above two types of training-inference discrepancies, we launch a thorough preliminary study.Based on our observations, we propose two simple yet effective methods to bridge the gaps mentioned above, named Distance Penalty and Adaptive Decay Sampling.Extensive experiments on 6 generation tasks confirm the superiority of our methods, which can achieve 100× → 200× speedup with better performance.Our code is available at https://github.com/CODINNLG/Bridge_Gap_Diffusion.
Disentangling Text Representation With Counter-Template For Unsupervised Opinion Summarization

Approaches for unsupervised opinion summarization are generally based on the reconstruction model and generate a summary by decoding the aggregated representation of inputs.Recent work has shown that aggregating via simple average leads to vector degeneration, generating the generic summary.To tackle the challenge, some approaches select the inputs before aggregating.However, we argue that the selection is too coarse as not all information in each input is equally essential for the summary.For example, the content information such as "great coffee maker, easy to set up" is more valuable than the pattern such as "this is a great product".Therefore, we propose a novel framework for unsupervised opinion summarization based on text representation disentanglement with counter-template.In specific, a disentangling module is added to the encoder-decoder architecture which decouples the input text representation into two parts: content and pattern.To capture the pattern information, a countertemplate is utilized as supervision, which is automatically generated based on contrastive learning.Experimental results on two benchmark datasets show that the proposed approach outperforms the state-of-the-art baselines on both quality and stability.
The HW-TSC’s Speech-to-Speech Translation System for IWSLT 2023

This paper describes our work on the IWSLT2023 Speech-to-Speech task.Our proposed cascaded system consists of an ensemble of Conformer and S2T-Transformer-based ASR models, a Transformer-based MT model, and a Diffusion-based TTS model.Our primary focus in this competition was to investigate the modeling ability of the Diffusion model for TTS tasks in high-resource scenarios and the role of TTS in the overall S2S task.To this end, we proposed DTS, an end-to-end diffusionbased TTS model that takes raw text as input and generates waveform by iteratively denoising on pure Gaussian noise.Compared to previous TTS models, the speech generated by DTS is more natural and performs better in codeswitching scenarios.As the training process is end-to-end, it is relatively straightforward.Our experiments demonstrate that DTS outperforms other TTS models on the GigaS2S benchmark, and also brings positive gain for the entire S2S system.