Reviewerly/exHarmony huggingface dataset:
End-to-End Task-Oriented Dialogue Systems Based on Schema

This paper presents a schema-aware end-toend neural network model for handling taskoriented dialogues based on a dynamic set of slots within a schema.Contrary to existing studies that proposed end-to-end approaches for task-oriented dialogue systems by relying on a unified schema across domains, we design our approach to support a domain covering multiple services where diverse schemas are available.To enable better generalizability among services and domains with different schemas, we supply the schema's context information including slot descriptions and value constraints to the model.The experimental results on a well-known Schema-Guided Dialogue (SGD) dataset demonstrated the performance improvement by the proposed model compared to state-of-the-art baselines in terms of end-to-end modeling, dialogue state tracking task, and generalization on new services and domains using a limited number of dialogues. 1
Federated Domain Adaptation for Named Entity Recognition via Distilling with Heterogeneous Tag Sets

Federated learning involves collaborative training with private data from multiple platforms, while not violating data privacy. We study the problem of federated domain adaptation for Named Entity Recognition (NER), where we seek to transfer knowledge across different platforms with data of multiple domains. In addition, we consider a practical and challenging scenario, where NER datasets of different platforms of federated learning are annotated with heterogeneous tag sets, i.e., different sets of entity types. The goal is to train a global model with federated learning, such that it can predict with a complete tag set, i.e., with all the occurring entity types for data across all platforms. To cope with the heterogeneous tag sets in a multi-domain setting, we propose a distillation approach along with a mechanism of instance weighting to facilitate knowledge transfer across platforms. Besides, we release two re-annotated clinic NER datasets, for testing the proposed method in the clinic domain. Our method shows superior empirical performance for NER with federated learning.
MeetingBank: A Benchmark Dataset for Meeting Summarization

As the number of recorded meetings increases, it becomes increasingly important to utilize summarization technology to create useful summaries of these recordings.However, there is a crucial lack of annotated meeting corpora for developing this technology, as it can be hard to collect meetings, especially when the topics discussed are confidential.Furthermore, meeting summaries written by experienced writers are scarce, making it hard for abstractive summarizers to produce sensible output without a reliable reference.This lack of annotated corpora has hindered the development of meeting summarization technology.In this paper, we present MeetingBank, a new benchmark dataset of city council meetings over the past decade.Meeting-Bank is unique among other meeting corpora due to its divide-and-conquer approach, which involves dividing professionally written meeting minutes into shorter passages and aligning them with specific segments of the meeting.This breaks down the process of summarizing a lengthy meeting into smaller, more manageable tasks.The dataset provides a new testbed of various meeting summarization systems and also allows the public to gain insight into how council decisions are made.We make the collection, including meeting video links, transcripts, reference summaries, agenda, and other metadata, publicly available to facilitate the development of better meeting summarization techniques. 1
Disambiguated Lexically Constrained Neural Machine Translation

Lexically constrained neural machine translation (LCNMT), which controls the translation generation with pre-specified constraints, is important in many practical applications.Current approaches to LCNMT typically assume that the pre-specified lexical constraints are contextually appropriate.This assumption limits their application to real-world scenarios where a source lexicon may have multiple target constraints, and disambiguation is needed to select the most suitable one.In this paper, we propose disambiguated LCNMT (D-LCNMT) to solve the problem.D-LCNMT is a robust and effective two-stage framework that disambiguates the constraints based on contexts at first, then integrates the disambiguated constraints into LCNMT.Experimental results show that our approach outperforms strong baselines including existing data augmentation based approaches on benchmark datasets, and comprehensive experiments in scenarios where a source lexicon corresponds to multiple target constraints demonstrate the constraint disambiguation superiority of our approach.
SAE-NTM: Sentence-Aware Encoder for Neural Topic Modeling

Incorporating external knowledge, such as pretrained language models (PLMs), into neural topic modeling has achieved great success in recent years.However, employing PLMs for topic modeling generally ignores the maximum sequence length of PLMs and the interaction between external knowledge and bag-of-words (BOW).To this end, we propose a sentenceaware encoder for neural topic modeling, which adopts fine-grained sentence embeddings as external knowledge to entirely utilize the semantic information of input documents.We introduce sentence-aware attention for document representation, where BOW enables the model to attend on topical sentences that convey topic-related cues.Experiments on three benchmark datasets show that our framework outperforms other state-of-the-art neural topic models in topic coherence.Further, we demonstrate that the proposed approach can yield better latent document-topic features through improvement on the document classification.
Prosody-TTS: Improving Prosody with Masked Autoencoder and Conditional Diffusion Model For Expressive Text-to-Speech

Expressive text-to-speech aims to generate high-quality samples with rich and diverse prosody, which is hampered by dual challenges: 1) prosodic attributes in highly dynamic voices are difficult to capture and model without intonation; and 2) highly multimodal prosodic representations cannot be well learned by simple regression (e.g., MSE) objectives, which causes blurry and over-smoothing predictions.This paper proposes Prosody-TTS, a two-stage pipeline that enhances prosody modeling and sampling by introducing several components: 1) a self-supervised masked autoencoder to model the prosodic representation without relying on text transcriptions or local prosody attributes, which ensures to cover diverse speaking voices with superior generalization; and 2) a diffusion model to sample diverse prosodic patterns within the latent space, which prevents TTS models from generating samples with dull prosodic performance.Experimental results show that Prosody-TTS achieves new state-of-the-art in text-to-speech with natural and expressive synthesis.Both subjective and objective evaluation demonstrate that it exhibits superior audio quality and prosody naturalness with rich and diverse prosodic attributes.
Dynamic Structured Neural Topic Model with Self-Attention Mechanism

This study presents a dynamic structured neural topic model, which can handle the timeseries development of topics while capturing their dependencies.Our model captures the topic branching and merging processes by modeling topic dependencies based on a selfattention mechanism.Additionally, we introduce citation regularization, which induces attention weights to represent citation relations by modeling text and citations jointly.Our model outperforms a prior dynamic embedded topic model (Dieng et al., 2019) regarding perplexity and coherence, while maintaining sufficient diversity across topics.Furthermore, we confirm that our model can potentially predict emerging topics from academic literature.
Understanding and Bridging the Modality Gap for Speech Translation

How to achieve better end-to-end speech translation (ST) by leveraging (text) machine translation (MT) data?Among various existing techniques, multi-task learning is one of the effective ways to share knowledge between ST and MT in which additional MT data can help to learn source-to-target mapping.However, due to the differences between speech and text, there is always a gap between ST and MT.In this paper, we first aim to understand this modality gap from the target-side representation differences, and link the modality gap to another well-known problem in neural machine translation: exposure bias.We find that the modality gap is relatively small during training except for some difficult cases, but keeps increasing during inference due to the cascading effect.To address these problems, we propose the Cross-modal Regularization with Scheduled Sampling (CRESS) method.Specifically, we regularize the output predictions of ST and MT, whose target-side contexts are derived by sampling between ground truth words and self-generated words with a varying probability.Furthermore, we introduce token-level adaptive training which assigns different training weights to target tokens to handle difficult cases with large modality gaps.Experiments and analysis show that our approach effectively bridges the modality gap, and achieves promising results in all eight directions of the MuST-C dataset.
Causal Intervention for Mitigating Name Bias in Machine Reading Comprehension

Machine Reading Comprehension (MRC) is to answer questions based on a given passage, which has made great achievements using pretrained Language Models (LMs).We study the robustness of MRC models to names which is flexible and repeatability.MRC models based on LMs may overuse the name information to make predictions, which causes the representation of names to be non-interchangeable, called name bias.In this paper, we propose a novel Causal Interventional paradigm for MRC (CI4MRC) to mitigate name bias.Specifically, we uncover that the pre-trained knowledge concerning names is indeed a confounder by analyzing the causalities among the pre-trained knowledge, context representation and answers based on a Structural Causal Model (SCM).We develop effective CI4MRC algorithmic implementations to constrain the confounder based on the neuron-wise and token-wise adjustments.Experiments demonstrate that our proposed CI4MRC effectively mitigates the name bias and achieves competitive performance on the original SQuAD.Moreover, our method is general to various pre-trained LMs and performs robustly on the adversarial datasets.
Enhanced Chart Understanding via Visual Language Pre-training on Plot Table Pairs

Building cross-model intelligence that can understand charts and communicate the salient information hidden behind them is an appealing challenge in the vision and language (V+L) community.The capability to uncover the underlined table data of chart figures is a critical key to automatic chart understanding.We introduce ChartT5, a V+L model that learns how to interpret table information from chart images via cross-modal pre-training on plot table pairs.Specifically, we propose two novel pre-training objectives: Masked Header Prediction (MHP) and Masked Value Prediction (MVP) to facilitate the model with different skills to interpret the table information.We have conducted extensive experiments on chart question answering and chart summarization to verify the effectiveness of the proposed pre-training strategies.In particular, on the ChartQA benchmark, our ChartT5 outperforms the state-of-the-art nonpretraining methods by over 8% performance gains.
WeCheck: Strong Factual Consistency Checker via Weakly Supervised Learning

A crucial issue of current text generation models is that they often uncontrollably generate text that is factually inconsistent with inputs.Due to lack of annotated data, existing factual consistency metrics usually train evaluation models on synthetic texts or directly transfer from other related tasks, such as question answering (QA) and natural language inference (NLI).Bias in synthetic text or upstream tasks makes them perform poorly on text actually generated by language models, especially for general evaluation for various tasks.To alleviate this problem, we propose a weakly supervised framework named WeCheck that is directly trained on actual generated samples from language models with weakly annotated labels.WeCheck first utilizes a generative model to infer the factual labels of generated samples by aggregating weak labels from multiple resources.Next, we train a simple noise-aware classification model as the target metric using the inferred weakly supervised information.Comprehensive experiments on various tasks demonstrate the strong performance of WeCheck, achieving an average absolute improvement of 3.3% on the TRUE benchmark over 11B state-of-the-art methods using only 435M parameters.Furthermore, it is up to 30× faster than previous evaluation methods, greatly improving the accuracy and efficiency of factual consistency evaluation.
A Measure-Theoretic Characterization of Tight Language Models

Language modeling, a central task in natural language processing, involves estimating a probability distribution over strings.In most cases, the estimated distribution sums to 1 over all finite strings.However, in some pathological cases, probability mass can "leak" onto the set of infinite sequences.In order to characterize the notion of leakage more precisely, this paper offers a measure-theoretic treatment of language modeling.We prove that many popular language model families are in fact tight, meaning that they will not leak in this sense.We also generalize characterizations of tightness proposed in previous works.
Modeling Appropriate Language in Argumentation

Online discussion moderators must make adhoc decisions about whether the contributions of discussion participants are appropriate or should be removed to maintain civility.Existing research on offensive language and the resulting tools cover only one aspect among many involved in such decisions.The question of what is considered appropriate in a controversial discussion has not yet been systematically addressed.In this paper, we operationalize appropriate language in argumentation for the first time.In particular, we model appropriateness through the absence of flaws, grounded in research on argument quality assessment, especially in aspects from rhetoric.From these, we derive a new taxonomy of 14 dimensions that determine inappropriate language in online discussions.Building on three argument quality corpora, we then create a corpus of 2191 arguments annotated for the 14 dimensions.Empirical analyses support that the taxonomy covers the concept of appropriateness comprehensively, showing several plausible correlations with argument quality dimensions.Moreover, results of baseline approaches to assessing appropriateness suggest that all dimensions can be modeled computationally on the corpus.
CTC-based Non-autoregressive Speech Translation

Combining end-to-end speech translation (ST) and non-autoregressive (NAR) generation is promising in language and speech processing for their advantages of less error propagation and low latency.In this paper, we investigate the potential of connectionist temporal classification (CTC) for non-autoregressive speech translation (NAST).In particular, we develop a model consisting of two encoders that are guided by CTC to predict the source and target texts, respectively.Introducing CTC into NAST on both language sides has obvious challenges: 1) the conditional independent generation somewhat breaks the interdependency among tokens, and 2) the monotonic alignment assumption in standard CTC does not hold in translation tasks.In response, we develop a prediction-aware encoding approach and a cross-layer attention approach to address these issues.We also use curriculum learning to improve convergence of training.Experiments on the MuST-C ST benchmarks show that our NAST model achieves an average BLEU score of 29.5 with a speed-up of 5.67×, which is comparable to the autoregressive counterpart and even outperforms the previous best result of 0.9 BLEU points 1 .
Bridging the Granularity Gap for Acoustic Modeling

While Transformer has become the de-facto standard for speech, modeling upon the finegrained frame-level features remains an open challenge of capturing long-distance dependencies and distributing the attention weights.We propose Progressive Down-Sampling (PDS) which gradually compresses the acoustic features into coarser-grained units containing more complete semantic information, like textlevel representation.In addition, we develop a representation fusion method to alleviate information loss that occurs inevitably during high compression.In this way, we compress the acoustic features into 1/32 of the initial length while achieving better or comparable performances on the speech recognition task.And as a bonus, it yields inference speedups ranging from 1.20× to 1.47×.By reducing the modeling burden, we also achieve competitive results when training on the more challenging speech translation task 1 .
G-Tuning: Improving Generalization of Pre-trained Language Models with Generative Adversarial Network

The generalization ability of pre-trained language models (PLMs) in downstream tasks is heavily influenced by fine-tuning.The objective of fine-tuning is to transform the latent representation of PLMs from a universal space to a target space, allowing the model to be applied to downstream tasks with the capability of generalizing to unseen samples.However, the effect of PLMs will be diminished when the training data coverage is insufficient, in which fine-tuning is inadequate to learn the complete mapping.In this study, we propose a new finetuning framework, referred to as G-Tuning, that aims to preserve the generalization ability of PLMs in downstream tasks.Specifically, we integrate a generative adversarial network into the fine-tuning process to aid in the transformation of the latent representation in the entire space.Empirical evaluations on the GLUE benchmark, as well as two additional demanding scenarios involving domain and language generalization, demonstrate that G-Tuning can accurately map the universal representation to the target space, thus effectively enhancing the generalization performance of PLMs across various downstream tasks.
DePA: Improving Non-autoregressive Translation with Dependency-Aware Decoder

Non-autoregressive machine translation (NAT) models have lower translation quality than autoregressive translation (AT) models because NAT decoders do not depend on previous target tokens in the decoder input.We propose a novel and general Dependency-Aware Decoder (DePA) to enhance target dependency modeling in the decoder of fully NAT models from two perspectives: decoder self-attention and decoder input.First, we propose an autoregressive forward-backward pre-training phase before NAT training, which enables the NAT decoder to gradually learn bidirectional target dependencies for the final NAT training.Second, we transform the decoder input from the source language representation space to the target language representation space through a novel attentive transformation process, which enables the decoder to better capture target dependencies.DePA can be applied to any fully NAT models.Extensive experiments show that DePA consistently improves highly competitive and state-of-the-art fully NAT models on widely used WMT and IWSLT benchmarks by up to 1.88 BLEU gain, while maintaining the inference latency comparable to other fully NAT models.
ON-TRAC Consortium Systems for the IWSLT 2023 Dialectal and Low-resource Speech Translation Tasks

This paper describes the ON-TRAC consortium speech translation systems developed for IWSLT 2023 evaluation campaign.Overall, we participated in three speech translation tracks featured in the low-resource and dialect speech translation shared tasks, namely; i) spoken Tamasheq to written French, ii) spoken Pashto to written French, and iii) spoken Tunisian to written English.All our primary submissions are based on the end-to-end speech-to-text neural architecture using a pre-trained SAMU-XLSR model as a speech encoder and an mbart model as a decoder.The SAMU-XLSR model is built from the XLS-R 128 in order to generate language agnostic sentence-level embeddings.This building is driven by the LaBSE model trained on a multilingual text dataset.This architecture allows us to improve the input speech representations and achieve significant improvements compared to conventional endto-end speech translation systems.
The NPU-MSXF Speech-to-Speech Translation System for IWSLT 2023 Speech-to-Speech Translation Task

This paper describes the NPU-MSXF system for the IWSLT 2023 speech-to-speech translation (S2ST) task which aims to translate from English speech of multi-source to Chinese speech.The system is built in a cascaded manner consisting of automatic speech recognition (ASR), machine translation (MT), and text-to-speech (TTS).We make tremendous efforts to handle the challenging multi-source input.Specifically, to improve the robustness to multi-source speech input, we adopt various data augmentation strategies and a ROVERbased score fusion on multiple ASR model outputs.To better handle the noisy ASR transcripts, we introduce a three-stage fine-tuning strategy to improve translation accuracy.Finally, we build a TTS model with high naturalness and sound quality, which leverages a two-stage framework, using network bottleneck features as a robust intermediate representation for speaker timbre and linguistic content disentanglement.Based on the two-stage framework, pre-trained speaker embedding is leveraged as a condition to transfer the speaker timbre in the source English speech to the translated Chinese speech.Experimental results show that our system has high translation accuracy, speech naturalness, sound quality, and speaker similarity.Moreover, it shows good robustness to multi-source data.
FolkScope: Intention Knowledge Graph Construction for E-commerce Commonsense Discovery

Understanding users' intentions in e-commerce platforms requires commonsense knowledge.In this paper, we present FolkScope, an intention knowledge graph construction framework to reveal the structure of humans' minds about purchasing items.As commonsense knowledge is usually ineffable and not expressed explicitly, it is challenging to perform information extraction.Thus, we propose a new approach that leverages the generation power of large language models (LLMs) and humanin-the-loop annotation to semi-automatically construct the knowledge graph.LLMs first generate intention assertions via e-commercespecific prompts to explain shopping behaviors, where the intention can be an open reason or a predicate falling into one of 18 categories aligning with ConceptNet, e.g., IsA, MadeOf, UsedFor, etc. Then we annotate plausibility and typicality labels of sampled intentions as training data in order to populate human judgments to all automatic generations.Last, to structurize the assertions, we propose pattern mining and conceptualization to form more condensed and abstract knowledge.Extensive evaluations and studies demonstrate that our constructed knowledge graph can well model ecommerce knowledge and have many potential applications.Our codes and datasets are publicly available at