Reviewerly/exHarmony huggingface dataset:
Are Intermediate Layers and Labels Really Necessary? A General Language Model Distillation Method

The large scale of pre-trained language models poses a challenge for their deployment on various devices, with a growing emphasis on methods to compress these models, particularly knowledge distillation. However, current knowledge distillation methods rely on the model’s intermediate layer features and the golden labels (also called hard labels), which usually require aligned model architecture and enough labeled data respectively. Moreover, the parameters of vocabulary are usually neglected in existing methods. To address these problems, we propose a general language model distillation (GLMD) method that performs two-stage word prediction distillation and vocabulary compression, which is simple and surprisingly shows extremely strong performance. Specifically, GLMD supports more general application scenarios by eliminating the constraints of dimension and structure between models and the need for labeled datasets through the absence of intermediate layers and golden labels. Meanwhile, based on the long-tailed distribution of word frequencies in the data, GLMD designs a strategy of vocabulary compression through decreasing vocabulary size instead of dimensionality. Experimental results show that our method outperforms 25 state-of-the-art methods on the SuperGLUE benchmark, achieving an average score that surpasses the best method by 3%.
Dual-Gated Fusion with Prefix-Tuning for Multi-Modal Relation Extraction

Multi-Modal Relation Extraction (MMRE) aims at identifying the relation between two entities in texts that contain visual clues.Rich visual content is valuable for the MMRE task, but existing works cannot well model finer associations among different modalities, failing to capture the truly helpful visual information and thus limiting relation extraction performance.In this paper, we propose a novel MMRE framework to better capture the deeper correlations of text, entity pair, and image/objects, so as to mine more helpful information for the task, termed as DGF-PT.We first propose a promptbased autoregressive encoder, which builds the associations of intra-modal and inter-modal features related to the task, respectively by entityoriented and object-oriented prefixes.To better integrate helpful visual information, we design a dual-gated fusion module to distinguish the importance of image/objects and further enrich text representations.In addition, a generative decoder is introduced with entity type restriction on relations, better filtering out candidates.Extensive experiments conducted on the benchmark dataset show that our approach achieves excellent performance compared to strong competitors, even in the few-shot situation.
SPC: Soft Prompt Construction for Cross Domain Generalization

Recent advances in prompt tuning have proven effective as a new language modeling paradigm for various natural language understanding tasks. However, it is challenging to adapt the soft prompt embeddings to different domains or generalize to low-data settings when learning soft prompts itself is unstable, task-specific, and bias-prone. This paper proposes a principled learning framework—soft prompt construction (SPC)—to facilitate learning domain-adaptable soft prompts. Derived from the SPC framework is a simple loss that can plug into various models and tuning approaches to improve their cross-domain performance. We show SPC can improve upon SOTA for contextual query rewriting, summarization, and paraphrase detection by up to 5%, 19%, and 16%, respectively.
Large Language Models respond to Influence like Humans

Two studies tested the hypothesis that a Large Language Model (LLM) can be used to model psychological change following exposure to influential input.The first study tested a generic mode of influence -the Illusory Truth Effect (ITE) -where earlier exposure to a statement boosts a later truthfulness test rating.Analysis of newly collected data from human and LLMsimulated subjects (1000 of each) showed the same pattern of effects in both populations; although with greater per statement variability for the LLM.The second study concerns a specific mode of influence -populist framing of news to increase its persuasion and political mobilization.Newly collected data from simulated subjects was compared to previously published data from a 15-country experiment on 7286 human participants.Several effects from the human study were replicated by the simulated study, including ones that surprised the authors of the human study by contradicting their theoretical expectations; but some significant relationships found in human data were not present in the LLM data.Together the two studies support the view that LLMs have potential to act as models of the effect of influence.
Graph Reasoning for Question Answering with Triplet Retrieval

Answering complex questions often requires reasoning over knowledge graphs (KGs).Stateof-the-art methods often utilize entities in questions to retrieve local subgraphs, which are then fed into KG encoder, e.g.graph neural networks (GNNs), to model their local structures and integrated into language models for question answering.However, this paradigm constrains retrieved knowledge in local subgraphs and discards more diverse triplets buried in KGs that are disconnected but useful for question answering.In this paper, we propose a simple yet effective method to first retrieve the most relevant triplets from KGs and then rerank them, which are then concatenated with questions to be fed into language models.Extensive results on both CommonsenseQA and Open-bookQA datasets show that our method can outperform state-of-the-art up to 4.6% absolute accuracy.
World-to-Words: Grounded Open Vocabulary Acquisition through Fast Mapping in Vision-Language Models

The ability to connect language units to their referents in the physical world, referred to as grounding, is crucial to learning and understanding grounded meanings of words.While humans demonstrate fast mapping in new word learning, it remains unclear whether modern vision-language models can truly represent language with their grounded meanings, and how grounding may further bootstrap new word learning.To this end, we introduce Grounded Open Vocabulary Acquisition (GOVA) to examine grounding and bootstrapping in openworld language learning.As an initial attempt, we propose World-to-Words (W2W), a novel visually-grounded language model by pre-training on image-text pairs highlighting grounding as an objective.Through extensive experiments and analysis, we demonstrate that W2W is a more coherent and fast grounded word learner, and that the grounding ability acquired during pre-training helps the model to learn unseen words more rapidly and robustly.
WebIE: Faithful and Robust Information Extraction on the Web

Extracting structured and grounded fact triples from raw text is a fundamental task in Information Extraction (IE).Existing IE datasets are typically collected from Wikipedia articles, using hyperlinks to link entities to the Wikidata knowledge base.However, models trained only on Wikipedia have limitations when applied to web domains, which often contain noisy text or text that does not have any factual information.We present WEBIE, the first large-scale, entity-linked closed IE dataset consisting of 1.6M sentences automatically collected from the English Common Crawl corpus.WEBIE also includes negative examples, i.e. sentences without fact triples, to better reflect the data on the web.We annotate ∼21K triples from WEBIE through crowdsourcing and introduce mWEBIE, a translation of the annotated set in four other languages: French, Spanish, Portuguese, and Hindi.We evaluate the in-domain, out-of-domain, and zero-shot cross-lingual performance of generative IE models and find models trained on WEBIE show better generalisability.We also propose three training strategies that use entity linking as an auxiliary task.Our experiments show that adding Entity-Linking objectives improves the faithfulness of our generative IE models 1 .
Early Exit with Disentangled Representation and Equiangular Tight Frame

Dynamic early exit has demonstrated great potential in coping with the sharply increasing number of pre-trained language model parameters, which can achieve a good trade-off between performance and efficiency.The existing early exit paradigm relies on training parametrical internal classifiers at each intermediate layer to complete specific tasks.Based on the predictions of these internal classifiers, different methods are designed to decide when to exit.Under this circumstance, each intermediate layer takes on both generic language representation learning and task-specific feature extraction, which makes each intermediate layer struggle to balance two types of backward loss signals during training.To break this dilemma, we propose an adapter method to decouple the two distinct types of representation and further introduce a non-parametric simplex equiangular tight frame classifier (ETF) for improvement.Extensive experiments on monolingual and multilingual tasks demonstrate that our method gains significant improvements over strong PLM backbones and early exit methods.
Noisy Positive-Unlabeled Learning with Self-Training for Speculative Knowledge Graph Reasoning

This paper studies speculative reasoning task on real-world knowledge graphs (KG) that contain both false negative issue (i.e., potential true facts being excluded) and false positive issue (i.e., unreliable or outdated facts being included).State-of-the-art methods fall short in the speculative reasoning ability, as they assume the correctness of a fact is solely determined by its presence in KG, making them vulnerable to false negative/positive issues.The new reasoning task is formulated as a noisy Positive-Unlabeled learning problem.We propose a variational framework, namely nPU-Graph, that jointly estimates the correctness of both collected and uncollected facts (which we call label posterior) and updates model parameters during training.The label posterior estimation facilitates speculative reasoning from two perspectives.First, it improves the robustness of a label posterior-aware graph encoder against false positive links.Second, it identifies missing facts to provide high-quality grounds of reasoning.They are unified in a simple yet effective self-training procedure.Empirically, extensive experiments on three benchmark KG and one Twitter dataset with various degrees of false negative/positive cases demonstrate the effectiveness of nPUGraph.
Small Data, Big Impact: Leveraging Minimal Data for Effective Machine Translation

For many languages, machine translation progress is hindered by the lack of reliable training data.Models are trained on whatever preexisting datasets may be available and then augmented with synthetic data, because it is often not economical to pay for the creation of largescale datasets.But for the case of low-resource languages, would the creation of a few thousand professionally translated sentence pairs give any benefit?In this paper, we show that it does.We describe a broad data collection effort involving around 6k professionally translated sentence pairs for each of 39 low-resource languages, which we make publicly available.We analyse the gains of models trained on this small but high-quality data, showing that it has significant impact even when larger but lower quality pre-existing corpora are used, or when data is augmented with millions of sentences through backtranslation.
ReCode: Robustness Evaluation of Code Generation Models

Code generation models have achieved impressive performance.However, they tend to be brittle as slight edits to a prompt could lead to very different generations; these robustness properties, critical for user experience when deployed in real-life applications, are not well understood.Most existing works on robustness in text or code tasks have focused on classification, while robustness in generation tasks is an uncharted area and to date there is no comprehensive benchmark for robustness in code generation.In this paper, we propose ReCode, a comprehensive robustness evaluation benchmark for code generation models.We customize over 30 transformations specifically for code on docstrings, function and variable names, code syntax, and code format.They are carefully designed to be natural in real-life coding practice, preserve the original semantic meaning, and thus provide multifaceted assessments of a model's robustness performance.With human annotators, we verified that over 90% of the perturbed prompts do not alter the semantic meaning of the original prompt.In addition, we define robustness metrics for code generation models considering the worst-case behavior under each type of perturbation, taking advantage of the fact that executing the generated code can serve as objective evaluation.We demonstrate ReCode on SOTA models using HumanEval, MBPP, as well as function completion tasks derived from them.Interesting observations include: better robustness for CodeGen over In-Coder and GPT-J; models are most sensitive to syntax perturbations; more challenging robustness evaluation on MBPP over HumanEval.
Enhanced Training Methods for Multiple Languages

Document-grounded dialogue generation based on multilingual is a challenging and realistic task. Unlike previous tasks, it need to tackle with multiple high-resource languages facilitating low-resource languages. This paper summarizes our research based on a three-stage pipeline that includes retrieval, re-rank and generation where each component is individually optimized. In different languages with limited data scenarios, we mainly improve the robustness of the pipeline through data augmentation and embedding perturbation with purpose of improving the performance designing three training methods: cross-language enhancement training, weighted training with neighborhood distribution augmentation, and ensemble adversarial training, all of that can be used as plug and play modules. Through experiments with different settings, it has been shown that our methods can effectively improve the generalization performance of pipeline with score ranking 6th among the public submissions on leaderboards.
CSS: A Large-scale Cross-schema Chinese Text-to-SQL Medical Dataset

The cross-domain text-to-SQL task aims to build a system that can parse user questions into SQL on complete unseen databases, and the single-domain text-to-SQL task evaluates the performance on identical databases.Both of these setups confront unavoidable difficulties in real-world applications.To this end, we introduce the cross-schema text-to-SQL task, where the databases of evaluation data are different from that in the training data but come from the same domain.Furthermore, we present CSS 1 , a large-scale CrosS-Schema Chinese text-to-SQL dataset, to carry on corresponding studies.CSS originally consisted of 4,340 question/SQL pairs across 2 databases.In order to generalize models to different medical systems, we extend CSS and create 19 new databases along with 29,280 corresponding dataset examples.Moreover, CSS is also a large corpus for single-domain Chinese textto-SQL studies.We present the data collection approach and a series of analyses of the data statistics.To show the potential and usefulness of CSS, benchmarking baselines have been conducted and reported.
LABO: Towards Learning Optimal Label Regularization via Bi-level Optimization

Regularization techniques are crucial to improving the generalization performance and training efficiency of deep neural networks.Many deep learning algorithms rely on weight decay, dropout, batch/layer normalization to converge faster and generalize.Label Smoothing (LS) is another simple, versatile and efficient regularization which can be applied to various supervised classification tasks.Conventional LS, however, regardless of the training instance assumes that each non-target class is equally likely.In this work, we present a general framework for training with label regularization, which includes conventional LS but can also model instance-specific variants.Based on this formulation, we propose an efficient way of learning LAbel regularization by devising a Bi-level Optimization (LABO) problem.We derive a deterministic and interpretable solution of the inner loop as the optimal label smoothing without the need to store the parameters or the output of a trained model.Finally, we conduct extensive experiments and demonstrate our LABO consistently yields improvement over conventional label regularization on various fields, including seven machine translation and three image classification tasks across various neural network architectures while maintaining training efficiency.
Towards Better Entity Linking with Multi-View Enhanced Distillation

Dense retrieval is widely used for entity linking to retrieve entities from large-scale knowledge bases.Mainstream techniques are based on a dual-encoder framework, which encodes mentions and entities independently and calculates their relevances via rough interaction metrics, resulting in difficulty in explicitly modeling multiple mention-relevant parts within entities to match divergent mentions.Aiming at learning entity representations that can match divergent mentions, this paper proposes a Multi-View Enhanced Distillation (MVD) framework, which can effectively transfer knowledge of multiple fine-grained and mention-relevant parts within entities from cross-encoders to dual-encoders.Each entity is split into multiple views to avoid irrelevant information being over-squashed into the mention-relevant view.We further design cross-alignment and self-alignment mechanisms for this framework to facilitate fine-grained knowledge distillation from the teacher model to the student model.Meanwhile, we reserve a global-view that embeds the entity as a whole to prevent dispersal of uniform information.Experiments show our method achieves state-of-the-art performance on several entity linking benchmarks 1 .
FluentSpeech: Stutter-Oriented Automatic Speech Editing with Context-Aware Diffusion Models

Stutter removal is an essential scenario in the field of speech editing.However, when the speech recording contains stutters, the existing text-based speech editing approaches still suffer from: 1) the over-smoothing problem in the edited speech; 2) lack of robustness due to the noise introduced by stutter; 3) to remove the stutters, users are required to determine the edited region manually.To tackle the challenges in stutter removal, we propose Flu-entSpeech, a stutter-oriented automatic speech editing model.Specifically, 1) we propose a context-aware diffusion model that iteratively refines the modified mel-spectrogram with the guidance of context features; 2) we introduce a stutter predictor module to inject the stutter information into the hidden sequence; 3) we also propose a stutter-oriented automatic speech editing (SASE) dataset that contains spontaneous speech recordings with time-aligned stutter labels to train the automatic stutter localization model.Experimental results on VCTK and LibriTTS datasets demonstrate that our model achieves state-of-the-art performance on speech editing.Further experiments on our SASE dataset show that FluentSpeech can effectively improve the fluency of stuttering speech in terms of objective and subjective metrics.Code and audio samples can be found at https:
Tokenization and the Noiseless Channel

Subword tokenization is a key part of many NLP pipelines.However, little is known about why some tokenizer and hyperparameter combinations lead to better downstream model performance than others.We propose that good tokenizers lead to efficient channel usage, where the channel is the means by which some input is conveyed to the model and efficiency can be quantified in information-theoretic terms as the ratio of the Shannon entropy to the maximum possible entropy of the token distribution.Yet, an optimal encoding according to Shannon entropy assigns extremely long codes to low-frequency tokens and very short codes to high-frequency tokens.Defining efficiency in terms of Rényi entropy, on the other hand, penalizes distributions with either very high or very low-frequency tokens.In machine translation, we find that across multiple tokenizers, the Rényi entropy with α = 2.5 has a very strong correlation with BLEU: 0.78 in comparison to just -0.32 for compressed length.
A Call for Standardization and Validation of Text Style Transfer Evaluation

Text Style Transfer (TST) evaluation is, in practice, inconsistent.Therefore, we conduct a meta-analysis on human and automated TST evaluation and experimentation that thoroughly examines existing literature in the field.The meta-analysis reveals a substantial standardization gap in human and automated evaluation.In addition, we also find a validation gap: only few automated metrics have been validated using human experiments.To this end, we thoroughly scrutinize both the standardization and validation gap and reveal the resulting pitfalls.This work also paves the way to close the standardization and validation gap in TST evaluation by calling out requirements to be met by future research.
MIL-Decoding: Detoxifying Language Models at Token-Level via Multiple Instance Learning

Despite advances in large pre-trained neural language models, they are prone to generating toxic language, which brings security risks to their applications.We introduce MIL-Decoding, which detoxifies language models at token-level by interpolating it with a trained multiple instance learning (MIL) network.MIL model is trained on a corpus with a toxicity label for each text to predict the overall toxicity and the toxicity of each token in its context.Intuitively, the MIL network computes a toxicity distribution over next tokens according to the generated context which supplements the original language model to avoid toxicity.We evaluate MIL-Decoding with automatic metrics and human evaluation, where MIL-Decoding outperforms other baselines in detoxification while it only hurts generation fluency a little bit.
TAM of SCNU at SemEval-2023 Task 1: FCLL: A Fine-grained Contrastive Language-Image Learning Model for Cross-language Visual Word Sense Disambiguation

Visual Word Sense Disambiguation (WSD), as a fine-grained image-text retrieval task, aims to identify the images that are relevant to ambiguous target words or phrases. However, the difficulties of limited contextual information and cross-linguistic background knowledge in text processing make this task challenging. To alleviate this issue, we propose a Fine-grained Contrastive Language-Image Learning (FCLL) model, which learns fine-grained image-text knowledge by employing a new fine-grained contrastive learning mechanism and enriches contextual information by establishing relationship between concepts and sentences. In addition, a new multimodal-multilingual knowledge base involving ambiguous target words is constructed for visual WSD. Experiment results on the benchmark datasets from SemEval-2023 Task 1 show that our FCLL ranks at the first in overall evaluation with an average H@1 of 72.56\% and an average MRR of 82.22\%. The results demonstrate that FCLL is effective in inference on fine-grained language-vision knowledge. Source codes and the knowledge base are publicly available at https://github.com/CharlesYang030/FCLL.