Reviewerly/exHarmony huggingface dataset:
A Survey on Asking Clarification Questions Datasets in Conversational Systems

The ability to understand a user's underlying needs is critical for conversational systems, especially with limited input from users in a conversation.Thus, in such a domain, Asking Clarification Questions (ACQs) to reveal users' true intent from their queries or utterances arise as an essential task.However, it is noticeable that a key limitation of the existing ACQs studies is their incomparability, from inconsistent use of data, distinct experimental setups and evaluation strategies.Therefore, in this paper, to assist the development of ACQs techniques, we comprehensively analyse the current ACQs research status, which offers a detailed comparison of publicly available datasets, and discusses the applied evaluation metrics, joined with benchmarks for multiple ACQs-related tasks.In particular, given a thorough analysis of the ACQs task, we discuss a number of corresponding research directions for the investigation of ACQs as well as the development of conversational systems.
Reinforced Active Learning for Low-Resource, Domain-Specific, Multi-Label Text Classification

Text classification datasets from specialised or technical domains are in high demand, especially in industrial applications.However, due to the high cost of annotation such datasets are usually expensive to create.While Active Learning (AL) can reduce the labeling cost, required AL strategies are often only tested on general knowledge domains and tend to use information sources that are not consistent across tasks.We propose Reinforced Active Learning (RAL) to train a Reinforcement Learning policy that utilizes many different aspects of the data and the task in order to select the most informative unlabeled subset dynamically over the course of the AL procedure.We demonstrate the superior performance of the proposed RAL framework compared to strong AL baselines across four intricate multi-class, multilabel text classification datasets taken from specialised domains.In addition, we experiment with a unique data augmentation approach to further reduce the number of samples RAL needs to annotate.
Friendly Neighbors: Contextualized Sequence-to-Sequence Link Prediction

We propose KGT5-context, a simple sequenceto-sequence model for link prediction (LP) in knowledge graphs (KG).Our work expands on KGT5, a recent LP model that exploits textual features of the KG, has small model size, and is scalable.To reach good predictive performance, however, KGT5 relies on an ensemble with a knowledge graph embedding model, which itself is excessively large and costly to use.In this short paper, we show empirically that adding contextual information-i.e., information about the direct neighborhood of the query entity-alleviates the need for a separate KGE model to obtain good performance.The resulting KGT5-context model is simple, reduces model size significantly, and obtains state-of-the-art performance in our experimental study.
Zemi: Learning Zero-Shot Semi-Parametric Language Models from Multiple Tasks

Although large language models have exhibited impressive zero-shot ability, the huge model size generally incurs high cost.Recently, semiparametric language models, which augment a smaller language model with retrieved related background knowledge, alleviate the need for storing everything into the model parameters.Although existing semi-parametric language models have demonstrated promising language modeling capabilities, it remains unclear whether they can exhibit competitive zeroshot abilities as their fully-parametric counterparts.In this work, we introduce Zemi, a semi-parametric language model for zero-shot task generalization.To our best knowledge, this is the first semi-parametric language model that can demonstrate strong zeroshot performance on a wide range of heldout unseen tasks.We train Zemi with semiparametric multitask training, which shows significant improvement compared with the parametric multitask training as proposed by T0 (Sanh et al., 2021).Specifically, during both training and inference, Zemi is equipped with a retrieval system based on the unlabeled pretraining corpus of our backbone model.To address the unique challenges from large-scale retrieval, we further propose a novel retrieval-augmentation fusion module that can effectively incorporate noisy retrieved documents.Finally, we show detailed analysis and ablation studies on the key ingredients towards building effective zero-shot semiparametric language models.Notably, our proposed Zemi LARGE model outperforms T0-3B by 16% across seven diverse evaluation tasks while being 3.8x smaller in scale.
TEPrompt: Task Enlightenment Prompt Learning for Implicit Discourse Relation Recognition

Implicit Discourse Relation Recognition (IDRR) aims at classifying the relation sense between two arguments without an explicit connective.Recently, the ConnPrompt (Xiang et al., 2022b) has leveraged the powerful prompt learning for IDRR based on the fusion of multi-prompt decisions from three different yet much similar connective prediction templates.Instead of multi-prompt ensembling, we propose to design auxiliary tasks with enlightened prompt learning for the IDRR task.Although an auxiliary task is not used to directly output final prediction, we argue that during the joint training some of its learned features can be useful to boost the main task.In light of such motivations, we propose a task enlightenment prompt learning model, called TEPrompt, to fuse learned features from three related tasks for IDRR.In particular, the TEPrompt contains three tasks, viz., Discourse Relation Recognition (DRR), Sense Semantics Classification (SSC) and Annotated Connective Prediction (ACP), each with a unique prompt template and an answer space.In the training phase, we jointly train three prompt learning tasks with shared argument representation.In the testing phase, we only take the DRR output with fused features as the final IDRR decision.Experiments with the same conditions have shown that the proposed TEPrompt outperforms the ConnPrompt.This can be attributed to the promoted decision features and language models benefited from joint-training of auxiliary tasks.
GEC-DePenD: Non-Autoregressive Grammatical Error Correction with Decoupled Permutation and Decoding

Grammatical error correction (GEC) is an important NLP task that is currently usually solved with autoregressive sequence-tosequence models.However, approaches of this class are inherently slow due to one-byone token generation, so non-autoregressive alternatives are needed.In this work, we propose a novel non-autoregressive approach to GEC that decouples the architecture into a permutation network that outputs a self-attention weight matrix that can be used in beam search to find the best permutation of input tokens (with auxiliary ins tokens) and a decoder network based on a step-unrolled denoising autoencoder that fills in specific tokens.This allows us to find the token permutation after only one forward pass of the permutation network, avoiding autoregressive constructions.We show that the resulting network improves over previously known non-autoregressive methods for GEC and reaches the level of autoregressive methods that do not use language-specific synthetic data generation methods.Our results are supported by a comprehensive experimental validation on the ConLL-2014 and Write&Improve+LOCNESS datasets and an extensive ablation study that supports our architectural and algorithmic choices.
Causal Matching with Text Embeddings: A Case Study in Estimating the Causal Effects of Peer Review Policies

A promising approach to estimate the causal effects of peer review policies is to analyze data from publication venues that shift policies from single-blind to double-blind from one year to the next.However, in these settings the content of the manuscript is a confounding variableeach year has a different distribution of scientific content which may naturally affect the distribution of reviewer scores.To address this textual confounding, we extend variable ratio nearest neighbor matching to incorporate text embeddings.We compare this matching method to a widely-used causal method of stratified propensity score matching and a baseline of randomly selected matches.For our case study of the ICLR conference shifting from singleto double-blind review from 2017 to 2018, we find human judges prefer manuscript matches from our method in 70% of cases.While the unadjusted estimate of the average causal effect of reviewers' scores is -0.25, our method shifts the estimate to -0.17, a slightly smaller difference between the outcomes of single-and doubleblind policies.We hope this case study enables exploration of additional text-based causal estimation methods and domains in the future.
Self-Evolution Learning for Discriminative Language Model Pretraining

Masked language modeling, widely used in discriminative language model (e.g., BERT) pretraining, commonly adopts a random masking strategy.However, random masking does not consider the importance of the different words in the sentence meaning, where some of them are more worthy to be predicted.Therefore, various masking strategies (e.g., entitylevel masking) are proposed, but most of them require expensive prior knowledge and generally train from scratch without reusing existing model weights.In this paper, we present Self-Evolution learning (SE), a simple and effective token masking and learning method to fully and wisely exploit the knowledge from data.SE focuses on learning the informative yet underexplored tokens and adaptively regularizes the training by introducing a novel Token-specific Label Smoothing approach.Experiments on 10 tasks show that our SE brings consistent and significant improvements (+1.43∼2.12average scores) upon different PLMs.In-depth analyses demonstrate that SE improves linguistic knowledge learning and generalization.
LightFormer: Light-weight Transformer Using SVD-based Weight Transfer and Parameter Sharing

Transformer has become an important technique for natural language processing tasks with great success.However, it usually requires huge storage space and computational cost, making it difficult to be deployed on resourceconstrained edge devices.To compress and accelerate Transformer, we propose LightFormer, which adopts a low-rank factorization initialized by SVD-based weight transfer and parameter sharing.The SVD-based weight transfer can effectively utilize the well-trained Transformer parameter knowledge to speed up the model convergence, and effectively alleviate the lowrank bottleneck problem combined with parameter sharing.We validate this method on machine translation, text summarization, and text classification tasks.Experiments show that on IWSLT'14 De-En and WMT'14 En-De, LightFormer achieves similar performance to the baseline Transformer with 3.8× and 1.8× fewer parameters, and achieves 2.3× speedup and 1.5× speedup respectively, generally outperforming recent light-weight Transformers.
Understanding Programs by Exploiting (Fuzzing) Test Cases

Semantic understanding of programs has attracted great attention in the community. Inspired by recent successes of large language models (LLMs) in natural language understanding, tremendous progress has been made by treating programming language as another sort of natural language and training LLMs on corpora of program code. However, programs are essentially different from texts after all, in a sense that they are normally heavily structured and syntax-strict. In particular, programs and their basic units (i.e., functions and subroutines) are designed to demonstrate a variety of behaviors and/or provide possible outputs, given different inputs. The relationship between inputs and possible outputs/behaviors represents the functions/subroutines and profiles the program as a whole. Hence, we propose to incorporate such a relationship into learning, for achieving a deeper semantic understanding of programs. To obtain inputs that are representative enough to trigger the execution of most part of the code, we resort to fuzz testing and propose fuzz tuning to boost the performance of program understanding and code representation learning, given a pre-trained LLM. The effectiveness of the proposed method is verified on two program understanding tasks including code clone detection and code classification, and it outperforms current state-of-the-arts by large margins. Code is available at https://github.com/rabbitjy/FuzzTuning.
Numeric Magnitude Comparison Effects in Large Language Models

Large Language Models (LLMs) do not differentially represent numbers, which are pervasive in text.In contrast, neuroscience research has identified distinct neural representations for numbers and words.In this work, we investigate how well popular LLMs capture the magnitudes of numbers (e.g., that 4 < 5) from a behavioral lens.Prior research on the representational capabilities of LLMs evaluates whether they show human-level performance, for instance, high overall accuracy on standard benchmarks.Here, we ask a different question, one inspired by cognitive science: How closely do the number representations of LLMs correspond to those of human language users, who typically demonstrate the distance, size, and ratio effects?We depend on a linking hypothesis to map the similarities among the model embeddings of number words and digits to human response times.The results reveal surprisingly human-like representations across language models of different architectures, despite the absence of the neural circuitry that directly supports these representations in the human brain.This research shows the utility of understanding LLMs using behavioral benchmarks and points the way to future work on the number of representations of LLMs and their cognitive plausibility.
KGA: A General Machine Unlearning Framework Based on Knowledge Gap Alignment

Recent legislation of the "right to be forgotten" has led to the interest in machine unlearning, where the learned models are endowed with the function to forget information about specific training instances as if they have never existed in the training set.Previous work mainly focuses on computer vision scenarios and largely ignores the essentials of unlearning in NLP field, where text data contains more explicit and sensitive personal information than images.In this paper, we propose a general unlearning framework called KGA to induce forgetfulness.Different from previous work that tries to recover gradients or forces models to perform close to one specific distribution, KGA maintains distribution differences (i.e., knowledge gap).This relaxes the distribution assumption.Furthermore, we first apply the unlearning method to various NLP tasks (i.e., classification, translation, response generation) and propose several unlearning evaluation metrics with pertinence.Experiments on large-scale datasets show that KGA yields comprehensive improvements over baselines, where extensive analyses further validate the effectiveness of KGA and provide insight into unlearning for NLP tasks 1 .
Dynamic and Efficient Inference for Text Generation via BERT Family

Despite the excellent performance of Pretrained Language Models on many text generation tasks, they suffer from inefficient inference on computation and memory due to their largescale parameters and the universal autoregressive decoding paradigm.In this work, we propose a novel fine-tuning method DEER, which can make a single pre-trained model support Dynamic and Efficient infERence and achieve an adaptive trade-off between model performance and latency.In particular, our critical insight is to jointly utilize the non-autoregressive (NAR) generation and dynamic parameter pruning techniques, which can flexibly control the decoding iteration steps and model sizes according to memory and latency limitations.Besides, we also explore the effectiveness of the pre-trained MLMs (i.e., the BERT family) for text generation tasks since their bidirectional attention nature is more suitable for the NAR training objective.Extensive experiments on both monolingual and multilingual pre-trained MLMs demonstrate the effectiveness of our proposed DEER method by consistently achieving (1) higher BLEU scores than the strong autoregressive Transformer model on three neural machine translation tasks with 3 → 12 times speedup, (2) competitive performance (but with much faster inference speed) compared with the BART model on four GLGE benchmark tasks.
Zhegu at SemEval-2023 Task 9: Exponential Penalty Mean Squared Loss for Multilingual Tweet Intimacy Analysis

We present the system description of our team Zhegu in SemEval-2023 Task 9 Multilingual Tweet Intimacy Analysis.We propose EPM (Exponential Penalty Mean Squared Loss) for the purpose of enhancing the ability of learning difficult samples during the training process.Meanwhile, we also apply several methods (frozen Tuning & contrastive learning based on Language) on the XLM-R multilingual language model for fine-tuning and model ensemble.The results in our experiments provide strong faithful evidence of the effectiveness of our methods.Eventually, we achieved a Pearson score of 0.567 on the test set.
The HW-TSC’s Simultaneous Speech-to-Text Translation System for IWSLT 2023 Evaluation

Jiaxin Guo, Daimeng Wei, Zhanglin Wu, Zongyao Li, Zhiqiang Rao, Minghan Wang, Hengchao Shang, Xiaoyu Chen, Zhengzhe Yu, Shaojun Li, Yuhao Xie, Lizhi Lei, Hao Yang. Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023). 2023.
Multimodal Prompt Learning for Product Title Generation with Extremely Limited Labels

Generating an informative and attractive title for the product is a crucial task for e-commerce.Most existing works follow the standard multimodal natural language generation approaches, e.g., image captioning, and employ the large scale of human-labelled datasets to train desirable models.However, for novel products, especially in a different domain, there are few existing labelled data.In this paper, we propose a prompt-based approach, i.e., the Multimodal Prompt Learning framework, to accurately and efficiently generate titles for novel products with limited labels.We observe that the core challenges of novel product title generation are the understanding of novel product characteristics and the generation of titles in a novel writing style.To this end, we build a set of multimodal prompts from different modalities to preserve the corresponding characteristics and writing styles of novel products.As a result, with extremely limited labels for training, the proposed method can retrieve the multimodal prompts to generate desirable titles for novel products.The experiments and analyses are conducted on five novel product categories under both the in-domain and out-of-domain experimental settings.The results show that, with only 1% of downstream labelled data for training, our proposed approach achieves the best few-shot results and even achieves competitive results with fully-supervised methods trained on 100% of training data; With the full labelled data for training, our method achieves state-of-the-art results.
Constructing Procedural Graphs with Multiple Dependency Relations: A New Dataset and Baseline

Current structured and semi-structured knowledge bases mainly focus on representing descriptive knowledge but ignore another commonsense knowledge (Procedural Knowledge).To structure the procedural knowledge, existing methods are proposed to automatically generate flow graphs from procedural documents.They focus on extracting sequential dependency between sentences but neglect another two important dependencies (i.e., inclusion dependency and constraint dependency) in procedural documents.In our paper, we explore a problem of automatically generating procedural graph with multiple dependency relations to extend the flow graph constructed by existing methods and propose a procedural graph construction method with syntactic information and discourse structures.A new dataset (WHPG) is built and extensive experiments are conducted to evaluate the effectiveness of our proposed model.
UniEX: An Effective and Efficient Framework for Unified Information Extraction via a Span-extractive Perspective

We propose a new paradigm for universal information extraction (IE) that is compatible with any schema format and applicable to a list of IE tasks, such as named entity recognition, relation extraction, event extraction and sentiment analysis.Our approach converts the text-based IE tasks as the token-pair problem, which uniformly disassembles all extraction targets into joint span detection, classification and association problems with a unified extractive framework, namely UniEX.UniEX can synchronously encode schema-based prompt and textual information, and collaboratively learn the generalized knowledge from pre-defined information using the auto-encoder language models.We develop a traffine attention mechanism to integrate heterogeneous factors including tasks, labels and inside tokens, and obtain the extraction target via a scoring matrix.Experiment results show that UniEX can outperform generative universal IE models in terms of performance and inference-speed on 14 benchmarks IE datasets with the supervised setting.The state-of-the-art performance in low-resource scenarios also verifies the transferability and effectiveness of UniEX.
Assisting Language Learners: Automated Trans-Lingual Definition Generation via Contrastive Prompt Learning

The standard definition generation task requires to automatically produce mono-lingual definitions (e.g., English definitions for English words), but ignores that the generated definitions may also consist of unfamiliar words for language learners.In this work, we propose a novel task of Trans-Lingual Definition Generation (TLDG), which aims to generate definitions in another language, i.e., the native speaker's language.Initially, we explore the unsupervised manner of this task and build up a simple implementation of fine-tuning the multilingual machine translation model.Then, we develop two novel methods, Prompt Combination and Contrastive Prompt Learning, for further enhancing the quality of the generation.Our methods are evaluated against the baseline Pipeline method in both rich-and low-resource settings, and we empirically establish its superiority in generating higher-quality trans-lingual definitions.The ablation studies and further analysis are also conducted to provide more hints on this new task.
Multi-target Backdoor Attacks for Code Pre-trained Models

Backdoor attacks for neural code models have gained considerable attention due to the advancement of code intelligence.However, most existing works insert triggers into task-specific data for code-related downstream tasks, thereby limiting the scope of attacks.Moreover, the majority of attacks for pre-trained models are designed for understanding tasks.In this paper, we propose task-agnostic backdoor attacks for code pre-trained models.Our backdoored model is pre-trained with two learning strategies (i.e., Poisoned Seq2Seq learning and token representation learning) to support the multitarget attack of downstream code understanding and generation tasks.During the deployment phase, the implanted backdoors in the victim models can be activated by the designed triggers to achieve the targeted attack.We evaluate our approach on two code understanding tasks and three code generation tasks over seven datasets.Extensive experiments demonstrate that our approach can effectively and stealthily attack code-related downstream tasks.