Reviewerly/exHarmony huggingface dataset:
AMR-TST: Abstract Meaning Representation-based Text Style Transfer

Meaning Representation (AMR) is a semantic representation that can enhance natural language generation (NLG) by providing a logical semantic input.In this paper, we propose the AMR-TST, an AMR-based text style transfer (TST) technique.The AMR-TST converts the source text to an AMR graph and generates the transferred text based on the AMR graph modified by a TST policy named style rewriting.Our method combines both the explainability and diversity of explicit and implicit TST methods.The experiments show that the proposed method achieves state-of-theart results compared with other baseline models in automatic and human evaluations.The generated transferred text in qualitative evaluation proves the AMR-TST have significant advantages in keeping semantic features and reducing hallucinations.To the best of our knowledge, this work is the first to apply the AMR method focusing on node-level features to the TST task.
Diversity-Aware Coherence Loss for Improving Neural Topic Models

The standard approach for neural topic modeling uses a variational autoencoder (VAE) framework that jointly minimizes the KL divergence between the estimated posterior and prior, in addition to the reconstruction loss.Since neural topic models are trained by recreating individual input documents, they do not explicitly capture the coherence between topic words on the corpus level.In this work, we propose a novel diversity-aware coherence loss that encourages the model to learn corpus-level coherence scores while maintaining a high diversity between topics.Experimental results on multiple datasets show that our method significantly improves the performance of neural topic models without requiring any pretraining or additional parameters.
mPMR: A Multilingual Pre-trained Machine Reader at Scale

We present multilingual Pre-trained Machine Reader (mPMR), a novel method for multilingual machine reading comprehension (MRC)style pre-training.mPMR aims to guide multilingual pre-trained language models (mPLMs) to perform natural language understanding (NLU) including both sequence classification and span extraction in multiple languages.To achieve cross-lingual generalization when only source-language fine-tuning data is available, existing mPLMs solely transfer NLU capability from a source language to target languages.In contrast, mPMR allows the direct inheritance of multilingual NLU capability from the MRCstyle pre-training to downstream tasks.Therefore, mPMR acquires better NLU capability for target languages.mPMR also provides a unified solver for tackling cross-lingual span extraction and sequence classification, thereby enabling the extraction of rationales to explain the sentence-pair classification process. 1
Enhancing Cross-lingual Prompting with Dual Prompt Augmentation

Prompting shows promising results in few-shot scenarios. However, its strength for multilingual/cross-lingual problems has not been fully exploited. hao and Schütze (2021) made initial explorations in this direction by presenting that cross-lingual prompting outperforms cross-lingual finetuning. In this paper, we conduct an empirical exploration on the effect of each component in cross-lingual prompting and derive Universal Prompting, which helps alleviate the discrepancies between source-language training and target-language inference. Based on this, we propose DPA, a dual prompt augmentation framework, aiming at relieving the data scarcity issue in few-shot cross-lingual prompting. Notably, for XNLI, our method achieves 46.54% with only 16 English training examples per class, significantly better than 34.99% of fine-tuning. Our code is available at https://github.com/DAMO-NLP-SG/DPA.
Enhancing Neural Topic Model with Multi-Level Supervisions from Seed Words

Efforts have been made to apply topic seed words to improve the topic interpretability of topic models.However, due to the semantic diversity of natural language, supervisions from seed words could be ambiguous, making it hard to be incorporated into the current neural topic models.In this paper, we propose SeededNTM, a neural topic model enhanced with supervisions from seed words on both word and document levels.We introduce a context-dependency assumption to alleviate the ambiguities with context document information, and an auto-adaptation mechanism to automatically balance between multi-level information.Moreover, an intra-sample consistency regularizer is proposed to deal with noisy supervisions via encouraging perturbation and semantic consistency.Extensive experiments on multiple datasets show that SeededNTM can derive semantically meaningful topics and outperforms the state-of-the-art seeded topic models in terms of topic quality and classification accuracy.
JUAGE at SemEval-2023 Task 10: Parameter Efficient Classification

Using pre-trained language models to implement classifiers from small to modest amounts of training data is an area of active research.The ability of large language models to generalize from few-shot examples and to produce strong classifiers is extended using the engineering approach of parameter-efficient tuning.Using the Explainable Detection of Online Sexism (EDOS) training data and a small number of trainable weights to create a tuned prompt vector, a competitive model for this task was built, which was top-ranked in Subtask B.
Team ISCL_WINTER at SemEval-2023 Task 12:AfriSenti-SemEval: Sentiment Analysis for Low-resource African Languages using Twitter Dataset

This paper presents a study on the effectiveness of various approaches for addressing the challenge of multilingual sentiment analysis in lowresource African languages.The study focuses on Task 12 of the SemEval-2023 Competition, which aims to promote interest in these languages and develop efficient models for their analysis.The approaches evaluated in the study include Support Vector Machines (SVM), translation, and an ensemble of pre-trained multilingual sentimental model methods.The paper provides a detailed analysis of the performance of each approach based on experimental results.In our findings, we suggest that the ensemble method is the most effective with an F1 Score of 0.68 on the final testing.This system ranked 19 out of 33 participants in the competition.
Target-Based Offensive Language Identification

We present TBO, a new dataset for Targetbased Offensive language identification.TBO contains post-level annotations regarding the harmfulness of an offensive post and tokenlevel annotations comprising of the target and the offensive argument expression.Popular offensive language identification datasets for social media focus on annotation taxonomies only at the post level and more recently, some datasets have been released that feature only token-level annotations.TBO is an important resource that bridges the gap between postlevel and token-level annotation datasets by introducing a single comprehensive unified annotation taxonomy.We use the TBO taxonomy to annotate post-level and token-level offensive language on English Twitter posts.We release an initial dataset of over 4,500 instances collected from Twitter and we carry out multiple experiments to compare the performance of different models trained and tested on TBO.
Large Scale Sequence-to-Sequence Models for Clinical Note Generation from Patient-Doctor Conversations

We present our work on building large scale sequence-to-sequence models for generating clinical note from patient-doctor conversation.This is formulated as an abstractive summarization task for which we use encoder-decoder transformer model with pointer-generator.We discuss various modeling enhancements to this baseline model which include using subword and multiword tokenization scheme, prefixing the targets with a chain-of-clinical-facts, and training with contrastive loss that is defined over various candidate summaries.We also use flash attention during training and query chunked attention during inference to be able to process long input and output sequences and to improve computational efficiency.Experiments are conducted on a proprietary dataset containing about 900K encounters in U.S. English from around 1800 healthcare providers covering 27 specialties.The results are broken down into primary care and non-primary care specialties.Consistent accuracy improvements are observed across both of these categories.
Balancing the Effect of Training Dataset Distribution of Multiple Styles for Multi-Style Text Transfer

Text style transfer is an exciting task within the field of natural language generation that is often plagued by the need for high-quality paired datasets.Furthermore, training a model for multi-attribute text style transfer requires datasets with sufficient support across all combinations of the considered stylistic attributes, adding to the challenges of training a style transfer model.This paper explores the impact of training data input diversity on the quality of the generated text from the multi-style transfer model.We construct a pseudo-parallel dataset by devising heuristics to adjust the style distribution in the training samples.We balance our training dataset using marginal and joint distributions to train our style transfer models.We observe that a balanced dataset produces more effective control effects over multiple styles than an imbalanced or skewed one.Through quantitative analysis, we explore the impact of multiple style distributions in training data on style-transferred output.These findings will better inform the design of style-transfer datasets.
Using Domain Knowledge to Guide Dialog Structure Induction via Neural Probabilistic Soft Logic

Dialog Structure Induction (DSI) is the task of inferring the latent dialog structure (i.e., a set of dialog states and their temporal transitions) of a given goal-oriented dialog.It is a critical component for modern dialog system design and discourse analysis.Existing DSI approaches are often purely data-driven, deploy models that infer latent states without access to domain knowledge, underperform when the training corpus is limited/noisy, or have difficulty when test dialogs exhibit distributional shifts from the training domain.This work explores a neural-symbolic approach as a potential solution to these problems.We introduce Neural Probabilistic Soft Logic Dialogue Structure Induction (NEUPSL DSI), a principled approach that injects symbolic knowledge into the latent space of a generative neural model.We conduct a thorough empirical investigation on the effect of NEUPSL DSI learning on hidden representation quality, few-shot learning, and out-of-domain generalization performance.Over three dialog structure induction datasets and across unsupervised and semi-supervised settings for standard and cross-domain generalization, the injection of symbolic knowledge using NEUPSL DSI provides a consistent boost in performance over the canonical baselines.
Text Style Transfer Back-Translation

Back Translation (BT) is widely used in the field of machine translation, as it has been proved effective for enhancing translation quality.However, BT mainly improves the translation of inputs that share a similar style (to be more specific, translation-like inputs), since the source side of BT data is machine-translated.For natural inputs, BT brings only slight improvements and sometimes even adverse effects.To address this issue, we propose Text Style Transfer Back Translation (TST BT), which uses a style transfer model to modify the source side of BT data.By making the style of source-side text more natural, we aim to improve the translation of natural inputs.Our experiments on various language pairs, including both high-resource and low-resource ones, demonstrate that TST BT significantly improves translation performance against popular BT benchmarks.In addition, TST BT is proved to be effective in domain adaptation so this strategy can be regarded as a general data augmentation method.Our training code and text style transfer model are opensourced.
CUPID: Curriculum Learning Based Real-Time Prediction using Distillation

Relevance in E-commerce Product Search is crucial for providing customers with accurate results that match their query intent.With recent advancements in NLP and Deep Learning, Transformers have become the default choice for relevance classification tasks.In such a setting, the relevance model uses query text and product title as input features, and estimates if the product is relevant for the customer query.While cross-attention in Transformers enables a more accurate relevance prediction in such a setting, its high evaluation latency makes it unsuitable for real-time predictions in which thousands of products must be evaluated against a user query within few milliseconds.To address this issue, we propose CUPID: a CUrriculum learning based real-time PredIction using Distillation that utilizes knowledge distillation within a curriculum learning setting to learn a simpler architecture that can be evaluated within low latency budgets.In a bi-lingual relevance prediction task, our approach shows an 302 bps improvement on English and 676 bps improvement for low-resource Arabic, while maintaining the low evaluation latency on CPUs.
DrBERT: A Robust Pre-trained Model in French for Biomedical and Clinical domains

In recent years, pre-trained language models (PLMs) achieve the best performance on a wide range of natural language processing (NLP) tasks.While the first models were trained on general domain data, specialized ones have emerged to more effectively treat specific domains.In this paper, we propose an original study of PLMs in the medical domain on French language.We compare, for the first time, the performance of PLMs trained on both public data from the web and private data from healthcare establishments.We also evaluate different learning strategies on a set of biomedical tasks.In particular, we show that we can take advantage of already existing biomedical PLMs in a foreign language by further pre-train it on our targeted data.Finally, we release the first specialized PLMs for the biomedical field in French, called DrBERT, as well as the largest corpus of medical data under free license on which these models are trained.
The HW-TSC’s Simultaneous Speech-to-Speech Translation System for IWSLT 2023 Evaluation

Hengchao Shang, Zhiqiang Rao, Zongyao Li, Zhanglin Wu, Jiaxin Guo, Minghan Wang, Daimeng Wei, Shaojun Li, Zhengzhe Yu, Xiaoyu Chen, Lizhi Lei, Hao Yang. Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023). 2023.
Intent Discovery with Frame-guided Semantic Regularization and Augmentation

Most intent discovery methods leverage representation learning and clustering to transfer the prior knowledge of known intents to unknown ones.The learned representations are limited to the syntactic forms of sentences, therefore, fall short of recognizing adequate variations under the same meaning of unknown intents.This paper proposes an approach utilizing frame knowledge as conceptual semantic guidance to bridge the gap between known intents representation learning and unknown intents clustering.Specifically, we employ semantic regularization to minimize the bidirectional KL divergence between model predictions for framebased and sentence-based samples.Moreover, we construct a frame-guided data augmenter to capture intent-friendly semantic information and implement contrastive clustering learning for unsupervised sentence embedding.Extensive experiments on two benchmark datasets show that our method achieves substantial improvements in accuracy (5%+) compared to solid baselines.
CoAD: Automatic Diagnosis through Symptom and Disease Collaborative Generation

Automatic diagnosis (AD), a critical application of AI in healthcare, employs machine learning techniques to assist doctors in gathering patient symptom information for precise disease diagnosis.The Transformer-based method utilizes an input symptom sequence, predicts itself through auto-regression, and employs the hidden state of the final symptom to determine the disease.Despite its simplicity and superior performance demonstrated, a decline in disease diagnosis accuracy is observed caused by 1) a mismatch between symptoms observed during training and generation, and 2) the effect of different symptom orders on disease prediction.To address the above obstacles, we introduce the CoAD, a novel disease and symptom collaborative generation framework, which incorporates several key innovations to improve AD: 1) aligning sentence-level disease labels with multiple possible symptom inquiry steps to bridge the gap between training and generation; 2) expanding symptom labels for each sub-sequence of symptoms to enhance annotation and eliminate the effect of symptom order; 3) developing a repeated symptom input schema to effectively and efficiently learn the expanded disease and symptom labels.We evaluate the CoAD framework using four datasets, including three public and one private, and demonstrate that it achieves an average 2.3% improvement over previous state-of-theart results in automatic disease diagnosis.For reproducibility, we release the code and data at https://github.com/KwanWaiChung/coad.
Towards Robust Personalized Dialogue Generation via Order-Insensitive Representation Regularization

Generating persona consistent dialogue response is important for developing an intelligent conversational agent.Recent works typically fine-tune large-scale pre-trained models on this task by concatenating persona texts and dialogue history as a single input sequence to generate the target response.While simple and effective, our analysis shows that this popular practice is seriously affected by Order Sensitivity where different input orders of persona sentences significantly impact the quality and consistency of generated response, resulting in severe performance fluctuations (i.e., 29.4% on GPT2 and 83.2% on BART).To mitigate the order sensitivity problem, we propose a model-agnostic framework, ORder Insensitive Generation (ORIG), which enables dialogue models to learn robust representation under different persona orders and improve the consistency of response generation.Experiments on Persona-Chat dataset justify the effectiveness and superiority of our method with two dominant pre-trained models (GPT2 and BART). 1
Few-shot Low-resource Knowledge Graph Completion with Reinforced Task Generation

Despite becoming a prevailing paradigm for organizing knowledge, most knowledge graphs (KGs) suffer from the low-resource issue due to the deficiency of data sources.The enrichment of KGs by automatic knowledge graph completion is impeded by the intrinsic long-tail property of KGs.In spite of their prosperity, existing few-shot learning-based models have difficulty alleviating the impact of the longtail issue on low-resource KGs because of the lack of training tasks.To tackle the challenging long-tail issue on low-resource KG completion, in this paper, we propose a novel fewshot low-resource knowledge graph completion framework, which is composed of three components, i.e., few-shot learner, task generator, and task selector.The key idea is to generate and then select the beneficial few-shot tasks that complement the current tasks and enable the optimization of the few-shot learner using the selected few-shot tasks.Extensive experiments conducted on several real-world knowledge graphs validate the effectiveness of our proposed method.
Few-shot Query-Focused Summarization with Prefix-Merging

Query-focused summarization has been considered as an important extension for text summarization.It aims to generate a concise highlight for a given query.Different from text summarization, query-focused summarization has long been plagued by the problem of lacking high-quality large-scale datasets.In this paper, we investigate the idea that whether we can integrate and transfer the knowledge of text summarization and question answering to assist the few-shot learning in query-focused summarization.Here, we propose prefix-merging, a prefix-based pretraining strategy for fewshot learning in query-focused summarization.Drawn inspiration from prefix-tuning, we are allowed to integrate the task knowledge from text summarization and question answering into a properly designed prefix and apply the merged prefix to query-focused summarization.With only a small amount of trainable parameters, prefix-merging outperforms fine-tuning on query-focused summarization.We further discuss the influence of different prefix designs and propose a visualized explanation for how prefix-merging works.
Investigating the Characteristics of a Transformer in a Few-Shot Setup: Does Freezing Layers in RoBERTa Help?

Transformer based language models have been widely adopted by industrial and research organisations in developing machine learning applications in the presence of limited annotated data.While these models show remarkable results, their functioning in few-shot settings is still poorly understood.Hence, we perform an investigative study to understand the characteristics of such models fine-tuned in few-shot setups.Specifically, we compare the intermediate layer representations obtained from a few-shot model and a pre-trained language model.We observe that pre-trained and few-shot models show similar representations over initial layers, whereas the later layers show a stark deviation.Based on these observations, we propose to freeze the initial Transformer layers to fine-tune the model in a constrained text classification setup with K annotated data points per class, where K ranges from 8 to 64.In our experiments across six benchmark sentence classification tasks, we discover that freezing initial 50% Transformer layers not only reduces training time but also surprisingly improves Macro F1 (upto 8%) when compared to fully trainable layers in few-shot setup.We also observe that this idea of layer freezing can very well be generalized to state-of-the-art few-shot text classification techniques, like DNNC and LM-BFF, leading to significant reduction in training time while maintaining comparable performance.
Rethinking the Authorship Verification Experimental Setups

One of the main drivers of the recent advances in authorship verification is the PAN large-scale authorship dataset.Despite generating significant progress in the field, inconsistent performance differences between the closed and open test sets have been reported.To this end, we improve the experimental setup by proposing five new public splits over the PAN dataset, specifically designed to isolate and identify biases related to the text topic and to the author's writing style.We evaluate several BERT-like baselines on these splits, showing that such models are competitive with authorship verification state-of-the-art methods.Furthermore, using explainable AI, we find that these baselines are biased towards named entities.We show that models trained without the named entities obtain better results and generalize better when tested on DarkReddit, our new dataset for authorship verification.