Reviewerly/exHarmony huggingface dataset:
EvolveMT: an Ensemble MT Engine Improving Itself with Usage Only

This paper presents EvolveMT for efficiently combining multiple machine translation (MT) engines.The proposed system selects the output from a single engine for each segment by utilizing online learning techniques to predict the most suitable system for every translation request.A neural quality estimation metric supervises the method without requiring reference translations.The online learning capability of this system allows for dynamic adaptation to alterations in the domain or machine translation engines, thereby obviating the necessity for additional training.EvolveMT selects a subset of translation engines to be called based on the source sentence features.The degree of exploration is configurable according to the desired quality-cost trade-off.Results from custom datasets demonstrate that EvolveMT achieves similar translation accuracy at a lower cost than selecting the best translation of each segment from all translations using an MT quality estimator.To our knowledge, EvolveMT is the first meta MT system that adapts itself after deployment to incoming translation requests from the production environment without needing costly retraining on human feedback.
Word-level Prefix/Suffix Sense Detection: A Case Study on Negation Sense with Few-shot Learning

Morphological analysis is an important research issue in the field of natural language processing.In this study, we propose a context-free morphological analysis task, namely word-level prefix/suffix sense detection, which deals with the ambiguity of sense expressed by prefix/suffix.To research this novel task, we first annotate a corpus with prefixes/suffixes expressing negation (e.g., il-, un-, -less) and then propose a novel fewshot learning approach that applies an inputaugmentation prompt to a token-replaced detection pre-training model.Empirical studies demonstrate the effectiveness of the proposed approach to word-level prefix/suffix negation sense detection.
Witcherses at SemEval-2023 Task 12: Ensemble Learning for African Sentiment Analysis

This paper describes our system submission for SemEval-2023 Task 12 AfriSenti-SemEval: Sentiment Analysis for African Languages.We propose an XGBoost-based ensemble model trained on emoticon frequency-based features and the predictions of several statistical models such as SVMs, Logistic Regression, Random Forests, and BERT-based pre-trained language models such as AfriBERTa and AfroXLMR.We also report results from additional experiments not in the system.Our system achieves a mixed bag of results, achieving a best rank of 7th in three of the languages -Igbo, Twi, and Yoruba.
Matching Pairs: Attributing Fine-Tuned Models to their Pre-Trained Large Language Models

The wide applicability and adaptability of generative large language models (LLMs) has enabled their rapid adoption.While the pretrained models can perform many tasks, such models are often fine-tuned to improve their performance on various downstream applications.However, this leads to issues over violation of model licenses, model theft, and copyright infringement.Moreover, recent advances show that generative technology is capable of producing harmful content which exacerbates the problems of accountability within model supply chains.Thus, we need a method to investigate how a model was trained or a piece of text was generated and what their pre-trained base model was.In this paper we take the first step to address this open problem by tracing back the origin of a given fine-tuned LLM to its corresponding pre-trained base model.We consider different knowledge levels and attribution strategies, and find that we can correctly trace back 8 out of the 10 fine tuned models with our best method.
Improving Formality-Sensitive Machine Translation Using Data-Centric Approaches and Prompt Engineering

In this paper, we present the KU x Upstage team's submission for the Special Task on Formality Control on Spoken Language Translation, which involves translating English into four languages with diverse grammatical formality markers.Our methodology comprises two primary components: 1) a language-specific data-driven approach, and 2) the generation of synthetic data through the employment of large-scale language models and empirically-grounded prompt engineering.By adapting methodologies and models to accommodate the unique linguistic properties of each language, we observe a notable enhancement in performance relative to the baseline, substantiating the heightened efficacy of data-driven approaches.Moreover, our devised prompt engineering strategy yields superior synthetic translation instances.
FACTUAL: A Benchmark for Faithful and Consistent Textual Scene Graph Parsing

Textual scene graph parsing has become increasingly important in various vision-language applications, including image caption evaluation and image retrieval.However, existing scene graph parsers that convert image captions into scene graphs often suffer from two types of errors.First, the generated scene graphs fail to capture the true semantics of the captions or the corresponding images, resulting in a lack of faithfulness.Second, the generated scene graphs have high inconsistency, with the same semantics represented by different annotations.To address these challenges, we propose a novel dataset, which involves re-annotating the captions in Visual Genome (VG) using a new intermediate representation called FACTUAL-MR.FACTUAL-MR can be directly converted into faithful and consistent scene graph annotations.Our experimental results clearly demonstrate that the parser trained on our dataset outperforms existing approaches in terms of faithfulness and consistency.This improvement leads to a significant performance boost in both image caption evaluation and zero-shot image retrieval tasks.Furthermore, we introduce a novel metric for measuring scene graph similarity, which, when combined with the improved scene graph parser, achieves stateof-the-art (SOTA) results on multiple benchmark datasets for the aforementioned tasks.
KingsmanTrio at SemEval-2023 Task 10: Analyzing the Effectiveness of Transfer Learning Models for Explainable Online Sexism Detection

Online social platforms are now propagating sexist content endangering the involvement and inclusion of women on these platforms.Sexism refers to hostility, bigotry, or discrimination based on gender, typically against women.The proliferation of such notions deters women from engaging in social media spontaneously.Hence, detecting sexist content is critical to ensure a safe online platform where women can participate without the fear of being a target of sexism.This paper describes our participation in subtask A of SemEval-2023 Task 10: Explainable Detection of Online Sexism (EDOS).This subtask requires classifying textual content as sexist or not sexist.We incorporate a RoBERTa-based architecture and further finetune the hyperparameters to entail better performance.The procured results depict the competitive performance of our approach among the other participants.
PAL: Persona-Augmented Emotional Support Conversation Generation

Due to the lack of human resources for mental health support, there is an increasing demand for employing conversational agents for support.Recent work has demonstrated the effectiveness of dialogue models in providing emotional support.As previous studies have demonstrated that seekers' persona is an important factor for effective support, we investigate whether there are benefits to modeling such information in dialogue models for support.In this paper, our empirical analysis verifies that persona has an important impact on emotional support.Therefore, we propose a framework for dynamically inferring and modeling seekers' persona.We first train a model for inferring the seeker's persona from the conversation history.Accordingly, we propose PAL, a model that leverages persona information and, in conjunction with our strategybased controllable generation method, provides personalized emotional support.Automatic and manual evaluations demonstrate that PAL achieves state-of-the-art results, outperforming the baselines on the studied benchmark.
DUTIR at SemEval-2023 Task 10: Semi-supervised Learning for Sexism Detection in English

Sexism is an injustice afflicting women and has become a common form of oppression in social media.In recent years, the automatic detection of sexist instances has been utilized to combat this oppression.The Subtask A of SemEval-2023 Task 10, Explainable Detection of Online Sexism, aims to detect whether an Englishlanguage post is sexist.In this paper, we describe our system for the competition.The structure of the classification model is based on RoBERTa, and we further pre-train it on the domain corpus.For fine-tuning, we adopt Unsupervised Data Augmentation (UDA), a semisupervised learning approach, to improve the robustness of the system.Specifically, we employ Easy Data Augmentation (EDA) method as the noising operation for consistency training.We train multiple models based on different hyperparameter settings and adopt the majority voting method to predict the labels of test entries.Our proposed system achieves a Macro-F1 score of 0.8352 and a ranking of 41/84 on the leaderboard of Subtask A.
Sabrina Spellman at SemEval-2023 Task 5: Discover the Shocking Truth Behind this Composite Approach to Clickbait Spoiling!

This paper describes an approach to automatically close the knowledge gap of Clickbait-Posts via a transformer model trained for Question-Answering, augmented by a taskspecific post-processing step.This was part of the SemEval 2023 Clickbait shared task (Fröbe et al., 2023a) -specifically task 5. We devised strategies to improve the existing model to fit the task better, e.g. with different special models and a post-processor tailored to different inherent challenges of the task.Furthermore, we explored the possibility of expanding the original training data by using strategies from Heuristic Labeling and Semi-Supervised Learning.With those adjustments, we were able to improve the baseline by 9.8 percentage points to a BLEU-4 score of 48.0%.
StFX-NLP at SemEval-2023 Task 4: Unsupervised and Supervised Approaches to Detecting Human Values in Arguments

In this paper, we discuss our models applied to Task 4: Human Value Detection of SemEval 2023, which incorporated two different embedding techniques to interpret the data.Preliminary experiments were conducted to observe important word types.Subsequently, we explored an XGBoost model, an unsupervised learning model, and two Ensemble learning models were then explored.The best performing model, an ensemble model employing a soft voting technique, secured the 34 th spot out of 39 teams, on a class imbalanced dataset.We explored the inclusion of different parts of the provided knowledge resource and found that considering only specific parts assisted our models.
Improving Cross-task Generalization of Unified Table-to-text Models with Compositional Task Configurations

There has been great progress in unifying various table-to-text tasks using a single encoderdecoder model trained via multi-task learning (Xie et al., 2022).However, existing methods typically encode task information with a simple dataset name as a prefix to the encoder.This not only limits the effectiveness of multitask learning, but also hinders the model's ability to generalize to new domains or tasks that were not seen during training, which is crucial for real-world applications.In this paper, we propose compositional task configurations, a set of prompts prepended to the encoder to improve cross-task generalization of unified models.We design the task configurations to explicitly specify the task type, dataset name, as well as its input and output types.We show that this not only allows the model to better learn shared knowledge across different tasks at training, but also allows us to control the model by composing new configurations that apply novel input-output combinations in a zero-shot manner.We demonstrate via experiments over ten table-to-text tasks that our method outperforms the UnifiedSKG baseline by noticeable margins in both in-domain and zero-shot settings, with average improvements of +0.5 and +12.6 from using a T5-large backbone, respectively.
Open-World Factually Consistent Question Generation

Question generation methods based on pretrained language models often suffer from factual inconsistencies and incorrect entities and are not answerable from the input paragraph.Domain shift -where the test data is from a different domain than the training data -further exacerbates the problem of hallucination.This is a critical issue for any natural language application doing question generation.In this work, we propose an effective data processing technique based on de-lexicalization for consistent question generation across domains.Unlike existing approaches for remedying hallucination, the proposed approach does not filter training data and is generic across question-generation models.Experimental results across six benchmark datasets show that our model is robust to domain shift and produces entity-level factually consistent questions without significant impact on traditional metrics.
Improving the Transferability of Clinical Note Section Classification Models with BERT and Large Language Model Ensembles

Text in electronic health records is organized into sections, and classifying those sections into section categories is useful for downstream tasks.In this work, we attempt to improve the transferability of section classification models by combining the dataset-specific knowledge in supervised learning models with the world knowledge inside large language models (LLMs).Surprisingly, we find that zero-shot LLMs out-perform supervised BERT-based models applied to out-of-domain data.We also find that their strengths are synergistic, so that a simple ensemble technique leads to additional performance gains.
PanwarJayant at SemEval-2023 Task 10: Exploring the Effectiveness of Conventional Machine Learning Techniques for Online Sexism Detection

The rapid growth of online communication using social media platforms has led to an increase in the presence of hate speech, especially in terms of sexist language online. The proliferation of such hate speech has a significant impact on the mental health and well-being of the users and hence the need for automated systems to detect and filter such texts. In this study, we explore the effectiveness of conventional machine learning techniques for detecting sexist text. We explore five conventional classifiers, namely, Logistic Regression, Decision Tree, XGBoost, Support Vector Machines, and Random Forest. The results show that different classifiers perform differently on each task due to their different inherent architectures which may be suited to a certain problem more. These models are trained on the shared task dataset, which includes both sexist and non-sexist texts.All in all, this study explores the potential of conventional machine learning techniques in detecting online sexist content. The results of this study highlight the strengths and weaknesses of all classifiers with respect to all subtasks. The results of this study will be useful for researchers and practitioners interested in developing systems for detecting or filtering online hate speech.
Hidden Schema Networks

Large, pretrained language models infer powerful representations that encode rich semantic and syntactic content, albeit implicitly.In this work we introduce a novel neural language model that enforces, via inductive biases, explicit relational structures which allow for compositionality onto the output representations of pretrained language models.Specifically, the model encodes sentences into sequences of symbols (composed representations), which correspond to the nodes visited by biased random walkers on a global latent graph, and infers the posterior distribution of the latter.We first demonstrate that the model is able to uncover ground-truth graphs from artificially generated datasets of random token sequences.Next, we leverage pretrained BERT and GPT-2 language models as encoder and decoder, respectively, to infer networks of symbols (schemata) from natural language datasets.Our experiments show that (i) the inferred symbols can be interpreted as encoding different aspects of language, as e.g.topics or sentiments, and that (ii) GPT-like models can effectively be conditioned on symbolic representations.Finally, we explore training autoregressive, random walk "reasoning" models on schema networks inferred from commonsense knowledge databases, and using the sampled paths to enhance the performance of pretrained language models on commonsense If-Then reasoning tasks.
Effectiveness of Data Augmentation for Parameter Efficient Tuning with Limited Data

Recent work has demonstrated that using parameter efficient tuning techniques such as prefix tuning (or P-tuning) on pretrained language models can yield performance that is comparable or superior to fine-tuning while dramatically reducing trainable parameters.Nevertheless, the effectiveness of such methods under the context of data augmentation, a common strategy to improve learning under low data regimes, has not been fully explored.In this paper, we examine the effectiveness of several popular task-agnostic data augmentation techniques, i.e., EDA, Back Translation, and Mixup, when using two general parameter efficient tuning methods, P-tuning v2 and LoRA, under data scarcity.We show that data augmentation can be used to boost the performance of P-tuning and LoRA models, but the effectiveness of each technique varies and certain methods can lead to a notable degradation in performance, particularly when using larger models and on harder tasks.We further analyze the sentence representations of P-tuning compared to fine-tuning to help understand the above behaviour, and reveal how P-tuning generally presents a more limited ability to separate the sentence embeddings from different classes of augmented data.In addition, it displays poorer performance on heavily altered data.However, we demonstrate that by adding a simple contrastive loss function it can help mitigate such issues for prefix tuning, resulting in sizable improvements to augmented data performance.
GKD: A General Knowledge Distillation Framework for Large-scale Pre-trained Language Model

Currently, the reduction in the parameter scale of large-scale pre-trained language models (PLMs) through knowledge distillation has greatly facilitated their widespread deployment on various devices.However, the deployment of knowledge distillation systems faces great challenges in real-world industrial-strength applications, which require the use of complex distillation methods on even larger-scale PLMs (over 10B), limited by memory on GPUs and the switching of methods.To overcome these challenges, we propose GKD, a general knowledge distillation framework that supports distillation on larger-scale PLMs using various distillation methods.With GKD, developers can build larger distillation models on memorylimited GPUs and easily switch and combine different distillation methods within a single framework.Experimental results show that GKD can support the distillation of at least 100B-scale PLMs and 25 mainstream methods on 8 NVIDIA A100 (40GB) GPUs.
Improving Named Entity Recognition via Bridge-based Domain Adaptation

Recent studies have shown remarkable success in cross-domain named entity recognition (cross-domain NER).Despite the promising results, existing methods mainly utilize pretraining language models like BERT to represent words.As such, the original chaotic representations may challenge them to distinguish entity types of entities, leading to entity type misclassification.To this end, we attempt to utilize contrastive learning to refine the original representations and propose a model-agnostic framework named MoCL for cross-domain NER.Additionally, we respectively combine MoCL with two distinctive cross-domain NER methods and two pre-training language models to explore its generalization ability.Empirical results on six domains show the effectiveness and good generalization ability of MoCL.
A Universal Discriminator for Zero-Shot Generalization

Generative modeling has been the dominant approach for large-scale pretraining and zeroshot generalization.In this work, we challenge this convention by showing that discriminative approaches perform substantially better than generative ones on a large number of NLP tasks.Technically, we train a single discriminator to predict whether a text sample comes from the true data distribution, similar to GANs.Since many NLP tasks can be formulated as selecting from a few options, we use this discriminator to predict the concatenation of input and which option has the highest probability of coming from the true data distribution.This simple formulation achieves state-of-theart zero-shot results on the T0 benchmark, outperforming T0 by 16.0%, 7.8%, and 11.5% respectively on different scales.In the finetuning setting, our approach also achieves new stateof-the-art results on a wide range of NLP tasks, with only 1/4 parameters of previous methods.Meanwhile, our approach requires minimal prompting efforts, which largely improves robustness and is essential for real-world applications.Furthermore, we also jointly train a generalized UD in combination with generative tasks, which maintains its advantage on discriminative tasks and simultaneously works on generative tasks.
CausalDialogue: Modeling Utterance-level Causality in Conversations

Despite their widespread adoption, neural conversation models have yet to exhibit natural chat capabilities with humans.In this research, we examine user utterances as causes and generated responses as effects, recognizing that changes in a cause should produce a different effect.To further explore this concept, we have compiled and expanded upon a new dataset called CausalDialogue through crowdsourcing.This dataset includes multiple causeeffect pairs within a directed acyclic graph (DAG) structure.Our analysis reveals that traditional loss functions struggle to effectively incorporate the DAG structure, leading us to propose a causality-enhanced method called Exponential Maximum Average Treatment Effect (ExMATE) to enhance the impact of causality at the utterance level in training neural conversation models.To evaluate the needs of considering causality in dialogue generation, we built a comprehensive benchmark on Causal-Dialogue dataset using different models, inference, and training methods.Through experiments, we find that a causality-inspired loss like ExMATE can improve the diversity and agility of conventional loss function and there is still room for improvement to reach human-level quality on this new dataset. 1
NormMark: A Weakly Supervised Markov Model for Socio-cultural Norm Discovery

Norms, which are culturally accepted guidelines for behaviours, can be integrated into conversational models to generate utterances that are appropriate for the socio-cultural context.Existing methods for norm recognition tend to focus only on surface-level features of dialogues and do not take into account the interactions within a conversation.To address this issue, we propose NORMMARK, a probabilistic generative Markov model to carry the latent features throughout a dialogue.These features are captured by discrete and continuous latent variables conditioned on the conversation history, and improve the model's ability in norm recognition.The model is trainable on weakly annotated data using the variational technique.On a dataset with limited norm annotations, we show that our approach achieves higher F1 score, outperforming current stateof-the-art methods, including GPT3.