Reviewerly/exHarmony huggingface dataset:
SESCORE2: Learning Text Generation Evaluation via Synthesizing Realistic Mistakes

Is it possible to train a general metric for evaluating text generation quality without humanannotated ratings?Existing learned metrics either perform unsatisfactorily across text generation tasks or require human ratings for training on specific tasks.In this paper, we propose SESCORE2, a self-supervised approach for training a model-based metric for text generation evaluation.The key concept is to synthesize realistic model mistakes by perturbing sentences retrieved from a corpus.The primary advantage of the SESCORE2 is its ease of extension to many other languages while providing reliable severity estimation.We evaluate SESCORE2 and previous methods on four text generation tasks across three languages.SESCORE2 outperforms unsupervised metric PRISM on four text generation evaluation benchmarks, with a Kendall improvement of 0.078.Surprisingly, SESCORE2 even outperforms the supervised BLEURT and COMET on multiple text generation tasks.The code and
The USTC’s Dialect Speech Translation System for IWSLT 2023

This paper presents the USTC system for the IWSLT 2023 Dialectal and Low-resource shared task, which involves translation from Tunisian Arabic to English.We aim to investigate the mutual transfer between Tunisian Arabic and Modern Standard Arabic (MSA) to enhance the performance of speech translation (ST) by following standard pre-training and fine-tuning pipelines.We synthesize a substantial amount of pseudo Tunisian-English paired data using a multi-step pre-training approach.Integrating a Tunisian-MSA translation module into the end-to-end ST model enables the transfer from Tunisian to MSA and facilitates linguistic normalization of the dialect.To increase the robustness of the ST system, we optimize the model's ability to adapt to ASR errors and propose a model ensemble method.Results indicate that applying the dialect transfer method can increase the BLEU score of dialectal ST.It is shown that the optimal system ensembles both cascaded and end-to-end ST models, achieving BLEU improvements of 2.4 and 2.8 in test1 and test2 sets, respectively, compared to the best published system.
Free Lunch: Robust Cross-Lingual Transfer via Model Checkpoint Averaging

Massively multilingual language models have displayed strong performance in zero-shot (ZS-XLT) and few-shot (FS-XLT) cross-lingual transfer setups, where models fine-tuned on task data in a source language are transferred without any or with only a few annotated instances to the target language(s).However, current work typically overestimates model performance as fine-tuned models are frequently evaluated at model checkpoints that generalize best to validation instances in the target languages.This effectively violates the main assumptions of 'true' ZS-XLT and FS-XLT.Such XLT setups require robust methods that do not depend on labeled target language data for validation and model selection.In this work, aiming to improve the robustness of 'true' ZS-XLT and FS-XLT, we propose a simple and effective method that averages different checkpoints (i.e., model snapshots) during task fine-tuning.We conduct exhaustive ZS-XLT and FS-XLT experiments across higher-level semantic tasks (NLI, extractive QA) and lower-level token classification tasks (NER, POS).The results indicate that averaging model checkpoints yields systematic and consistent performance gains across diverse target languages in all tasks.Importantly, it simultaneously substantially desensitizes XLT to varying hyperparameter choices in the absence of target language validation.We also show that checkpoint averaging benefits performance when further combined with run averaging (i.e., averaging the parameters of models fine-tuned over independent runs).
Self-Edit: Fault-Aware Code Editor for Code Generation

Large language models (LLMs) have demonstrated an impressive ability to generate codes on competitive programming tasks.However, with limited sample numbers, LLMs still suffer from poor accuracy.Inspired by the process of human programming, we propose a generateand-edit approach named Self-Edit that utilizes execution results of the generated code from LLMs to improve the code quality on the competitive programming task.We execute the generated code on the example test case provided in the question and wrap execution results into a supplementary comment.Utilizing this comment as guidance, our fault-aware code editor is employed to correct errors in the generated code.We perform extensive evaluations across two competitive programming datasets with nine different LLMs.Compared to directly generating from LLMs, our approach can improve the average of pass@1 by 89% on APPS-dev, 31% on APPS-test, and 48% on HumanEval over nine popular code generation LLMs with parameter sizes ranging from 110M to 175B.Compared to other post-processing methods, our method demonstrates superior accuracy and efficiency.
YNUNLP at SemEval-2023 Task 2: The Pseudo Twin Tower Pre-training Model for Chinese Named Entity Recognition

This paper introduces our method of developing a system for SemEval 2023 Task 2: Multi-CoNER II Multilingual Complex Named Entity Recognition, Track 9-Chinese.In this task, we need to identify entity boundaries and category labels for the six identified categories.The focus of this task is to detect fine-grained named entities whose data set has a fine-grained taxonomy of 36 NE classes, representing a realistic challenge for NER.We use BERT embedding to represent each character in the original sentence and train CRF-Rdrop to predict named entity categories using the data set provided by the organizer.Our best submission, with a macro average f1 score of 0.5657, ranked 15th out of 22 teams.
MURMUR: Modular Multi-Step Reasoning for Semi-Structured Data-to-Text Generation

Prompting large language models has enabled significant recent progress in multi-step reasoning over text.However, when applied to text generation from semi-structured data (e.g., graphs or tables), these methods typically suffer from low semantic coverage, hallucination, and logical inconsistency.We propose MUR-MUR, a neuro-symbolic modular approach to text generation from semi-structured data with multi-step reasoning.MURMUR is a best-first search method that generates reasoning paths using: (1) neural and symbolic modules with specific linguistic and logical skills, (2) a grammar whose production rules define valid compositions of modules, and (3) value functions that assess the quality of each reasoning step.We conduct experiments on two diverse datato-text generation tasks like WebNLG and Log-icNLG.The tasks differ in their data representations (graphs and tables) and span multiple linguistic and logical skills.MURMUR obtains significant improvements over recent few-shot baselines like direct prompting and chain-ofthought prompting, while also achieving comparable performance to fine-tuned GPT-2 on out-of-domain data.Moreover, human evaluation shows that MURMUR generates highly faithful and correct reasoning paths that lead to 26% more logically consistent summaries on LogicNLG, compared to direct prompting. 1
Towards Generative Event Factuality Prediction

We present a novel end-to-end generative task and system for predicting event factuality holders, targets, and their associated factuality values. We perform the first experiments using all sources and targets of factuality statements from the FactBank corpus. We perform multi-task learning with other tasks and event-factuality corpora to improve on the FactBank source and target task. We argue that careful domain specific target text output format in generative systems is important and verify this with multiple experiments on target text output structure. We redo previous state-of-the-art author-only event factuality experiments and also offer insights towards a generative paradigm for the author-only event factuality prediction task.
Dialogue Summarization with Static-Dynamic Structure Fusion Graph

Dialogue summarization, one of the most challenging and intriguing text summarization tasks, has attracted increasing attention in recent years.Since dialogue possesses dynamic interaction nature and presumably inconsistent information flow scattered across multiple utterances by different interlocutors, many researchers address this task by modeling dialogue with pre-computed static graph structure using external linguistic toolkits.However, such methods heavily depend on the reliability of external tools and the static graph construction is disjoint with the graph representation learning phase, which could not make the graph dynamically adapt to the downstream summarization task.In this paper, we propose a Static-Dynamic graph-based Dialogue Summarization model (SDDS) * , which fuses prior knowledge from human expertise and implicit knowledge from a PLM, and adaptively adjusts the graph weight, and learns the graph structure in an end-to-end learning fashion from the supervision of summarization task.To verify the effectiveness of SDDS, we conduct extensive experiments on three benchmark datasets (SAMSum, MediaSum, and DialogSum) and observe significant improvement over strong baselines.
JUST_ONE at SemEval-2023 Task 10: Explainable Detection of Online Sexism (EDOS)

The problem of online sexism, which refers to offensive content targeting women based on their gender or the intersection of their gender with one or more additional identity characteristics, such as race or religion, has become a widespread phenomenon on social media.This can include sexist comments and memes.To address this issue, the SemEval-2023 international workshop introduced the "Explainable Detection of Online Sexism Challenge", which aims to explain the classifications given by AI models for detecting sexism.In this paper, we present the contributions of our team, JUST_ONE, to all three sub-tasks of the challenge: subtask A, a binary classification task; subtask B, a four-class classification task; and subtask C, a fine-grained classification task.To accomplish this, we utilized pretrained language models, specifically BERT and RoBERTa from Hugging Face, and a selective ensemble method in task 10 of the Se-mEval 2023 competition.As a result, our team achieved the following rankings and scores in different tasks: 19th out of 84 with a Macro-F1 score of 0.8538 in task A, 22nd out of 69 with a Macro-F1 score of 0.6417 in task B, and 14th out of 63 with a Macro-F1 score of 0.4774 in task C.
Prompted LLMs as Chatbot Modules for Long Open-domain Conversation

In this paper, we propose MPC (Modular Prompted Chatbot), a new approach for creating high-quality conversational agents without the need for fine-tuning. Our method utilizes pre-trained large language models (LLMs) as individual modules for long-term consistency and flexibility, by using techniques such as few-shot prompting, chain-of-thought (CoT), and external memory. Our human evaluation results show that MPC is on par with fine-tuned chatbot models in open-domain conversations, making it an effective solution for creating consistent and engaging chatbots.
UKP-SQuARE: An Interactive Tool for Teaching Question Answering

The exponential growth of question answering (QA) has made it an indispensable topic in any Natural Language Processing (NLP) course.Additionally, the breadth of QA derived from this exponential growth makes it an ideal scenario for teaching related NLP topics such as information retrieval, explainability, and adversarial attacks among others.In this paper, we introduce UKP-SQuARE as a platform for QA education.This platform provides an interactive environment where students can run, compare, and analyze various QA models from different perspectives, such as general behavior, explainability, and robustness.Therefore, students can get a first-hand experience in different QA techniques during the class.Thanks to this, we propose a learner-centered approach for QA education in which students proactively learn theoretical concepts and acquire problem-solving skills through interactive exploration, experimentation, and practical assignments, rather than solely relying on traditional lectures.To evaluate the effectiveness of UKP-SQuARE in teaching scenarios, we adopted it in a postgraduate NLP course and surveyed the students after the course.Their positive feedback shows the platform's effectiveness in their course and invites a wider adoption.
Rethinking Round-Trip Translation for Machine Translation Evaluation

Automatic evaluation methods for translation often require model training, and thus the availability of parallel corpora limits their applicability to low-resource settings.Round-trip translation is a potential workaround, which can reframe bilingual evaluation into a much simpler monolingual task.Early results from the era of statistical machine translation (SMT) raised fundamental concerns about the utility of this approach, based on poor correlation with human translation quality judgments.In this paper, we revisit this technique with modern neural translation (NMT) and show that round-trip translation does allow for accurate automatic evaluation without the need for reference translations.These opposite findings can be explained through the copy mechanism in SMT that is absent in NMT.We demonstrate that round-trip translation benefits multiple machine translation evaluation tasks: i) predicting forward translation scores; ii) improving the performance of a quality estimation model; and iii) identifying adversarial competitors in shared tasks via cross-system verification.
The Coreference under Transformation Labeling Dataset: Entity Tracking in Procedural Texts Using Event Models

We demonstrate that coreference resolution in procedural texts is significantly improved when performing transformation-based entity linking prior to coreference relation identification.When events in the text introduce changes to the state of participating entities, it is often impossible to accurately link entities in anaphoric and coreference relations without an understanding of the transformations those entities undergo.We show how adding event semantics helps to better model entity coreference.We argue that all transformation predicates, not just creation verbs, introduce a new entity into the discourse, as a kind of generalized Result Role, which is typically not textually mentioned.This allows us to model procedural texts as process graphs and to compute the coreference type for any two entities in the recipe.We present our annotation methodology and the corpus generated as well as describe experiments on coreference resolution of entity mentions under a process-oriented model of events.
Responsibility Perspective Transfer for Italian Femicide News

Different ways of linguistically expressing the same real-world event can lead to different perceptions of what happened.Previous work has shown that different descriptions of genderbased violence (GBV) influence the reader's perception of who is to blame for the violence, possibly reinforcing stereotypes which see the victim as partly responsible, too.As a contribution to raise awareness on perspectivebased writing, and to facilitate access to alternative perspectives, we introduce the novel task of automatically rewriting GBV descriptions as a means to alter the perceived level of responsibility on the perpetrator.We present a quasi-parallel dataset of sentences with low and high perceived responsibility levels for the perpetrator, and experiment with unsupervised (mBART-based), zero-shot and few-shot (GPT3-based) methods for rewriting sentences.We evaluate our models using a questionnaire study and a suite of automatic metrics.
CEFR-based Contextual Lexical Complexity Classifier in English and French

This paper describes a CEFR-based classifier of single-word and multi-word lexical complexity in context from a second language learner perspective in English and in French, developed as an analytical tool for the pedagogical team of the language learning application Mauril.We provide an overview of the required corpora and the way we transformed it into rich contextual representations that allow the disambiguation and accurate labelling in context of polysemous occurrences of a given lexical item.We report evaluation results for all models, including two multi-lingual lexical classifiers evaluated on novel French datasets created for this experiment.Finally, we share the perspective of Mauril's pedagogical team on the limitations of such systems.
Detoxifying Online Discourse: A Guided Response Generation Approach for Reducing Toxicity in User-Generated Text

The expression of opinions, stances, and moral foundations on social media often coincide with toxic, divisive, or inflammatory language that can make constructive discourse across communities difficult.Natural language generation methods could provide a means to reframe or reword such expressions in a way that fosters more civil discourse, yet current Large Language Model (LLM) methods tend towards language that is too generic or formal to seem authentic for social media discussions.We present preliminary work on training LLMs to maintain authenticity while presenting a community's ideas and values in a constructive, non-toxic manner.
Graph Propagation based Data Augmentation for Named Entity Recognition

Data augmentation is an effective solution to improve model performance and robustness for low-resource named entity recognition (NER).However, synthetic data often suffer from poor diversity, which leads to performance limitations.In this paper, we propose a novel Graph Propagated Data Augmentation (GPDA) framework for Named Entity Recognition (NER), leveraging graph propagation to build relationships between labeled data and unlabeled natural texts.By projecting the annotations from the labeled text to the unlabeled text, the unlabeled texts are partially labeled, which has more diversity rather than synthetic annotated data.To strengthen the propagation precision, a simple search engine built on Wikipedia is utilized to fetch related texts of labeled data and to propagate the entity labels to them in the light of the anchor links.Besides, we construct and perform experiments on a real-world lowresource dataset of the E-commerce domain, which will be publicly available to facilitate the low-resource NER research.Experimental results show that GPDA presents substantial improvements over previous data augmentation methods on multiple low-resource NER datasets.
Generating EDU Extracts for Plan-Guided Summary Re-Ranking

Two-step approaches, in which summary candidates are generated-then-reranked to return a single summary, can improve ROUGE scores over the standard single-step approach.Yet, standard decoding methods (i.e., beam search, nucleus sampling, and diverse beam search) produce candidates with redundant, and often low quality, content.In this paper, we design a novel method to generate candidates for re-ranking that addresses these issues.We ground each candidate abstract on its own unique content plan and generate distinct plan-guided abstracts using a model's top beam.More concretely, a standard language model (a BART LM) auto-regressively generates elemental discourse unit (EDU) content plans with an extractive copy mechanism.The top K beams from the content plan generator are then used to guide a separate LM, which produces a single abstractive candidate for each distinct plan.We apply an existing re-ranker (BRIO) to abstractive candidates generated from our method, as well as baseline decoding methods.We show large relevance improvements over previously published methods on widely used single document news article corpora, with ROUGE-2 F1 gains of 0.88, 2.01, and 0.38 on CNN / Dailymail, NYT, and Xsum, respectively.A human evaluation on CNN / DM validates these results.Similarly, on 1k samples from CNN / DM, we show that prompting GPT-3 to follow EDU plans outperforms sampling-based methods by 1.05 ROUGE-2 F1 points.Code to generate and realize plans is available at https: //github.com/griff4692/edu-sum.
ML Mob at SemEval-2023 Task 1: Probing CLIP on Visual Word-Sense Disambiguation

Successful word sense disambiguation (WSD) is a fundamental element of natural language understanding.As part of SemEval-2023 Task 1, we investigate WSD in a multimodal setting, where ambiguous words are to be matched with candidate images representing word senses.We compare multiple systems based on pre-trained CLIP models.In our experiments, we find CLIP to have solid zero-shot performance on monolingual and multilingual data.By employing different fine-tuning techniques, we are able to further enhance performance.However, transferring knowledge between data distributions proves to be more challenging.
Quintilian at SemEval-2023 Task 4: Grouped BERT for Multi-Label Classification

In this paper, we initially discuss about the ValueEval (Kiesel et al., 2023) task and the challenges involved in multi-label classification tasks.We tried to approach this task using Natural Language Inference and proposed a Grouped-BERT 1 architecture which leverages commonality between the classes for a multilabel classification task.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation

Cross-modal contrastive learning in vision language pretraining (VLP) faces the challenge of (partial) false negatives.In this paper, we study this problem from the perspective of Mutual Information (MI) optimization.It is common sense that InfoNCE loss used in contrastive learning will maximize the lower bound of MI between anchors and their positives, while we theoretically prove that MI involving negatives also matters when noises commonly exist.Guided by a more general lower bound form for optimization, we propose a contrastive learning strategy regulated by progressively refined cross-modal similarity, to more accurately optimize MI between an image/text anchor and its negative texts/images instead of improperly minimizing it.Our method performs competitively on four downstream cross-modal tasks and systematically balances the beneficial and harmful effects of (partial) false negative samples under theoretical guidance.
Reliable and Interpretable Drift Detection in Streams of Short Texts

Data drift is the change in model input data that is one of the key factors leading to machine learning models performance degradation over time.Monitoring drift helps detecting these issues and preventing their harmful consequences.Meaningful drift interpretation is a fundamental step towards effective retraining of the model.In this study we propose an end-to-end framework for reliable modelagnostic change-point detection and interpretation in large task-oriented dialog systems, proven effective in multiple customer deployments.We evaluate our approach and demonstrate its benefits with a novel variant of intent classification training dataset, simulating customer requests to a dialog system.We make the data publicly available.
MLlab4CS at SemEval-2023 Task 2: Named Entity Recognition in Low-resource Language Bangla Using Multilingual Language Models

Extracting of NERs from low-resource languages and recognizing their types is one of the important tasks in the entity extraction domain.Recently many studies have been conducted in this area of research.In our study, we introduce a system for identifying complex entities and recognizing their types from low-resource language Bangla, which was published in SemEval Task 2 MulitCoNER II 2023.For this sequence labeling task, we use a pretrained language model built on a natural language processing framework.Our team name in this competition is MLlab4CS.Our model MuRIL CRF Bangla produces a macro average Fscore of 76.27%, which is a comparable result for this competition.