Reviewerly/exHarmony huggingface dataset:
NLP-LTU at SemEval-2023 Task 10: The Impact of Data Augmentation and Semi-Supervised Learning Techniques on Text Classification Performance on an Imbalanced Dataset

In this paper, we propose a methodology for task 10 of SemEval23, focusing on detecting and classifying online sexism in social media posts.The task is tackling a serious issue, as detecting harmful content on social media platforms is crucial for mitigating the harm of these posts on users.Our solution for this task is based on an ensemble of finetuned transformer-based models (BERTweet, RoBERTa, and DeBERTa).To alleviate problems related to class imbalance, and to improve the generalization capability of our model, we also experiment with data augmentation and semi-supervised learning.In particular, for data augmentation, we use back-translation, either on all classes, or on the underrepresented classes only.We analyze the impact of these strategies on the overall performance of the pipeline through extensive experiments.while for semi-supervised learning, we found that with a substantial amount of unlabelled, indomain data available, semi-supervised learning can enhance the performance of certain models.Our proposed method (for which the source code is available on Github 12 ) attains an F 1-score of 0.8613 for sub-taskA, which ranked us 10th in the competition.
Open-WikiTable : Dataset for Open Domain Question Answering with Complex Reasoning over Table

Despite recent interest in open domain question answering (ODQA) over tables, many studies still rely on datasets that are not truly optimal for the task with respect to utilizing structural nature of table.These datasets assume answers reside as a single cell value and do not necessitate exploring over multiple cells such as aggregation, comparison, and sorting.Thus, we release Open-WikiTable, the first ODQA dataset that requires complex reasoning over tables.Open-WikiTable is built upon Wik-iSQL and WikiTableQuestions to be applicable in the open-domain setting.As each question is coupled with both textual answers and SQL queries, Open-WikiTable opens up a wide range of possibilities for future research, as both reader and parser methods can be applied.The dataset and code are publicly available 1 .
SRCB at SemEval-2023 Task 2: A System of Complex Named Entity Recognition with External Knowledge

The MultiCoNER II shared task aims at detecting semantically ambiguous and complex named entities in short and low-context settings for multiple languages.The lack of context makes the recognition of ambiguous named entities challenging.To alleviate this issue, our team SRCB proposes an external knowledge based system, where we utilize 3 different types of external knowledge retrieved in different ways.Given an original text, our system retrieves the possible labels and the descriptions for each potential entity detected by a mention detection model.And we also retrieve a related document as extra context from Wikipedia for each original text.We concatenate the original text with the external knowledge as the input of NER models.The informative contextual representations with external knowledge significantly improve the NER performance in both Chinese and English tracks.Our system win the 3rd place in the Chinese track and the 6th place in the English track.
Teamwork Is Not Always Good: An Empirical Study of Classifier Drift in Class-incremental Information Extraction

Class-incremental learning (CIL) aims to develop a learning system that can continually learn new classes from a data stream without forgetting previously learned classes.When learning classes incrementally, the classifier must be constantly updated to incorporate new classes, and the drift in decision boundary may lead to severe forgetting.This fundamental challenge, however, has not yet been studied extensively, especially in the setting where no samples from old classes are stored for rehearsal.In this paper, we take a closer look at how the drift in the classifier leads to forgetting, and accordingly, design four simple yet (super-) effective solutions to alleviate the classifier drift: an Individual Classifiers with Frozen Feature Extractor (ICE) framework where we individually train a classifier for each learning session, and its three variants ICE-PL, ICE-O and ICE-PL&O which further take the logits of previously learned classes from old sessions or a constant logit of an Other class as constraint to the learning of new classifiers.Extensive experiments and analysis on 6 class-incremental information extraction tasks demonstrate that our solutions, especially ICE-O, consistently show significant improvement over the previous state-of-the-art approaches with up to 44.7% absolute F-score gain, providing a strong baseline and insights for future research on class-incremental learning. 1
Reducing cohort bias in natural language understanding systems with targeted self-training scheme

Bias in machine learning models can be an issue when the models are trained on particular types of data that do not generalize well, causing under performance in certain groups of users. In this work, we focus on reducing the bias related to new customers in a digital voice assistant system. It is observed that natural language understanding models often have lower performance when dealing with requests coming from new users rather than experienced users. To mitigate this problem, we propose a framework that consists of two phases (1) a fixing phase with four active learning strategies used to identify important samples coming from new users, and (2) a self training phase where a teacher model trained from the first phase is used to annotate semi-supervised samples to expand the training data with relevant cohort utterances. We explain practical strategies that involve an identification of representative cohort-based samples through density clustering as well as employing implicit customer feedbacks to improve new customers’ experience. We demonstrate the effectiveness of our approach in a real world large scale voice assistant system for two languages, German and French through both offline experiments as well as A/B testings.
Low-Resource Formality Controlled NMT Using Pre-trained LM

This paper describes the UCSC's submission to the shared task on formality control for spoken language translation at IWSLT 2023.For this task, we explored the use of "additive style intervention" using a pre-trained multilingual translation model, namely mBART.Compared to prior approaches where a single style-vector was added to all tokens in the encoder output, we explored an alternative approach in which we learn a unique style-vector for each input token.We believe this approach, which we call "style embedding intervention," is better suited for formality control as it can potentially learn which specific input tokens to modify during decoding.While the proposed approach obtained similar performance to "additive style intervention" for the supervised English-to-Vietnamese task, it performed significantly better for English-to-Korean, in which it achieved an average matched accuracy of 90.6 compared to 85.2 for the baseline.When we constrained the model further to only perform style intervention on the <bos> (beginning of sentence) token, the average matched accuracy improved further to 92.0, indicating that the model could learn to control the formality of the translation output based solely on the embedding of the <bos> token.
Few-shot Adaptation Works with UnpredicTable Data

Prior work on language models (LMs) shows that training on a large number of diverse tasks improves few-shot learning (FSL) performance on new tasks.We take this to the extreme, automatically extracting 413,299 tasks from internet tables -orders of magnitude more than the next-largest public datasets.Finetuning on the resulting dataset leads to improved FSL performance on Natural Language Processing (NLP) tasks, but not proportionally to dataset scale.In fact, we find that narrow subsets of our dataset sometimes outperform more diverse datasets.For example, finetuning on software documentation from support.google.com raises FSL performance by a mean of +7.5% on 52 downstream tasks, which beats training on 40 humancurated NLP datasets (+6.7%).Finetuning on various narrow datasets leads to similar broad improvements across test tasks, suggesting that the gains are not from domain adaptation but adapting to FSL in general.We do not observe clear patterns between the datasets that lead to FSL gains, leaving open questions about why certain data helps with FSL.
Training-free Neural Architecture Search for RNNs and Transformers

Neural architecture search (NAS) has allowed for the automatic creation of new and effective neural network architectures, offering an alternative to the laborious process of manually designing complex architectures.However, traditional NAS algorithms are slow and require immense amounts of computing power.Recent research has investigated training-free NAS metrics for image classification architectures, drastically speeding up search algorithms.In this paper, we investigate trainingfree NAS metrics for recurrent neural network (RNN) and BERT-based transformer architectures, targeted towards language modeling tasks.First, we develop a new trainingfree metric, named hidden covariance, that predicts the trained performance of an RNN architecture and significantly outperforms existing training-free metrics.We experimentally evaluate the effectiveness of the hidden covariance metric on the NAS-Bench-NLP benchmark.Second, we find that the current search space paradigm for transformer architectures is not optimized for training-free neural architecture search.Instead, a simple qualitative analysis can effectively shrink the search space to the best performing architectures.This conclusion is based on our investigation of existing training-free metrics and new metrics developed from recent transformer pruning literature, evaluated on our own benchmark of trained BERT architectures.Ultimately, our analysis shows that the architecture search space and the training-free metric must be developed together in order to achieve effective results.
Target-Side Augmentation for Document-Level Machine Translation

Document-level machine translation faces the challenge of data sparsity due to its long input length and a small amount of training data, increasing the risk of learning spurious patterns.To address this challenge, we propose a target-side augmentation method, introducing a data augmentation (DA) model to generate many potential translations for each source document.Learning on these wider range translations, an MT model can learn a smoothed distribution, thereby reducing the risk of data sparsity.We demonstrate that the DA model, which estimates the posterior distribution, largely improves the MT performance, outperforming the previous best system by 2.30 s-BLEU on News and achieving new state-of-the-art on News and Europarl benchmarks.Our code is available at https://github.com/baoguangsheng/target-side-augmentation.
ELiRF-VRAIN at BioNLP Task 1B: Radiology Report Summarization

This paper presents our system at the Radiology Report Summarization Shared Task-1B of the 22nd BioNLP Workshop 2023.Inspired by the work of the BioBART model, we continuously pre-trained a general domain BART model with biomedical data to adapt it to this specific domain.In the pre-training phase, several pre-training tasks are aggregated to inject linguistic knowledge and increase the abstractivity of the generated summaries.We present the results of our models, and also, we have carried out an additional study on the lengths of the generated summaries, which has provided us with interesting information.
Towards Flow Graph Prediction of Open-Domain Procedural Texts

Machine comprehension of procedural texts is essential for reasoning about the steps and automating the procedures.However, this requires identifying entities within a text and resolving the relationships between the entities.Previous work focused on the cooking domain and proposed a framework to convert a recipe text into a flow graph (FG) representation.In this work, we propose a framework based on the recipe FG for flow graph prediction of opendomain procedural texts.To investigate flow graph prediction performance in non-cooking domains, we introduce the wikiHow-FG corpus from articles on wikiHow, a website of how-to instruction articles.In experiments, we consider using the existing recipe corpus and performing domain adaptation from the cooking to the target domain.Experimental results show that the domain adaptation models achieve higher performance than those trained only on the cooking or target domain data.
Pre-trained Personalized Review Summarization with Effective Salience Estimation

Personalized review summarization in recommender systems is a challenging task of generating condensed summaries for product reviews while preserving the salient content of reviews.Recently, Pretrained Language Models (PLMs) have become a new paradigm in text generation for the strong ability of natural language comprehension.However, it is nontrivial to apply PLMs in personalized review summarization directly since there are rich personalized information (e.g., user preferences and product characteristics) to be considered, which is crucial to the salience estimation of input review.In this paper, we propose a pre-trained personalized review summarization method, which aims to effectively incorporate the personalized information of users and products into the salience estimation of the input reviews.We design a personalized encoder that could identify the salient contents of the input sequence by jointly considering the semantic and personalized information respectively (i.e., ratings, user and product IDs, and linguistic features), yielding personalized representations for the input reviews and history summaries separately.Moreover, we design an interactive information selection mechanism that further identifies the salient contents of the input reviews and selects relative information from the history summaries.The results on real-world datasets show that our method performs better than the state-of-the-art baselines and could generate more readable summaries.
Janko at SemEval-2023 Task 2: Bidirectional LSTM Model Based on Pre-training for Chinese Named Entity Recognition

This paper describes the method we submitted as the Janko team in the SemEval-2023 Task 2, Multilingual Complex Named Entity Recognition (MultiCoNER 2).We only participated in the Chinese track.In this paper, we implement the BERT-BiLSTM-RDrop model.We use the fine-tuned BERT models, take the output of BERT as the input of the BiLSTM network, and finally use R-Drop technology to optimize the loss function.Our submission achieved a macro-averaged F1 score of 0.579 on the test set.
ACCENT: An Automatic Event Commonsense Evaluation Metric for Open-Domain Dialogue Systems

Commonsense reasoning is omnipresent in human communications and thus is an important feature for open-domain dialogue systems.However, evaluating commonsense in dialogue systems is still an open challenge.We take the first step by focusing on event commonsense that considers events and their relations, and is crucial in both dialogues and general commonsense reasoning.We propose AC-CENT, an event commonsense evaluation metric empowered by commonsense knowledge bases (CSKBs).ACCENT first extracts eventrelation tuples from a dialogue, and then evaluates the response by scoring the tuples in terms of their compatibility with the CSKB.To evaluate ACCENT, we construct the first public event commonsense evaluation dataset for open-domain dialogues.Our experiments show that ACCENT is an efficient metric for event commonsense evaluation, which achieves higher correlations with human judgments than existing baselines.
Contrastive Novelty-Augmented Learning: Anticipating Outliers with Large Language Models

In many task settings, text classification models are likely to encounter examples from novel classes on which they cannot predict correctly.Selective prediction, in which models abstain on low-confidence examples, provides a possible solution, but existing models are often overly confident on unseen classes.To remedy this overconfidence, we introduce Contrastive Novelty-Augmented Learning (CoNAL), a twostep method that generates OOD examples representative of novel classes, then trains to decrease confidence on them.First, we generate OOD examples by prompting a large language model twice: we prompt it to enumerate relevant novel classes, then generate examples from each novel class matching the task format.Second, we train a classifier with a novel contrastive objective that encourages lower confidence on generated OOD examples than training examples.When trained with CoNAL, classifiers improve in their ability to detect and abstain on novel class examples over prior methods by an average of 2.3% in terms of accuracy under the accuracy-coverage curve (AUAC) and 5.5% AUROC across 4 NLP datasets, with no cost to in-distribution accuracy.
ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation

Large pre-trained language models achieve impressive results across many tasks.However, recent works point out that pre-trained language models may memorize a considerable fraction of their training data, leading to the privacy risk of information leakage.In this paper, we propose a method named ETHICIST for targeted training data Extraction THrough loss smoothed soft prompting and calIbrated ConfIdence eSTimation, investigating how to recover the suffix in the training data when given a prefix.To elicit memorization in the attacked model, we tune soft prompt embeddings while keeping the model fixed.We further propose a smoothing loss that smooths the loss distribution of the suffix tokens to make it easier to sample the correct suffix.In order to select the most probable suffix from a collection of sampled suffixes and estimate the prediction confidence, we propose a calibrated confidence estimation method, which normalizes the confidence of the generated suffixes with a local estimation.We show that ETHICIST significantly improves the extraction performance on a recently proposed public benchmark.We also investigate several factors influencing the data extraction performance, including decoding strategy, model scale, prefix length, and suffix length.
HHS at SemEval-2023 Task 10: A Comparative Analysis of Sexism Detection Based on the RoBERTa Model

This paper describes the methods and models applied by our team HHS in SubTask-A of SemEval-2023 Task 10 about sexism detection.In this task, we trained with the officially released data and analyzed the performance of five models, TextCNN, BERT, RoBERTa, XLNet, and Sup-SimCSE-RoBERTa.The experiments show that most of the models can achieve good results.Then, we tried data augmentation, model ensemble, dropout, and other operations on several of these models, and compared the results for analysis.In the end, the most effective approach that yielded the best results on the test set involved the following steps: enhancing the sexist data using dropout, feeding it as input to the Sup-SimCSE-RoBERTa model, and providing the raw data as input to the XLNet model.Then, combining the outputs of the two methods led to even better results.This method yielded a Macro-F1 score of 0.823 in the final evaluation phase of the SubTask-A of the competition.
UniBoe’s at SemEval-2023 Task 10: Model-Agnostic Strategies for the Improvement of Hate-Tuned and Generative Models in the Classification of Sexist Posts

We present our submission to SemEval-2023 Task 10: Explainable Detection of Online Sexism (EDOS).We address all three tasks: Task A consists of identifying whether a post is sexist.If so, Task B attempts to assign it one of four classes: threats, derogation, animosity, and prejudiced discussions.Task C aims for an even more fine-grained classification, divided among 11 classes.We experiment with finetuning of hate-tuned Transformer-based models and priming for generative models.In addition, we explore model-agnostic strategies, such as data augmentation techniques combined with active learning, as well as obfuscation of identity terms.Our official submissions obtain an F 1 score of 0.83 for Task A, 0.58 for Task B and 0.32 for Task C.
Unsupervised Selective Rationalization with Noise Injection

A major issue with using deep learning models in sensitive applications is that they provide no explanation for their output.To address this problem, unsupervised selective rationalization produces rationales alongside predictions by chaining two jointly-trained components, a rationale generator and a predictor.Although this architecture guarantees that the prediction relies solely on the rationale, it does not ensure that the rationale contains a plausible explanation for the prediction.We introduce a novel training technique that effectively limits generation of implausible rationales by injecting noise between the generator and the predictor.Furthermore, we propose a new benchmark for evaluating unsupervised selective rationalization models using movie reviews from existing datasets.We achieve sizeable improvements in rationale plausibility and task accuracy over the state-of-the-art across a variety of tasks, including our new benchmark, while maintaining or improving model faithfulness. 1
INTapt: Information-Theoretic Adversarial Prompt Tuning for Enhanced Non-Native Speech Recognition

Automatic Speech Recognition (ASR) systems have attained unprecedented performance with large speech models pre-trained based on self-supervised speech representation learning.However, these pre-trained speech models suffer from representational bias as they tend to better represent those prominent accents (i.e., native (L1) English accent) in the pre-training speech corpus than less represented accents, resulting in a deteriorated performance for nonnative (L2) English accents.Although there have been some approaches to mitigate this issue, all of these methods require updating the pre-trained model weights.In this paper, we propose Information Theoretic Adversarial Prompt Tuning (INTapt), which introduces prompts concatenated to the original input that can re-modulate the attention of the pre-trained model such that the corresponding input resembles a native (L1) English speech without updating the backbone weights.INTapt is trained simultaneously in the following two manners:(1) adversarial training to reduce accent feature dependence between the original input and the prompt-concatenated input and (2) training to minimize CTC loss for improving ASR performance to a prompt-concatenated input.Experimental results show that INTapt improves the performance of L2 English and increases feature similarity between L2 and L1 accents.