Reviewerly/exHarmony huggingface dataset:
CNLP-NITS at SemEval-2023 Task 10: Online sexism prediction, PREDHATE!

Online sexism is a rising issue that threatens women's safety, fosters hostile situations, and upholds social inequities.We describe a task SemEval-2023 Task 10 for creating Englishlanguage models that can precisely identify and categorize sexist content on internet forums and social platforms like Gab and Reddit as well to provide an explainability in order to address this problem.The problem is divided into three hierarchically organized subtasks: binary sexism detection, sexism by category, and sexism by fine-grained vector.The dataset consists of 20,000 labelled entries.For Task A, pertained models like Convolutional Neural Network (CNN) and Bidirectional Long Short-Term Memory (BiLSTM), which is called CNN-BiLSTM and Generative Pretrained Transformer 2 (GPT-2) models were used, as well as the GPT-2 model for Task B and C, and have provided experimental configurations.According to our findings, the GPT-2 model performs better than the CNN-BiLSTM model for Task A, while GPT-2 is highly accurate for Tasks B and C on the training, validation and testing splits of the training data provided in the task.Our proposed models allow researchers to create more precise and understandable models for identifying and categorizing sexist content in online forums, thereby empowering users and moderators.
Kb at SemEval-2023 Task 3: On Multitask Hierarchical BERT Base Neural Network for Multi-label Persuasion Techniques Detection

This paper presents a solution for Semeval 2023 subtask3 of task3: persuasion techniques in paragraphs detection.The aim of this task is to identify all persuasion techniques in each paragraph of given news articles.We use hierarchical multitask neural networks combined with transformers.Span detection is an auxiliary task that helps in the main task: identify propaganda techniques.Our experiments show that if we change the index of BERT embedding from the first token of the whole input to the first token of the identified span it can improve the performance.Span and label detection can be performed using one network so we save data and, when data is limited, we can use more of it for training.
Don’t Parse, Choose Spans! Continuous and Discontinuous Constituency Parsing via Autoregressive Span Selection

We present a simple and unified approach for both continuous and discontinuous constituency parsing via autoregressive span selection.Constituency parsing aims to produce a set of non-crossing spans so that they can form a constituency parse tree.We sort gold spans in a predefined order and train a pointer network to autoregressively select spans by that order.To deal with a discontinuous span, we consecutively select its subspans from left to right, label all but the last subspans with a special discontinuous label, and label the last subspan with the whole discontinuous span's label.We use a simple heuristic to output valid trees from selected spans so that our approach is able to predict all possible continuous and discontinuous constituency trees without sacrificing data coverage and without the need to use expensive chart-based parsing algorithms.Extensive experiments show that our model achieves stateof-the-art or competitive performance on all benchmarks of continuous and discontinuous constituency parsing .
Do Models Really Learn to Follow Instructions? An Empirical Study of Instruction Tuning

Recent works on instruction tuning (IT) have achieved great performance with zero-shot generalizability to unseen tasks.With additional context (e.g., task definition, examples) provided to models for fine-tuning, they achieved much higher performance than untuned models.Despite impressive performance gains, what models learn from IT remains understudied.In this work, we analyze how models utilize instructions during IT by comparing model training with altered vs. original instructions.Specifically, we create simplified task definitions by removing all semantic components and only leaving the output space information, and delusive examples that contain incorrect input-output mapping.Our experiments show that models trained on simplified task definition or delusive examples can achieve comparable performance to the ones trained on the original instructions and examples.Furthermore, we introduce a random baseline to perform zeroshot classification tasks, and find it achieves similar performance (42.6% exact-match) as IT does (43% exact-match) in low resource setting, while both methods outperform naive T5 significantly (30% per exact-match).Our analysis provides evidence that the impressive performance gain of current IT models can come from picking up superficial patterns, such as learning the output format and guessing.Our study highlights the urgent need for more reliable IT methods and evaluation.
Exploring Robust Overfitting for Pre-trained Language Models

We identify the robust overfitting issue for pretrained language models by showing that the robust test loss increases as the epoch grows.Through comprehensive exploration of the robust loss on the training set, we attribute robust overfitting to the model's memorization of the adversarial training data.We attempt to mitigate robust overfitting by combining regularization methods with adversarial training.Following the philosophy to prevent the model from memorizing the adversarial data, we find that flooding, a regularization method with loss scaling, can mitigate robust overfitting for pretrained language models.Eventually, we investigate the effect of flooding levels and evaluate the models' adversarial robustness under textual adversarial attacks.Extensive experiments demonstrate that our method can mitigate robust overfitting upon three top adversarial training methods and further promote adversarial robustness.
Cost-effective Distillation of Large Language Models

Knowledge distillation (KD) involves training a small "student" model to replicate the strong performance of a high-capacity "teacher" model, enabling efficient deployment in resource-constrained settings.Topperforming methods tend to be task-or architecture-specific and lack generalizability.Several existing approaches require pretraining of the teacher on task-specific datasets, which can be costly for large and unstable for small datasets.Here we propose an approach for improving KD through a novel distillation loss agnostic to the task and model architecture.We successfully apply our method to the distillation of the BERT-base and achieve highly competitive results from the distilled student across a range of GLUE tasks, especially for tasks with smaller datasets. 1
C-XNLI: Croatian Extension of XNLI Dataset

Comprehensive multilingual evaluations have been encouraged by emerging cross-lingual benchmarks and constrained by existing parallel datasets.To partially mitigate this limitation, we extended the Cross-lingual Natural Language Inference (XNLI) corpus with Croatian.The development and test sets were translated by a professional translator, and we show that Croatian is consistent with other XNLI dubs.The train set is translated using Facebook's 1.2B parameter m2m_100 model.We thoroughly analyze the Croatian train set and compare its quality with the existing machinetranslated German set.The comparison is based on 2000 manually scored sentences per language using a variant of the Direct Assessment (DA) score commonly used at the Conference on Machine Translation (WMT).Our findings reveal that a less-resourced language like Croatian is still lacking in translation quality of longer sentences compared to German.However, both sets have a substantial amount of poor quality translations, which should be considered in translation-based training or evaluation setups.
Brainstormers_msec at SemEval-2023 Task 10: Detection of sexism related comments in social media using deep learning

Social media is the media through which people share their thoughts and opinions.This has both its pros and cons which depends on the type of information being conveyed.If any information conveyed over social media hurts or affects a person, such information can be removed as it may disturb their mental health and may decrease their self confidence.During the last decade, hateful and sexist content towards women is being increasingly spread on social networks.The exposure to sexist speech has serious consequences to women's life and limits their freedom of speech.Sexism is expressed in very different forms: it includes subtle stereotypes and attitudes that, although frequently unnoticed, are extremely harmful for both women and society.Sexist comments have a major impact on women being subjected to it.We as a team participated in the shared task Explainable Detection of Online Sexism (EDOS) at Se-mEval 2023 and have proposed a model which identifies the sexist comments and its type from English social media posts using the data set shared for the task.Different tranformer models like BERT, DistilBERT, and RoBERTa are used by the proposed model for implementing all the three tasks shared by EDOS.On using the BERT model, macro F1 score of 0.8073, 0.5876 and 0.3729 are achieved for Task A, Task B and Task C respectively.
(QA)2: Question Answering with Questionable Assumptions: Question Answering with Questionable Assumptions

Naturally occurring information-seeking questions often contain questionable assumptions-assumptions that are false or unverifiable.Questions containing questionable assumptions are challenging because they require a distinct answer strategy that deviates from typical answers for information-seeking questions.For instance, the question When did Marie Curie discover Uranium?cannot be answered as a typical when question without addressing the false assumption Marie Curie discovered Uranium.In this work, we propose (QA) 2 (Question Answering with Questionable Assumptions), an open-domain evaluation dataset consisting of naturally occurring search engine queries that may or may not contain questionable assumptions.To be successful on (QA) 2 , systems must be able to detect questionable assumptions and also be able to produce adequate responses for both typical information-seeking questions and ones with questionable assumptions.Through human rater acceptability on end-to-end QA with (QA) 2 , we find that current models do struggle with handling questionable assumptions, leaving substantial headroom for progress.* Equal contribution, corresponding authors ∆ Work partly done at NYU before joining BU. δ Work done
Large Scale Generative Multimodal Attribute Extraction for E-commerce Attributes

E-commerce websites (e.g.Amazon) have a plethora of structured and unstructured information (text and images) present on the product pages.Sellers often either don't label or mislabel values of the attributes (e.g.color, size etc.) for their products.Automatically identifying these attribute values from an eCommerce product page that contains both text and images is a challenging task, especially when the attribute value is not explicitly mentioned in the catalog.In this paper, we present a scalable solution for this problem where we pose attribute extraction problem as a question-answering task, which we solve using MXT, consisting of three key components: (i) MAG (Multimodal Adaptation Gate), (ii) Xception network, and (iii) T5 encoder-decoder.Our system consists of a generative model that generates attribute-values for a given product by using both textual and visual characteristics (e.g.images) of the product.We show that our system is capable of handling zero-shot attribute prediction (when attribute value is not seen in training data) and value-absent prediction (when attribute value is not mentioned in the text) which are missing in traditional classification-based and NERbased models respectively.We have trained our models using distant supervision, removing dependency on human labeling, thus making them practical for real-world applications.With this framework, we are able to train a single model for 1000s of (product-type, attribute) pairs, thus reducing the overhead of training and maintaining separate models.Extensive experiments on two real world datasets show that our framework improves the absolute recall@90P by 10.16% and 6.9% from the existing state of the art models.In a popular e- * This work was done while author was in International Machine Learning team.commerce store, we have deployed our models for 1000s of (product-type, attribute) pairs.
ArgU: A Controllable Factual Argument Generator

Effective argumentation is essential towards a purposeful conversation with a satisfactory outcome.For example, persuading someone to reconsider smoking might involve empathetic, well founded arguments based on facts and expert opinions about its ill-effects and the consequences on one's family.However, the automatic generation of high-quality factual arguments can be challenging.Addressing existing controllability issues can make the recent advances in computational models for argument generation a potential solution.In this paper, we introduce ArgU: a neural argument generator capable of producing factual arguments from input facts and real-world concepts that can be explicitly controlled for stance and argument structure using Walton's argument scheme-based control codes.Unfortunately, computational argument generation is a relatively new field and lacks datasets conducive to training.Hence, we have compiled and released an annotated corpora of 69,428 arguments spanning six topics and six argument schemes, making it the largest publicly available corpus for identifying argument schemes; the paper details our annotation and dataset creation framework.We further experiment with an argument generation strategy that establishes an inference strategy by generating an "argument template" before actual argument generation.Our results demonstrate that it is possible to automatically generate diverse arguments exhibiting different inference patterns for the same set of facts by using control codes based on argument schemes and stance.
One-Shot Exemplification Modeling via Latent Sense Representations

Exemplification modeling is a recently proposed task that aims to produce a viable sentence using a target word that takes on a specific meaning.This task can be particularly challenging for polysemous words since they can have multiple meanings.In this paper, we propose a one-shot variant of the exemplification modeling task such that labeled data is not needed during training, making it possible to train our system using a raw text corpus.Given one example at test time, our proposed approach can generate diverse and fluent examples where the target word accurately matches its intended meaning.We compare our approach to a fully-supervised baseline trained with different amounts of data and focus our evaluation on polysemous words.We use both automatic and human evaluations to demonstrate how each model performs on both seen and unseen words.Our proposed approach performs similarly to the fully-supervised baseline despite not using labeled data during training.
TextVerifier: Robustness Verification for Textual Classifiers with Certifiable Guarantees

When textual classifiers are deployed in safetycritical workflows, they must withstand the onslaught of AI-enabled model confusion caused by adversarial examples with minor alterations.In this paper, the main objective is to provide a formal verification framework, called TextVerifier, with certifiable guarantees on deep neural networks in natural language processing against word-level alteration attacks.We aim to provide an approximation of the maximal safe radius by deriving provable bounds both mathematically and automatically, where a minimum word-level L 0 distance is quantified as a guarantee for the classification invariance of victim models.Here, we illustrate three strengths of our strategy: i) certifiable guarantee: effective verification with convergence to ensure approximation of maximal safe radius with tight bounds ultimately; ii) high-efficiency: it yields an efficient speed edge by a novel parallelization strategy that can process a set of candidate texts simultaneously on GPUs; and iii) reliable anytime estimation: the verification can return intermediate bounds, and robustness estimates that are gradually, but strictly, improved as the computation proceeds.Furthermore, experiments are conducted on text classification on four datasets over three victim models to demonstrate the validity of tightening bounds.Our tool TextVerifier is available at https: //github.com/TrustAI/TextVerifer.
Incomplete Utterance Rewriting by A Two-Phase Locate-and-Fill Regime

Rewriting incomplete and ambiguous utterances can improve dialogue models' understanding of the context and help them generate better results.However, the existing end-toend models will have the problem of too large search space, resulting in poor quality of rewriting results.We propose a 2-phase rewriting framework which first predicts the empty slots in the utterance that need to be completed, and then generate the part to be filled into each positions.Our framework is simple to implement, fast to run, and achieves the state-of-the-art results on several public rewriting datasets.
Eliciting Rich Positive Emotions in Dialogue Generation

Positive emotion elicitation aims at evoking positive emotion states in human users in open-domain dialogue generation. However, most work focuses on inducing a single-dimension of positive sentiment using human annotated datasets, which limits the scale of the training dataset. In this paper, we propose to model various emotions in large unannotated conversations, such as joy, trust and anticipation, by leveraging a latent variable to control the emotional intention of the response. Our proposed emotion-eliciting-Conditional-Variational-AutoEncoder (EE-CVAE) model generates more diverse and emotionally-intelligent responses compared to single-dimension baseline models in human evaluation.
Enhancing Video Translation Context with Object Labels

We present a simple yet efficient method to enhance the quality of machine translation models trained on multimodal corpora by augmenting the training text with labels of detected objects in the corresponding video segments.We then test the effects of label augmentation in both baseline and two automatic speech recognition (ASR) conditions.In contrast with multimodal techniques that merge visual and textual features, our modular method is easy to implement and the results are more interpretable.Comparisons are made with Transformer translation architectures trained with baseline and augmented labels, showing improvements of up to +1.0 BLEU on the How2 dataset.
From the One, Judge of the Whole: Typed Entailment Graph Construction with Predicate Generation

Entailment Graphs (EGs) have been constructed based on extracted corpora as a strong and explainable form to indicate contextindependent entailment relations in natural languages.However, EGs built by previous methods often suffer from the severe sparsity issues, due to limited corpora available and the longtail phenomenon of predicate distributions.In this paper, we propose a multi-stage method, Typed Predicate-Entailment Graph Generator (TP-EGG), to tackle this problem.Given several seed predicates, TP-EGG builds the graphs by generating new predicates and detecting entailment relations among them.The generative nature of TP-EGG helps us leverage the recent advances from large pretrained language models (PLMs), while avoiding the reliance on carefully prepared corpora.Experiments on benchmark datasets show that TP-EGG can generate high-quality and scale-controllable entailment graphs, achieving significant in-domain improvement over state-of-the-art EGs and boosting the performance of down-stream inference tasks 1 .
2*n is better than is better than n2: Decomposing Event Coreference Resolution into Two Tractable Problems: Decomposing Event Coreference Resolution into Two Tractable Problems

Event Coreference Resolution (ECR) is the task of linking mentions of the same event either within or across documents.Most mention pairs are not coreferent, yet many that are coreferent can be identified through simple techniques such as lemma matching of the event triggers or the sentences in which they appear.Existing methods for training coreference systems sample from a largely skewed distribution, making it difficult for the algorithm to learn coreference beyond surface matching.Additionally, these methods are intractable because of the quadratic operations needed.To address these challenges, we break the problem of ECR into two parts: a) a heuristic to efficiently filter out a large number of non-coreferent pairs, and b) a training approach on a balanced set of coreferent and non-coreferent mention pairs.By following this approach, we show that we get comparable results to the state of the art on two popular ECR datasets while significantly reducing compute requirements.We also analyze the mention pairs that are "hard" to accurately classify as coreferent or non-coreferent 1 .
DiffuSum: Generation Enhanced Extractive Summarization with Diffusion

Extractive summarization aims to form a summary by directly extracting sentences from the source document.Existing works mostly formulate it as a sequence labeling problem by making individual sentence label predictions.This paper proposes DiffuSum, a novel paradigm for extractive summarization, by directly generating the desired summary sentence representations with diffusion models and extracting sentences based on sentence representation matching.In addition, DiffuSum jointly optimizes a contrastive sentence encoder with a matching loss for sentence representation alignment and a multi-class contrastive loss for representation diversity.Experimental results show that DiffuSum achieves the new state-ofthe-art extractive results on CNN/DailyMail with ROUGE scores of 44.83/22.56/40.56.Experiments on the other two datasets with different summary lengths also demonstrate the effectiveness of DiffuSum.The strong performance of our framework shows the great potential of adapting generative models for extractive summarization.To encourage more following work in the future, we have released our codes at https://github.com/hpzhang94/DiffuSum
HAUSER: Towards Holistic and Automatic Evaluation of Simile Generation

Similes play an imperative role in creative writing such as story and dialogue generation.Proper evaluation metrics are like a beacon guiding the research of simile generation (SG).However, it remains under-explored as to what criteria should be considered, how to quantify each criterion into metrics, and whether the metrics are effective for comprehensive, efficient, and reliable SG evaluation.To address the issues, we establish HAUSER, a holistic and automatic evaluation system for the SG task, which consists of five criteria from three perspectives and automatic metrics for each criterion.Through extensive experiments, we verify that our metrics are significantly more correlated with human ratings from each perspective compared with prior automatic metrics.