Reviewerly/exHarmony huggingface dataset:
KDDIE at SemEval-2023 Task 2: External Knowledge Injection for Named Entity Recognition

This paper introduces our system for the Se-mEval 2023 Task 2: Multilingual Complex Named Entity Recognition (MultiCoNER II) competition.Our team (KDDIE) focused on the sub-task of Named Entity Recognition (NER) for the language of English in the competition and reported our results.To achieve our goal, we utilized transfer learning by finetuning pre-trained language models (PLMs) on the competition dataset.Our approach involved combining a BERT-based PLM with external knowledge to provide additional context to the model.In this report, we present our findings and results.
Sentence Ordering with a Coherence Verifier

This paper presents a novel sentence ordering method by plugging a coherence verifier (CoVer) into pair-wise ranking-based and sequence generation-based methods. It does not change the model parameters of the baseline, and only verifies the coherence of candidate (partial) orders produced by the baseline and reranks them in beam search. We also propose a coherence model as CoVer with a novel graph formulation and a novel data construction strategy for contrastive pre-training independently of the sentence ordering task. Experimental results on four benchmarks demonstrate the effectiveness of our method with topological sorting-based and pointer network-based methods as the baselines. Detailed analyses illustrate how CoVer improves the baselines and confirm the importance of its graph formulation and training strategy. Our code is available at https://github.com/SN-Jia/SO_with_CoVer.
Adversarial Textual Robustness on Visual Dialog

Adversarial robustness evaluates the worst-case performance scenario of a machine learning model to ensure its safety and reliability.For example, cases where the user input contains a minimal change, e.g. a synonym, which causes the previously correct model to return a wrong answer.Using this scenario, this study is the first to investigate the robustness of visually grounded dialog models towards textual attacks.We first aim to understand how multimodal input components contribute to model robustness.Our results show that models which encode dialog history are more robust by providing redundant information.This is in contrast to prior work which finds that dialog history is negligible for model performance on this task.We also evaluate how to generate adversarial test examples which successfully fool the model but remain undetected by the user/software designer.Our analysis shows that the textual, as well as the visual context are important to generate plausible attacks.
Interpretable Math Word Problem Solution Generation via Step-by-step Planning

Solutions to math word problems (MWPs) with step-by-step explanations are valuable, especially in education, to help students better comprehend problem-solving strategies.Most existing approaches only focus on obtaining the final correct answer.A few recent approaches leverage intermediate solution steps to improve final answer correctness but often cannot generate coherent steps with a clear solution strategy.Contrary to existing work, we focus on improving the correctness and coherence of the intermediate solutions steps.We propose a step-by-step planning approach for intermediate solution generation, which strategically plans the generation of the next solution step based on the MWP and the previous solution steps.Our approach first plans the next step by predicting the necessary math operation needed to proceed, given history steps, then generates the next step, token-by-token, by prompting a language model with the predicted math operation.Experiments on the GSM8K dataset demonstrate that our approach improves the accuracy and interpretability of the solution on both automatic metrics and human evaluation.
Hexatagging: Projective Dependency Parsing as Tagging

We introduce a novel dependency parser, the hexatagger, that constructs dependency trees by tagging the words in a sentence with elements from a finite set of possible tags. In contrast to many approaches to dependency parsing, our approach is fully parallelizable at training time, i.e., the structure-building actions needed to build a dependency parse can be predicted in parallel to each other. Additionally, exact decoding is linear in time and space complexity. Furthermore, we derive a probabilistic dependency parser that predicts hexatags using no more than a linear model with features from a pretrained language model, i.e., we forsake a bespoke architecture explicitly designed for the task. Despite the generality and simplicity of our approach, we achieve state-of-the-art performance of 96.4 LAS and 97.4 UAS on the Penn Treebank test set. Additionally, our parser’s linear time complexity and parallelism significantly improve computational efficiency, with a roughly 10-times speed-up over previous state-of-the-art models during decoding.
Diane Simmons at SemEval-2023 Task 5: Is it possible to make good clickbait spoilers using a Zero-Shot approach? Check it out!

In this paper, we present a possible solution to the SemEval23 shared task of generating spoilers for clickbait headlines.Using a Zero-Shot approach with two different Transformer architectures, BLOOM and RoBERTa, we generate three different types of spoilers: phrase, passage and multi.We found, RoBERTa pretrained for Question-Answering to perform better than BLOOM for causal language modelling, however both architectures proved promising for future attempts at such tasks.
Peeking inside the black box: A Commonsense-aware Generative Framework for Explainable Complaint Detection

Complaining is an illocutionary act in which the speaker communicates his/her dissatisfaction with a set of circumstances and holds the hearer (the complainee) answerable, directly or indirectly.Considering breakthroughs in machine learning approaches, the complaint detection task has piqued the interest of the natural language processing (NLP) community.Most of the earlier studies failed to justify their findings, necessitating the adoption of interpretable models that can explain the model's output in real-time.We introduce an explainable complaint dataset, X-CI, the first benchmark dataset for explainable complaint detection.Each instance in the X-CI dataset is annotated with five labels: complaint label, emotion label, polarity label, complaint severity level, and rationale (explainability), i.e., the causal span explaining the reason for the complaint/noncomplaint label.We address the task of explainable complaint detection and propose a commonsense-aware unified generative framework by reframing the multitask problem as a text-to-text generation task.Our framework can predict the complaint cause, severity level, emotion, and polarity of the text in addition to detecting whether it is a complaint or not.We further establish the advantages of our proposed model on various evaluation metrics over the state-of-the-art models and other baselines when applied to the X-CI dataset in both full and few-shot settings 1 .
Multi-Source Test-Time Adaptation as Dueling Bandits for Extractive Question Answering

In this work, we study multi-source test-time model adaptation from user feedback, where K distinct models are established for adaptation.To allow efficient adaptation, we cast the problem as a stochastic decision-making process, aiming to determine the best adapted model after adaptation.We discuss two frameworks: multi-armed bandit learning and multi-armed dueling bandits.Compared to multi-armed bandit learning, the dueling framework allows pairwise collaboration among K models, which is solved by a novel method named Co-UCB proposed in this work.Experiments on six datasets of extractive question answering (QA) show that the dueling framework using Co-UCB is more effective than other strong baselines for our studied problem 1 .
DS4DH at MEDIQA-Chat 2023: Leveraging SVM and GPT-3 Prompt Engineering for Medical Dialogue Classification and Summarization

This paper presents the results of the Data Science for Digital Health (DS4DH) group in the MEDIQA-Chat Tasks at ACL-ClinicalNLP 2023.Our study combines the power of a classical machine learning method, Support Vector Machine, for classifying medical dialogues, along with the implementation of oneshot prompts using GPT-3.5.We employ dialogues and summaries from the same category as prompts to generate summaries for novel dialogues.Our findings exceed the average benchmark score, offering a robust reference for assessing performance in this field.
DecompX: Explaining Transformers Decisions by Propagating Token Decomposition

An emerging solution for explainingTransformer-based models is to use vectorbased analysis on how the representations are formed.However, providing a faithful vector-based explanation for a multi-layer model could be challenging in three aspects: (1) Incorporating all components into the analysis, (2) Aggregating the layer dynamics to determine the information flow and mixture throughout the entire model, and (3) Identifying the connection between the vector-based analysis and the model's predictions.In this paper, we present DecompX to tackle these challenges.DecompX is based on the construction of decomposed token representations and their successive propagation throughout the model without mixing them in between layers.Additionally, our proposal provides multiple advantages over existing solutions for its inclusion of all encoder components (especially nonlinear feed-forward networks) and the classification head.The former allows acquiring precise vectors while the latter transforms the decomposition into meaningful prediction-based values, eliminating the need for norm-or summation-based vector aggregation.According to the standard faithfulness evaluations, DecompX consistently outperforms existing gradient-based and vector-based approaches on various datasets.Our code is available at github.com/mohsenfayyaz/DecompX.
ECG-QALM: Entity-Controlled Synthetic Text Generation using Contextual Q&amp;A for NER

Named Entity Recognition (NER) state-ofthe-art methods requires high-quality labeled datasets.Issues such as scarcity of labeled data, under-representation of entities, and privacy concerns with using sensitive data for training, can be significant barriers.Generating synthetic data to train models is a promising solution to mitigate these problems.We propose ECG-QALM, a contextual question and answering approach using pre-trained language models to synthetically generate entitycontrolled text.Generated text is then used to augment small labeled datasets for downstream NER tasks.We evaluate our method on two publicly available datasets.We find ECG-QALM is capable of producing full text samples with desired entities appearing in a controllable way, while retaining sentence coherence closest to the real world data.Evaluations on NER tasks show significant improvements (75% -140%) in low-labeled data regimes.
Controlling Learned Effects to Reduce Spurious Correlations in Text Classifiers

To address the problem of NLP classifiers learning spurious correlations between training features and target labels, a common approach is to make the model's predictions invariant to these features.However, this can be counterproductive when the features have a non-zero causal effect on the target label and thus are important for prediction.Therefore, using methods from the causal inference literature, we propose an algorithm to regularize the learnt effect of the features on the model's prediction to the estimated effect of feature on label.This results in an automated augmentation method that leverages the estimated effect of a feature to appropriately change the labels for new augmented inputs.On toxicity and IMDB review datasets, the proposed algorithm minimises spurious correlations and improves the minority group (i.e., samples breaking spurious correlations) accuracy, while also improving the total accuracy compared to standard training. 1
HyperT5: Towards Compute-Efficient Korean Language Modeling

Pretraining and fine-tuning language models have become the standard practice in industrial natural language processing (NLP), but developing and deploying general-purpose language models without the abundant computation or data resources is a real-world issue faced by smaller organizations or communities whose main focus is languages with less accessible resources (e.g., non-English).This paper explores the sequence-to-sequence (seq2seq) language model architecture as a more practical and compute-efficient alternative to the decoder-oriented approach (e.g., GPT-3), accompanied by novel findings in compute-optimality analyses.We successfully trained billion-scale Korean-language seq2seq language models that strongly outperform other competitive models in Korean benchmarks.Moreover, we demonstrate that such language models can be more efficiently utilized by employing a heavy pre-finetuning strategy, by showcasing a case study on dialog-task adaptation.Our case study shows that adopting language models with more readily available domain-specific unlabeled data greatly improves fine-tuning data efficiency in lowresource settings.
Improving the Robustness of Summarization Systems with Dual Augmentation

A robust summarization system should be able to capture the gist of the document, regardless of the specific word choices or noise in the input.In this work, we first explore the summarization models' robustness against perturbations including word-level synonym substitution and noise.To create semanticconsistent substitutes, we propose a SummAttacker, which is an efficient approach to generating adversarial samples based on language models.Experimental results show that stateof-the-art summarization models have a significant decrease in performance on adversarial and noisy test sets.Next, we analyze the vulnerability of the summarization systems and explore improving the robustness by data augmentation.Specifically, the first brittleness factor we found is the poor understanding of infrequent words in the input.Correspondingly, we feed the encoder with more diverse cases created by SummAttacker in the input space.The other factor is in the latent space, where the attacked inputs bring more variations to the hidden states.Hence, we construct adversarial decoder input and devise manifold softmixing operation in hidden space to introduce more diversity.Experimental results on Gigaword and CNN/DM datasets demonstrate that our approach achieves significant improvements over strong baselines and exhibits higher robustness on noisy, attacked, and clean datasets 1 .
MOSPC: MOS Prediction Based on Pairwise Comparison

As a subjective metric to evaluate the quality of synthesized speech, Mean opinion score (MOS) usually requires multiple annotators to score the same speech.Such an annotation approach requires a lot of manpower and is also timeconsuming.MOS prediction model for automatic evaluation can significantly reduce labor cost.In previous works, it is difficult to accurately rank the quality of speech when the MOS scores are close.However, in practical applications, it is more important to correctly rank the quality of synthesis systems or sentences than simply predicting MOS scores.Meanwhile, as each annotator scores multiple audios during annotation, the score is probably a relative value based on the first or the first few speech scores given by the annotator.Motivated by the above two points, we propose a general framework for MOS prediction based on pair comparison (MOSPC), and we utilize C-Mixup algorithm to enhance the generalization performance of MOSPC.The experiments on BVCC and VCC2018 show that our framework outperforms the baselines on most of the correlation coefficient metrics, especially on the metric KTAU related to quality ranking.And our framework also surpasses the strong baseline in ranking accuracy on each fine-grained segment.These results indicate that our framework contributes to improving the ranking accuracy of speech quality.
Define, Evaluate, and Improve Task-Oriented Cognitive Capabilities for Instruction Generation Models

Recent work studies the cognitive capabilities of language models through psychological tests designed for humans.While these studies are helpful for understanding the general capabilities of these models, there is no guarantee that a model possessing sufficient capabilities to pass those tests would actually use those capabilities in performing real-life tasks.In this work, we formulate task-oriented cognitive capabilities, which are human-like cognitive capabilities that language models leverage to perform tasks.These capabilities are (i) the ability to quickly generate good candidate utterances (the search capability) (ii) the ability to predict how a listener interprets those utterances and choose the most appropriate one (the pragmatic capability).We design an evaluation scheme for comparing these capabilities of a language model with those of a human.Applying this scheme to examine various models in a navigation instruction generation problem, we find that their pragmatic capability is severely lacking.This insight leads us to augment them with better models of the listener and obtain a significant boost of 11% in success rate in guiding real humans.Our work advocates for having a principled procedure for aligning language models with humans that involves (i) formulating task-oriented capabilities, (ii) devising a method to quantify their deficiency, and (iii) iteratively improving them.
Mars: Modeling Context &amp; State Representations with Contrastive Learning for End-to-End Task-Oriented Dialog

Traditional end-to-end task-oriented dialog systems first convert dialog context into belief state and action state before generating the system response.The system response performance is significantly affected by the quality of the belief state and action state.We first explore what dialog context representation is beneficial to improving the quality of the belief state and action state, which further enhances the generated response quality.To tackle our exploration, we propose Mars, an end-to-end task-oriented dialog system with two contrastive learning strategies to model the relationship between dialog context and belief/action state representations.Empirical results show dialog context representations, which are more different from semantic state representations, are more conducive to multi-turn task-oriented dialog.Moreover, our proposed Mars achieves state-of-theart performance on the MultiWOZ 2.0, CamRest676, and CrossWOZ 1 .
Digging out Discrimination Information from Generated Samples for Robust Visual Question Answering

Visual Question Answering (VQA) aims to answer a textual question based on a given image.Nevertheless, recent studies have shown that VQA models tend to capture the biases to answer the question, instead of using the reasoning ability, resulting in poor generalisation ability.To alleviate the issue, some existing methods consider the natural distribution of the data, and construct samples to balance the dataset, achieving remarkable performance.However, these methods may encounter some limitations: 1) rely on additional annotations, 2) the generated samples may be inaccurate, e.g., assigned wrong answers, and 3) ignore the power of positive samples.In this paper, we propose a method to Dig out Discrimination information from Generated samples (DDG) to address the above limitations.Specifically, we first construct positive and negative samples in vision and language modalities, without using additional annotations.Then, we introduce a knowledge distillation mechanism to promote the learning of the original samples by the positive samples.Moreover, we impel the VQA models to focus on vision and language modalities using the negative samples.Experimental results on the VQA-CP v2 and VQA v2 datasets show the effectiveness of our DDG.
What are the Desired Characteristics of Calibration Sets? Identifying Correlates on Long Form Scientific Summarization

Summarization models often generate text that is poorly calibrated to quality metrics because they are trained to maximize the likelihood of a single reference (MLE).To address this, recent work has added a calibration step, which exposes a model to its own ranked outputs to improve relevance or, in a separate line of work, contrasts positive and negative sets to improve faithfulness.While effective, much of this work has focused on how to generate and optimize these sets.Less is known about why one setup is more effective than another.In this work, we uncover the underlying characteristics of effective sets.For each training instance, we form a large, diverse pool of candidates and systematically vary the subsets used for calibration fine-tuning.Each selection strategy targets distinct aspects of the sets, such as lexical diversity or the size of the gap between positive and negatives.On three diverse scientific long-form summarization datasets (spanning biomedical, clinical, and chemical domains), we find, among others, that faithfulness calibration is optimal when the negative sets are extractive and more likely to be generated, whereas for relevance calibration, the metric margin between candidates should be maximized and surprise-the disagreement between model and metric defined candidate rankings-minimized.Code to create, select, and optimize calibration sets is available at https://github.com/griff4692/calibrating-summaries.
On the Efficacy of Sampling Adapters

Sampling is a common strategy for generating text from probabilistic models, yet standard ancestral sampling often results in text that is incoherent or ungrammatical.To alleviate this issue, various modifications to a model's sampling distribution, such as nucleus or top-k sampling, have been introduced and are now ubiquitously used in language generation systems.We propose a unified framework for understanding these techniques, which we term sampling adapters.Sampling adapters often lead to qualitatively better text, which raises the question: From a formal perspective, how are they changing the (sub)word-level distributions of language generation models?And why do these local changes lead to higher-quality text?We argue that the shift they enforce can be viewed as a trade-off between precision and recall: while the model loses its ability to produce certain strings, its precision rate on desirable text increases.While this trade-off is not reflected in standard metrics of distribution quality (such as perplexity), we find that several precision-emphasizing measures indeed indicate that sampling adapters can lead to probability distributions more aligned with the true distribution.Further, these measures correlate with higher sequence-level quality scores, specifically, MAUVE.https://github.com/rycolab/sampling-adapters
Coco at SemEval-2023 Task 10: Explainable Detection of Online Sexism

Sexism is a growing concern on social media platforms because it affects the overall health of the internet and can have negative impacts on society.This paper describes the coco system that participated in SemEval-2023 Task 10, Explainable Detection of Online Sexism (EDOS), which aims at sexism detection in various settings of natural language understanding.We develop a novel neural framework for sexism detection and misogyny that can combine text representations obtained using pre-trained language model models such as Bidirectional Encoder Representations from Transformers and using BiLSTM architecture to obtain the local and global semantic information.Further, considering that the EDOS dataset is relatively small and extremely unbalanced, we conducted data augmentation and introduced two datasets in the field of sexism detection.Moreover, we introduced Focal Loss which is a loss function in order to improve the performance of processing imbalanced data classification.Our system achieved an F1 score of 78.95% on Task Abinary sexism.