Reviewerly/exHarmony huggingface dataset:
MaChAmp at SemEval-2023 tasks 2, 3, 4, 5, 7, 8, 9, 10, 11, and 12: On the Effectiveness of Intermediate Training on an Uncurated Collection of Datasets.

To improve the ability of language models to handle Natural Language Processing (NLP) tasks and intermediate step of pre-training has recently been introduced.In this setup, one takes a pre-trained language model, trains it on a (set of) NLP dataset(s), and then finetunes it for a target task.It is known that the selection of relevant transfer tasks is important, but recently some work has shown substantial performance gains by doing intermediate training on a very large set of datasets.Most previous work uses generative language models or only focuses on one or a couple of tasks and uses a carefully curated setup.We compare intermediate training with one or many tasks in a setup where the choice of datasets is more arbitrary; we use all SemEval 2023 text-based tasks.We reach performance improvements for most tasks when using intermediate training.Gains are higher when doing intermediate training on single tasks than all tasks if the right transfer task is identified.Dataset smoothing and heterogeneous batching did not lead to robust gains in our setup. 1
Entity Contrastive Learning in a Large-Scale Virtual Assistant System

Conversational agents are typically made up of domain (DC) and intent classifiers (IC) that identify the general subject an utterance belongs to and the specific action a user wishes to achieve.In addition, named entity recognition (NER) performs per token labeling to identify specific entities of interest in a spoken utterance.We investigate improving joint IC and NER models using entity contrastive learning that attempts to cluster similar entities together in a learned representation space.We compare a full virtual assistant system trained using entity contrastive learning to a baseline system that does not use contrastive learning.We present both offline results, using retrospective test sets, as well as online results from an A/B test that compared the two systems.In both the offline and online settings, entity contrastive training improved overall performance against baseline systems.Furthermore, we provide a detailed analysis of learned entity embeddings, including both qualitative analysis via dimensionalityreduced visualizations and quantitative analysis by computing alignment and uniformity metrics.We show that entity contrastive learning improves alignment metrics and produces wellformed embedding clusters in representation space.
Incorporating Graph Information in Transformer-based AMR Parsing

Meaning Representation (AMR) is a Semantic Parsing formalism that aims at providing a semantic graph abstraction representing a given text.Current approaches are based on autoregressive language models such as BART or T5, fine-tuned through Teacher Forcing to obtain a linearized version of the AMR graph from a sentence.In this paper, we present LeakDistill, a model and method that explores a modification to the Transformer architecture, using structural adapters to explicitly incorporate graph information into the learned representations and improve AMR parsing performance.Our experiments show how, by employing word-to-node alignment to embed graph structural information into the encoder at training time, we can obtain state-of-the-art AMR parsing through self-knowledge distillation, even without the use of additional data.We release the code at http://www.github.
Evaluating and Improving Automatic Speech Recognition using Severity

A common metric for evaluating Automatic Speech Recognition (ASR) is Word Error Rate (WER) which solely takes into account discrepancies at the word-level.Although useful, WER is not guaranteed to correlate well with human judgment or performance on downstream tasks that use ASR.Meaningful assessment of ASR mistakes becomes even more important in high-stake scenarios such as healthcare.We propose 2 general measures to evaluate the severity of mistakes made by ASR systems, one based on sentiment analysis and another based on text embeddings.We evaluate these measures on simulated patient-doctor conversations using 5 ASR systems.Results show that these measures capture characteristics of ASR errors that WER does not.Furthermore, we train an ASR system incorporating severity and demonstrate the potential for using severity not only in the evaluation, but in the development of ASR.Advantages and limitations of this methodology are analyzed and discussed.
AVEN-GR: Attribute Value Extraction and Normalization using product GRaphs

Getting a good understanding of the user intent is vital for e-commerce applications to surface the right product to a given customer query.Query Understanding (QU) systems are essential for this purpose, and many e-commerce providers are working on complex solutions that need to be data efficient and able to capture early emerging market trends.Query Attribute Understanding (QAU) is a sub-component of QU that involves extracting named attributes from user queries and linking them to existing e-commerce entities such as brand, material, color, etc.While extracting named entities from text has been extensively explored in the literature, QAU requires specific attention due to the nature of the queries, which are often short, noisy, ambiguous, and constantly evolving.This paper makes three contributions to QAU.First, we propose a novel end-to-end approach that jointly solves Named Entity Recognition (NER) and Entity Linking (NEL) and enables open-world reasoning for QAU.Second, we introduce a novel method for utilizing product graphs to enhance the representation of query entities.Finally, we present a new dataset constructed from public sources that can be used to evaluate the performance of future QAU systems.
IREL at SemEval-2023 Task 11: User Conditioned Modelling for Toxicity Detection in Subjective Tasks

This paper describes our system used in the SemEval-2023 Task 11 Learning With Disagreements (Le-Wi-Di).This is a subjective task since it deals with detecting hate speech, misogyny and offensive language.Thus, disagreement among annotators is expected.We experiment with different settings like loss functions specific for subjective tasks and include anonymized annotator-specific information to help us understand the level of disagreement.We perform an in-depth analysis of the performance discrepancy of these different modelling choices.Our system achieves a cross-entropy of 0.58, 4.01 and 3.70 on the test sets of HS-Brexit, ArMIS and MD-Agreement, respectively.Our code implementation is publicly available.
The Whole Truth and Nothing But the Truth: Faithful and Controllable Dialogue Response Generation with Dataflow Transduction and Constrained Decoding

In a real-world dialogue system, generated text must be truthful and informative while remaining fluent and adhering to a prescribed style.Satisfying these constraints simultaneously is difficult for the two predominant paradigms in language generation: neural language modeling and rule-based generation.We describe a hybrid architecture for dialogue response generation that combines the strengths of both paradigms.The first component of this architecture is a rule-based content selection model defined using a new formal framework called dataflow transduction, which uses declarative rules to transduce a dialogue agent's actions and their results (represented as dataflow graphs) into context-free grammars representing the space of contextually acceptable responses.The second component is a constrained decoding procedure that uses these grammars to constrain the output of a neural language model, which selects fluent utterances.Our experiments show that this system outperforms both rule-based and learned approaches in human evaluations of fluency, relevance, and truthfulness.* Equal contribution.
UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Units

Direct speech-to-speech translation (S2ST), in which all components can be optimized jointly, is advantageous over cascaded approaches to achieve fast inference with a simplified pipeline.We present a novel two-pass direct S2ST architecture, UnitY, which first generates textual representations and predicts discrete acoustic units subsequently.We enhance the model performance by subword prediction in the first-pass decoder, advanced two-pass decoder architecture design and search strategy, and better training regularization.To leverage large amounts of unlabeled text data, we pre-train the firstpass text decoder based on the self-supervised denoising auto-encoding task.Experimental evaluations on benchmark datasets at various data scales demonstrate that UnitY outperforms a single-pass speech-to-unit translation model by 2.5-4.2ASR-BLEU with 2.83× decoding speed-up.We show that the proposed methods boost the performance even when predicting spectrogram in the second pass.However, predicting discrete units achieves 2.51× decoding speed-up compared to that case.
Math Word Problem Solving by Generating Linguistic Variants of Problem Statements

The art of mathematical reasoning stands as a fundamental pillar of intellectual progress and is a central catalyst in cultivating human ingenuity.Researchers have recently published a plethora of works centered around the task of solving Math Word Problems (MWP) -a crucial stride towards general AI.These existing models are susceptible to dependency on shallow heuristics and spurious correlations to derive the solution expressions.In order to ameliorate this issue, in this paper, we propose a framework for MWP solvers based on the generation of linguistic variants of the problem text.The approach involves solving each of the variant problems and electing the predicted expression with the majority of the votes.We use DeBERTa (Decoding-enhanced BERT with disentangled attention) as the encoder to leverage its rich textual representations and enhanced mask decoder to construct the solution expressions.Furthermore, we introduce a challenging dataset, PARAMAWPS, consisting of paraphrased, adversarial, and inverse variants of selectively sampled MWPs from the benchmark MAWPS dataset.We extensively experiment on this dataset along with other benchmark datasets using some baseline MWP solver models.We show that training on linguistic variants of problem statements and voting on candidate predictions improve the mathematical reasoning and robustness of the model.We make our code and data publicly available.
A Customized Text Sanitization Mechanism with Differential Privacy

As privacy issues are receiving increasing attention within the Natural Language Processing (NLP) community, numerous methods have been proposed to sanitize texts subject to differential privacy.However, the state-of-theart text sanitization mechanisms based on metric local differential privacy (MLDP) do not apply to non-metric semantic similarity measures and cannot achieve good trade-offs between privacy and utility.To address the above limitations, we propose a novel Customized Text (CusText) sanitization mechanism based on the original ϵ-differential privacy (DP) definition, which is compatible with any similarity measure.Furthermore, CusText assigns each input token a customized output set of tokens to provide more advanced privacy protection at the token level.Extensive experiments on several benchmark datasets show that CusText achieves a better trade-off between privacy and utility than existing mechanisms.
Learning Answer Generation using Supervision from Automatic Question Answering Evaluators

Recent studies show that sentence-level extractive QA, i.e., based on Answer Sentence Selection (AS2), is outperformed by Generationbased QA (GenQA) models, which generate answers using the top-k answer sentences ranked by AS2 models (a la retrieval-augmented generation style).In this paper, we propose a novel training paradigm for GenQA using supervision from automatic QA evaluation models (GAVA).Specifically, we propose three strategies to transfer knowledge from these QA evaluation models to a GenQA model: (i) augmenting training data with answers generated by the GenQA model and labelled by GAVA (either statically, before training, or (ii) dynamically, at every training epoch); and (iii) using the GAVA score for weighting the generator loss during the learning of the GenQA model.We evaluate our proposed methods on two academic and one industrial dataset, obtaining a significant improvement in answering accuracy over the previous state of the art.
HITSZQ at SemEval-2023 Task 10: Category-aware Sexism Detection Model with Self-training Strategy

This paper describes our system used in the SemEval-2023 Task 10 Explainable Detection of Online Sexism (EDOS).Specifically, we participated in subtask B: a 4-class sexism classification task, and subtask C: a more fine-grained (11-class) sexism classification task, where it is necessary to predict the category of sexism.We treat these two subtasks as one multi-label hierarchical text classification problem and propose an integrated sexism detection model for improving the performance of the sexism detection task.More concretely, we use the pretrained BERT model to encode the text and class label and a hierarchy-relevant structure encoder is employed to model the relationship between classes of subtasks B and C. Additionally, a self-training strategy is designed to alleviate the imbalanced problem of distribution classes.Extensive experiments on subtasks B and C demonstrate the effectiveness of our proposed approach.
University of Hildesheim at SemEval-2023 Task 1: Combining Pre-trained Multimodal and Generative Models for Image Disambiguation

Multimodal ambiguity is a challenge for understanding text and images.Large pre-trained models have reached a high level of quality already.This paper presents an implementation for solving an image disambiguation task relying solely on the knowledge captured in multimodal and language models.Within task 1 of SemEval 2023 (Visual Word Sense Disambiguation), this approach managed to achieve an MRR of 0.738 using CLIP-Large and the OPT model for generating text.Applying a generative model to create more text given a phrase with an ambiguous word leads to an improvement in our results.The performance gain from a bigger language model is larger than the performance gain from using the larger CLIP model.
Entity Tracking in Language Models

Keeping track of how states of entities change as a text or dialog unfolds is a key prerequisite to discourse understanding.Yet, there have been few systematic investigations into the ability of large language models (LLMs) to track discourse entities.In this work, we present a task probing to what extent a language model can infer the final state of an entity given an English description of the initial state and a series of state-changing operations.We use this task to first investigate whether Flan-T5, GPT-3 and GPT-3.5 can track the state of entities, and find that only GPT-3.5 models, which have been pretrained on large amounts of code, exhibit this ability.We then investigate whether smaller models pretrained primarily on text can learn to track entities, through finetuning T5 on several training/evaluation splits.While performance degrades for more complex splits, we find that even when evaluated on a different set of entities from training or longer operation sequences, a finetuned model can perform nontrivial entity tracking.Taken together, these results suggest that language models can learn to track entities but pretraining on text corpora alone does not make this capacity surface.* Equal contribution.Author order was determined by a fair process of letting Cookie the cat pick a colored ball.
MeetingQA: Extractive Question-Answering on Meeting Transcripts

With the ubiquitous use of online meeting platforms and robust automatic speech recognition systems, meeting transcripts have emerged as a promising domain for natural language tasks.Most recent works on meeting transcripts primarily focus on summarization and extraction of action items.However, meeting discussions also have a useful question-answering (QA) component, crucial to understanding the discourse or meeting content, and can be used to build interactive interfaces on top of long transcripts.Hence, in this work, we leverage this inherent QA component of meeting discussions and introduce MEETINGQA, an extractive QA dataset comprising of questions asked by meeting participants and corresponding responses.As a result, questions can be open-ended and actively seek discussions, while the answers can be multi-span and distributed across multiple speakers.Our comprehensive empirical study of several robust baselines including long-context language models and recent instruction-tuned models reveals that models perform poorly on this task (F1 = 57.3)and severely lag behind human performance (F1 = 84.6),thus presenting a challenging new task for the community to improve upon. 1
Contextualized Soft Prompts for Extraction of Event Arguments

Event argument extraction (EAE) is a subtask of event extraction where the goal is to identify roles of entity mentions for events in text.The current state-of-the-art approaches for this problem explore prompt-based methods to prompt pre-trained language models for arguments over input context.However, existing prompt-based methods mainly rely on discrete and manually-designed prompts that cannot exploit specific context for each example to improve customization for optimal performance.In addition, the discrete nature of current prompts prevents the incorporation of relevant context from multiple external documents to enrich prompts for EAE.To this end, we propose a novel prompt-based method for EAE that introduces soft prompts to facilitate the encoding of individual example context and multiple relevant documents to boost EAE.We extensively evaluate the proposed method on benchmark datasets for EAE to demonstrate its benefits with state-of-the-art performance.
A Class-Rebalancing Self-Training Framework for Distantly-Supervised Named Entity Recognition

Distant supervision reduces the reliance on human annotation in the named entity recognition tasks.The class-level imbalanced distant annotation is a realistic and unexplored problem, and the popular method of self-training can not handle class-level imbalanced learning.More importantly, self-training is dominated by the high-performance class in selecting candidates, and deteriorates the low-performance class with the bias of generated pseudo label.To address the class-level imbalance performance, we propose a class-rebalancing selftraining framework for improving the distantlysupervised named entity recognition.In candidate selection, a class-wise flexible threshold is designed to fully explore other classes besides the high-performance class.In label generation, injecting the distant label, a hybrid pseudo label is adopted to provide straight semantic information for the low-performance class.Experiments on five flat and two nested datasets show that our model achieves state-of-the-art results.We also conduct extensive research to analyze the effectiveness of the flexible threshold and the hybrid pseudo label.
Fine-tuning Sentence-RoBERTa to Construct Word Embeddings for Low-resource Languages from Bilingual Dictionaries

Conventional approaches to learning word embeddings (Mikolov et al., 2013;Pennington et al., 2014) are limited to relatively few languages with sufficiently large training corpora.To address this limitation, we propose an alternative approach to deriving word embeddings for Wolastoqey and Mi'kmaq that leverages definitions from a bilingual dictionary.More specifically, following Bear and Cook (2022), we experiment with encoding English definitions of Wolastoqey and Mi'kmaq words into vector representations using English sequence representation models.For this, we consider using and fine-tuning sentence-RoBERTa models (Reimers and Gurevych, 2019).We evaluate our word embeddings using a similar methodology to that of Bear and Cook using evaluations based on word classification, clustering and reverse dictionary search.We additionally construct word embeddings for higher-resource languages -English, German and Spanish -using our methods and evaluate our embeddings on existing word-similarity datasets.Our findings indicate that our word embedding methods can be used to produce meaningful vector representations for low-resource languages such as Wolastoqey and Mi'kmaq and for higher-resource languages.
Common Law Annotations: Investigating the Stability of Dialog System Output Annotations

Metrics for Inter-Annotator Agreement (IAA), like Cohen's Kappa, are crucial for validating annotated datasets.Although high agreement is often used to show the reliability of annotation procedures, it is insufficient to ensure validity or reproducibility.While researchers are encouraged to increase annotator agreement, this can lead to specific and tailored annotation guidelines.We hypothesize that this may result in diverging annotations from different groups.To study this, we first propose the Lee et al.Protocol (LEAP), a standardized and codified annotation protocol.LEAP strictly enforces transparency in the annotation process, which ensures reproducibility of annotation guidelines.Using LEAP to annotate a dialog dataset, we empirically show that while research groups may create reliable guidelines by raising agreement, this can cause divergent annotations across different research groups, thus questioning the validity of the annotations.Therefore, we caution NLP researchers against using reliability as a proxy for reproducibility and validity.
Unpacking Ambiguous Structure: A Dataset for Ambiguous Implicit Discourse Relations for English and Egyptian Arabic

In this paper, we present principles of constructing and resolving ambiguity in implicit discourse relations.Following these principles, we created a dataset in both English and Egyptian Arabic that controls for semantic disambiguation, enabling the investigation of prosodic features in future work.In these datasets, examples are two-part sentences with an implicit discourse relation that can be ambiguously read as either causal or concessive, paired with two different preceding context sentences forcing either the causal or the concessive reading.We also validated both datasets by humans and language models (LMs) to study whether context can help humans or LMs resolve ambiguities of implicit relations and identify the intended relation.As a result, this task posed no difficulty for humans, but proved challenging for BERT/CamelBERT and ELEC-TRA/AraELECTRA models.
Let Me Check the Examples: Enhancing Demonstration Learning via Explicit Imitation

Demonstration learning aims to guide the prompt prediction by providing answered demonstrations in the few shot settings.Despite achieving promising results, existing work only concatenates the answered examples as demonstrations to the prompt template (including the raw context) without any additional operation, neglecting the prompt-demonstration dependencies.Besides, prior research found that randomly replacing the labels of demonstrations marginally hurts performance, illustrating that the model could not properly learn the knowledge brought by the demonstrations.Inspired by the human learning process, in this paper, we introduce Imitation DEMOnstration learning (Imitation-Demo) to strengthen demonstration learning via explicitly imitating human review behaviour, which includes: (1) contrastive learning mechanism to concentrate on similar demonstrations.(2)demonstration-label re-prediction method to consolidate known knowledge.Experiment results show that our proposed method achieves state-of-the-art performance on 5 out of 14 classification corpus.Further studies also prove that Imitation-Demo strengthens the associations between the prompt and demonstrations, which could provide the basis for exploring how demonstration learning works.