Reviewerly/exHarmony huggingface dataset:
GENEVA: Benchmarking Generalizability for Event Argument Extraction with Hundreds of Event Types and Argument Roles

Recent works in Event Argument Extraction (EAE) have focused on improving model generalizability to cater to new events and domains.However, standard benchmarking datasets like ACE and ERE cover less than 40 event types and 25 entity-centric argument roles.Limited diversity and coverage hinder these datasets from adequately evaluating the generalizability of EAE models.In this paper, we first contribute by creating a large and diverse EAE ontology.This ontology is created by transforming FrameNet, a comprehensive semantic role labeling (SRL) dataset for EAE, by exploiting the similarity between these two tasks.Then, exhaustive human expert annotations are collected to build the ontology, concluding with 115 events and 220 argument roles, with a significant portion of roles not being entities.We utilize this ontology to further introduce GENEVA, a diverse generalizability benchmarking dataset comprising four test suites, aimed at evaluating models' ability to handle limited data and unseen event type generalization.We benchmark six EAE models from various families.The results show that owing to non-entity argument roles, even the best-performing model can only achieve 39% F1 score, indicating how GENEVA provides new challenges for generalization in EAE.Overall, our large and diverse EAE ontology can aid in creating more comprehensive future resources, while GENEVA is a challenging benchmarking dataset encouraging further research for improving generalizability in EAE.The code and data can be found at https: //github.com/PlusLabNLP/GENEVA.
Difficulties in Handling Mathematical Expressions in Universal Dependencies

In this paper, we give a brief survey of the difficulties in handling the syntax of mathematical expressions in Universal Dependencies, focusing on examples from English language corpora.We first examine the prevalence and current handling of mathematical expressions in UD corpora.We then examine several strategies for how to approach the handling of syntactic dependencies for such expressions: as multi-word expressions, as a domain appropriate for code-switching, or as approximate to other types of natural language.Ultimately, we argue that mathematical expressions should primarily be analyzed as natural language, and we offer recommendations for the treatment of basic mathematical expressions as analogous to English natural language.
Covering Uncommon Ground: Gap-Focused Question Generation for Answer Assessment

Human communication often involves information gaps between the interlocutors.For example, in an educational dialogue, a student often provides an answer that is incomplete, and there is a gap between this answer and the perfect one expected by the teacher.Successful dialogue then hinges on the teacher asking about this gap in an effective manner, thus creating a rich and interactive educational experience.We focus on the problem of generating such gap-focused questions (GFQs) automatically.We define the task, highlight key desired aspects of a good GFQ, and propose a model that satisfies these.Finally, we provide an evaluation by human annotators of our generated questions compared against human generated ones, demonstrating competitive performance.
Tab-CQA: A Tabular Conversational Question Answering Dataset on Financial Reports

Existing conversational question answering (CQA) datasets have been usually constructed from unstructured texts in English.In this paper, we propose Tab-CQA, a tabular CQA dataset created from Chinese financial reports that are extracted from listed companies in a wide range of different sectors in the past 30 years.From these reports, we select 2,463 tables, and manually generate 2,463 conversations with 35,494 QA pairs.Additionally, we select 4,578 tables, from which 4,578 conversations with 73,595 QA pairs are automatically created via a template-based method.With the manually-and automatically-generated conversations, Tab-CQA contains answerable and unanswerable questions.For the answerable questions, we further diversify them to cover a wide range of skills, e.g., table retrieval, fact checking, numerical reasoning, so as to accommodate real-world scenarios.We further propose two different tabular CQA models, a text-based model and an operation-based model, and evaluate them on Tab-CQA.Experiment results show that Tab-CQA is a very challenging dataset, where a huge performance gap exists between human and neural models.In order to promote further research on Chinese tabular CQA, we release the dataset as a benchmark testbed at https://github.com/tjunlp-lab/Tab-CQA.
Disambiguating Numeral Sequences to Decipher Ancient Accounting Corpora

A numeration system encodes abstract numeric quantities as concrete strings of written characters.The numeration systems used by modern scripts tend to be precise and unambiguous, but this was not so for the ancient and partially-deciphered proto-Elamite (PE) script, where written numerals can have up to four distinct readings depending on the system that is used to read them.We consider the task of disambiguating between these readings in order to determine the values of the numeric quantities recorded in this corpus.We algorithmically extract a list of possible readings for each PE numeral notation, and contribute two disambiguation techniques based on structural properties of the original documents and classifiers learned with the bootstrapping algorithm.We also contribute a test set for evaluating disambiguation techniques, as well as a novel approach to cautious rule selection for bootstrapped classifiers.Our analysis confirms existing intuitions about this script and reveals previously-unknown correlations between tablet content and numeral magnitude.This work is crucial to understanding and deciphering PE, as the corpus is heavily accountingfocused and contains many more numeric tokens than tokens of text.
Mapping Brains with Language Models: A Survey

Over the years, many researchers have seemingly made the same observation: Brain and language model activations exhibit some structural similarities, enabling linear partial mappings between features extracted from neural recordings and computational language models.In an attempt to evaluate how much evidence has been accumulated for this observation, we survey over 30 studies spanning 10 datasets and 8 metrics.How much evidence has been accumulated, and what, if anything, is missing before we can draw conclusions?Our analysis of the evaluation methods used in the literature reveals that some of the metrics are less conservative.We also find that the accumulated evidence, for now, remains ambiguous, but correlations with model size and quality provide grounds for cautious optimism.
Is a Knowledge-based Response Engaging?: An Analysis on Knowledge-Grounded Dialogue with Information Source Annotation

Currently, most knowledge-grounded dialogue response generation models focus on reflecting given external knowledge.However, even when conveying external knowledge, humans integrate their own knowledge, experiences, and opinions with external knowledge to make their utterances engaging.In this study, we analyze such human behavior by annotating the utterances in an existing knowledge-grounded dialogue corpus.Each entity in the corpus is annotated with its information source, either derived from external knowledge (database-derived) or the speaker's own knowledge, experiences, and opinions (speaker-derived).Our analysis shows that the presence of speaker-derived information in the utterance improves dialogue engagingness.We also confirm that responses generated by an existing model, which is trained to reflect the given knowledge, cannot include speakerderived information in responses as often as humans do.
shefnlp at SemEval-2023 Task 10: Compute-Efficient Category Adapters

As social media platforms grow, so too does the volume of hate speech and negative sentiment expressed towards particular social groups.In this paper, we describe our approach to SemEval-2023 Task 10, involving the detection and classification of online sexism (abuse directed towards women), with fine-grained categorisations intended to facilitate the development of a more nuanced understanding of the ideologies and processes through which online sexism is expressed.We experiment with several approaches involving language model finetuning, class-specific adapters, and pseudolabelling.Our best-performing models involve the training of adapters specific to each subtask category (combined via fusion layers) using a weighted loss function, in addition to performing naive pseudo-labelling on a large quantity of unlabelled data.We successfully outperform the baseline models on all 3 subtasks, placing 56th (of 84) on Task A, 43rd (of 69) on Task B, and 37th (of 63) on Task C.
What does the Failure to Reason with “Respectively” in Zero/Few-Shot Settings Tell Us about Language Models?

Humans can effortlessly understand the coordinate structure of sentences such as “Niels Bohr and Kurt Cobain were born in Copenhagen and Seattle, *respectively*”. In the context of natural language inference (NLI), we examine how language models (LMs) reason with respective readings (Gawron and Kehler, 2004) from two perspectives: syntactic-semantic and commonsense-world knowledge. We propose a controlled synthetic dataset WikiResNLI and a naturally occurring dataset NatResNLI to encompass various explicit and implicit realizations of “respectively”. We show that fine-tuned NLI models struggle with understanding such readings without explicit supervision. While few-shot learning is easy in the presence of explicit cues, longer training is required when the reading is evoked implicitly, leaving models to rely on common sense inferences. Furthermore, our fine-grained analysis indicates models fail to generalize across different constructions. To conclude, we demonstrate that LMs still lag behind humans in generalizing to the long tail of linguistic constructions.
Characterizing and Measuring Linguistic Dataset Drift

NLP models often degrade in performance when real world data distributions differ markedly from training data.However, existing dataset drift metrics in NLP have generally not considered specific dimensions of linguistic drift that affect model performance, and they have not been validated in their ability to predict model performance at the individual example level, where such metrics are often used in practice.In this paper, we propose three dimensions of linguistic dataset drift: vocabulary, structural, and semantic drift.These dimensions correspond to content word frequency divergences, syntactic divergences, and meaning changes not captured by word frequencies (e.g.lexical semantic change).We propose interpretable metrics for all three drift dimensions, and we modify past performance prediction methods to predict model performance at both the example and dataset level for English sentiment classification and natural language inference.We find that our drift metrics are more effective than previous metrics at predicting out-of-domain model accuracies (mean 16.8% root mean square error decrease), particularly when compared to popular fine-tuned embedding distances (mean 47.7% error decrease).Fine-tuned embedding distances are much more effective at ranking individual examples by expected performance, but decomposing into vocabulary, structural, and semantic drift produces the best example rankings of all considered model-agnostic drift metrics (mean 6.7% ROC AUC increase).
Mr-Fosdick at SemEval-2023 Task 5: Comparing Dataset Expansion Techniques for Non-Transformer and Transformer Models: Improving Model Performance through Data Augmentation

In supervised learning, a significant amount of data is essential.To achieve this, we generated and evaluated datasets based on a provided dataset using transformer and non-transformer models.By utilizing these generated datasets during the training of new models, we attain a higher balanced accuracy during validation compared to using only the original dataset.
A Dialogue System for Assessing Activities of Daily Living: Improving Consistency with Grounded Knowledge

In healthcare, the ability to care for oneself is reflected in the "Activities of Daily Living (ADL)," which serve as a measure of functional ability (functioning).A lack of functioning may lead to poor living conditions requiring personal care and assistance.To accurately identify those in need of support, assistance programs continuously evaluate participants' functioning across various domains.However, the assessment process may encounter consistency issues when multiple assessors with varying levels of expertise are involved.Novice assessors, in particular, may lack the necessary preparation for real-world interactions with participants.To address this issue, we developed a dialogue system that simulates interactions between assessors and individuals of varying functioning in a natural and reproducible way.The dialogue system consists of two major modules, one for natural language understanding (NLU) and one for natural language generation (NLG), respectively.In order to generate responses consistent with the underlying knowledge base, the dialogue system requires both an understanding of the user's query and of biographical details of an individual being simulated.To fulfill this requirement, we experimented with query classification and generated responses based on those biographical details using some recently released InstructGPT-like models.
Retrieving Multimodal Prompts for Generative Visual Question Answering

Recent years have witnessed impressive results of pre-trained vision-language models on knowledge-intensive tasks such as visual question answering (VQA).Despite the recent advances in VQA, existing methods mainly adopt a discriminative formulation that predicts answers within a pre-defined label set, leading to easy overfitting on low-resource domains with limited labeled data (e.g., medicine) and poor generalization under domain shift to another dataset.To tackle this limitation, we propose a novel generative model enhanced by multimodal prompt retrieval (MPR) that integrates retrieved prompts and multimodal features to generate answers in free text.Our generative model enables rapid zero-shot dataset adaptation to unseen data distributions and open-set answer labels across datasets.Our experiments on medical VQA tasks show that MPR outperforms its non-retrieval counterpart by up to 30% accuracy points in a few-shot domain adaptation setting. 1
HiFi: High-Information Attention Heads Hold for Parameter-Efficient Model Adaptation

To fully leverage the advantages of large-scale pre-trained language models (PLMs) on downstream tasks, it has become a ubiquitous adaptation paradigm to fine-tune the entire parameters of PLMs.However, this paradigm poses issues of inefficient updating and resource overconsuming for fine-tuning in data-scarce and resource-limited scenarios, because of the large scale of parameters in PLMs.To alleviate these concerns, in this paper, we propose a parameterefficient fine-tuning method HiFi, that is, only the highly informative and strongly correlated attention heads for the specific task are finetuned.To search for those significant attention heads, we develop a novel framework to analyze the effectiveness of heads.Specifically, we first model the relationship between heads into a graph from two perspectives of information richness and correlation, and then apply PageRank algorithm to determine the relative importance of each head.Extensive experiments on the GLUE benchmark demonstrate the effectiveness of our method, and show that HiFi obtains state-of-the-art performance over the prior baselines.
SemEval-2023 Task 8: Causal Medical Claim Identification and Related PIO Frame Extraction from Social Media Posts

Identification of medical claims from usergenerated text data is an onerous but essential step for various tasks including content moderation, and hypothesis generation.SemEval-2023 Task 8 is an effort towards building those capabilities and motivating further research in this direction.This paper summarizes the details and results of shared task 8 at SemEval-2023 which involved identifying causal medical claims and extracting related Populations, Interventions, and Outcomes ("PIO") frames from social media (Reddit) text. 1 This shared task comprised two subtasks: (1) Causal claim identification; and ( 2) PIO frame extraction.In total, seven teams participated in the task.Of the seven, six provided system descriptions which we summarize here.For the first subtask, the best approach yielded a macro-averaged F-1 score of 78.40, and for the second subtask, the best approach achieved token-level F-1 scores of 40.55 for Populations, 49.71 for Interventions, and 30.08 for Outcome frames.
Improving Contrastive Learning of Sentence Embeddings from AI Feedback

Contrastive learning has become a popular approach in natural language processing, particularly for the learning of sentence embeddings.However, the discrete nature of natural language makes it difficult to ensure the quality of positive and negative sample pairs generated through data augmentation methods.Although supervised contrastive learning can produce more accurate sample pairs with human feedback labels, it still lacks fine-grained training signals.In this paper, we propose to improve Contrastive Learning of sentence embeddings from AI Feedback (CLAIF).Our method utilizes AI feedback from large pre-trained language models (LLMs) to construct sample pairs with fine-grained sample similarity scores to improve contrastive learning.Besides, we combine human feedback and AI feedback to provide better supervision signals for supervised contrastive learning of sentence embeddings.Experimental results show that our method achieves state-of-the-art performance on several semantic textual similarity (STS) and transfer learning tasks compared to other unsupervised and supervised contrastive learning methods. 1
Preserving the Authenticity of Handwritten Learner Language: Annotation Guidelines for Creating Transcripts Retaining Orthographic Features

Handwritten texts produced by young learners often contain orthographic features like spelling errors, capitalization errors, punctuation errors, and impurities such as strikethroughs, inserts, and smudges.All of those are typically normalized or ignored in existing transcriptions.For applications like handwriting recognition with the goal of automatically analyzing a learner's language performance, however, retaining such features would be necessary.To address this, we present transcription guidelines that retain the features addressed above.Our guidelines were developed iteratively and include numerous example images to illustrate the various issues.On a subset of about 90 double-transcribed texts, we compute inter-annotator agreement and show that our guidelines can be applied with high levels of percentage agreement of about .98.Overall, we transcribed 1,350 learner texts, which is about the same size as the widely adopted handwriting recognition datasets IAM (1,500 pages) and CVL (1,600 pages).Our final corpus can be used to train a handwriting recognition system that transcribes closely to the real productions by young learners.Such a system is a prerequisite for applying automatic orthography feedback systems to handwritten texts in the future.
Going Beyond Sentence Embeddings: A Token-Level Matching Algorithm for Calculating Semantic Textual Similarity

Semantic Textual Similarity (STS) measures the degree to which the underlying semantics of paired sentences are equivalent.State-of-theart methods for STS task use language models to encode sentences into embeddings.However, these embeddings are limited in representing semantics because they mix all the semantic information together in fixed-length vectors, which are difficult to recover and lack explainability.This paper presents a token-level matching inference algorithm, which can be applied on top of any language model to improve its performance on STS task.Our method calculates pairwise token-level similarity and token matching scores, and then aggregates them with pretrained token weights to produce sentence similarity.Experimental results on seven STS datasets show that our method improves the performance of almost all language models, with up to 12.7% gain in Spearman's correlation.We also demonstrate that our method is highly explainable and computationally efficient.
Arguably at SemEval-2023 Task 11: Learning the disagreements using unsupervised behavioral clustering and language models

We describe SemEval-2023 Task 11, on behavioral segregation of annotations to find the similarities and contextual thinking of a group of annotators.We have utilized a behavioral segmentation analysis on the annotators to model them independently and combine the results to yield soft and hard scores.Our team focused on experimenting with hierarchical clustering with various distance metrics for similarity, dissimilarity, and reliability.We modeled the clusters and assigned weightage to find the soft and hard scores.Our team was able to find out hidden behavioral patterns among the judgments of annotators after rigorous experiments.The proposed system is made available 1
MDACE: MIMIC Documents Annotated with Code Evidence

We introduce a dataset for evidence/rationale extraction on an extreme multi-label classification task over long medical documents.One such task is Computer-Assisted Coding (CAC) which has improved significantly in recent years thanks to advances in machine learning technologies.However, simply predicting a set of final codes for a patient encounter is insufficient, as CAC systems are required to provide supporting textual evidence to justify the billing codes.A model able to produce accurate and reliable supporting evidence for each code would be a tremendous benefit.However, a human-annotated code evidence corpus is extremely difficult to create because it requires specialized knowledge.In this paper, we introduce MDACE, the first publicly available code evidence dataset, which is built on a subset of the MIMIC-III (English) clinical records.The dataset -annotated by professional medical coders -consists of 302 Inpatient charts with 3,934 evidence spans and 52 Profee charts with 5,563 evidence spans.We implemented several evidence extraction methods based on the EffectiveCAN model (Liu et al., 2021) to establish baseline performance on this dataset.MDACE can be used to evaluate code evidence extraction methods for CAC systems, as well as the accuracy and interpretability of deep learning models for multi-label classification.We believe that the release of MDACE will greatly improve the understanding and application of deep learning technologies for medical coding and document classification.
Improving Translation Quality Estimation with Bias Mitigation

State-of-the-art translation Quality Estimation (QE) models are proven to be biased.More specifically, they over-rely on monolingual features while ignoring the bilingual semantic alignment.In this work, we propose a novel method to mitigate the bias of the QE model and improve estimation performance.Our method is based on the contrastive learning between clean and noisy sentence pairs.We first introduce noise to the target side of the parallel sentence pair, forming the negative samples.With the original parallel pairs as the positive sample, the QE model is contrastively trained to distinguish the positive samples from the negative ones.This objective is jointly trained with the regression-style quality estimation, so as to prevent the QE model from overfitting to monolingual features.Experiments on WMT QE evaluation datasets demonstrate that our method improves the estimation performance by a large margin while mitigating the bias 1 .