Reviewerly/exHarmony huggingface dataset:
WikiBio: a Semantic Resource for the Intersectional Analysis of Biographical Events

Biographical event detection is a relevant task for the exploration and comparison of the ways in which people's lives are told and represented.In this sense, it may support several applications in digital humanities and in works aimed at exploring bias about minoritized groups.Despite that, there are no corpora and models specifically designed for this task.In this paper we fill this gap by presenting a new corpus annotated for biographical event detection.The corpus, which includes 20 Wikipedia biographies, was compared with five existing corpora to train a model for the biographical event detection task.The model was able to detect all mentions of the target-entity in a biography with an F-score of 0.808 and the entity-related events with an F-score of 0.859.Finally, the model was used for performing an analysis of biases about women and non-Western people in Wikipedia biographies.
NormBank: A Knowledge Bank of Situational Social Norms

We present NORMBANK, a knowledge bank of 155k situational norms.This resource is designed to ground flexible normative reasoning for interactive, assistive, and collaborative AI systems.Unlike prior commonsense resources, NORMBANK grounds each inference within a multivalent sociocultural frame, which includes the setting (e.g., restaurant), the agents' contingent roles (waiter, customer), their attributes (age, gender), and other physical, social, and cultural constraints (e.g., the temperature or the country of operation).In total, NORMBANK contains 63k unique constraints from a taxonomy that we introduce and iteratively refine here.Constraints then apply in different combinations to frame social norms.Under these manipulations, norms are non-monotonic -one can cancel an inference by updating its frame even slightly.Still, we find evidence that neural models can help reliably extend the scope and coverage of NORMBANK.We further demonstrate the utility of this resource with a series of transfer experiments.For data and code, see https: //github.com/SALT-NLP/normbank
TeAST: Temporal Knowledge Graph Embedding via Archimedean Spiral Timeline

Temporal knowledge graph embedding (TKGE) models are commonly utilized to infer the missing facts and facilitate reasoning and decision-making in temporal knowledge graph based systems.However, existing methods fuse temporal information into entities, potentially leading to the evolution of entity information and limiting the link prediction performance of TKG.Meanwhile, current TKGE models often lack the ability to simultaneously model important relation patterns and provide interpretability, which hinders their effectiveness and potential applications.To address these limitations, we propose a novel TKGE model which encodes Temporal knowledge graph embeddings via Archimedean Spiral Timeline (TeAST), which maps relations onto the corresponding Archimedean spiral timeline and transforms the quadruples completion to 3th-order tensor completion problem.Specifically, the Archimedean spiral timeline ensures that relations that occur simultaneously are placed on the same timeline, and all relations evolve over time.Meanwhile, we present a novel temporal spiral regularizer to make the spiral timeline orderly.In addition, we provide mathematical proofs to demonstrate the ability of TeAST to encode various relation patterns.Experimental results show that our proposed model significantly outperforms existing TKGE methods.Our code is available at https://github.com/IMU-MachineLearningSXD/TeAST.
Multilingual Clinical NER: Translation or Cross-lingual Transfer?

Natural language tasks like Named Entity Recognition (NER) in the clinical domain on non-English texts can be very time-consuming and expensive due to the lack of annotated data.Cross-lingual transfer (CLT) is a way to circumvent this issue thanks to the ability of multilingual large language models to be fine-tuned on a specific task in one language and to provide high accuracy for the same task in another language.However, other methods leveraging translation models can be used to perform NER without annotated data in the target language, by either translating the training set or test set.This paper compares cross-lingual transfer with these two alternative methods, to perform clinical NER in French and in German without any training data in those languages.To this end, we release MedNERF a medical NER test set extracted from French drug prescriptions and annotated with the same guidelines as an English dataset.Through extensive experiments on this dataset and on a German medical dataset (Frei and Kramer, 2021), we show that translation-based methods can achieve similar performance to CLT but require more care in their design.And while they can take advantage of monolingual clinical language models, those do not guarantee better results than large general-purpose multilingual models, whether with cross-lingual transfer or translation.
A Study on Annotation Interfaces for Summary Comparison

The task of summarisation is notoriously difficult to evaluate, with agreement even between expert raters unlikely to be perfect.One technique for summary evaluation relies on collecting comparison data by presenting annotators with generated summaries and tasking them with selecting the best one.This paradigm is currently being exploited in reinforcement learning using human feedback, whereby a reward function is trained using pairwise choice data.Comparisons are an easier way to elicit human feedback for summarisation, however, such decisions can be bottle necked by the usability of the annotator interface.In this paper, we present the results of a pilot study exploring how the user interface impacts annotator agreement when judging summary quality.
MANNER: A Variational Memory-Augmented Model for Cross Domain Few-Shot Named Entity Recognition

This paper focuses on the task of cross domain few-shot named entity recognition (NER), which aims to adapt the knowledge learned from source domain to recognize named entities in target domain with only a few labeled examples.To address this challenging task, we propose MANNER, a variational memoryaugmented few-shot NER model.Specifically, MANNER uses a memory module to store information from the source domain and then retrieve relevant information from the memory to augment few-shot tasks in the target domain.In order to effectively utilize the information from memory, MANNER uses optimal transport to retrieve and process information from memory, which can explicitly adapt the retrieved information from source domain to target domain and improve the performance in the cross domain few-shot setting.We conduct experiments on both English and Chinese cross domain fewshot NER datasets, and the experimental results demonstrate that MANNER can achieve superior performance 1 .
FORK: A Bite-Sized Test Set for Probing Culinary Cultural Biases in Commonsense Reasoning Models

It is common sense that one should prefer to eat a salad with a fork rather than with a chainsaw.However, for eating a bowl of rice, the choice between a fork and a pair of chopsticks is culturally relative.We introduce FORK, a small, manually-curated set of CommonsenseQA-style questions for probing cultural biases and assumptions present in commonsense reasoning systems, with a specific focus on food-related customs.We test several CommonsenseQA systems on FORK, and while we see high performance on questions about the US culture, the poor performance of these systems on questions about non-US cultures highlights systematic cultural biases aligned with US over non-US cultures.
Rethinking Masked Language Modeling for Chinese Spelling Correction

In this paper, we study Chinese Spelling Correction (CSC) as a joint decision made by two separate models: a language model and an error model.Through empirical analysis, we find that fine-tuning BERT tends to over-fit the error model while under-fit the language model, resulting in poor generalization to outof-distribution error patterns.Given that BERT is the backbone of most CSC models, this phenomenon has a significant negative impact.To address this issue, we are releasing a multidomain benchmark LEMON, with higher quality and diversity than existing benchmarks, to allow a comprehensive assessment of the open domain generalization of CSC models.Then, we demonstrate that a very simple strategyrandomly masking 20% non-error tokens from the input sequence during fine-tuning -is sufficient for learning a much better language model without sacrificing the error model.This technique can be applied to any model architecture and achieves new state-of-the-art results on SIGHAN, ECSpell, and LEMON 1 .
Overview of the RadSum23 Shared Task on Multi-modal and Multi-anatomical Radiology Report Summarization

Radiology report summarization is a growing area of research.Given the Findings and/or Background sections of a radiology report, the goal is to generate a summary (called an Impression section) that highlights the key observations and conclusions of the radiology study.Recent efforts have released systems that achieve promising performance as measured by widely used summarization metrics such as BLEU and ROUGE.However, the research area of radiology report summarization currently faces two important limitations.First, most of the results are reported on private datasets.This limitation prevents the ability to reproduce results and fairly compare different systems and solutions.Secondly, to the best of our knowledge, most research is carried out on chest X-rays.To palliate these two limitations, we propose a radiology report summarization (RadSum) challenge on i) a new dataset of eleven different modalities and anatomies pairs based on the MIMIC-III database ii) a multimodal report summarization dataset based on MIMIC-CXR enhanced with a brand-new test-set from Stanford Hospital.In total, we received 112 submissions across 11 teams.
DiffuDetox: A Mixed Diffusion Model for Text Detoxification

Text detoxification is a conditional text generation task aiming to remove offensive content from toxic text.It is highly useful for online forums and social media, where offensive content is frequently encountered.Intuitively, there are diverse ways to detoxify sentences while preserving their meanings, and we can select from detoxified sentences before displaying text to users.Conditional diffusion models are particularly suitable for this task given their demonstrated higher generative diversity than existing conditional text generation models based on language models.Nonetheless, text fluency declines when they are trained with insufficient data, which is the case for this task.In this work, we propose DiffuDetox 1 , a mixed conditional and unconditional diffusion model for text detoxification.The conditional model takes toxic text as the condition and reduces its toxicity, yielding a diverse set of detoxified sentences.The unconditional model is trained to recover the input text, which allows the introduction of additional fluent text for training and thus ensures text fluency.Extensive experimental results and in-depth analysis demonstrate the effectiveness of our proposed DiffuDetox.
RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs

Despite their unprecedented success, even the largest language models make mistakes.Similar to how humans learn and improve using feedback, previous work proposed providing language models with natural language feedback to guide them in repairing their outputs.Because human-generated critiques are expensive to obtain, researchers have devised learned critique generators in lieu of human critics while assuming one can train downstream models to utilize generated feedback.However, this approach does not apply to black-box or limited access models such as ChatGPT, as they cannot be fine-tuned.Moreover, in the era of large general-purpose language agents, fine-tuning is neither computationally nor spatially efficient as it results in multiple copies of the network.In this work, we introduce RL4F (Reinforcement Learning for Feedback), a multi-agent collaborative framework where the critique generator is trained to maximize end-task performance of GPT-3, a fixed model more than 200 times its size.RL4F produces critiques that help GPT-3 revise its outputs.We study three datasets for action planning, summarization and alphabetization and show relative improvements up to 10% in multiple text similarity metrics over other learned, retrievalaugmented or prompting-based critique generators.
UniTRec: A Unified Text-to-Text Transformer and Joint Contrastive Learning Framework for Text-based Recommendation

Prior study has shown that pretrained language models (PLM) can boost the performance of text-based recommendation.In contrast to previous works that either use PLM to encode user history as a whole input text, or impose an additional aggregation network to fuse multiturn history representations, we propose a unified local-and global-attention Transformer encoder to better model two-level contexts of user history.Moreover, conditioned on user history encoded by Transformer encoders, our framework leverages Transformer decoders to estimate the language perplexity of candidate text items, which can serve as a straightforward yet significant contrastive signal for user-item text matching.Based on this, our framework, UniTRec, unifies the contrastive objectives of discriminative matching scores and candidate text perplexity to jointly enhance text-based recommendation.Extensive evaluation shows that UniTRec delivers SOTA performance on three text-based recommendation tasks.
UMASS_BioNLP at MEDIQA-Chat 2023: Can LLMs generate high-quality synthetic note-oriented doctor-patient conversations?

This paper presents UMASS_BioNLP team participation in the MEDIQA-Chat 2023 shared task for Task-A and Task-C.We focus especially on Task-C and propose a novel LLMs cooperation system named a doctor-patient loop to generate high-quality conversation data sets.The experiment results demonstrate that our approaches yield reasonable performance as evaluated by automatic metrics such as ROUGE, medical concept recall, BLEU, and Self-BLEU.Furthermore, we conducted a comparative analysis between our proposed method and Chat-GPT and GPT-4.This analysis also investigates the potential of utilizing cooperation LLMs to generate high-quality datasets.
Adversarial Clean Label Backdoor Attacks and Defenses on Text Classification Systems

Clean-label (CL) attack is a form of data poisoning attack where an adversary modifies only the textual input of the training data, without requiring access to the labeling function.CL attacks are relatively unexplored in NLP, as compared to label flipping (LF) attacks, where the latter additionally requires access to the labeling function as well.While CL attacks are more resilient to data sanitization and manual relabeling methods than LF attacks, they often demand as high as ten times the poisoning budget than LF attacks.In this work, we first introduce an Adversarial Clean Label attack which can adversarially perturb in-class training examples for poisoning the training set.We then show that an adversary can significantly bring down the data requirements for a CL attack, using the aforementioned approach, to as low as 20 % of the data otherwise required.We then systematically benchmark and analyze a number of defense methods, for both LF and CL attacks, some previously employed solely for LF attacks in the textual domain and others adapted from computer vision.We find that text-specific defenses greatly vary in their effectiveness depending on their properties.
AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities

CLIP (Contrastive Language-Image Pretraining) is an English multimodal representation model learned from a massive amount of English text-image pairs and has achieved great success in various downstream tasks, including image classification, text-to-image retrieval, and image generation.When extending CLIP to other languages, the major problem is the lack of good-quality text-image pairs.In this work, we present AltCLIP, a simple and lowresource method to build a strong multilingual multimodal representation model.Instead of training a model from scratch on multilingual text-image pairs, we take the original CLIP model trained on English text-image pairs and alter its text encoder with a pre-trained multilingual text encoder (XLM-R).We then align text and image representations by a two-stage training schema consisting of teacher learning and contrastive learning.Our method utilizes the existence of rich parallel text data and pre-trained multilingual language models.We present extensive experimental evaluations to demonstrate the effectiveness of our proposed method.Our model sets new state-ofthe-art zero-shot performances on a wide range of tasks in multilingual multimodal benchmarks, including ImageNet-CN/IT/JA/KO serials, Flicker30k-CN, COCO-CN, Multi30k, and XTD.Further, our model outperforms the original CLIP model on zero-shot crossmodal retrieval, Image Classification in the Wild (ICinW) tasks, and CLIP Benchmark.We open-source our code, pre-trained model weights, and evaluation toolkit of multilingual multimodal tasks, to facilitate research on multilingual multimodal representation learning.
SemEval-2023 Task 2: Fine-grained Multilingual Named Entity Recognition (MultiCoNER 2)

We present the findings of SemEval-2023 Task 2 on Fine-grained Multilingual Named Entity Recognition (MULTICONER 2). 1 Divided into 13 tracks, the task focused on methods to identify complex fine-grained named entities (like WRITTENWORK, VEHICLE, MUSICALGRP) across 12 languages, in both monolingual and multilingual scenarios, as well as noisy settings.The task used the MULTICONER V2 dataset, composed of 2.2 million instances in Bangla, Chinese, English, Farsi, French, German, Hindi, Italian., Portuguese, Spanish, Swedish, and Ukrainian.MULTICONER 2 was one of the most popular tasks of SemEval-2023.It attracted 842 submissions from 47 teams, and 34 teams submitted system papers.Results showed that complex entity types such as media titles and product names were the most challenging.Methods fusing external knowledge into transformer models achieved the best performance, and the largest gains were on the Creative Work and Group classes, which are still challenging even with external knowledge.Some fine-grained classes proved to be more challenging than others, such as SCIENTIST, ARTWORK, and PRIVATECORP.We also observed that noisy data has a significant impact on model performance, with an average drop of 10% on the noisy subset.The task highlights the need for future research on improving NER robustness on noisy data containing complex entities.
Rarely a problem? Language models exhibit inverse scaling in their predictions following few-type quantifiers

How well do language models deal with quantification?In this study, we focus on few-type quantifiers, as in few children like toys, which might pose a particular challenge for language models because the sentence components without the quantifier are likely to co-occur, and few-type quantifiers are rare.We present 960 English sentence stimuli from two human neurolinguistic experiments to 22 autoregressive transformer models of differing sizes.Not only do all the models perform poorly on few-type quantifiers, but overall the larger the model, the worse its performance.This inverse scaling is consistent with previous work suggesting that larger models increasingly reflect online rather than offline human processing, and we argue that the decreasing performance of larger models may challenge uses of language models as the basis for natural language systems.
On Degrees of Freedom in Defining and Testing Natural Language Understanding

Natural language understanding (NLU) studies often exaggerate or underestimate the capabilities of systems, thereby limiting the reproducibility of their findings.These erroneous evaluations can be attributed to the difficulty of defining and testing NLU adequately.In this position paper, we reconsider this challenge by identifying two types of researcher degrees of freedom.We revisit Turing's original interpretation of the Turing test and indicate that an NLU test does not provide an operational definition; it merely provides inductive evidence that the test subject understands the language sufficiently well to meet stakeholder objectives.In other words, stakeholders are free to arbitrarily define NLU through their objectives.To use the test results as inductive evidence, stakeholders must carefully assess if the interpretation of test scores is valid or not.However, designing and using NLU tests involve other degrees of freedom, such as specifying target skills and defining evaluation metrics.As a result, achieving consensus among stakeholders becomes difficult.To resolve this issue, we propose a validity argument, which is a framework comprising a series of validation criteria across test components.By demonstrating that current practices in NLU studies can be associated with those criteria and organizing them into a comprehensive checklist, we prove that the validity argument can serve as a coherent guideline for designing credible test sets and facilitating scientific communication.
ByGPT5: End-to-End Style-conditioned Poetry Generation with Token-free Language Models

State-of-the-art poetry generation systems are often complex.They either consist of taskspecific model pipelines, incorporate prior knowledge in the form of manually created constraints, or both.In contrast, end-to-end models would not suffer from the overhead of having to model prior knowledge and could learn the nuances of poetry from data alone, reducing the degree of human supervision required.In this work, we investigate end-to-end poetry generation conditioned on styles such as rhyme, meter, and alliteration.We identify and address lack of training data and mismatching tokenization algorithms as possible limitations of past attempts.In particular, we successfully pre-train ByGPT5, a new token-free decoder-only language model, and fine-tune it on a large custom corpus of English and German quatrains annotated with our styles.We show that ByGPT5 outperforms other models such as mT5, ByT5, GPT-2 and ChatGPT, while also being more parameter efficient and performing favorably compared to humans.In addition, we analyze its runtime performance and demonstrate that it is not prone to memorization.We make our code, models, and datasets publicly available.1
Focus-aware Response Generation in Inquiry Conversation

Inquiry conversation is a common form of conversation that aims to complete the investigation (e.g., court hearing, medical consultation and police interrogation) during which a series of focus shifts occurs.While many models have been proposed to generate a smooth response to a given conversation history, neglecting the focus can limit performance in inquiry conversation where the order of the focuses plays there a key role.In this paper, we investigate the problem of response generation in inquiry conversation by taking the focus into consideration.We propose a novel Focusaware Response Generation (FRG) method by jointly optimizing a multi-level encoder and a set of focal decoders to generate several candidate responses that correspond to different focuses.Additionally, a focus ranking module is proposed to predict the next focus and rank the candidate responses.Experiments on two orthogonal inquiry conversation datasets (judicial, medical domain) demonstrate that our method generates results significantly better in automatic metrics and human evaluation compared to the state-of-the-art approaches.