Reviewerly/exHarmony huggingface dataset:
Making Pre-trained Language Models both Task-solvers and Self-calibrators

Pre-trained language models (PLMs) serve as backbones for various real-world systems.For high-stake applications, it's equally essential to have reasonable confidence estimations in predictions.While the vanilla confidence scores of PLMs can already be effectively utilized, PLMs consistently become overconfident in their wrong predictions, which is not desirable in practice.Previous work shows that introducing an extra calibration task can mitigate this issue.The basic idea involves acquiring additional data to train models in predicting the confidence of their initial predictions.However, it only demonstrates the feasibility of this kind of method, assuming that there are abundant extra available samples for the introduced calibration task.In this work, we consider the practical scenario that we need to effectively utilize training samples to make PLMs both task-solvers and self-calibrators.Three challenges are presented, including limited training samples, data imbalance, and distribution shifts.We first conduct pilot experiments to quantify various decisive factors in the calibration task.Based on the empirical analysis results, we propose a training algorithm LM-TOAST to tackle the challenges.Experimental results show that LM-TOAST can effectively utilize the training data to make PLMs have reasonable confidence estimations while maintaining the original task performance.Further, we consider three downstream applications, namely selective classification, adversarial defense, and model cascading, to show the practical usefulness of LM-TOAST.The code will be made public at https://github.com/Yangyi-Chen/LM-TOAST. Vanilla Confidence Estimation LM-TOASTAverage Confidence Difference: 42.46 Average Confidence Difference: 15.18 utilize their strong capacities (Bommasani et al., 2021;Han et al., 2021).Beyond the performance evaluation, an essential requirement in high-stake applications for PLMs is to assign reasonable confidence to their predictions.This can enable the decision-makers to better handle the low-confident predictions, e.g., directly abstain to give predictions or transfer the input to human experts.The original predictive probability of PLMs can be effectively utilized for ranking predictions.This simple strategy can reasonably give relatively higher confidence to correct predictions compared to the wrong ones (Hendrycks et al., 2020a).However, relying on the vanilla confidence scores cannot well distinguish between correct and wrong predictions.PLMs consistently assign high confidence in their predictions, no matter
A Statistical Exploration of Text Partition Into Constituents: The Case of the Priestly Source in the Books of Genesis and Exodus

We present a pipeline for a statistical stylometric exploration of a hypothesized partition of a text. Given a parameterization of the text, our pipeline: (1) detects literary features yielding the optimal overlap between the hypothesized and unsupervised partitions, (2) performs a hypothesis-testing analysis to quantify the statistical significance of the optimal overlap, while conserving implicit correlations between units of text that are more likely to be grouped, and (3) extracts and quantifies the importance of features most responsible for the classification, estimates their statistical stability and cluster-wise abundance.We apply our pipeline to the first two books in the Bible, where one stylistic component stands out in the eyes of biblical scholars, namely, the Priestly component. We identify and explore statistically significant stylistic differences between the Priestly and non-Priestly components.
SlowBERT: Slow-down Attacks on Input-adaptive Multi-exit BERT

For pretrained language models such as Google’s BERT, recent research designs several input-adaptive inference mechanisms to improve the efficiency on cloud and edge devices. In this paper, we reveal a new attack surface on input-adaptive multi-exit BERT, where the adversary imperceptibly modifies the input texts to drastically increase the average inference cost. Our proposed slow-down attack called SlowBERT integrates a new rank-and-substitute adversarial text generation algorithm to efficiently search for the perturbation which maximally delays the exiting time. With no direct access to the model internals, we further devise a time-based approximation algorithm to infer the exit position as the loss oracle. Our extensive evaluation on two popular instances of multi-exit BERT for GLUE classification tasks validates the effectiveness of SlowBERT. In the worst case, SlowBERT increases the inference cost by 4.57×, which would strongly hurt the service quality of multi-exit BERT in practice, e.g., increasing the real-time cloud services’ response times for online users.
Know Where You’re Going: Meta-Learning for Parameter-Efficient Fine-Tuning

A recent family of techniques, dubbed lightweight fine-tuning methods, facilitates parameter-efficient transfer by updating only a small set of additional parameters while keeping the parameters of the original model frozen.While proven to be an effective approach, there are no existing studies on if and how such knowledge of the downstream fine-tuning approach calls for complementary measures after pre-training and before fine-tuning.In this work, we show that taking the ultimate choice of fine-tuning into consideration boosts the performance of parameter-efficient fine-tuning.By relying on optimization-based meta-learning using MAML with certain modifications for our distinct purpose, we prime the pre-trained model specifically for parameter-efficient finetuning, resulting in gains of up to 4.96 points on cross-lingual NER fine-tuning.Our ablation settings and analyses further reveal that the specific approach we take to meta-learning is crucial for the attained gains. 1
Improving Zero-shot Relation Classification via Automatically-acquired Entailment Templates

While fully supervised relation classification (RC) models perform well on large-scale datasets, their performance drops drastically in low-resource settings.As generating annotated examples are expensive, recent zero-shot methods have been proposed that reformulate RC into other NLP tasks for which supervision exists such as textual entailment.However, these methods rely on templates that are manually created which is costly and requires domain expertise.In this paper, we present a novel strategy for template generation for relation classification, which is based on adapting Harris' distributional similarity principle to templates encoded using contextualized representations.Further, we perform empirical evaluation of different strategies for combining the automatically acquired templates with manual templates.The experimental results on TACRED show that our approach not only performs better than the zero-shot RC methods that only use manual templates, but also that it achieves state-of-the-art performance for zero-shot TACRED at 64.3 F1 score.
Zero-shot Approach to Overcome Perturbation Sensitivity of Prompts

Recent studies have demonstrated that naturallanguage prompts can help to leverage the knowledge learned by pre-trained language models for the binary sentence-level sentiment classification task.Specifically, these methods utilize few-shot learning settings to finetune the sentiment classification model using manual or automatically generated prompts.However, the performance of these methods is sensitive to the perturbations of the utilized prompts.Furthermore, these methods depend on a few labeled instances for automatic prompt generation and prompt ranking.This study aims to find high-quality prompts for the given task in a zero-shot setting.Given a base prompt, our proposed approach automatically generates multiple prompts similar to the base prompt employing positional, reasoning, and paraphrasing techniques and then ranks the prompts using a novel metric.We empirically demonstrate that the top-ranked prompts are high-quality and significantly outperform the base prompt and the prompts generated using few-shot learning for the binary sentence-level sentiment classification task.
Advancing Multi-Criteria Chinese Word Segmentation Through Criterion Classification and Denoising

Recent research on multi-criteria Chinese word segmentation (MCCWS) mainly focuses on building complex private structures, adding more handcrafted features, or introducing complex optimization processes.In this work, we show that through a simple yet elegant inputhint-based MCCWS model, we can achieve state-of-the-art (SoTA) performances on several datasets simultaneously.We further propose a novel criterion-denoising objective that hurts slightly on F1 score but achieves SoTA recall on out-of-vocabulary words.Our result establishes a simple yet strong baseline for future MCCWS research.
Assessing Word Importance Using Models Trained for Semantic Tasks

Many NLP tasks require to automatically identify the most significant words in a text.In this work, we derive word significance from models trained to solve semantic task: Natural Language Inference and Paraphrase Identification.Using an attribution method aimed to explain the predictions of these models, we derive importance scores for each input token.We evaluate their relevance using a so-called crosstask evaluation: Analyzing the performance of one model on an input masked according to the other model's weight, we show that our method is robust with respect to the choice of the initial task.Additionally, we investigate the scores from the syntax point of view and observe interesting patterns, e.g.words closer to the root of a syntactic tree receive higher importance scores.Altogether, these observations suggest that our method can be used to identify important words in sentences without any explicit word importance labeling in training.
Being Right for Whose Right Reasons?

Explainability methods are used to benchmark the extent to which model predictions align with human rationales i.e., are 'right for the right reasons'.Previous work has failed to acknowledge, however, that what counts as a rationale is sometimes subjective.This paper presents what we think is a first of its kind, a collection of human rationale annotations augmented with the annotators demographic information.We cover three datasets spanning sentiment analysis and common-sense reasoning, and six demographic groups (balanced across age and ethnicity).Such data enables us to ask both what demographics our predictions align with and whose reasoning patterns our models' rationales align with.We find systematic inter-group annotator disagreement and show how 16 Transformer-based models align better with rationales provided by certain demographic groups: We find that models are biased towards aligning best with older and/or white annotators.We zoom in on the effects of model size and model distillation, finding -contrary to our expectations -negative correlations between model size and rationale agreement as well as no evidence that either model size or model distillation improves fairness.
An Open Dataset and Model for Language Identification

Language identification (LID) is a fundamental step in many natural language processing pipelines.However, current LID systems are far from perfect, particularly on lower-resource languages.We present a LID model which achieves a macro-average F1 score of 0.93 and a false positive rate of 0.033% across 201 languages, outperforming previous work.We achieve this by training on a curated dataset of monolingual data, the reliability of which we ensure by auditing a sample from each source and each language manually.We make both the model and the dataset available to the research community.Finally, we carry out detailed analysis into our model's performance, both in comparison to existing open models and by language class.
Understanding Native Language Identification for Brazilian Indigenous Languages

We investigate native language identification (LangID) for Brazilian Indigenous Languages (BILs), using the Bible as training data.Our research extends from previous work, by presenting two analyses on the generalization of Bible-based LangID in non-biblical data.First, with newly collected non-biblical datasets, we show that such a LangID can still provide quite reasonable accuracy in languages for which there are more established writing standards, such as Guarani Mbya and Kaigang, but there can be a quite drastic drop in accuracy depending on the language.Then, we applied the LangID on a large set of texts, about 13M sentences from the Portuguese Wikipedia, towards understanding the difficulty factors may come out of such task in practice.The main outcome is that the lack of handling other American indigenous languages can affect considerably the precision for BILs, suggesting the need of a joint effort with related languages from the Americas.
Parallel Data Helps Neural Entity Coreference Resolution

Coreference resolution is the task of finding expressions that refer to the same entity in a text.Coreference models are generally trained on monolingual annotated data but annotating coreference is expensive and challenging.Hardmeier et al. (2013) have shown that parallel data contains latent anaphoric knowledge, but it has not been explored in end-to-end neural models yet.In this paper, we propose a simple yet effective model to exploit coreference knowledge from parallel data.In addition to the conventional modules learning coreference from annotations, we introduce an unsupervised module to capture cross-lingual coreference knowledge.Our proposed cross-lingual model achieves consistent improvements, up to 1.74 percentage points, on the OntoNotes 5.0 English dataset using 9 different synthetic parallel datasets.These experimental results confirm that parallel data can provide additional coreference knowledge which is beneficial to coreference resolution tasks.
Images in Language Space: Exploring the Suitability of Large Language Models for Vision &amp; Language Tasks

Large language models have demonstrated robust performance on various language tasks using zero-shot or few-shot learning paradigms.While being actively researched, multimodal models that can additionally handle images as input have yet to catch up in size and generality with language-only models.In this work, we ask whether language-only models can be utilised for tasks that require visual input -but also, as we argue, often require a strong reasoning component.Similar to some recent related work, we make visual information accessible to the language model using separate verbalisation models.Specifically, we investigate the performance of open-source, open-access language models against GPT-3 on five visionlanguage tasks when given textually-encoded visual information.Our results suggest that language models are effective for solving visionlanguage tasks even with limited samples.This approach also enhances the interpretability of a model's output by providing a means of tracing the output back through the verbalised image content.
SRCB at SemEval-2023 Task 1: Prompt Based and Cross-Modal Retrieval Enhanced Visual Word Sense Disambiguation

The Visual Word Sense Disambiguation (VWSD) shared task aims at selecting the image among candidates that best interprets the semantics of a target word with a short-length phrase for English, Italian, and Farsi.The limited phrase context, which only contains 2-3 words, challenges the model's understanding ability, and the visual label requires image-text matching performance across different modalities.In this paper, we propose a prompt based and multimodal retrieval enhanced VWSD system, which uses the rich potential knowledge of large-scale pretrained models by prompting and additional text-image information from knowledge bases and open datasets.Under the English situation and given an input phrase, (1) the context retrieval module predicts the correct definition from sense inventory by matching phrase and context through a biencoder architecture.(2) The image retrieval module retrieves the relevant images from an image dataset.(3) The matching module decides that either text or image is used to pair with image labels by a rule-based strategy, then ranks the candidate images according to the similarity score.Our system ranks first in the English track and second in the average of all languages (English, Italian, and Farsi).
FLamE: Few-shot Learning from Natural Language Explanations

Natural language explanations have the potential to provide rich information that in principle guides model reasoning.Yet, recent work by Lampinen et al. ( 2022) has shown limited utility of natural language explanations in improving classification.To effectively learn from explanations, we present FLamE, a two-stage few-shot learning framework that first generates explanations using GPT-3, and then finetunes a smaller model (e.g., RoBERTa) with generated explanations.Our experiments on natural language inference demonstrate effectiveness over strong baselines, increasing accuracy by 17.6% over GPT-3 Babbage and 5.7% over GPT-3 Davinci in e-SNLI.Despite improving classification performance, human evaluation surprisingly reveals that the majority of generated explanations does not adequately justify classification decisions.Additional analyses point to the important role of label-specific cues (e.g., "not know" for the neutral label) in generated explanations.
Phrase Retrieval for Open Domain Conversational Question Answering with Conversational Dependency Modeling via Contrastive Learning

Open-Domain Conversational Question Answering (ODConvQA) aims at answering questions through a multi-turn conversation based on a retriever-reader pipeline, which retrieves passages and then predicts answers with them.However, such a pipeline approach not only makes the reader vulnerable to the errors propagated from the retriever, but also demands additional effort to develop both the retriever and the reader, which further makes it slower since they are not runnable in parallel.In this work, we propose a method to directly predict answers with a phrase retrieval scheme for a sequence of words, reducing the conventional two distinct subtasks into a single one.Also, for the first time, we study its capability for ODConvQA tasks.However, simply adopting it is largely problematic, due to the dependencies between previous and current turns in a conversation.To address this problem, we further introduce a novel contrastive learning strategy, making sure to reflect previous turns when retrieving the phrase for the current context, by maximizing representational similarities of consecutive turns in a conversation while minimizing irrelevant conversational contexts.We validate our model on two ODConvQA datasets, whose experimental results show that it substantially outperforms the relevant baselines with the retriever-reader.
An Investigation of Noise in Morphological Inflection

With a growing focus on morphological inflection systems for languages where high-quality data is scarce, training data noise is a serious but so far largely ignored concern.We aim at closing this gap by investigating the types of noise encountered within a pipeline for truly unsupervised morphological paradigm completion and its impact on morphological inflection systems: First, we propose an error taxonomy and annotation pipeline for inflection training data.Then, we compare the effect of different types of noise on multiple state-of-theart inflection models.Finally, we propose a novel character-level masked language modeling (CMLM) pretraining objective and explore its impact on the models' resistance to noise.Our experiments show that various architectures are impacted differently by separate types of noise, but encoder-decoders tend to be more robust to noise than models trained with a copy bias.CMLM pretraining helps transformers, but has lower impact on LSTMs.
SPM: A Split-Parsing Method for Joint Multi-Intent Detection and Slot Filling

In a task-oriented dialogue system, joint intent detection and slot filling for multi-intent utterances become meaningful since users tend to query more.The current state-of-the-art studies choose to process multi-intent utterances through a single joint model of sequence labelling and multi-label classification, which cannot generalize to utterances with more intents than training samples.Meanwhile, it lacks the ability to assign slots to each corresponding intent.To overcome these problems, we propose a Split-Parsing Method (SPM) for joint multiple intent detection and slot filling, which is a two-stage method.It first splits an input sentence into multiple sub-sentences which contain a single-intent, and then a joint single intent detection and slot filling model is applied to parse each sub-sentence recurrently.Finally, we integrate the parsed results.The sub-sentence split task is also treated as a sequence labelling problem with only one entitylabel, which can effectively generalize to a sentence with more intents unseen in the training set.Experimental results on three multi-intent datasets show that our method obtains substantial improvements over different baselines.
Rare Codes Count: Mining Inter-code Relations for Long-tail Clinical Text Classification

Multi-label clinical text classification, such as automatic ICD coding, has always been a challenging subject in Natural Language Processing, due to its long, domain-specific documents and long-tail distribution over a large label set.Existing methods adopt different model architectures to encode the clinical notes.Whereas without digging out the useful connections between labels, the model presents a huge gap in predicting performances between rare and frequent codes.In this work, we propose a novel method for further mining the helpful relations between different codes via a relationenhanced code encoder to improve the rare code performance.Starting from the simple code descriptions, the model reaches comparable, even better performances than models with heavy external knowledge.Our proposed method is evaluated on MIMIC-III, a common dataset in the medical domain.It outperforms the previous state-of-art models on both overall metrics and rare code performances.Moreover, the interpretation results further prove the effectiveness of our methods.Our code is publicly available 1 .
Few-shot Spanish-Aymara Machine Translation Using English-Aymara Lexicon

This paper presents the experiments to train a Spanish-Aymara machine translation model for the AmericasNLP 2023 Machine Translation shared task. We included the English-Aymara GlobalVoices corpus and an English-Aymara lexicon to train the model and limit our training resources to train the model in a \textit{few-shot} manner.