Reviewerly/exHarmony huggingface dataset:
Pre-trained language models in Spanish for health insurance coverage

The field of clinical natural language processing (NLP) can extract useful information from clinical text.Since 2017, the NLP field has shifted towards using pre-trained language models (PLMs), improving performance in several tasks.Most of the research in this field has focused on English text, but there are some available PLMs in Spanish.In this work, we use clinical PLMs to analyze text from admission and medical reports in Spanish for an insurance and health provider to give a probability of no coverage in a labor insurance process.Our results show that fine-tuning a PLM pretrained with the provider's data leads to better results, but this process is time-consuming and computationally expensive.At least for this task, fine-tuning publicly available clinical PLM leads to comparable results to a custom PLM, but in less time and with fewer resources.Analyzing large volumes of insurance requests is burdensome for employers, and models can ease this task by pre-classifying reports that are likely not to have coverage.Our approach of entirely using clinical-related text improves the current models while reinforcing the idea of clinical support systems that simplify human labor but do not replace it.To our knowledge, the clinical corpus collected for this study is the largest one reported for the Spanish language.
A Simple Yet Strong Domain-Agnostic De-bias Method for Zero-Shot Sentiment Classification

Zero-shot prompt-based learning has made much progress in sentiment analysis, and considerable effort has been dedicated to designing high-performing prompt templates.However, two problems exist; First, large language models are often biased to their pre-training data, leading to poor performance in prompt templates that models have rarely seen.Second, in order to adapt to different domains, redesigning prompt templates is usually required, which is time-consuming and inefficient.To remedy both shortcomings, we propose a simple yet strong data construction method to debias a given prompt template, yielding a large performance improvement in sentiment analysis tasks across different domains, pre-trained language models, and prompt templates.Also, we demonstrate the advantage of using domainagnostic generic responses over the in-domain ground-truth data.We release the code here 1 .
Do language models have coherent mental models of everyday things?

When people think of everyday things like an egg, they typically have a mental image associated with it.This allows them to correctly judge, for example, that "the yolk surrounds the shell" is a false statement.Do language models similarly have a coherent picture of such everyday things?To investigate this, we propose a benchmark dataset consisting of 100 everyday things, their parts, and the relationships between these parts, expressed as 11,720 "X relation Y?" true/false questions.Using these questions as probes, we observe that state-ofthe-art pre-trained language models (LMs) like GPT-3 and Macaw have fragments of knowledge about these everyday things, but do not have fully coherent "parts mental models" (54-59% accurate, 19-43% conditional constraint violation).We propose an extension where we add a constraint satisfaction layer on top of the LM's raw predictions to apply commonsense constraints.As well as removing inconsistencies, we find that this also significantly improves accuracy (by 16-20%), suggesting how the incoherence of the LM's pictures of everyday things can be significantly reduced. 1
Tsingriver at SemEval-2023 Task 10: Labeled Data Augmentation in Consistency Training

Semi-supervised learning has promising performance in deep learning, one of the approaches is consistency training on a large amount of unlabeled data to constrain model predictions to be invariant to input noise.However, The degree of correlation between unlabeled data and task objective directly affects model prediction performance.This paper describes our system designed for SemEval-2023 Task 10: Explainable Detection of Online Sexism.We utilize a consistency training framework and data augmentation as the main strategy to train a model.The score obtained by our method is 0.8180 in subtask A, ranking 57 in all the teams.
Model Interpretability and Rationale Extraction by Input Mask Optimization

Concurrent with the rapid progress in neural network-based models in NLP, the need for creating explanations for the predictions of these black-box models has risen steadily.Yet, especially for complex inputs like texts or images, existing interpretability methods still struggle with deriving easily interpretable explanations that also accurately represent the basis for the model's decision.To this end, we propose a new, model-agnostic method to generate extractive explanations for predictions made by neural networks, that is based on masking parts of the input which the model does not consider to be indicative of the respective class.The masking is done using gradient-based optimization combined with a new regularization scheme that enforces sufficiency, comprehensiveness, and compactness of the generated explanation.Our method achieves state-of-the-art results in a challenging paragraph-level rationale extraction task, showing that this task can be performed without training a specialized model.We further apply our method to image inputs and obtain high-quality explanations for image classifications, which indicates that the objectives for optimizing explanation masks in text generalize to inputs of other modalities.
Understanding Demonstration-based Learning from a Causal Perspective

Demonstration-based learning has shown impressive performance in exploiting pretrained language models under few-shot learning settings.It is interesting to see that demonstrations, even those composed of random tokens, can still improve performance.In this paper, we build a Structural Causal Model (SCM) to understand demonstration-based learning from causal perspectives and interpret random demonstrations as interventions on the demonstration variable within the causal model.We investigate the causal effects and find that the concurrence of specific words in the demonstration will induce bias, while randomly sampled tokens in the demonstration do not.Based on this finding, we further propose simple ways to construct random demonstrations, which even outperform hand-crafted, meaningful demonstrations on public sequence labeling benchmarks 1 .
Tokenization with Factorized Subword Encoding

In recent years, language models have become increasingly larger and more complex.However, the input representations for these models continue to rely on simple and greedy subword tokenization methods.In this paper, we propose a novel tokenization method that factorizes subwords onto discrete triplets using a VQ-VAE model.The effectiveness of the proposed tokenization method, referred to as the FACTORIZER, is evaluated on language modeling and morpho-syntactic tasks for 7 diverse languages.Results indicate that this method is more appropriate and robust for morphological tasks than the commonly used byte-pair encoding (BPE) tokenization algorithm.
ConKI: Contrastive Knowledge Injection for Multimodal Sentiment Analysis

Multimodal Sentiment Analysis leverages multimodal signals to detect the sentiment of a speaker.Previous approaches concentrate on performing multimodal fusion and representation learning based on general knowledge obtained from pretrained models, which neglects the effect of domain-specific knowledge.In this paper, we propose Contrastive Knowledge Injection (ConKI) for multimodal sentiment analysis, where specific-knowledge representations for each modality can be learned together with general knowledge representations via knowledge injection based on an adapter architecture.In addition, ConKI uses a hierarchical contrastive learning procedure performed between knowledge types within every single modality, across modalities within each sample, and across samples to facilitate the effective learning of the proposed representations, hence improving multimodal sentiment predictions.The experiments on three popular multimodal sentiment analysis benchmarks show that ConKI outperforms all prior methods on a variety of performance metrics.
Towards Adaptive Prefix Tuning for Parameter-Efficient Language Model Fine-tuning

Fine-tuning large pre-trained language models on various downstream tasks with whole parameters is prohibitively expensive.Hence, Parameter-efficient fine-tuning has attracted attention that only optimizes a few task-specific parameters with the frozen pre-trained model.In this work, we focus on prefix tuning, which only optimizes continuous prefix vectors (i.e.pseudo tokens) inserted into Transformer layers.Based on the observation that the learned syntax and semantics representation varies a lot at different layers, we argue that the adaptive prefix will be further tailored to each layer than the fixed one, enabling the fine-tuning more effective and efficient.Thus, we propose Adaptive Prefix Tuning (APT) to adjust the prefix in terms of both fine-grained token level and coarse-grained layer level with a gate mechanism.Experiments on the SuperGLUE and NER datasets show the effectiveness of APT.In addition, taking the gate as a probing, we validate the efficiency and effectiveness of the variable prefix.
Model-Based Simulation for Optimising Smart Reply

Smart Reply (SR) systems present a user with a set of replies, of which one can be selected in place of having to type out a response.To perform well at this task, a system should be able to effectively present the user with a diverse set of options, to maximise the chance that at least one of them conveys the user's desired response.This is a significant challenge, due to the lack of datasets containing sets of responses to learn from.Resultantly, previous work has focused largely on post-hoc diversification, rather than explicitly learning to predict sets of responses.Motivated by this problem, we present a novel method SIMSR, that employs model-based simulation to discover high-value response sets, through simulating possible user responses with a learned world model.Unlike previous approaches, this allows our method to directly optimise the end-goal of SR-maximising the relevance of at least one of the predicted replies.Empirically on two public datasets, when compared to SoTA baselines, our method achieves up to 21% and 18% improvement in ROUGE score and Self-ROUGE score respectively.
Reasoning in Large Language Models Through Symbolic Math Word Problems

Large language models (LLMs) have revolutionized NLP by solving downstream tasks with little to no labeled data.Despite their versatile abilities, the larger question of their ability to reason remains ill-understood.This paper addresses reasoning in math word problems (MWPs) by studying symbolic versions of the numeric problems, since a symbolic expression is a "concise explanation" of the numeric answer.We create and use a symbolic version of the SVAMP dataset and find that GPT-3's davinci-002 model also has good zeroshot accuracy on symbolic MWPs.To evaluate the faithfulness of the model's reasoning, we go beyond accuracy and additionally evaluate the alignment between the final answer and the outputted reasoning, which correspond to numeric and symbolic answers respectively for MWPs.We explore a self-prompting approach to encourage the symbolic reasoning to align with the numeric answer, thus equipping the LLM with the ability to provide a concise and verifiable reasoning and making it more interpretable.Surprisingly, self-prompting also improves the symbolic accuracy to be higher than both the numeric and symbolic accuracies, thus providing an ensembling effect.The SVAMP-Sym dataset will be released for future research on symbolic math problems.
TART: Improved Few-shot Text Classification Using Task-Adaptive Reference Transformation

Meta-learning has emerged as a trending technique to tackle few-shot text classification and achieve state-of-the-art performance.However, the performance of existing approaches heavily depends on the inter-class variance of the support set.As a result, it can perform well on tasks when the semantics of sampled classes are distinct while failing to differentiate classes with similar semantics.In this paper, we propose a novel Task-Adaptive Reference Transformation (TART) network, aiming to enhance the generalization by transforming the class prototypes to per-class fixed reference points in task-adaptive metric spaces.To further maximize divergence between transformed prototypes in task-adaptive metric spaces, TART introduces a discriminative reference regularization among transformed prototypes.Extensive experiments are conducted on four benchmark datasets and our method demonstrates clear superiority over the stateof-the-art models in all the datasets.In particular, our model surpasses the state-of-the-art method by 7.4% and 5.4% in 1-shot and 5-shot classification on the 20 Newsgroups dataset, respectively.Our code is available at https: //github.com/slei109/TART
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization

In long document controllable summarization, where labeled data is scarce, pretrained models struggle to adapt to the task and effectively respond to user queries.In this paper, we introduce SOCRATIC pretraining, a question-driven, unsupervised pretraining objective specifically designed to improve controllability in summarization tasks.By training a model to generate and answer relevant questions in a given context, SOCRATIC pretraining enables the model to more effectively adhere to user-provided queries and identify relevant content to be summarized.We demonstrate the effectiveness of this approach through extensive experimentation on two summarization domains, short stories and dialogue, and multiple control strategies: keywords, questions, and factoid QA pairs.Our pretraining method relies only on unlabeled documents and a question generation system and outperforms pre-finetuning approaches that use additional supervised data.Furthermore, our results show that SOCRATIC pretraining cuts task-specific labeled data requirements in half, is more faithful to userprovided queries, and achieves state-of-the-art performance on QMSum and SQuALITY.* Work done during internship at SalesforceIn a group discussion about a philosophical concept, Sarah used the Socratic method by asking and answering questions to stimulate critical thinking and clarify underlying assumptions.The method helped her and her classmates achieve a deeper understanding of the concept and address areas of disagreement.
UoR-NCL at SemEval-2023 Task 1: Learning Word-Sense and Image Embeddings for Word Sense Disambiguation

In SemEval-2023 Task 1, a task of applying Word Sense Disambiguation in an image retrieval system was introduced.To resolve this task, this work proposes three approaches: (1) an unsupervised approach considering similarities between word senses and image captions, (2) a supervised approach using a Siamese neural network, and (3) a self-supervised approach using a Bayesian personalized ranking framework.According to the results, both supervised and self-supervised approaches outperformed the unsupervised approach.They can effectively identify correct images of ambiguous words in the dataset provided in this task.
Ethical Considerations for Machine Translation of Indigenous Languages: Giving a Voice to the Speakers

In recent years machine translation has become very successful for high-resource language pairs.This has also sparked new interest in research on the automatic translation of lowresource languages, including Indigenous languages.However, the latter are deeply related to the ethnic and cultural groups that speak (or used to speak) them.The data collection, modeling and deploying machine translation systems thus result in new ethical questions that must be addressed.Motivated by this, we first survey the existing literature on ethical considerations for the documentation, translation, and general natural language processing for Indigenous languages.Afterward, we conduct and analyze an interview study to shed light on the positions of community leaders, teachers, and language activists regarding ethical concerns for the automatic translation of their languages.Our results show that the inclusion, at different degrees, of native speakers and community members is vital to performing better and more ethical research on Indigenous languages.
Towards Efficient Simultaneous Speech Translation: CUNI-KIT System for Simultaneous Track at IWSLT 2023

In this paper, we describe our submission to the Simultaneous Track at IWSLT 2023.This year, we continue with the successful setup from the last year, however, we adopt the latest methods that further improve the translation quality.Additionally, we propose a novel online policy for attentional encoder-decoder models.The policy prevents the model to generate translation beyond the current speech input by using an auxiliary CTC output layer.We show that the proposed simultaneous policy can be applied to both streaming blockwise models and offline encoder-decoder models.We observe significant improvements in quality (up to 1.1 BLEU) and the computational footprint (up to 45 % relative RTF).
Cross-task Knowledge Transfer for Extremely Weakly Supervised Text Classification

Text classification with extremely weak supervision (EWS) imposes stricter supervision constraints compared to regular weakly supervised classification.Absolutely no labeled training samples or hand-crafted rules specific to the evaluation data are allowed.Such restrictions limit state-of-the-art EWS classification methods to indirect weak labeling techniques that assign unnatural label uncertainty estimates.We present PLAT, a framework that creates weak labels by leveraging recent developments in zero-shot text classification.PLAT employs models trained for sub-tasks other than classification to label documents.Most importantly, PLAT refrains from assigning overly confident weak labels and improves soft-label training performance for downstream classifiers.Classifiers trained with PLAT significantly outperform those trained on weak labels generated by the previous state-of-the-art in extremely weakly supervised text classification.
Implicit Memory Transformer for Computationally Efficient Simultaneous Speech Translation

Simultaneous speech translation is an essential communication task difficult for humans whereby a translation is generated concurrently with oncoming speech inputs.For such a streaming task, transformers using block processing to break an input sequence into segments have achieved state-of-the-art performance at a reduced cost.Current methods to allow information to propagate across segments, including left context and memory banks, have faltered as they are both insufficient representations and unnecessarily expensive to compute.In this paper, we propose an Implicit Memory Transformer that implicitly retains memory through a new left context method, removing the need to explicitly represent memory with memory banks.We generate the left context from the attention output of the previous segment and include it in the keys and values of the current segment's attention calculation.Experiments on the MuST-C dataset show that the Implicit Memory Transformer provides a substantial speedup on the encoder forward pass with nearly identical translation quality when compared with the state-of-the-art approach that employs both left context and memory banks.
Revisit Few-shot Intent Classification with PLMs: Direct Fine-tuning vs. Continual Pre-training

We consider the task of few-shot intent detection, which involves training a deep learning model to classify utterances based on their underlying intents using only a small amount of labeled data.The current approach to address this problem is through continual pretraining, i.e., fine-tuning pre-trained language models (PLMs) on external resources (e.g., conversational corpora, public intent detection datasets, or natural language understanding datasets) before using them as utterance encoders for training an intent classifier.In this paper, we show that continual pre-training may not be essential, since the overfitting problem of PLMs on this task may not be as serious as expected.Specifically, we find that directly fine-tuning PLMs on only a handful of labeled examples already yields decent results compared to methods that employ continual pre-training, and the performance gap diminishes rapidly as the number of labeled data increases.To maximize the utilization of the limited available data, we propose a context augmentation method and leverage sequential self-distillation to boost performance.Comprehensive experiments on real-world benchmarks show that given only two or more labeled samples per class, direct fine-tuning outperforms many strong baselines that utilize external data sources for continual pre-training.
HuaSLIM: Human Attention Motivated Shortcut Learning Identification and Mitigation for Large Language models

Large language models have made remarkable progress on a variety of NLP tasks. However, it has been found that they tend to rely on shortcut features that spuriously correlate with labels for prediction, which weakens their generalization on out-of-distribution samples. In this paper, we propose a human attention guided approach to identifying and mitigating shortcut learning, which encourages the LLM-based target model to learn relevant features. We define an attention-based measurement to capture both model and data bias and identify shortcut tokens by exploring both human and neural attention. In a self-distillation framework, we mitigate shortcut learning by dynamically adjusting the distillation temperature according to the detected shortcut tokens and estimated shortcut degree. Additionally, we utilize human attention as a supervisory signal to constrain large language models to pay more attention to relevant tokens. Experimental results on multiple NLP tasks show that our proposed method can effectively identify shortcut tokens, and significantly improve the robustness of large language models on OOD samples, while not undermining the performance on IID data.
A Closer Look at k-Nearest Neighbors Grammatical Error Correction

In various natural language processing tasks, such as named entity recognition and machine translation, example-based approaches have been used to improve performance by leveraging existing knowledge.However, the effectiveness of this approach for Grammatical Error Correction (GEC) is unclear.In this work, we explore how an example-based approach affects the accuracy and interpretability of the output of GEC systems and the trade-offs involved.The approach we investigate has shown great promise in machine translation by using the k nearest translation examples to improve the results of a pretrained Transformer model.We find that using this technique increases precision by reducing the number of false positives, but recall suffers as the model becomes more conservative overall.Increasing the number of example sentences in the datastore does lead to better performing systems, but with diminishing returns and a high decoding cost.Synthetic data can be used as examples, but the effectiveness varies depending on the base model.Finally, we find that finetuning on a set of data may be more effective than using that data during decoding as examples.