Reviewerly/exHarmony huggingface dataset:
UTSA-NLP at RadSum23: Multi-modal Retrieval-Based Chest X-Ray Report Summarization

Radiology report summarization aims to automatically provide concise summaries of radiology findings, reducing time and errors in manual summaries.However, current methods solely summarize the text, which overlooks critical details in the images.Unfortunately, directly using the images in a multimodal model is difficult.Multimodal models are susceptible to overfitting due to their increased capacity, and modalities tend to overfit and generalize at different rates.Thus, we propose a novel retrieval-based approach that uses image similarities to generate additional text features.We further employ few-shot with chain-of-thought and ensemble techniques to boost performance.Overall, our method achieves state-of-the-art performance in the F1RadGraph score, which measures the factual correctness of summaries.We rank second place in both MIMIC-CXR and MIMIC-III hidden tests among 11 teams, with F1RadGraph scores of 42.86 and 36.31.
Position Matters! Empirical Study of Order Effect in Knowledge-grounded Dialogue

With the power of large pretrained language models, various research works have integrated knowledge into dialogue systems.The traditional techniques treat knowledge as part of the input sequence for the dialogue system, prepending a set of knowledge statements in front of dialogue history.However, such a mechanism forces knowledge sets to be concatenated in an ordered manner, making models implicitly pay imbalanced attention to the sets during training.In this paper, we first investigate how the order of the knowledge set can influence autoregressive dialogue systems' responses.We conduct experiments on two commonly used dialogue datasets with two types of transformer-based models and find that models view the input knowledge unequally.To this end, we propose a simple and novel technique to alleviate the order effect by modifying the position embeddings of knowledge input in these models.With the proposed position embedding method, the experimental results show that each knowledge statement is uniformly considered to generate responses.
COCKATIEL: COntinuous Concept ranKed ATtribution with Interpretable ELements for explaining neural net classifiers on NLP

Transformer architectures are complex and their use in NLP, while it has engendered many successes, makes their interpretability or explainability challenging.Recent debates have shown that attention maps and attribution methods are unreliable (Pruthi et al., 2019;Brunner et al., 2019).In this paper, we present some of their limitations and introduce COCKATIEL, which successfully addresses some of them.COCKATIEL is a novel, post-hoc, conceptbased, model-agnostic XAI technique that generates meaningful explanations from the last layer of a neural net model trained on an NLP classification task by using Non-Negative Matrix Factorization (NMF) to discover the concepts the model leverages to make predictions and by exploiting a Sensitivity Analysis to estimate accurately the importance of each of these concepts for the model.It does so without compromising the accuracy of the underlying model or requiring a new one to be trained.We conduct experiments in single and multiaspect sentiment analysis tasks and we show COCKATIEL's superior ability to discover concepts that align with humans' on Transformer models without any supervision, we objectively verify the faithfulness of its explanations through fidelity metrics, and we showcase its ability to provide meaningful explanations in two different datasets.
Controllable Mixed-Initiative Dialogue Generation through Prompting

Mixed-initiative dialogue tasks involve repeated exchanges of information and conversational control.Conversational agents gain control by generating responses that follow particular dialogue intents or strategies, prescribed by a policy planner.The standard approach has been fine-tuning pre-trained language models to perform generation conditioned on these intents.However, these supervised generation models are limited by the cost and quality of data annotation.We instead prompt large language models as a drop-in replacement to finetuning on conditional generation.We formalize prompt construction for controllable mixedinitiative dialogue.Our findings show improvements over fine-tuning and ground truth responses according to human evaluation and automatic metrics for two tasks: PersuasionFor-Good and Emotional Support Conversations.
GPL at SemEval-2023 Task 1: WordNet and CLIP to Disambiguate Images

Given a word in context, the task of Visual Word Sense Disambiguation consists of selecting the correct image among a set of candidates.To select the correct image, we propose a solution blending text augmentation and multimodal models.Text augmentation leverages the fine-grained semantic annotation from Word-Net to get a better representation of the textual component.We then compare this senseaugmented text to the set of image using pretrained multimodal models CLIP and ViLT.Our system has been ranked 16 th for the English language, achieving 68.5 points for hit rate and 79.2 for mean reciprocal rank.The code to this project is available on Github 1 .
DN at SemEval-2023 Task 12: Low-Resource Language Text Classification via Multilingual Pretrained Language Model Fine-tuning

In our work, a model is implemented that solves the task, based on multilingual pre-trained models. We also consider various methods of data preprocessing
Hybrid Models for Sentence Readability Assessment

Automatic readability assessment (ARA) predicts how difficult it is for the reader to understand a text.While ARA has traditionally been performed at the passage level, there has been increasing interest in ARA at the sentence level, given its applications in downstream tasks such as text simplification and language exercise generation.Recent research has suggested the effectiveness of hybrid approaches for ARA, but they have yet to be applied on the sentence level.We present the first study that compares neural and hybrid models for sentence-level ARA.We conducted experiments on graded sentences from the Wall Street Journal (WSJ) and a dataset derived from the OneStopEnglish corpus.Experimental results show that both neural and hybrid models outperform traditional classifiers trained on linguistic features.Hybrid models obtained the best accuracy on both datasets, surpassing the previous best result reported on the WSJ dataset by almost 13% absolute.
Prompt to be Consistent is Better than Self-Consistent? Few-Shot and Zero-Shot Fact Verification with Pre-trained Language Models

Few-shot or zero-shot fact verification only relies on a few or no labeled training examples.In this paper, we propose a novel method called ProToCo, to Prompt pre-trained language models (PLMs) To be Consistent, for improving the factuality assessment capability of PLMs in the few-shot and zero-shot settings.Given a claim-evidence pair, ProToCo generates multiple variants of the claim with different relations and frames a simple consistency mechanism as constraints for making compatible predictions across these variants.We update PLMs by using parameter-efficient fine-tuning (PEFT), leading to more accurate predictions in few-shot and zero-shot fact verification tasks.Our experiments on three public verification datasets show that ProToCo significantly outperforms state-of-the-art few-shot fact verification baselines.With a small number of unlabeled instances, ProToCo also outperforms the strong zero-shot learner T0 on zero-shot verification.Compared to large PLMs using incontext learning (ICL) method, ProToCo outperforms OPT-30B and the Self-Consistencyenabled OPT-6.7Bmodel in both few-and zeroshot settings.
Hints on the data for language modeling of synthetic languages with transformers

Language Models (LM) are becoming more and more useful for providing representations upon which to train Natural Language Processing applications.However, there is now clear evidence that attention-based transformers require a critical amount of language data to produce good enough LMs.The question we have addressed in this paper is to what extent the critical amount of data varies for languages of different morphological typology, in particular those that have a rich inflectional morphology, and whether the tokenization method to preprocess the data can make a difference.These details can be important for low-resource languages that need to plan the production of datasets.We evaluated intrinsically and extrinsically the differences of five different languages with different pretraining dataset sizes and three different tokenization methods for each.The results confirm that the size of the vocabulary due to morphological characteristics is directly correlated with both the LM perplexity and the performance of two typical downstream tasks such as NER identification and POS Tagging.The experiments also provide new evidence that a canonical tokenizer can reduce perplexity by more than a half for a polysynthetic language like Quechua as well as raising macro-F1 score from 0.8 to more than 0.9 in both downstream tasks with a LM trained with only 6M tokens. 1 .
Noam Chomsky at SemEval-2023 Task 4: Hierarchical Similarity-aware Model for Human Value Detection

This paper presents a hierarchical similarityaware approach for the SemEval-2023 task 4 human value detection behind arguments using SBERT.The approach takes similarity score as an additional source of information between the input arguments and the lower level of labels in a human value hierarchical dataset.Our similarity-aware model improved the similarityagnostic baseline model, especially showing a significant increase in the value categories with lowest scores by the baseline model.* Equal contribution.
Extractive is not Faithful: An Investigation of Broad Unfaithfulness Problems in Extractive Summarization

The problems of unfaithful summaries have been widely discussed under the context of abstractive summarization.Though extractive summarization is less prone to the common unfaithfulness issues of abstractive summaries, does that mean extractive is equal to faithful?Turns out that the answer is no.In this work, we define a typology with five types of broad unfaithfulness problems (including and beyond not-entailment) that can appear in extractive summaries, including incorrect coreference, incomplete coreference, incorrect discourse, incomplete discourse, as well as other misleading information.We ask humans to label these problems out of 1600 English summaries produced by 16 diverse extractive systems.We find that 30% of the summaries have at least one of the five issues.To automatically detect these problems, we find that 5 existing faithfulness evaluation metrics for summarization have poor correlations with human judgment.To remedy this, we propose a new metric, EXTEVAL, that is designed for detecting unfaithful extractive summaries and is shown to have the best performance.We hope our work can increase the awareness of unfaithfulness problems in extractive summarization and help future work to evaluate and resolve these issues.
Saliency Map Verbalization: Comparing Feature Importance Representations from Model-free and Instruction-based Methods

Saliency maps can explain a neural model's predictions by identifying important input features.They are difficult to interpret for laypeople, especially for instances with many features.In order to make them more accessible, we formalize the underexplored task of translating saliency maps into natural language and compare methods that address two key challenges of this approach -what and how to verbalize.In both automatic and human evaluation setups, using token-level attributions from text classification tasks, we compare two novel methods (search-based and instruction-based verbalizations) against conventional feature importance representations (heatmap visualizations and extractive rationales), measuring simulatability, faithfulness, helpfulness and ease of understanding.Instructing GPT-3.5 to generate saliency map verbalizations yields plausible explanations which include associations, abstractive summarization and commonsense reasoning, achieving by far the highest human ratings, but they are not faithfully capturing numeric information and are inconsistent in their interpretation of the task.In comparison, our searchbased, model-free verbalization approach efficiently completes templated verbalizations, is faithful by design, but falls short in helpfulness and simulatability.Our results suggest that saliency map verbalization makes feature attribution explanations more comprehensible and less cognitively challenging to humans than conventional representations. 1
A Needle in a Haystack: An Analysis of High-Agreement Workers on MTurk for Summarization

To prevent the costly and inefficient use of resources on low-quality annotations, we want a method for creating a pool of dependable annotators who can effectively complete difficult tasks, such as evaluating automatic summarization.Thus, we investigate the recruitment of high-quality Amazon Mechanical Turk workers via a two-step pipeline.We show that we can successfully filter out subpar workers before they carry out the evaluations and obtain highagreement annotations with similar constraints on resources.Although our workers demonstrate a strong consensus among themselves and CloudResearch workers, their alignment with expert judgments on a subset of the data is not as expected and needs further training in correctness.This paper still serves as a best practice for the recruitment of qualified annotators in other challenging annotation tasks.
Learning to Predict Persona Information for Dialogue Personalization without Explicit Persona Description

Personalizing dialogue agents is important for dialogue systems to generate more specific, consistent, and engaging responses.However, most current dialogue personalization approaches rely on explicit persona descriptions during inference, which severely restricts its application.In this paper, we propose a novel approach that learns to predict persona information based on the dialogue history to personalize the dialogue agent without relying on any explicit persona descriptions during inference.Experimental results on the PersonaChat dataset show that the proposed method can improve the consistency of generated responses when conditioning on the predicted profile of the dialogue agent (i.e."self persona"), and improve the engagingness of the generated responses when conditioning on the predicted persona of the dialogue partner (i.e."their persona").We also find that a trained persona prediction model can be successfully transferred to other datasets and help generate more relevant responses.
Don’t Retrain, Just Rewrite: Countering Adversarial Perturbations by Rewriting Text

Can language models transform inputs to protect text classifiers against adversarial attacks?In this work, we present ATINTER, a model that intercepts and learns to rewrite adversarial inputs to make them non-adversarial for a downstream text classifier.Our experiments on four datasets and five attack mechanisms reveal that ATINTER is effective at providing better adversarial robustness than existing defense approaches, without compromising task accuracy.For example, on sentiment classification using the SST-2 dataset, our method improves the adversarial accuracy over the best existing defense approach by more than 4% with a smaller decrease in task accuracy (0.5 % vs. 2.5%).Moreover, we show that ATINTER generalizes across multiple downstream tasks and classifiers without having to explicitly retrain it for those settings.For example, we find that when ATINTER is trained to remove adversarial perturbations for the sentiment classification task on the SST-2 dataset, it even transfers to a semantically different task of news classification (on AGNews) and improves the adversarial robustness by more than 10%. BERT-SST-2 RoBERTa-SST-2Aids warning over bushmeat barter: Meat from African wild animals ... in the UK is spreading a virus similar to HIV, a leading scientist warns.Aids warning over bushmeat trade: Meat from African wild animals ... in the UK is spreading a virus similar to HIV, a leading scientist warns.BusinessAdversarial input for BERT-SST-2 (incorrectly predicted positive)Adversarial input for RoBERTa-SST-2 (incorrectly predicted positive)Adversarial input for BERT-AGNews (incorrectly predicted Business)
TACR: A Table Alignment-based Cell Selection Method for HybridQA

Hybrid Question-Answering (HQA), which targets reasoning over tables and passages linked from table cells, has witnessed significant research in recent years.A common challenge in HQA and other passage-table QA datasets is that it is generally unrealistic to iterate over all table rows, columns, and linked passages to retrieve evidence.Such a challenge made it difficult for previous studies to show their reasoning ability in retrieving answers.To bridge this gap, we propose a novel Tablealignment-based Cell-selection and Reasoning model (TACR) for hybrid text and table QA, evaluated on the HybridQA and WikiTable-Questions datasets.In evidence retrieval, we design a table-question-alignment enhanced cellselection method to retrieve fine-grained evidence.In answer reasoning, we incorporate a QA module that treats the row containing selected cells as context.Experimental results over the HybridQA and WikiTableQuestions (WTQ) datasets show that TACR achieves stateof-the-art results on cell selection and outperforms fine-grained evidence retrieval baselines on HybridQA, while achieving competitive performance on WTQ.We also conducted a detailed analysis to demonstrate that being able to align questions to tables in the cell-selection stage can result in important gains from experiments of over 90% table row and column selection accuracy, meanwhile also improving output explainability.
Cross-domain German Medical Named Entity Recognition using a Pre-Trained Language Model and Unified Medical Semantic Types

Information extraction from clinical text has the potential to facilitate clinical research and personalized clinical care, but annotating large amounts of data for each set of target tasks is prohibitive.We present a German medical Named Entity Recognition (NER) system capable of cross-domain knowledge transferring.The system builds on a pre-trained German language model and a token-level binary classifier, employing semantic types sourced from the Unified Medical Language System (UMLS) as entity labels to identify corresponding entity spans within the input text.To enhance the system's performance and robustness, we pretrain it using a medical literature corpus that incorporates UMLS semantic term annotations.We evaluate the system's effectiveness on two German annotated datasets obtained from different clinics in zero-and few-shot settings.The results show that our approach outperforms task-specific Condition Random Fields (CRF) classifiers in terms of accuracy.Our work contributes to developing robust and transparent German medical NER models that can support the extraction of information from various clinical texts.
Making Better Use of Training Corpus: Retrieval-based Aspect Sentiment Triplet Extraction via Label Interpolation

In this paper, we aim to adapt the idea of retrieval-based neural approaches to the Aspect Sentiment Triplet Extraction (ASTE) task.Different from previous studies retrieving semantic similar neighbors, the ASTE task has its specialized challenges when adapting, i.e., the purpose includes predicting the sentiment polarity and it is usually aspect-dependent.Semantic similar neighbors with different polarities will be infeasible even counterproductive.To tackle this issue, we propose a retrieval-based neural ASTE approach, named RLI (Retrieval-based Aspect Sentiment Triplet Extraction via Label Interpolation), which exploits the label information of neighbors.Given an aspect-opinion term pair, we retrieve semantic similar triplets from the training corpus and interpolate their label information into the augmented representation of the target pair.The retriever is jointly trained with the whole ASTE framework, and neighbors with both similar semantics and sentiments can be recalled with the aid of this distant supervision.In addition, we design a simple yet effective pre-train method for the retriever that implicitly encodes the label similarities.Extensive experiments and analysis on two widely-used benchmarks show that the proposed model establishes a new state-of-the-art on ASTE.
Mutual Information Alleviates Hallucinations in Abstractive Summarization

Despite significant progress in the quality of language generated from abstractive summarization models, these models still exhibit the tendency to hallucinate, i.e., output content not supported by the source document. A number of works have tried to fix—or at least uncover the source of—the problem with limited success. In this paper, we identify a simple criterion under which models are significantly more likely to assign more probability to hallucinated content during generation: high model uncertainty. This finding offers a potential explanation for hallucinations: models default to favoring text with high marginal probability, i.e., high-frequency occurrences in the training set, when uncertain about a continuation. It also motivates possible routes for real-time intervention during decoding to prevent such hallucinations. We propose a decoding strategy that switches to optimizing for pointwise mutual information of the source and target token—rather than purely the probability of the target token—when the model exhibits uncertainty. Experiments on the dataset show that our method decreases the probability of hallucinated tokens while maintaining the Rouge and BERT-S scores of top-performing decoding strategies.
Ask-and-Verify: Span Candidate Generation and Verification for Attribute Value Extraction

The product attribute value extraction (AVE) task aims to capture key factual information from product profiles, and is useful for several downstream applications in e-Commerce platforms.Previous contributions usually formulate this task using sequence labeling or reading comprehension architectures.However, sequence labeling models tend to be conservative in their predictions resulting in a high false negative rate.Existing reading comprehension formulations, on the other hand, can over-generate attribute values which hinders precision.In the present work we address these limitations with a new end-to-end pipeline framework called Ask-and-Verify.Given a product and an attribute query, the Ask step detects the top-K span candidates (i.e., possible attribute values) from the product profiles, then the Verify step filters out false positive candidates.We evaluate Ask-and-Verify model on Amazon's product pages and AliExpress public dataset, and present a comparative analysis as well as a detailed ablation study.Despite its simplicity, we show that Ask-and-Verify outperforms recent state-of-the-art models by up to 3.1% F1 absolute improvement points, while also scaling to thousands of attributes.