GenAI4ELab/papercli-papers huggingface dataset:
COGEN: Abductive Commonsense Language Generation
Authors: Rohola Zandie; Danny Brahman; Mohammad Mahoor
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers))

Abstract: Reasoning is one of the most important elements in achieving Artificial General Intelligence (AGI), specifically when it comes to Abductive and counterfactual reasoning. In order to introduce these capabilities of reasoning in Natural Language Processing (NLP) models, there have been recent advances towards training NLP models to better perform on two main tasks - Abductive Natural Language Inference (alphaNLI) and Abductive Natural Language Generation Task (alphaNLG). This paper proposes CoGen, a model for both alphaNLI and alphaNLG tasks that employ a novel approach of combining the temporal commonsense reasoning for each observation (before and after a real hypothesis) from pre-trained models with contextual filtering for training. Additionally, we use state-of-the-art semantic entailment to filter out the contradictory hypothesis during the inference. Our experimental results show that CoGen outperforms current models and set a new state of the art in regards to alphaNLI and alphaNLG tasks. We make the source code of CoGen model publicly available for reproducibility and to facilitate relevant future research.
Multimodal Relation Extraction with Cross-Modal Retrieval and Synthesis
Authors: Xuming Hu; Zhijiang Guo; Zhiyang Teng; Irwin King; Philip S. Yu
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers))

Abstract: Multimodal relation extraction (MRE) is the task of identifying the semantic relationships between two entities based on the context of the sentence image pair. Existing retrieval-augmented approaches mainly focused on modeling the retrieved textual knowledge, but this may not be able to accurately identify complex relations. To improve the prediction, this research proposes to retrieve textual and visual evidence based on the object, sentence, and whole image. We further develop a novel approach to synthesize the object-level, image-level, and sentence-level information for better reasoning between the same and different modalities. Extensive experiments and analyses show that the proposed method is able to effectively select and compare evidence across modalities and significantly outperforms state-of-the-art models.
Characterization of Stigmatizing Language in Medical Records
Authors: Keith Harrigian; Ayah Zirikly; Brant Chee; Alya Ahmad; Anne Links; Somnath Saha; Mary Catherine Beach; Mark Dredze
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers))

Abstract: Widespread disparities in clinical outcomes exist between different demographic groups in the United States. A new line of work in medical sociology has demonstrated physicians often use stigmatizing language in electronic medical records within certain groups, such as black patients, which may exacerbate disparities. In this study, we characterize these instances at scale using a series of domain-informed NLP techniques. We highlight important differences between this task and analogous bias-related tasks studied within the NLP community (e.g., classifying microaggressions). Our study establishes a foundation for NLP researchers to contribute timely insights to a problem domain brought to the forefront by recent legislation regarding clinical documentation transparency. We release data, code, and models.
Abstractive Summarizers are Excellent Extractive Summarizers
Authors: Daniel Varab; Yumo Xu
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers))

Abstract: Extractive and abstractive summarization designs have historically been fragmented, limiting the benefits that often arise from compatible model architectures. In this paper, we explore the potential synergies of modeling extractive summarization with an abstractive summarization system and propose three novel inference algorithms using the sequence-to-sequence architecture. We evaluate them on the CNN & Dailymail dataset and show that recent advancements in abstractive system designs enable abstractive systems to not only compete, but even surpass the performance of extractive systems with custom architectures. To our surprise, abstractive systems achieve this without being exposed to extractive oracle summaries and, therefore, for the first time allow a single model to produce both abstractive and extractive summaries. This evidence questions our fundamental understanding of extractive system design, and the necessity for extractive labels while pathing the way for promising research directions in hybrid models.
Language Models Get a Gender Makeover: Mitigating Gender Bias with Few-Shot Data Interventions
Authors: Himanshu Thakur; Atishay Jain; Praneetha Vaddamanu; Paul Pu Liang; Louis-Philippe Morency
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers))

Abstract: Societal biases present in pre-trained large language models are a critical issue as these models have been shown to propagate biases in countless downstream applications, rendering them unfair towards specific groups of people. Since large-scale retraining of these models from scratch is both time and compute-expensive, a variety of approaches have been previously proposed that de-bias a pre-trained model. While the majority of current state-of-the-art debiasing methods focus on changes to the training regime, in this paper, we propose data intervention strategies as a powerful yet simple technique to reduce gender bias in pre-trained models. Specifically, we empirically show that by fine-tuning a pre-trained model on only 10 debiased (intervened) training examples, the tendency to favor any gender is significantly reduced. Since our proposed method only needs a few training examples, we argue that our few-shot de-biasing approach is highly feasible and practical. Through extensive experimentation, we show that our de-biasing technique performs better than competitive state-of-the-art baselines with minimal loss in language modeling ability.
PLUE: Language Understanding Evaluation Benchmark for Privacy Policies in English
Authors: Jianfeng Chi; Wasi Uddin Ahmad; Yuan Tian; Kai-Wei Chang
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers))

Abstract: Privacy policies provide individuals with information about their rights and how their personal information is handled. Natural language understanding (NLU) technologies can support individuals and practitioners to understand better privacy practices described in lengthy and complex documents. However, existing efforts that use NLU technologies are limited by processing the language in a way exclusive to a single task focusing on certain privacy practices. To this end, we introduce the Privacy Policy Language Understanding Evaluation (PLUE) benchmark, a multi-task benchmark for evaluating the privacy policy language understanding across various tasks. We also collect a large corpus of privacy policies to enable privacy policy domain-specific language model pre-training. We evaluate several generic pre-trained language models and continue pre-training them on the collected corpus. We demonstrate that domain-specific continual pre-training offers performance improvements across all tasks. The code and models are released at https://github.com/JFChi/PLUE.
Stop Pre-Training: Adapt Visual-Language Models to Unseen Languages
Authors: Yasmine Karoui; Rémi Lebret; Negar Foroutan Eghlidi; Karl Aberer
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers))

Abstract: Vision-Language Pre-training (VLP) has advanced the performance of many vision-language tasks, such as image-text retrieval, visual entailment, and visual reasoning. The pre-training mostly utilizes lexical databases and image queries in English. Previous work has demonstrated that the pre-training in English does not transfer well to other languages in a zero-shot setting. However, multilingual pre-trained language models (MPLM) have excelled at a variety of single-modal language tasks. In this paper, we propose a simple yet efficient approach to adapt VLP to unseen languages using MPLM.We utilize a cross-lingual contextualised token embeddings alignment approach to train text encoders for non-English languages. Our approach does not require image input and primarily uses machine translation, eliminating the need for target language data. Our evaluation across three distinct tasks (image-text retrieval, visual entailment, and natural language visual reasoning) demonstrates that this approach outperforms the state-of-the-art multilingual vision-language models without requiring large parallel corpora. Our code is available at https://github.com/Yasminekaroui/CliCoTea.
BUCA: A Binary Classification Approach to Unsupervised Commonsense Question Answering
Authors: Jie He; Simon U; Victor Gutierrez-Basulto; Jeff Pan
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers))

Abstract: Unsupervised commonsense reasoning (UCR) is becoming increasingly popular as the construction of commonsense reasoning datasets is expensive, and they are inevitably limited in their scope. A popular approach to UCR is to fine-tune language models with external knowledge (e.g., knowledge graphs), but this usually requires a large number of training examples. In this paper, we propose to transform the downstream multiple choice question answering task into a simpler binary classification task by ranking all candidate answers according to their reasonableness. To this end, for training the model, we convert the knowledge graph triples into reasonable and unreasonable texts. Extensive experimental results show the effectiveness of our approach on various multiple choice question answering benchmarks. Furthermore, compared with existing UCR approaches using KGs, ours is less data hungry.
Nichelle and Nancy: The Influence of Demographic Attributes and Tokenization Length on First Name Biases
Authors: Haozhe An; Rachel Rudinger
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers))

Abstract: Through the use of first name substitution experiments, prior research has demonstrated the tendency of social commonsense reasoning models to systematically exhibit social biases along the dimensions of race, ethnicity, and gender (An et al., 2023). Demographic attributes of first names, however, are strongly correlated with corpus frequency and tokenization length, which may influence model behavior independent of or in addition to demographic factors. In this paper, we conduct a new series of first name substitution experiments that measures the influence of these factors while controlling for the others. We find that demographic attributes of a name (race, ethnicity, and gender) and name tokenization length are both factors that systematically affect the behavior of social commonsense reasoning models.
Improving Syntactic Probing Correctness and Robustness with Control Tasks
Authors: Weicheng Ma; Brian Wang; Hefan Zhang; Lili Wang; Rolando Coto-Solano; Saeed Hassanpour; Soroush Vosoughi
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers))

Abstract: Syntactic probing methods have been used to examine whether and how pre-trained language models (PLMs) encode syntactic features. However, the probing methods are usually biased by the PLMs’ memorization of common word co-occurrences, even if they do not form syntactic relations. This paper presents a random-word-substitution and random-label-matching control task to reduce these biases and improve the robustness of syntactic probing methods. Our control tasks are also shown to notably improve the consistency of probing results between different probing methods and make the methods more robust with respect to the text attributes of the probing instances. Our control tasks make syntactic probing methods better at reconstructing syntactic features and more generalizable to unseen text domains. Our experiments show that our proposed control tasks are effective on different PLMs, probing methods, and syntactic features.
Split-NER: Named Entity Recognition via Two Question-Answering-based Classifications
Authors: Jatin Arora; Youngja Park
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers))

Abstract: In this work, we address the NER problem by splitting it into two logical sub-tasks: (1) Span Detection which simply extracts entity mention spans irrespective of entity type; (2) Span Classification which classifies the spans into their entity types. Further, we formulate both sub-tasks as question-answering (QA) problems and produce two leaner models which can be optimized separately for each sub-task. Experiments with four cross-domain datasets demonstrate that this two-step approach is both effective and time efficient. Our system, SplitNER outperforms baselines on OntoNotes5.0, WNUT17 and a cybersecurity dataset and gives on-par performance on BioNLP13CG. In all cases, it achieves a significant reduction in training time compared to its QA baseline counterpart. The effectiveness of our system stems from fine-tuning the BERT model twice, separately for span detection and classification. The source code can be found at https://github.com/c3sr/split-ner.
Credible without Credit: Domain Experts Assess Generative Language Models
Authors: Denis Peskoff; Brandon Stewart
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers))

Abstract: Language models have recently broken into the public consciousness with the release of the wildly popular ChatGPT. Commentators have argued that language models could replace search engines, make college essays obsolete, or even write academic research papers. All of these tasks rely on accuracy of specialized information which can be difficult to assess for non-experts. Using 10 domain experts across science and culture, we provide an initial assessment of the coherence, conciseness, accuracy, and sourcing of two language models across 100 expert-written questions. While we find the results are consistently cohesive and concise, we find that they are mixed in their accuracy. These results raise questions of the role language models should play in general-purpose and expert knowledge seeking.
Grokking of Hierarchical Structure in Vanilla Transformers
Authors: Shikhar Murty; Pratyusha Sharma; Jacob Andreas; Christopher Manning
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers))

Abstract: For humans, language production and comprehension is sensitive to the hierarchical structure of sentences. In natural language processing, past work has questioned how effectively neural sequence models like transformers capture this hierarchical structure when generalizing to structurally novel inputs. We show that transformer language models can learn to generalize hierarchically after training for extremely long periods—far beyond the point when in-domain accuracy has saturated. We call this phenomenon structural grokking. On multiple datasets, structural grokking exhibits inverted U-shaped scaling in model depth: intermediate-depth models generalize better than both very deep and very shallow transformers. When analyzing the relationship between model-internal properties and grokking, we find that optimal depth for grokking can be identified using the tree-structuredness metric of CITATION. Overall, our work provides strong evidence that, with extended training, vanilla transformers discover and use hierarchical structure.
Zero-shot Cross-lingual Transfer With Learned Projections Using Unlabeled Target-Language Data
Authors: Ujan Deb; Ridayesh Parab; Preethi Jyothi
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers))

Abstract: Adapters have emerged as a parameter-efficient Transformer-based framework for cross-lingual transfer by inserting lightweight language-specific modules (language adapters) and task-specific modules (task adapters) within pretrained multilingual models. Zero-shot transfer is enabled by pairing the language adapter in the target language with an appropriate task adapter in a source language. If our target languages are known apriori, we explore how zero-shot transfer can be further improved within the adapter framework by utilizing unlabeled text during task-specific finetuning. We construct language-specific subspaces using standard linear algebra constructs and selectively project source-language representations into the target language subspace during task-specific finetuning using two schemes. Our experiments on three cross-lingual tasks, Named Entity Recognition (NER), Question Answering (QA) and Natural Language Inference (NLI) yield consistent benefits compared to adapter baselines over a wide variety of target languages with up to 11% relative improvement in NER, 2% relative improvement in QA and 5% relative improvement in NLI.
Context-Aware Transformer Pre-Training for Answer Sentence Selection
Authors: Luca Di Liello; Siddhant Garg; Alessandro Moschitti
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers))

Abstract: Answer Sentence Selection (AS2) is a core component for building an accurate Question Answering pipeline. AS2 models rank a set of candidate sentences based on how likely they answer a given question. The state of the art in AS2 exploits pre-trained transformers by transferring them on large annotated datasets, while using local contextual information around the candidate sentence. In this paper, we propose three pre-training objectives designed to mimic the downstream fine-tuning task of contextual AS2. This allows for specializing LMs when fine-tuning for contextual AS2. Our experiments on three public and two large-scale industrial datasets show that our pre-training approaches (applied to RoBERTa and ELECTRA) can improve baseline contextual AS2 accuracy by up to 8% on some datasets.
Toward Expanding the Scope of Radiology Report Summarization to Multiple Anatomies and Modalities
Authors: Zhihong Chen; Maya Varma; Xiang Wan; Curtis Langlotz; Jean-Benoit Delbrouck
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers))

Abstract: Radiology report summarization (RRS) is a growing area of research. Given the Findings section of a radiology report, the goal is to generate a summary (called an Impression section) that highlights the key observations and conclusions of the radiology study. However, RRS currently faces essential limitations. First, many prior studies conduct experiments on private datasets, preventing reproduction of results and fair comparisons across different systems and solutions. Second, most prior approaches are evaluated solely on chest X-rays. To address these limitations, we propose a dataset (MIMIC-RRS) involving three new modalities and seven new anatomies based on the MIMIC-III and MIMIC-CXR datasets. We then conduct extensive experiments to evaluate the performance of models both within and across modality-anatomy pairs in MIMIC-RRS. In addition, we evaluate their clinical efficacy via RadGraph, a factual correctness metric.
Efficient Diagnosis Assignment Using Unstructured Clinical Notes
Authors: Louis Blankemeier; Jason Fries; Robert Tinn; Joseph Preston; Nigam Shah; Akshay Chaudhari
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers))

Abstract: Electronic phenotyping entails using electronic health records (EHRs) to identify patients with specific health outcomes and determine when those outcomes occurred. Unstructured clinical notes, which contain a vast amount of information, are a valuable resource for electronic phenotyping. However, traditional methods, such as rule-based labeling functions or neural networks, require significant manual effort to tune and may not generalize well to multiple indications. To address these challenges, we propose HyDE (hybrid diagnosis extractor). HyDE is a simple framework for electronic phenotyping that integrates labeling functions and a disease-agnostic neural network to assign diagnoses to patients. By training HyDE’s model to correct predictions made by labeling functions, we are able to disambiguate hypertension true positives and false positives with a supervised area under the precision-recall curve (AUPRC) of 0.85. We extend this hypertension-trained model to zero-shot evaluation of four other diseases, generating AUPRC values ranging from 0.82 - 0.95 and outperforming a labeling function baseline by 44 points in F1 score and a Word2Vec baseline by 24 points in F1 score on average. Furthermore, we demonstrate a speedup of >4x by pruning the length of inputs into our language model to ~2.3% of the full clinical notes, with negligible impact to the AUPRC. HyDE has the potential to improve the efficiency and efficacy of interpreting large-scale unstructured clinical notes for accurate EHR phenotyping.
MetaVL: Transferring In-Context Learning Ability From Language Models to Vision-Language Models
Authors: Masoud Monajatipoor; Liunian Harold Li; Mozhdeh Rouhsedaghat; Lin Yang; Kai-Wei Chang
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers))

Abstract: Large-scale language models have shown the ability to adapt to a new task via conditioning on a few demonstrations (i.e., in-context learning). However, in the vision-language domain, most large-scale pre-trained vision-language (VL) models do not possess the ability to conduct in-context learning. How can we enable in-context learning for VL models? In this paper, we study an interesting hypothesis: can we transfer the in-context learning ability from the language domain to the VL domain? Specifically, we first meta-trains a language model to perform in-context learning on NLP tasks (as in MetaICL); then we transfer this model to perform VL tasks by attaching a visual encoder. Our experiments suggest that indeed in-context learning ability can be transferred cross modalities: our model considerably improves the in-context learning capability on VL tasks and can even compensate for the size of the model significantly. On VQA, OK-VQA, and GQA, our method could outperform the baseline model while having ~20 times fewer parameters.
On the Interpretability and Significance of Bias Metrics in Texts: a PMI-based Approach
Authors: Francisco Valentini; Germán Rosati; Damián Blasi; Diego Fernandez Slezak; Edgar Altszyler
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers))

Abstract: In recent years, word embeddings have been widely used to measure biases in texts. Even if they have proven to be effective in detecting a wide variety of biases, metrics based on word embeddings lack transparency and interpretability. We analyze an alternative PMI-based metric to quantify biases in texts. It can be expressed as a function of conditional probabilities, which provides a simple interpretation in terms of word co-occurrences. We also prove that it can be approximated by an odds ratio, which allows estimating confidence intervals and statistical significance of textual biases. This approach produces similar results to metrics based on word embeddings when capturing gender gaps of the real world embedded in large corpora.