GenAI4ELab/papercli-papers huggingface dataset:
Implications of Annotation Artifacts in Edge Probing Test Datasets
Authors: Sagnik Ray Choudhury; Jushaan Kalra
Venue: EMNLP 2023 (Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL))

Abstract: Edge probing tests are classification tasks that test for grammatical knowledge encoded in token representations coming from contextual encoders such as large language models (LLMs). Many LLM encoders have shown high performance in EP tests, leading to conjectures about their ability to encode linguistic knowledge. However, a large body of research claims that the tests necessarily do not measure the LLM’s capacity to encode knowledge, but rather reflect the classifiers’ ability to learn the problem. Much of this criticism stems from the fact that often the classifiers have very similar accuracy when an LLM vs a random encoder is used. Consequently, several modifications to the tests have been suggested, including information theoretic probes. We show that commonly used edge probing test datasets have various biases including memorization. When these biases are removed, the LLM encoders do show a significant difference from the random ones, even with the simple non-information theoretic probes.
REFER: An End-to-end Rationale Extraction Framework for Explanation Regularization
Authors: Mohammad Reza Ghasemi Madani; Pasquale Minervini
Venue: EMNLP 2023 (Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL))

Abstract: Human-annotated textual explanations are becoming increasingly important in Explainable Natural Language Processing. Rationale extraction aims to provide faithful (i.e. reflective of the behavior of the model) and plausible (i.e. convincing to humans) explanations by highlighting the inputs that had the largest impact on the prediction without compromising the performance of the task model. In recent works, the focus of training rationale extractors was primarily on optimizing for plausibility using human highlights, while the task model was trained on jointly optimizing for task predictive accuracy and faithfulness. We propose REFER, a framework that employs a differentiable rationale extractor that allows to back-propagate through the rationale extraction process. We analyze the impact of using human highlights during training by jointly training the task model and the rationale extractor. In our experiments, REFER yields significantly better results in terms of faithfulness, plausibility, and downstream task accuracy on both in-distribution and out-of-distribution data. On both e-SNLI and CoS-E, our best setting produces better results in terms of composite normalized relative gain than the previous baselines by 11% and 3%, respectively.
Findings of the Second Shared Task on Multilingual Coreference Resolution
Authors: Zdeněk Žabokrtský; Miloslav Konopik; Anna Nedoluzhko; Michal Novák; Maciej Ogrodniczuk; Martin Popel; Ondrej Prazak; Jakub Sido; Daniel Zeman
Venue: EMNLP 2023 (Proceedings of the CRAC 2023 Shared Task on Multilingual Coreference Resolution)

Abstract: This paper summarizes the second edition of the shared task on multilingual coreference resolution, held with the CRAC 2023 workshop. Just like last year, participants of the shared task were to create trainable systems that detect mentions and group them based on identity coreference; however, this year’s edition uses a slightly different primary evaluation score, and is also broader in terms of covered languages: version 1.1 of the multilingual collection of harmonized coreference resources CorefUD was used as the source of training and evaluation data this time, with 17 datasets for 12 languages. 7 systems competed in this shared task.
Multilingual coreference resolution: Adapt and Generate
Authors: Natalia Skachkova; Tatiana Anikina; Anna Mokhova
Venue: EMNLP 2023 (Proceedings of the CRAC 2023 Shared Task on Multilingual Coreference Resolution)

Abstract: The paper presents two multilingual coreference resolution systems submitted for the CRAC Shared Task 2023. The DFKI-Adapt system achieves 61.86 F1 score on the shared task test data, outperforming the official baseline by 4.9 F1 points. This system uses a combination of different features and training settings, including character embeddings, adapter modules, joint pre-training and loss-based re-training. We provide evaluation for each of the settings on 12 different datasets and compare the results. The other submission DFKI-MPrompt uses a novel approach that involves prompting for mention generation. Although the scores achieved by this model are lower compared to the baseline, the method shows a new way of approaching the coreference task and provides good results with just five epochs of training.
Neural End-to-End Coreference Resolution using Morphological Information
Authors: Tuğba Pamay Arslan; Kutay Acar; Gülşen Eryiğit
Venue: EMNLP 2023 (Proceedings of the CRAC 2023 Shared Task on Multilingual Coreference Resolution)

Abstract: In morphologically rich languages, words consist of morphemes containing deeper information in morphology, and thus such languages may necessitate the use of morpheme-level representations as well as word representations. This study introduces a neural multilingual end-to-end coreference resolution system by incorporating morphological information in transformer-based word embeddings on the baseline model. This proposed model participated in the Sixth Workshop on Computational Models of Reference, Anaphora and Coreference (CRAC 2023). Including morphological information explicitly into the coreference resolution improves the performance, especially in morphologically rich languages (e.g., Catalan, Hungarian, and Turkish). The introduced model outperforms the baseline system by 2.57 percentage points on average by obtaining 59.53% CoNLL F-score.
ÚFAL CorPipe at CRAC 2023: Larger Context Improves Multilingual Coreference Resolution
Authors: Milan Straka
Venue: EMNLP 2023 (Proceedings of the CRAC 2023 Shared Task on Multilingual Coreference Resolution)

Abstract: We present CorPipe, the winning entry to the CRAC 2023 Shared Task on Multilingual Coreference Resolution. Our system is an improved version of our earlier multilingual coreference pipeline, and it surpasses other participants by a large margin of 4.5 percent points. CorPipe first performs mention detection, followed by coreference linking via an antecedent-maximization approach on the retrieved spans. Both tasks are trained jointly on all available corpora using a shared pretrained language model. Our main improvements comprise inputs larger than 512 subwords and changing the mention decoding to support ensembling. The source code is available at https://github.com/ufal/crac2023-corpipe.
McGill at CRAC 2023: Multilingual Generalization of Entity-Ranking Coreference Resolution Models
Authors: Ian Porada; Jackie Chi Kit Cheung
Venue: EMNLP 2023 (Proceedings of the CRAC 2023 Shared Task on Multilingual Coreference Resolution)

Abstract: Our submission to the CRAC 2023 shared task, described herein, is an adapted entity-ranking model jointly trained on all 17 datasets spanning 12 languages. Our model outperforms the shared task baselines by a difference in F1 score of +8.47, achieving an ultimate F1 score of 65.43 and fourth place in the shared task. We explore design decisions related to data preprocessing, the pretrained encoder, and data mixing.
Contextualizing the Limits of Model & Evaluation Dataset Curation on Semantic Similarity Classification Tasks
Authors: Daniel Theron
Venue: EMNLP 2023 (Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM))

Abstract: This paper demonstrates how the limitations of pre-trained models and open evaluation datasets factor into assessing the performance of binary semantic similarity classification tasks. As (1) end-user-facing documentation around the curation of these datasets and pre-trained model training regimes is often not easily accessible and (2) given the lower friction and higher demand to quickly deploy such systems in real-world contexts, our study reinforces prior work showing performance disparities across datasets, embedding techniques and distance metrics, while highlighting the importance of understanding how data is collected, curated and analyzed in semantic similarity classification.
Dialogue Quality and Emotion Annotations for Customer Support Conversations
Authors: John Mendonca; Patrícia Pereira; Miguel Menezes; Vera Cabarrão; Ana C Farinha; Helena Moniz; Alon Lavie; Isabel Trancoso
Venue: EMNLP 2023 (Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM))

Abstract: Task-oriented conversational datasets often lack topic variability and linguistic diversity. However, with the advent of Large Language Models (LLMs) pretrained on extensive, multilingual and diverse text data, these limitations seem overcome. Nevertheless, their generalisability to different languages and domains in dialogue applications remains uncertain without benchmarking datasets. This paper presents a holistic annotation approach for emotion and conversational quality in the context of bilingual customer support conversations. By performing annotations that take into consideration the complete instances that compose a conversation, one can form a broader perspective of the dialogue as a whole. Furthermore, it provides a unique and valuable resource for the development of text classification models. To this end, we present benchmarks for Emotion Recognition and Dialogue Quality Estimation and show that further research is needed to leverage these models in a production setting.
Formalizing content creation and evaluation methods for AI-generated social media content
Authors: Christian Jensen; Axel Højmark
Venue: EMNLP 2023 (Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM))

Abstract: This study explores the use of large language models (LLMs), such as ChatGPT and GPT-4, in creating high-quality text-based social media content for businesses on LinkedIn. We introduce a novel architecture incorporating external knowledge bases and a multi-step writing approach, which extracts facts from company websites to form a knowledge graph. Our method’s efficacy is assessed using the “Long-LinkedIn” evaluation dataset designed for long-form post generation. Results indicate that our iterative refinement significantly improves content quality. However, knowledge-enhanced prompts occasionally reduced quality due to potential formulation issues. LLM-based evaluations, particularly using ChatGPT, showcased potential as a less resource-intensive alternative to human assessments, with a notable alignment between the two evaluation techniques.
Automatic Evaluation of Generative Models with Instruction Tuning
Authors: Shuhaib Mehri; Vered Shwartz
Venue: EMNLP 2023 (Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM))

Abstract: Automatic evaluation of natural language generation has long been an elusive goal in NLP. A recent paradigm fine-tunes pre-trained language models to emulate human judgements for a particular task and evaluation criterion. Inspired by the generalization ability of instruction-tuned models, we propose a learned metric based on instruction tuning. To test our approach, we collected HEAP, a dataset of human judgements across various NLG tasks and evaluation criteria. Our findings demonstrate that instruction tuning language models on HEAP yields good performance on many evaluation tasks, though some criteria are less trivial to learn than others. Further, jointly training on multiple tasks can yield additional performance improvements, which can be beneficial for future tasks with little to no human annotated data.
Effective Proxy for Human Labeling: Ensemble Disagreement Scores in Large Language Models for Industrial NLP
Authors: Wei Du; Laksh Advani; Yashmeet Gambhir; Daniel Perry; Prashant Shiralkar; Zhengzheng Xing; Aaron Colak
Venue: EMNLP 2023 (Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM))

Abstract: Large language models (LLMs) have demonstrated significant capability to generalize across a large number of NLP tasks. For industry applications, it is imperative to assess the performance of the LLM on unlabeled production data from time to time to validate for a real-world setting. Human labeling to assess model error requires considerable expense and time delay. Here we demonstrate that ensemble disagreement scores work well as a proxy for human labeling for language models in zero-shot, few-shot, and fine-tuned settings, per our evaluation on keyphrase extraction (KPE) task. We measure fidelity of the results by comparing to true error measured from human labeled ground truth. We contrast with the alternative of using another LLM as a source of machine labels, or ‘silver labels’. Results across various languages and domains show disagreement scores provide a better estimation of model performance with mean average error (MAE) as low as 0.4% and on average 13.8% better than using silver labels.
Automatic Reflection Generation for Peer-to-Peer Counseling
Authors: Emma O’neil; João Sedoc; Diyi Yang; Haiyi Zhu; Lyle Ungar
Venue: EMNLP 2023 (Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM))

Abstract: Online peer counseling platforms enable conversations between millions of people seeking and offering mental health support. Among counseling skills, reflective listening, i.e., capturing and returning to the client something the client has said, is important for positive therapeutic outcomes. We introduce a reflection generation system for online mental health support conversations leveraging GPT-3, a large language model. We compare few-shot learning against fine-tuning and assess the impact of the quality of training examples as measured by fluency, reflection resemblance, and overall preference. Fine-tuned GPT-3 generates responses that human evaluators rate as comparable in reflection quality to responses used for tuning. Models based on high-quality responses generate substantially better reflections than ones tuned on actual responses from a large online counseling service–and better reflections than the actual counselor responses. These results suggest the care needed in selecting examples for tuning generative models.
One-Shot and Few-Shot Exemplification Modeling
Authors: John Harvill; Hee Suk Yoon; Eunseop Yoon; Mark Hasegawa-Johnson; Chang Yoo
Venue: EMNLP 2023 (Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM))

Abstract: Exemplification modeling is a task where the goal is to produce a viable example sentence that uses a target word with a target definition. The task is non-trivial for polysemous words, and previous works have only explored settings where ample labeled training data is available. In this paper, we demonstrate that exemplification modeling can be performed without a large labeled training corpus by either changing the format of the task (one-shot) or prompting large language models (few-shot), and ablate key components of our proposed one-shot and few-shot systems. We provide extensive automatic and human evaluations of model performance and find that our proposed one-shot and few-shot approaches perform similarly to a fully supervised baseline. We compare and contrast each method in terms of labeled training dataset size, performance, and model size, and find that each technique has at least one tradeoff that another approach does not.
Leveraging Large Language Models for Enhanced Product Descriptions in eCommerce
Authors: Jianghong Zhou; Bo Liu; Jhalak Acharya; Yao Hong; Kuang-Chih Lee; Musen Wen
Venue: EMNLP 2023 (Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM))

Abstract: In the dynamic field of eCommerce, the quality and comprehensiveness of product descriptions are pivotal for enhancing search visibility and customer engagement. Effective product descriptions can address the ‘cold start’ problem, align with market trends, and ultimately lead to increased click-through rates. Traditional methods for crafting these descriptions often involve significant human effort and may lack both consistency and scalability. This paper introduces a novel methodology for automating product description generation using the LLAMA 2.0 7B language model. We train the model on a dataset of authentic product descriptions from Walmart, one of the largest eCommerce platforms. The model is then fine-tuned for domain-specific language features and eCommerce nuances to enhance its utility in sales and user engagement. We employ multiple evaluation metrics—including NDCG, customer click-through rates, and human assessments—to validate the effectiveness of our approach. Our findings reveal that the system is not only scalable but also significantly reduces the human workload involved in creating product descriptions. This study underscores the considerable potential of large language models like LLAMA 2.0 7B in automating and optimizing various facets of eCommerce platforms, offering significant business impact, including improved search functionality and increased sales.
QAMPARI: A Benchmark for Open-domain Questions with Many Answers
Authors: Samuel Amouyal; Tomer Wolfson; Ohad Rubin; Ori Yoran; Jonathan Herzig; Jonathan Berant
Venue: EMNLP 2023 (Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM))

Abstract: Existing benchmarks for open-domain question answering (ODQA) typically focus on questions whose answers are all in a single paragraph. By contrast, many natural questions, such as “What players were drafted by the Brooklyn Nets?” have a long list of answers extracted from multiple paragraphs. Answering such questions requires retrieving and reading many passages from a large corpus. We introduce QAMPARI, an ODQA benchmark, where answers are lists of entities, spread across many paragraphs. We created QAMPARI by (a) generating questions with multiple answers from Wikipedia’s knowledge graph and tables, (b) automatically pairing answers with supporting evidence in Wikipedia paragraphs, and (c) manually paraphrasing questions and validating each answer. Across a wide range of ODQA models, we find that QAMPARI is challenging in terms of both passage retrieval and answer generation, with models reaching an F1 score of 32.8 at best. We view QAMPARI as a valuable resource for ODQA research, which will aid to develop models that handle a broad range of question types, including single and multi-answer questions.
Unveiling Safety Vulnerabilities of Large Language Models
Authors: George Kour; Marcel Zalmanovici; Naama Zwerdling; Esther Goldbraich; Ora Fandina; Ateret Anaby Tavor; Orna Raz; Eitan Farchi
Venue: EMNLP 2023 (Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM))

Abstract: As large language models become more prevalent, their possible harmful or inappropriate responses are a cause for concern. This paper introduces a unique dataset containing adversarial examples in the form of questions, we call AttaQ, designed to provoke such harmful or inappropriate responses. We assess the efficacy of our dataset by analyzing the vulnerabilities of various models when subjected to it. Additionally, we introduce a novel automatic approach for identifying and naming vulnerable semantic regions — input semantic areas for which the model is likely to produce harmful outputs. This is achieved through the application of specialized clustering techniques that consider both the semantic similarity of the input attacks and the harmfulness of the model’s responses.Automatically identifying vulnerable semantic regions enhances the evaluation of model weaknesses, facilitating targeted improvements to its safety mechanisms and overall reliability.
Adapting Pre-trained Generative Models for Extractive Question Answering
Authors: Prabir Mallick; Tapas Nayak; Indrajit Bhattacharya
Venue: EMNLP 2023 (Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM))

Abstract: Pre-trained Generative models such as BART, T5, etc. have gained prominence as a preferred method for text generation in various natural language processing tasks, including abstractive long-form question answering (QA) and summarization. However, the potential of generative models in extractive QA tasks, where discriminative models are commonly employed, remains largely unexplored. Discriminative models often encounter challenges associated with label sparsity, particularly when only a small portion of the context contains the answer. The challenge is more pronounced for multi-span answers. In this work, we introduce a novel approach that uses the power of pre-trained generative models to address extractive QA tasks by generating indexes corresponding to context tokens or sentences that form part of the answer. Through comprehensive evaluations on multiple extractive QA datasets, including MultiSpanQA, BioASQ, MASHQA, and WikiQA, we demonstrate the superior performance of our proposed approach compared to existing state-of-the-art models.
Predicting Question-Answering Performance of Large Language Models through Semantic Consistency
Authors: Ella Rabinovich; Samuel Ackerman; Orna Raz; Eitan Farchi; Ateret Anaby Tavor
Venue: EMNLP 2023 (Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM))

Abstract: Semantic consistency of a language model is broadly defined as the model’s ability to produce semantically-equivalent outputs, given semantically-equivalent inputs. We address the task of assessing question-answering (QA) semantic consistency of contemporary large language models (LLMs) by manually creating a benchmark dataset with high-quality paraphrases for factual questions, and release the dataset to the community.We further combine the semantic consistency metric with additional measurements suggested in prior work as correlating with LLM QA accuracy, for building and evaluating a framework for factual QA reference-less performance prediction – predicting the likelihood of a language model to accurately answer a question. Evaluating the framework on five contemporary LLMs, we demonstrate encouraging, significantly outperforming baselines, results.
Towards Effective Long-Form QA with Evidence Augmentation
Authors: Mengxia Yu; Sara Rosenthal; Mihaela Bornea; Avi Sil
Venue: EMNLP 2023 (Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM))

Abstract: In this study, we focus on the challenge of improving Long-form Question Answering (LFQA) by extracting and effectively utilizing knowledge from a large set of retrieved passages. We first demonstrate the importance of accurate evidence retrieval for LFQA, showing that optimal extracted knowledge from passages significantly benefits the generation. We also show that the choice of generative models impacts the system’s ability to leverage the evidence and produce answers that are grounded in the retrieved passages. We propose a Mixture of Experts (MoE) model as an alternative to the Fusion in Decoder (FiD) used in state-of-the-art LFQA systems and we compare these two models in our experiments.