GenAI4ELab/papercli-papers huggingface dataset:
Exploring Retrieval Augmented Generation For Real-world Claim Verification
Authors: Omar Adjali
Venue: EMNLP 2024 (Proceedings of the Seventh Fact Extraction and VERification Workshop (FEVER))

Abstract: Automated Fact-Checking (AFC) has recently gained considerable attention to address the increasing misinformation spreading in the web and social media. The recently introduced AVeriTeC dataset alleviates some limitations of existing AFC benchmarks. In this paper, we propose to explore Retrieval Augmented Generation (RAG) and describe the system (UPS participant) we implemented to solve the AVeriTeC shared task.Our end-to-end system integrates retrieval and generation in a joint training setup to enhance evidence retrieval and question generation. Our system operates as follows: First, we conduct dense retrieval of evidence by encoding candidate evidence sentences from the provided knowledge store documents. Next, we perform a secondary retrieval of question-answer pairs from the training set, encoding these into dense vectors to support question generation with relevant in-context examples. During training, the question generator is optimized to generate questions based on retrieved or gold evidence. In preliminary automatic evaluation, our system achieved respectively 0.198 and 0.210 AVeriTeC scores on the dev and test sets.
GProofT: A Multi-dimension Multi-round Fact Checking Framework Based on Claim Fact Extraction
Authors: Jiayu Liu; Junhao Tang; Hanwen Wang; Baixuan Xu; Haochen Shi; Weiqi Wang; Yangqiu Song
Venue: EMNLP 2024 (Proceedings of the Seventh Fact Extraction and VERification Workshop (FEVER))

Abstract: In the information era, the vast proliferation of online content poses significant challenges, particularly concerning the trustworthiness of these digital statements, which can have profound societal implications. Although it is possible to manually annotate and verify the authenticity of such content, the sheer volume and rapid pace of information generation render this approach impractical, both in terms of time and cost. Therefore, it is imperative to develop automated systems capable of validating online claims, ensuring that users can use the wealth of information available on the Internet effectively and reliably. Using primarily ChatGPT and the Google search API, GProofT fact checking framework generates question-answer pairs to systematically extract and verify the facts within claims. Based on the outcomes of these QA pairs, claims are subsequently labeled as Supported, Conflicted Evidence/Cherry-Picking, or Refuted. Shown by extensive experiments, GProofT Retrieval generally performs effectively in fact-checking and makes a substantial contribution to the task. Our code is released on https://github.com/HKUST-KnowComp/GProofT.
HerO at AVeriTeC: The Herd of Open Large Language Models for Verifying Real-World Claims
Authors: Yejun Yoon; Jaeyoon Jung; Seunghyun Yoon; Kunwoo Park
Venue: EMNLP 2024 (Proceedings of the Seventh Fact Extraction and VERification Workshop (FEVER))

Abstract: To tackle the AVeriTeC shared task hosted by the FEVER-24, we introduce a system that only employs publicly available large language models (LLMs) for each step of automated fact-checking, dubbed the Herd of Open LLMs for verifying real-world claims (HerO). HerO employs multiple LLMs for each step of automated fact-checking. For evidence retrieval, a language model is used to enhance a query by generating hypothetical documents that check the veracity of a claim. We fine-tune LLMs for question generation and veracity prediction by crafting prompts with retrieved in-context samples. HerO achieved 2nd place on the leaderboard with the AVeriTeC score of 0.57, suggesting the potential of open LLMs for verifying real-world claims. For future research, we make our code publicly available at https://github.com/ssu-humane/HerO.
AIC CTU system at AVeriTeC: Re-framing automated fact-checking as a simple RAG task
Authors: Herbert Ullrich; Tomáš Mlynář; Jan Drchal
Venue: EMNLP 2024 (Proceedings of the Seventh Fact Extraction and VERification Workshop (FEVER))

Abstract: This paper describes our 3rd place submission in the AVeriTeC shared task in which we attempted to address the challenge of fact-checking with evidence retrieved in the wild using a simple scheme of Retrieval-Augmented Generation (RAG) designed for the task, leveraging the predictive power of Large Language Models.We release our codebase and explain its two modules - the Retriever and the Evidence & Label generator - in detail, justifying their features such as MMR-reranking and Likert-scale confidence estimation.We evaluate our solution on AVeriTeC dev and test set and interpret the results, picking the GPT-4o as the most appropriate model for our pipeline at the time of our publication, with Llama 3.1 70B being a promising open-source alternative.We perform an empirical error analysis to see that faults in our predictions often coincide with noise in the data or ambiguous fact-checks, provoking further research and data augmentation.
Enhancing Fact Verification with Causal Knowledge Graphs and Transformer-Based Retrieval for Deductive Reasoning
Authors: Fiona Anting Tan; Jay Desai; Srinivasan H. Sengamedu
Venue: EMNLP 2024 (Proceedings of the Seventh Fact Extraction and VERification Workshop (FEVER))

Abstract: The ability to extract and verify factual information from free-form text is critical in an era where vast amounts of unstructured data are available, yet unreliable sources abound. This paper focuses on enhancing causal deductive reasoning, a key component of factual verification, through the lens of accident investigation, where determining the probable causes of events is paramount. Deductive reasoning refers to the task of drawing conclusions based on a premise. While some deductive reasoning benchmarks exist, none focus on causal deductive reasoning and are from real-world applications. Recently, large language models (LLMs) used with prompt engineering techniques like retrieval-augmented generation (RAG) have demonstrated remarkable performance across various natural language processing benchmarks. However, adapting these techniques to handle scenarios with no knowledge bases and to different data structures, such as graphs, remains an ongoing challenge. In our study, we introduce a novel framework leveraging LLMs’ decent ability to detect and infer causal relations to construct a causal Knowledge Graph (KG) which represents knowledge that the LLM recognizes. Additionally, we propose a RoBERTa-based Transformer Graph Neural Network (RoTG) specifically designed to select relevant nodes within this KG. Integrating RoTG-retrieved causal chains into prompts effectively enhances LLM performance, demonstrating usefulness of our approach in advancing LLMs’ causal deductive reasoning capabilities.
Numerical Claim Detection in Finance: A New Financial Dataset, Weak-Supervision Model, and Market Analysis
Authors: Agam Shah; Arnav Hiray; Pratvi Shah; Arkaprabha Banerjee; Anushka Singh; Dheeraj Deepak Eidnani; Sahasra Chava; Bhaskar Chaudhury; Sudheer Chava
Venue: EMNLP 2024 (Proceedings of the Seventh Fact Extraction and VERification Workshop (FEVER))

Abstract: In this paper, we investigate the influence of claims in analyst reports and earnings calls on financial market returns, considering them as significant quarterly events for publicly traded companies. To facilitate a comprehensive analysis, we construct a new financial dataset for the claim detection task in the financial domain. We benchmark various language models on this dataset and propose a novel weak-supervision model that incorporates the knowledge of subject matter experts (SMEs) in the aggregation function, outperforming existing approaches. We also demonstrate the practical utility of our proposed model by constructing a novel measure of *optimism*. Here, we observe the dependence of earnings surprise and return on our optimism measure. Our dataset, models, and code are publicly (under CC BY 4.0 license) available on GitHub.
Streamlining Conformal Information Retrieval via Score Refinement
Authors: Yotam Intrator; Regev Cohen; Ori Kelner; Roman Goldenberg; Ehud Rivlin; Daniel Freedman
Venue: EMNLP 2024 (Proceedings of the Seventh Fact Extraction and VERification Workshop (FEVER))

Abstract: Information retrieval (IR) methods, like retrieval augmented generation, are fundamental to modern applications but often lack statistical guarantees. Conformal prediction addresses this by retrieving sets guaranteed to include relevant information, yet existing approaches produce large-sized sets, incurring high computational costs and slow response times. In this work, we introduce a score refinement method that applies a simple monotone transformation to retrieval scores, leading to significantly smaller conformal sets while maintaining their statistical guarantees. Experiments on various BEIR benchmarks validate the effectiveness of our approach in producing compact sets containing relevant information.
Improving Explainable Fact-Checking via Sentence-Level Factual Reasoning
Authors: Francielle Vargas; Isadora Salles; Diego Alves; Ameeta Agrawal; Thiago A. S. Pardo; Fabrício Benevenuto
Venue: EMNLP 2024 (Proceedings of the Seventh Fact Extraction and VERification Workshop (FEVER))

Abstract: Most existing fact-checking systems are unable to explain their decisions by providing relevant rationales (justifications) for their predictions. It highlights a lack of transparency that poses significant risks, such as the prevalence of unexpected biases, which may increase political polarization due to limitations in impartiality. To address this critical gap, we introduce SEntence-Level FActual Reasoning (SELFAR), aimed at improving explainable fact-checking. SELFAR relies on fact extraction and verification by predicting the news source reliability and factuality (veracity) of news articles or claims at the sentence level, generating post-hoc explanations using SHAP/LIME and zero-shot prompts. Our experiments show that unreliable news stories predominantly consist of subjective statements, in contrast to reliable ones. Consequently, predicting unreliable news articles at the sentence level by analyzing impartiality and subjectivity is a promising approach for fact extraction and improving explainable fact-checking. Furthermore, LIME outperforms SHAP in explaining predictions on reliability. Additionally, while zero-shot prompts provide highly readable explanations and achieve an accuracy of 0.71 in predicting factuality, their tendency to hallucinate remains a challenge. Lastly, this paper also presents the first study on explainable fact-checking in the Portuguese language.
Fast Evidence Extraction for Grounded Language Model Outputs
Authors: Pranav Mani; Davis Liang; Zachary Chase Lipton
Venue: EMNLP 2024 (Proceedings of the Seventh Fact Extraction and VERification Workshop (FEVER))

Abstract: Summarizing documents with Large Language Models (LLMs) warrants a rigorous inspection of the resulting outputs by humans. However, unaided verification of generated outputs is time-intensive and intractable at scale. For high-stakes applications like healthcare where verification is necessary, expediting this step can unlock massive gains in productivity. In this paper, we focus on the task of evidence extraction for abstractive summarization: for each summary line, extract the corresponding evidence spans from a source document. Viewing this evidence extraction problem through the lens of extractive question answering, we train a set of fast and scalable hierarchical architectures: EarlyFusion, MidFusion, and LateFusion. Our experiments show that (i) our method outperforms the state-of-the-art by 1.4% relative F1-Score; (ii) our model architecture reduces latency by 4x over a RoBERTa-Large baseline; and (iii) pretraining on an extractive QA corpus confers positive transfer to evidence extraction, especially in low-resource regimes.
Question-Based Retrieval using Atomic Units for Enterprise RAG
Authors: Vatsal Raina; Mark Gales
Venue: EMNLP 2024 (Proceedings of the Seventh Fact Extraction and VERification Workshop (FEVER))

Abstract: Enterprise retrieval augmented generation (RAG) offers a highly flexible framework for combining powerful large language models (LLMs) with internal, possibly temporally changing, documents. In RAG, documents are first chunked. Relevant chunks are then retrieved for a user query, which are passed as context to a synthesizer LLM to generate the query response. However, the retrieval step can limit performance, as incorrect chunks can lead the synthesizer LLM to generate a false response. This work applies a zero-shot adaptation of standard dense retrieval steps for more accurate chunk recall. Specifically, a chunk is first decomposed into atomic statements. A set of synthetic questions are then generated on these atoms (with the chunk as the context). Dense retrieval involves finding the closest set of synthetic questions, and associated chunks, to the user query. It is found that retrieval with the atoms leads to higher recall than retrieval with chunks. Further performance gain is observed with retrieval using the synthetic questions generated over the atoms. Higher recall at the retrieval step enables higher performance of the enterprise LLM using the RAG pipeline.
AMREx: AMR for Explainable Fact Verification
Authors: Chathuri Jayaweera; Sangpil Youm; Bonnie J Dorr
Venue: EMNLP 2024 (Proceedings of the Seventh Fact Extraction and VERification Workshop (FEVER))

Abstract: With the advent of social media networks and the vast amount of information circulating through them, automatic fact verification is an essential component to prevent the spread of misinformation. It is even more useful to have fact verification systems that provide explanations along with their classifications to ensure accurate predictions. To address both of these requirements, we implement AMREx, an Abstract Meaning Representation (AMR)-based veracity prediction and explanation system for fact verification using a combination of Smatch, an AMR evaluation metric to measure meaning containment and textual similarity, and demonstrate its effectiveness in producing partially explainable justifications using two community standard fact verification datasets, FEVER and AVeriTeC. AMREx surpasses the AVeriTec baseline accuracy showing the effectiveness of our approach for real-world claim verification. It follows an interpretable pipeline and returns an explainable AMR node mapping to clarify the system’s veracity predictions when applicable. We further demonstrate that AMREx output can be used to prompt LLMs to generate natural-language explanations using the AMR mappings as a guide to lessen the probability of hallucinations.
Claim Check-Worthiness Detection: How Well do LLMs Grasp Annotation Guidelines?
Authors: Laura Majer; Jan Šnajder
Venue: EMNLP 2024 (Proceedings of the Seventh Fact Extraction and VERification Workshop (FEVER))

Abstract: The rising threat of disinformation underscores the need to fully or partially automate the fact-checking process. Identifying text segments requiring fact-checking is known as claim detection (CD) and claim check-worthiness detection (CW), the latter incorporating complex domain-specific criteria of worthiness and often framed as a ranking task. Zero- and few-shot LLM prompting is an attractive option for both tasks, as it bypasses the need for labeled datasets and allows verbalized claim and worthiness criteria to be directly used for prompting. We evaluate the LLMs’ predictive accuracy on five CD/CW datasets from diverse domains, using corresponding annotation guidelines in prompts. We examine two key aspects: (1) how to best distill factuality and worthiness criteria into a prompt, and (2) how much context to provide for each claim. To this end, we experiment with different levels of prompt verbosity and varying amounts of contextual information given to the model. We additionally evaluate the top-performing models with ranking metrics, resembling prioritization done by fact-checkers. Our results show that optimal prompt verbosity varies, meta-data alone adds more performance boost than co-text, and confidence scores can be directly used to produce reliable check-worthiness rankings.
Contrastive Learning to Improve Retrieval for Real-World Fact Checking
Authors: Aniruddh Sriram; Fangyuan Xu; Eunsol Choi; Greg Durrett
Venue: EMNLP 2024 (Proceedings of the Seventh Fact Extraction and VERification Workshop (FEVER))

Abstract: Recent work on fact-checking addresses a realistic setting where models incorporate evidence retrieved from the web to decide the veracity of claims. A bottleneck in this pipeline is in retrieving relevant evidence: traditional methods may surface documents directly related to a claim, but fact-checking complex claims requires more inferences. For instance, a document about how a vaccine was developed is relevant to addressing claims about what it might contain, even if it does not address them directly. We present Contrastive Fact-Checking Reranker (CFR), an improved retriever for this setting. By leveraging the AVeriTeC dataset, which annotates subquestions for claims with human written answers from evidence documents, we fine-tune Contriever with a contrastive objective based on multiple training signals, including distillation from GPT-4, evaluating subquestion answers, and gold labels in the dataset. We evaluate our model on both retrieval and end-to-end veracity judgments about claims. On the AVeriTeC dataset, we find a 6% improvement in veracity classification accuracy. We also show our gains can be transferred to FEVER, ClaimDecomp, HotpotQA, and a synthetic dataset requiring retrievers to make inferences.
RAGAR, Your Falsehood Radar: RAG-Augmented Reasoning for Political Fact-Checking using Multimodal Large Language Models
Authors: Mohammed Abdul Khaliq; Paul Yu-Chun Chang; Mingyang Ma; Bernhard Pflugfelder; Filip Miletić
Venue: EMNLP 2024 (Proceedings of the Seventh Fact Extraction and VERification Workshop (FEVER))

Abstract: The escalating challenge of misinformation, particularly in political discourse, requires advanced fact-checking solutions; this is even clearer in the more complex scenario of multimodal claims. We tackle this issue using a multimodal large language model in conjunction with retrieval-augmented generation (RAG), and introduce two novel reasoning techniques: Chain of RAG (CoRAG) and Tree of RAG (ToRAG). They fact-check multimodal claims by extracting both textual and image content, retrieving external information, and reasoning subsequent questions to be answered based on prior evidence. We achieve a weighted F1-score of 0.85, surpassing a baseline reasoning technique by 0.14 points. Human evaluation confirms that the vast majority of our generated fact-check explanations contain all information from gold standard data.
FactGenius: Combining Zero-Shot Prompting and Fuzzy Relation Mining to Improve Fact Verification with Knowledge Graphs
Authors: Sushant Gautam; Roxana Pop
Venue: EMNLP 2024 (Proceedings of the Seventh Fact Extraction and VERification Workshop (FEVER))

Abstract: Fact-checking is a crucial natural language processing (NLP) task that verifies the truthfulness of claims by considering reliable evidence. Traditional methods are labour- intensive, and most automatic approaches focus on using documents as evidence. In this paper, we focus on the relatively understudied fact-checking with Knowledge Graph data as evidence and experiment on the recently introduced FactKG benchmark. We present FactGenius, a novel method that enhances fact- checking by combining zero-shot prompting of large language models (LLMs) with fuzzy text matching on knowledge graphs (KGs). Our method employs LLMs for filtering relevant connections from the graph and validates these connections via distance-based matching. The evaluation of FactGenius on an existing benchmark demonstrates its effectiveness, as we show it significantly outperforms state-of- the-art methods. The code and materials are available at https://github.com/SushantGautam/FactGenius.
Fact or Fiction? Improving Fact Verification with Knowledge Graphs through Simplified Subgraph Retrievals
Authors: Tobias Aanderaa Opsahl
Venue: EMNLP 2024 (Proceedings of the Seventh Fact Extraction and VERification Workshop (FEVER))

Abstract: Despite recent success in natural language processing (NLP), fact verification still remains a difficult task. Due to misinformation spreading increasingly fast, attention has been directed towards automatically verifying the correctness of claims. In the domain of NLP, this is usually done by training supervised machine learning models to verify claims by utilizing evidence from trustworthy corpora. We present efficient methods for verifying claims on a dataset where the evidence is in the form of structured knowledge graphs. We use the FactKG dataset, which is constructed from the DBpedia knowledge graph extracted from Wikipedia. By simplifying the evidence retrieval process, from fine-tuned language models to simple logical retrievals, we are able to construct models that both require less computational resources and achieve better test-set accuracy.
BERTrend: Neural Topic Modeling for Emerging Trends Detection
Authors: Allaa Boutaleb; Jerome Picault; Guillaume Grosjean
Venue: EMNLP 2024 (Proceedings of the Workshop on the Future of Event Detection (FuturED))

Abstract: enter abstract here
An Incremental Clustering Baseline for Event Detection on Twitter
Authors: Marjolaine Ray; Qi Wang; Frédérique Mélanie-Becquet; Thierry Poibeau; Béatrice Mazoyer
Venue: EMNLP 2024 (Proceedings of the Workshop on the Future of Event Detection (FuturED))

Abstract: enter abstract here
DEGREEˆ2: Efficient Extraction of Multiple Events Using Language Models
Authors: Philip Blair; Kfir Bar
Venue: EMNLP 2024 (Proceedings of the Workshop on the Future of Event Detection (FuturED))

Abstract: enter abstract here
MUMOSA, Interactive Dashboard for MUlti-MOdal Situation Awareness
Authors: Stephanie M. Lukin; Shawn Bowser; Reece Suchocki; Douglas Summers-Stay; Francis Ferraro; Cynthia Matuszek; Clare Voss
Venue: EMNLP 2024 (Proceedings of the Workshop on the Future of Event Detection (FuturED))

Abstract: enter abstract here
Reasoning and Tools for Human-Level Forecasting
Authors: Elvis Hsieh; Preston Fu; Jonathan Chen
Venue: EMNLP 2024 (Proceedings of the Workshop on the Future of Event Detection (FuturED))

Abstract: enter abstract here
A Comprehensive Survey on Document-Level Information Extraction
Authors: Hanwen Zheng; Sijia Wang; Lifu Huang
Venue: EMNLP 2024 (Proceedings of the Workshop on the Future of Event Detection (FuturED))

Abstract: Document-level information extraction (doc-IE) plays a pivotal role in the realm of natural language processing (NLP). This paper embarks on a comprehensive review and discussion of contemporary literature related to doc-IE. In addition, we conduct a thorough error analysis using state-of-the-art algorithms, shedding light on their limitations and remaining challenges for tackling the task of doc-IE. Our findings demonstrate that issues like entity coreference resolution and the lack of robust reasoning significantly hinder the effectiveness of document-level information extraction (doc-IE). Additionally, we uncover new challenges, including labeling noise and relation transitivity. The overarching objective of this survey paper is to provide valuable insights that can empower NLP researchers to further advance the performance of doc-IE.
Generative Approaches to Event Extraction: Survey and Outlook
Authors: Étienne Simon; Helene Olsen; Huiling You; Samia Touileb; Lilja Øvrelid; Erik Velldal
Venue: EMNLP 2024 (Proceedings of the Workshop on the Future of Event Detection (FuturED))

Abstract: enter abstract here