GenAI4ELab/papercli-papers huggingface dataset:
OpenResearcher: Unleashing AI for Accelerated Scientific Research
Authors: Yuxiang Zheng; Shichao Sun; Lin Qiu; Dongyu Ru; Cheng Jiayang; Xuefeng Li; Jifan Lin; Binjie Wang; Yun Luo; Renjie Pan; Yang Xu; Qingkai Min; Zizhao Zhang; Yiwen Wang; Wenjie Li; Pengfei Liu
Venue: EMNLP 2024 (Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: The rapid growth of scientific literature imposes significant challenges for researchers endeavoring to stay updated with the latest advancements in their fields and delve into new areas. We introduce OpenResearcher, an innovative platform that leverages Artificial Intelligence (AI) techniques to accelerate the research process by answering diverse questions from researchers. OpenResearcher is built based on Retrieval-Augmented Generation (RAG) to integrate Large Language Models (LLMs) with up-to-date, domain-specific knowledge. Moreover, we develop various tools for OpenResearcher to understand researchers’ queries, search from the scientific literature, filter retrieved information, provide accurate and comprehensive answers, and self-refine these answers. OpenResearcher can flexibly use these tools to balance efficiency and effectiveness. As a result, OpenResearcher enables researchers to save time and increase their potential to discover new insights and drive scientific breakthroughs. Demo, video, and code are available at: https://github.com/GAIR-NLP/OpenResearcher.
OpenFactCheck: A Unified Framework for Factuality Evaluation of LLMs
Authors: Hasan Iqbal; Yuxia Wang; Minghan Wang; Georgi Nenkov Georgiev; Jiahui Geng; Iryna Gurevych; Preslav Nakov
Venue: EMNLP 2024 (Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: The increased use of large language models (LLMs) across a variety of real-world applications calls for automatic tools to check the factual accuracy of their outputs, as LLMs often hallucinate. This is difficult as it requires assessing the factuality of free-form open-domain responses. While there has been a lot of research on this topic, different papers use different evaluation benchmarks and measures,which makes them hard to compare and hampers future progress. To mitigate these issues, we developed OpenFactCheck, a unified framework, with three modules: (i) RESPONSEEVAL, which allows users to easily customize an automatic fact-checking system and to assess the factuality of all claims in an input document using that system, (ii) LLMEVAL, which assesses the overall factuality of an LLM, and (iii) CHECKEREVAL, a module to evaluate automatic fact-checking systems. OpenFactCheck is open-sourced (https://github.com/mbzuai-nlp/openfactcheck) and publicly released as a Python library (https://pypi.org/project/openfactcheck/) and also as a web service (http://app.openfactcheck.com). A video describing the system is available at https://youtu.be/-i9VKL0HleI.
ULLME: A Unified Framework for Large Language Model Embeddings with Generation-Augmented Learning
Authors: Hieu Man; Nghia Trung Ngo; Franck Dernoncourt; Thien Huu Nguyen
Venue: EMNLP 2024 (Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: Large Language Models (LLMs) excel in various natural language processing tasks, but leveraging them for dense passage embedding remains challenging. This is due to their causal attention mechanism and the misalignment between their pre-training objectives and the text ranking tasks. Despite some recent efforts to address these issues, existing frameworks for LLM-based text embeddings have been limited by their support for only a limited range of LLM architectures and fine-tuning strategies, limiting their practical application and versatility. In this work, we introduce the Unified framework for Large Language Model Embedding (ULLME), a flexible, plug-and-play implementation that enables bidirectional attention across various LLMs and supports a range of fine-tuning strategies. We also propose Generation-augmented Representation Learning (GRL), a novel fine-tuning method to boost LLMs for text embedding tasks. GRL enforces consistency between representation-based and generation-based relevance scores, leveraging LLMs’ powerful generative abilities for learning passage embeddings. To showcase our framework’s flexibility and effectiveness, we release three pre-trained models from ULLME with different backbone architectures, ranging from 1.5B to 8B parameters, all of which demonstrate strong performance on the Massive Text Embedding Benchmark. Our framework is publicly available at: https://github.com/nlp-uoregon/ullme. A demo video for ULLME can also be found at https://rb.gy/ws1ile.
To the Globe (TTG): Towards Language-Driven Guaranteed Travel Planning
Authors: Da Ju; Song Jiang; Andrew Cohen; Aaron Foss; Sasha Mitts; Arman Zharmagambetov; Brandon Amos; Xian Li; Justine T Kao; Maryam Fazel-Zarandi; Yuandong Tian
Venue: EMNLP 2024 (Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: Travel planning is a challenging and time-consuming task that aims to find an itinerary which satisfies multiple, interdependent constraints regarding flights, accommodations, attractions, and other travel arrangements. In this paper, we propose To the Globe (TTG), a real-time demo system that takes natural language requests from users, translates it to symbolic form via a fine-tuned Large Language Model, and produces optimal travel itineraries with Mixed Integer Linear Programming solvers. The overall system takes ~5seconds to reply to the user request with guaranteed itineraries. To train TTG, we develop a synthetic data pipeline that generates userrequests, flight and hotel information in symbolic form without human annotations, based on the statistics of real-world datasets, and fine-tune an LLM to translate NL user requests to their symbolic form, which is sent to the symbolic solver to compute optimal itineraries. Our NL-symbolic translation achieves ~91% exact match in a backtranslation metric (i.e., whether the estimated symbolic form of generated natural language matches the groundtruth), and its returned itineraries have a ratio of 0.979 compared to the optimal cost of the ground truth user request. When evaluated by users, TTG achieves consistently high Net Promoter Scores (NPS) of 35-40% on generated itinerary.
MATSA: Multi-Agent Table Structure Attribution
Authors: Puneet Mathur; Alexa Siu; Nedim Lipka; Tong Sun
Venue: EMNLP 2024 (Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: Large Language Models (LLMs) have significantly advanced QA tasks through in-context learning but often suffer from hallucinations. Attributing supporting evidence grounded in source documents has been explored for unstructured text in the past. However, tabular data present unique challenges for attribution due to ambiguities (e.g., abbreviations, domain-specific terms), complex header hierarchies, and the difficulty in interpreting individual table cells without row and column context. We introduce a new task, Fine-grained Structured Table Attribution (FAST-Tab), to generate row and column-level attributions supporting LLM-generated answers. We present MATSA, a novel LLM-based Multi-Agent system capable of post-hoc Table Structure Attribution to help users visually interpret factual claims derived from tables. MATSA augments tabular entities with descriptive context about structure, metadata, and numerical trends to semantically retrieve relevant rows and columns corresponding to facts in an answer. Additionally, we propose TabCite, a diverse benchmark designed to evaluate the FAST-Tab task on tables with complex layouts sourced from Wikipedia and business PDF documents. Extensive experiments demonstrate that MATSA significantly outperforms SOTA baselines on TabCite, achieving an 8-13% improvement in F1 score. Qualitative user studies show that MATSA helps increase user trust in Generative AI by providing enhanced explainability for LLM-assisted table QA and enables professionals to be more productive by saving time on fact-checking LLM-generated answers.
OpenT2T: An Open-Source Toolkit for Table-to-Text Generation
Authors: Haowei Zhang; Shengyun Si; Yilun Zhao; Lujing Xie; Zhijian Xu; Lyuhao Chen; Linyong Nan; Pengcheng Wang; Xiangru Tang; Arman Cohan
Venue: EMNLP 2024 (Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: Table data is pervasive in various industries, and its comprehension and manipulation demand significant time and effort for users seeking to extract relevant information. Consequently, an increasing number of studies have been directed towards table-to-text generation tasks. However, most existing methods are benchmarked solely on a limited number of datasets with varying configurations, leading to a lack of unified, standardized, fair, and comprehensive comparison between methods. This paper presents OpenT2T, the first open-source toolkit for table-to-text generation, designed to reproduce existing large language models (LLMs) for performance comparison and expedite the development of new models.We have implemented and compared a wide range of LLMs under zero- and few-shot settings on 9 table-to-text generation datasets, covering data insight generation, table summarization, and free-form table question answering. Additionally, we maintain a public leaderboard to provide insights for future work into how to choose appropriate table-to-text generation systems for real-world scenarios.
ChatHF: Collecting Rich Human Feedback from Real-time Conversations
Authors: Andrew Li; Zhenduo Wang; Ethan Mendes; Duong Minh Le; Wei Xu; Alan Ritter
Venue: EMNLP 2024 (Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: We introduce ChatHF, an interactive annotation framework for chatbot evaluation, which integrates configurable annotation within a chat interface. ChatHF can be flexibly configured to accommodate various chatbot evaluation tasks, for example detecting offensive content, identifying incorrect or misleading information in chatbot responses, and chatbot responses that might compromise privacy. It supports post-editing of chatbot outputs and supports visual inputs, in addition to an optional voice interface. ChatHF is suitable for collection and annotation of NLP datasets, and Human-Computer Interaction studies, as demonstrated in case studies on image geolocation and assisting older adults with daily activities. ChatHF is publicly accessible at https://chat-hf.com.
KMatrix: A Flexible Heterogeneous Knowledge Enhancement Toolkit for Large Language Model
Authors: Shun Wu; Di Wu; Kun Luo; XueYou Zhang; Jun Zhao; Kang Liu
Venue: EMNLP 2024 (Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: Knowledge-Enhanced Large Language Models (K-LLMs) system enhances Large Language Models (LLMs) abilities using external knowledge. Existing K-LLMs toolkits mainly focus on free-textual knowledge, lacking support for heterogeneous knowledge like tables and knowledge graphs, and fall short in comprehensive datasets, models, and user-friendly experience. To address this gap, we introduce KMatrix: a flexible heterogeneous knowledge enhancement toolkit for LLMs including verbalizing-retrieval and parsing-query methods. Our modularity and control-logic flow diagram design flexibly supports the entire lifecycle of various complex K-LLMs systems, including training, evaluation, and deployment. To assist K-LLMs system research, a series of related knowledge, datasets, and models are integrated into our toolkit, along with performance analyses of K-LLMs systems enhanced by different types of knowledge. Using our toolkit, developers can rapidly build, evaluate, and deploy their own K-LLMs systems.
Xinference: Making Large Model Serving Easy
Authors: Weizheng Lu; Lingfeng Xiong; Feng Zhang; Xuye Qin; Yueguo Chen
Venue: EMNLP 2024 (Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: The proliferation of open-source large models necessitates dedicated tools for deployment and accessibility. To mitigate the complexities of model serving, we develop Xinference, an open-source library designed to simplify the deployment and management of large models. Xinference effectively simplifies deployment complexities for users by (a) preventing users from writing code and providing built-in support for various models and OpenAI-compatible APIs; (b) enabling full model serving lifecycle management; (c) guaranteeing efficient and scalable inference and achieving high throughput and low latency. In comparative experiments with similar products like BentoML and Ray Serve, Xinference outperforms these tools and offers superior ease of use.Xinference is available at https://github.com/xorbitsai/inference.
RETAIN: Interactive Tool for Regression Testing Guided LLM Migration
Authors: Tanay Dixit; Daniel Lee; Sally Fang; Sai Sree Harsha; Anirudh Sureshan; Akash V Maharaj; Yunyao Li
Venue: EMNLP 2024 (Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: Large Language Models (LLMs) are increasingly integrated into diverse applications. The rapid evolution of LLMs presents opportunities for developers to enhance applications continuously. However, this constant adaptation can also lead to performance regressions during model migrations. While several interactive tools have been proposed to streamline the complexity of prompt engineering, few address the specific requirements of regression testing for LLM Migrations. To bridge this gap, we introduce RETAIN (REgression Testing guided LLM migrAtIoN), a tool designed explicitly for regression testing in LLM Migrations. RETAIN comprises two key components: an interactive interface tailored to regression testing needs during LLM migrations, and an error discovery module that facilitates understanding of differences in model behaviors. The error discovery module generates textual descriptions of various errors or differences between model outputs, providing actionable insights for prompt refinement. Our automatic evaluation and empirical user studies demonstrate that RETAIN, when compared to manual evaluation, enabled participants to identify twice as many errors, facilitated experimentation with 75% more prompts, and achieves 12% higher metric scores in a given time frame.
ClaimLens: Automated, Explainable Fact-Checking on Voting Claims Using Frame-Semantics
Authors: Jacob Devasier; Rishabh Mediratta; Phuong Anh Le; David Huang; Chengkai Li
Venue: EMNLP 2024 (Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: We present ClaimLens, an automated fact-checking system focused on voting-related factual claims. Existing fact-checking solutions often lack transparency, making it difficult for users to trust and understand the reasoning behind the outcomes. In this work, we address the critical need for transparent and explainable automated fact-checking solutions. We propose a novel approach that leverages frame-semantic parsing to provide structured and interpretable fact verification. By focusing on voting-related claims, we can utilize publicly available voting records from official United States congressional sources and the established Vote semantic frame to extract relevant information from claims. Furthermore, we propose novel data augmentation techniques for frame-semantic parsing, a task known to lack robust annotated data, which leads to a +9.5% macro F1 score on frame element identification over our baseline.
RAGViz: Diagnose and Visualize Retrieval-Augmented Generation
Authors: Tevin Wang; Jingyuan He; Chenyan Xiong
Venue: EMNLP 2024 (Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: Retrieval-augmented generation (RAG) combines knowledge from domain-specific sources into large language models to ground answer generation. Current RAG systems lack customizable visibility on the context documents and the model’s attentiveness towards such documents. We propose RAGViz, a RAG diagnosis tool that visualizes the attentiveness of the generated tokens in retrieved documents. With a built-in user interface, retrieval index, and Large Language Model (LLM) backbone, RAGViz provides two main functionalities: (1) token and document-level attention visualization, and (2) generation comparison upon context document addition and removal. As an open-source toolkit, RAGViz can be easily hosted with a custom embedding model and HuggingFace-supported LLM backbone. Using a hybrid ANN (Approximate Nearest Neighbor) index, memory-efficient LLM inference tool, and custom context snippet method, RAGViz operates efficiently with a median query time of about 5 seconds on a moderate GPU node. Our code is available at https://github.com/cxcscmu/RAGViz. A demo video of RAGViz can be found at https://youtu.be/cTAbuTu6ur4.
LLM-DetectAIve: a Tool for Fine-Grained Machine-Generated Text Detection
Authors: Mervat Abassy; Kareem Elozeiri; Alexander Aziz; Minh Ngoc Ta; Raj Vardhan Tomar; Bimarsha Adhikari; Saad El Dine Ahmed; Yuxia Wang; Osama Mohammed Afzal; Zhuohan Xie; Jonibek Mansurov; Ekaterina Artemova; Vladislav Mikhailov; Rui Xing; Jiahui Geng; Hasan Iqbal; Zain Muhammad Mujahid; Tarek Mahmoud; Akim Tsvigun; Alham Fikri Aji; Artem Shelmanov; Nizar Habash; Iryna Gurevych; Preslav Nakov
Venue: EMNLP 2024 (Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: The ease of access to large language models (LLMs) has enabled a widespread of machine-generated texts, and now it is often hard to tell whether a piece of text was human-written or machine-generated. This raises concerns about potential misuse, particularly within educational and academic domains. Thus, it is important to develop practical systems that can automate the process. Here, we present one such system, LLM-DetectAIve, designed for fine-grained detection. Unlike most previous work on machine-generated text detection, which focused on binary classification, LLM-DetectAIve supports four categories: (i) human-written, (ii) machine-generated, (iii) machine-written, then machine-humanized, and (iv) human-written, then machine-polished. Category (iii) aims to detect attempts to obfuscate the fact that a text was machine-generated, while category (iv) looks for cases where the LLM was used to polish a human-written text, which is typically acceptable in academic writing, but not in education. Our experiments show that LLM-DetectAIve can effectively identify the above four categories, which makes it a potentially useful tool in education, academia, and other domains.LLM-DetectAIve is publicly accessible at https://github.com/mbzuai-nlp/LLM-DetectAIve. The video describing our system is available at https://youtu.be/E8eT_bE7k8c.
Translation Canvas: An Explainable Interface to Pinpoint and Analyze Translation Systems
Authors: Chinmay Dandekar; Wenda Xu; Xi Xu; Siqi Ouyang; Lei Li
Venue: EMNLP 2024 (Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: With the rapid advancement of machine translation research, evaluation toolkits have become essential for benchmarking system progress. Tools like COMET and SacreBLEU offer single quality score assessments that are effective for pairwise system comparisons. However, these tools provide limited insights for fine-grained system-level comparisons and the analysis of instance-level defects. To address these limitations, we introduce Translation Canvas, an explainable interface designed to pinpoint and analyze translation systems’ performance: 1) Translation Canvas assists machine translation researchers in comprehending system-level model performance by identifying common errors (their frequency and severity) and analyzing relationships between different systems based on various evaluation metrics. 2) It supports fine-grained analysis by highlighting error spans with explanations and selectively displaying systems’ predictions. According to human evaluation, Translation Canvas demonstrates superior performance over COMET and SacreBLEU packages under enjoybility and understandbility criteria.
Debug Smarter, Not Harder: AI Agents for Error Resolution in Computational Notebooks
Authors: Konstantin Grotov; Artem Borzilov; Maksim Krivobok; Timofey Bryksin; Yaroslav Zharov
Venue: EMNLP 2024 (Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: Computational notebooks became indispensable tools for research-related development, offering unprecedented interactivity and flexibility in the development process. However, these benefits come at the cost of reproducibility and an increased potential for bugs.With the rise of code-fluent Large Language Models empowered with agentic techniques, smart bug-fixing tools with a high level of autonomy have emerged.However, those tools are tuned for classical script programming and still struggle with non-linear computational notebooks.In this paper, we present an AI agent designed specifically for error resolution in a computational notebook. We have developed an agentic system capable of exploring a notebook environment by interacting with it—similar to how a user would—and integrated the system into the JetBrains service for collaborative data science called Datalore.We evaluate our approach against the pre-existing single-action solution by comparing costs and conducting a user study. Users rate the error resolution capabilities of the agentic system higher but experience difficulties with UI. We share the results of the study and consider them valuable for further improving user-agent collaboration.
Schema-Guided Culture-Aware Complex Event Simulation with Multi-Agent Role-Play
Authors: Sha Li; Revanth Gangi Reddy; Khanh Duy Nguyen; Qingyun Wang; Yi Fung; Chi Han; Jiawei Han; Kartik Natarajan; Clare R. Voss; Heng Ji
Venue: EMNLP 2024 (Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: Complex news events, such as natural disasters and socio-political conflicts, require swift responses from the government and society. Relying on historical events to project the future is insufficient as such events are sparse and do not cover all possible conditions and nuanced situations. Simulation of these complex events can help better prepare and reduce the negative impact. We develop a controllable complex news event simulator guided by both the event schema representing domain knowledge about the scenario and user-provided assumptions representing case-specific conditions.As event dynamics depend on the fine-grained social and cultural context, we further introduce a geo-diverse commonsense and cultural norm-aware knowledge enhancement component.To enhance the coherence of the simulation, apart from the global timeline of events,we take an agent-based approach to simulate the individual character states, plans, and actions. By incorporating the schema and cultural norms, our generated simulations achieve much higher coherence and appropriateness and are received favorably by participants from a humanitarian assistance organization.