GenAI4ELab/papercli-papers huggingface dataset:
CHATREPORT: Democratizing Sustainability Disclosure Analysis through LLM-based Tools
Authors: Jingwei Ni; Julia Bingler; Chiara Colesanti-Senni; Mathias Kraus; Glen Gostlow; Tobias Schimanski; Dominik Stammbach; Saeid Ashraf Vaghefi; Qian Wang; Nicolas Webersinke; Tobias Wekhof; Tingyu Yu; Markus Leippold
Venue: EMNLP 2023 (Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: In the face of climate change, are companies really taking substantial steps toward more sustainable operations? A comprehensive answer lies in the dense, information-rich landscape of corporate sustainability reports. However, the sheer volume and complexity of these reports make human analysis very costly. Therefore, only a few entities worldwide have the resources to analyze these reports at scale, which leads to a lack of transparency in sustainability reporting. Empowering stakeholders with LLM-based automatic analysis tools can be a promising way to democratize sustainability report analysis. However, developing such tools is challenging due to (1) the hallucination of LLMs and (2) the inefficiency of bringing domain experts into the AI development loop. In this paper, we introduce ChatReport, a novel LLM-based system to automate the analysis of corporate sustainability reports, addressing existing challenges by (1) making the answers traceable to reduce the harm of hallucination and (2) actively involving domain experts in the development loop. We make our methodology, annotated datasets, and generated analyses of 1015 reports publicly available. Video Introduction: https://www.youtube.com/watch?v=Q5AzaKzPE4M Github: https://github.com/EdisonNi-hku/chatreport Live web app: reports.chatclimate.ai
RaLLe: A Framework for Developing and Evaluating Retrieval-Augmented Large Language Models
Authors: Yasuto Hoshi; Daisuke Miyashita; Youyang Ng; Kento Tatsuno; Yasuhiro Morioka; Osamu Torii; Jun Deguchi
Venue: EMNLP 2023 (Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: Retrieval-augmented large language models (R-LLMs) combine pre-trained large language models (LLMs) with information retrieval systems to improve the accuracy of factual question-answering. However, current libraries for building R-LLMs provide high-level abstractions without sufficient transparency for evaluating and optimizing prompts within specific inference processes such as retrieval and generation. To address this gap, we present RaLLe, an open-source framework designed to facilitate the development, evaluation, and optimization of R-LLMs for knowledge-intensive tasks. With RaLLe, developers can easily develop and evaluate R-LLMs, improving hand-crafted prompts, assessing individual inference processes, and objectively measuring overall system performance quantitatively. By leveraging these features, developers can enhance the performance and accuracy of their R-LLMs in knowledge-intensive generation tasks.
VIST5: An Adaptive, Retrieval-Augmented Language Model for Visualization-oriented Dialog
Authors: Henrik Voigt; Nuno Carvalhais; Monique Meuschke; Markus Reichstein; Sina Zarrieß; Kai Lawonn
Venue: EMNLP 2023 (Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: The advent of large language models has brought about new ways of interacting with data intuitively via natural language. In recent years, a variety of visualization systems have explored the use of natural language to create and modify visualizations through visualization-oriented dialog. However, the majority of these systems rely on tailored dialog agents to analyze domain-specific data and operate domain-specific visualization tools and libraries. This is a major challenge when trying to transfer functionalities between dialog interfaces of different visualization applications. To address this issue, we propose VIST5, a visualization-oriented dialog system that focuses on easy adaptability to an application domain as well as easy transferability of language-controllable visualization library functions between applications. Its architecture is based on a retrieval-augmented T5 language model that leverages few-shot learning capabilities to enable a rapid adaptation of the system.
H2O Open Ecosystem for State-of-the-art Large Language Models
Authors: Arno Candel; Jon McKinney; Philipp Singer; Pascal Pfeiffer; Maximilian Jeblick; Chun Ming Lee; Marcos Conde
Venue: EMNLP 2023 (Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: Large Language Models (LLMs) represent a revolution in AI. However, they also pose many significant risks, such as the presence of biased, private, copyrighted or harmful text. For this reason we need open, transparent and safe solutions. We introduce a complete open-source ecosystem for developing and testing LLMs. The goal of this project is to boost open alternatives to closed-source approaches. We release h2oGPT, a family of fine-tuned LLMs from 7 to 70 Billion parameters. We also introduce H2O LLM Studio, a framework and no-code GUI designed for efficient fine-tuning, evaluation, and deployment of LLMs using the most recent state-of-the-art techniques. Our code and models are licensed under fully permissive Apache 2.0 licenses. We believe open-source language models help to boost AI development and make it more accessible and trustworthy. Our demo is available at: https://gpt.h2o.ai/
Koala: An Index for Quantifying Overlaps with Pre-training Corpora
Authors: Thuy-Trang Vu; Xuanli He; Gholamreza Haffari; Ehsan Shareghi
Venue: EMNLP 2023 (Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: In very recent years more attention has been placed on probing the role of pre-training data in Large Language Models (LLMs) downstream behaviour. Despite the importance, there is no public tool that supports such analysis of pre-training corpora at large scale. To help research in this space, we launch Koala, a searchable index over large pre-training corpora using lossless compressed suffix arrays with highly efficient compression rate and search support. In its first release we index the public proportion of OPT 175B, GPT-3, GPT-Neo, GPT-Neo, LLaMA, BERT, ELECTRA, RoBERTA, XLNet pre-training corpora. Koala provides a framework to do forensic analysis on the current and future benchmarks as well as to assess the degree of memorization in the output from the LLMs. Koala is available for public use at https://koala-index.erc.monash.edu/.
Sudowoodo: A Chinese Lyric Imitation System with Source Lyrics
Authors: Yongzhu Chang; Rongsheng Zhang; Lin Jiang; Qihang Chen; Le Zhang; Jiashu Pu
Venue: EMNLP 2023 (Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: Lyrics generation is a well-known application in natural language generation research, with several previous studies focusing on generating accurate lyrics using precise control such as keywords, rhymes, etc. However, lyrics imitation, which involves writing new lyrics by imitating the style and content of the source lyrics, remains a challenging task due to the lack of a parallel corpus. In this paper, we introduce Sudowoodo, a Chinese lyrics imitation system that can generate new lyrics based on the text of source lyrics. To address the issue of lacking a parallel training corpus for lyrics imitation, we propose a novel framework to construct a parallel corpus based on a keyword-based lyrics model from source lyrics. Then the pairs (new lyrics, source lyrics) are used to train the lyrics imitation model. During the inference process, we utilize a post-processing module to filter and rank the generated lyrics, selecting the highest-quality ones. We incorporated audio information and aligned the lyrics with the audio to form the songs as a bonus. The human evaluation results show that our framework can perform better lyric imitation. Meanwhile, the Sudowoodo system and demo video of the system is available at Sudowoodo and https://youtu.be/u5BBT\_j1L5M
ConvLab-3: A Flexible Dialogue System Toolkit Based on a Unified Data Format
Authors: Qi Zhu; Christian Geishauser; Hsien-chin Lin; Carel van Niekerk; Baolin Peng; Zheng Zhang; Shutong Feng; Michael Heck; Nurul Lubis; Dazhen Wan; Xiaochen Zhu; Jianfeng Gao; Milica Gasic; Minlie Huang
Venue: EMNLP 2023 (Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: Task-oriented dialogue (TOD) systems function as digital assistants, guiding users through various tasks such as booking flights or finding restaurants. Existing toolkits for building TOD systems often fall short in delivering comprehensive arrays of data, model, and experimental environments with a user-friendly experience. We introduce ConvLab-3: a multifaceted dialogue system toolkit crafted to bridge this gap. Our unified data format simplifies the integration of diverse datasets and models, significantly reducing complexity and cost for studying generalization and transfer. Enhanced with robust reinforcement learning (RL) tools, featuring a streamlined training process, in-depth evaluation tools, and a selection of user simulators, ConvLab-3 supports the rapid development and evaluation of robust dialogue policies. Through an extensive study, we demonstrate the efficacy of transfer learning and RL and showcase that ConvLab-3 is not only a powerful tool for seasoned researchers but also an accessible platform for newcomers.
FLEEK: Factual Error Detection and Correction with Evidence Retrieved from External Knowledge
Authors: Farima Fatahi Bayat; Kun Qian; Benjamin Han; Yisi Sang; Anton Belyy; Samira Khorshidi; Fei Wu; Ihab Ilyas; Yunyao Li
Venue: EMNLP 2023 (Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: Detecting factual errors of textual information, whether generated by large language models (LLM) or curated by humans, is crucial for making informed decisions. LLMs’ inability to attribute their claims to external knowledge and their tendency to hallucinate makes it difficult to rely on their responses. Humans, too, are prone to factual errors in their writing. Since manual detection and correction of factual er- rors is labor-intensive, developing an automatic approach can greatly reduce human effort. We present a prototype tool that automatically extracts factual claims from text, gathers evidence from external knowledge sources, evaluates the factuality of each claim, and suggests revisions for identified errors using the collected evidence. Initial empirical evaluation on fact error detection (77-85% F1) shows the potential of our tool.
YATO: Yet Another deep learning based Text analysis Open toolkit
Authors: Zeqiang Wang; Yile Wang; Jiageng Wu; Zhiyang Teng; Jie Yang
Venue: EMNLP 2023 (Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: We introduce YATO, an open-source, easy-to-use toolkit for text analysis with deep learning. Different from existing heavily engineered toolkits and platforms, YATO is lightweight and user-friendly for researchers from cross-disciplinary areas. Designed in a hierarchical structure, YATO supports free combinations of three types of widely used features including 1) traditional neural networks (CNN, RNN, etc.); 2) pre-trained language models (BERT, RoBERTa, ELECTRA, etc.); and 3) user-customized neural features via a simple configurable file. Benefiting from the advantages of flexibility and ease of use, YATO can facilitate fast reproduction and refinement of state-of-the-art NLP models, and promote the cross-disciplinary applications of NLP techniques. The code, examples, and documentation are publicly available at https://github.com/jiesutd/YATO. A demo video is also available at https://www.youtube.com/playlist?list=PLJ0mhzMcRuDUlTkzBfAftOqiJRxYTTjXH.
Spacerini: Plug-and-play Search Engines with Pyserini and Hugging Face
Authors: Christopher Akiki; Odunayo Ogundepo; Aleksandra Piktus; Xinyu Zhang; Akintunde Oladipo; Jimmy Lin; Martin Potthast
Venue: EMNLP 2023 (Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: We present Spacerini, a tool that integrates the Pyserini toolkit for reproducible information retrieval research with Hugging Face to enable the seamless construction and deployment of interactive search engines. Spacerini makes state-of-the-art sparse and dense retrieval models more accessible to non-IR practitioners while minimizing deployment effort. This is useful for NLP researchers who want to better understand and validate their research by performing qualitative analyses of training corpora, for IR researchers who want to demonstrate new retrieval models integrated into the growing Pyserini ecosystem, and for third parties reproducing the work of other researchers. Spacerini is open source and includes utilities for loading, preprocessing, indexing, and deploying search engines locally and remotely. We demonstrate a portfolio of 13 search engines created with Spacerini for different use cases.
Adapters: A Unified Library for Parameter-Efficient and Modular Transfer Learning
Authors: Clifton Poth; Hannah Sterz; Indraneil Paul; Sukannya Purkayastha; Leon Engländer; Timo Imhof; Ivan Vulić; Sebastian Ruder; Iryna Gurevych; Jonas Pfeiffer
Venue: EMNLP 2023 (Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: We introduce Adapters, an open-source library that unifies parameter-efficient and modular transfer learning in large language models. By integrating 10 diverse adapter methods into a unified interface, Adapters offers ease of use and flexible configuration. Our library allows researchers and practitioners to leverage adapter modularity through composition blocks, enabling the design of complex adapter setups. We demonstrate the library’s efficacy by evaluating its performance against full fine-tuning on various NLP tasks. Adapters provides a powerful tool for addressing the challenges of conventional fine-tuning paradigms and promoting more efficient and modular transfer learning. The library is available via https://adapterhub.ml/adapters.
INTELMO: Enhancing Models’ Adoption of Interactive Interfaces
Authors: Chunxu Yang; Chien-Sheng Wu; Lidiya Murakhovs’ka; Philippe Laban; Xiang Chen
Venue: EMNLP 2023 (Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: This paper presents INTELMO, an easy-to-use library to help model developers adopt user-faced interactive interfaces and articles from real-time RSS sources for their language models. The library categorizes common NLP tasks and provides default style patterns, streamlining the process of creating interfaces with minimal code modifications while ensuring an intuitive user experience. Moreover, INTELMO employs a multi-granular hierarchical abstraction to provide developers with fine-grained and flexible control over user interfaces. INTELMO is under active development, with document available at https://intelmo.github.io.
Humanoid Agents: Platform for Simulating Human-like Generative Agents
Authors: Zhilin Wang; Yu Ying Chiu; Yu Cheung Chiu
Venue: EMNLP 2023 (Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: Just as computational simulations of atoms, molecules and cells have shaped the way we study the sciences, true-to-life simulations of human-like agents can be valuable tools for studying human behavior. We propose Humanoid Agents, a system that guides Generative Agents to behave more like humans by introducing three elements of System 1 processing: Basic needs (e.g. hunger, health and energy), Emotion and Closeness in Relationships. Humanoid Agents are able to use these dynamic elements to adapt their daily activities and conversations with other agents, as supported with empirical experiments. Our system is designed to be extensible to various settings, three of which we demonstrate, as well as to other elements influencing human behavior (e.g. empathy, moral values and cultural background). Our platform also includes a Unity WebGL game interface for visualization and an interactive analytics dashboard to show agent statuses over time. Our platform is available on https://www.humanoidagents.com/ and code is on https://github.com/HumanoidAgents/HumanoidAgents
TP-Detector: Detecting Turning Points in the Engineering Process of Large-scale Projects
Authors: Qi Wu; WenHan Chao; Xian Zhou; Zhunchen Luo
Venue: EMNLP 2023 (Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: This paper introduces a novel task of detecting turning points in the engineering process of large-scale projects, wherein the turning points signify significant transitions occurring between phases. Given the complexities involving diverse critical events and limited comprehension in individual news reports, we approach the problem by treating the sequence of related news streams as a window with multiple instances. To capture the evolution of changes effectively, we adopt a deep Multiple Instance Learning (MIL) framework and employ the multiple instance ranking loss to discern the transition patterns exhibited in the turning point window. Extensive experiments consistently demonstrate the effectiveness of our proposed approach on the constructed dataset compared to baseline methods. We deployed the proposed mode and provided a demonstration video to illustrate its functionality. The code and dataset are available on GitHub.
CLEVA: Chinese Language Models EVAluation Platform
Authors: Yanyang Li; Jianqiao Zhao; Duo Zheng; Zi-Yuan Hu; Zhi Chen; Xiaohui Su; Yongfeng Huang; Shijia Huang; Dahua Lin; Michael Lyu; Liwei Wang
Venue: EMNLP 2023 (Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: With the continuous emergence of Chinese Large Language Models (LLMs), how to evaluate a model’s capabilities has become an increasingly significant issue. The absence of a comprehensive Chinese benchmark that thoroughly assesses a model’s performance, the unstandardized and incomparable prompting procedure, and the prevalent risk of contamination pose major challenges in the current evaluation of Chinese LLMs. We present CLEVA, a user-friendly platform crafted to holistically evaluate Chinese LLMs. Our platform employs a standardized workflow to assess LLMs’ performance across various dimensions, regularly updating a competitive leaderboard. To alleviate contamination, CLEVA curates a significant proportion of new data and develops a sampling strategy that guarantees a unique subset for each leaderboard round. Empowered by an easy-to-use interface that requires just a few mouse clicks and a model API, users can conduct a thorough evaluation with minimal coding. Large-scale experiments featuring 23 Chinese LLMs have validated CLEVA’s efficacy.
DOPA METER – A Tool Suite for Metrical Document Profiling and Aggregation
Authors: Christina Lohr; Udo Hahn
Venue: EMNLP 2023 (Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: We present DOPA METER, a tool suite for the metrical investigation of written language, that provides diagnostic means for its division into discourse categories, such as registers, genres, and style. The quantitative basis of our system are 120 metrics covering a wide range of lexical, syntactic, and semantic features relevant for language profiling. The scores can be summarized, compared, and aggregated using visualization tools that can be tailored according to the users’ needs. We also showcase an application scenario for DOPA METER.
Muted: Multilingual Targeted Offensive Speech Identification and Visualization
Authors: Christoph Tillmann; Aashka Trivedi; Sara Rosenthal; Santosh Borse; Rong Zhang; Avirup Sil; Bishwaranjan Bhattacharjee
Venue: EMNLP 2023 (Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: Offensive language such as hate, abuse, and profanity (HAP) occurs in various content on the web. While previous work has mostly dealt with sentence level annotations, there have been a few recent attempts to identify offensive spans as well. We build upon this work and introduce MUTED, a system to identify multilingual HAP content by displaying offensive arguments and their targets using heat maps to indicate their intensity. MUTED can leverage any transformer-based HAP-classification model and its attention mechanism out-of-the-box to identify toxic spans, without further fine-tuning. In addition, we use the spaCy library to identify the specific targets and arguments for the words predicted by the attention heatmaps. We present the model’s performance on identifying offensive spans and their targets in existing datasets and present new annotations on German text. Finally, we demonstrate our proposed visualization tool on multilingual inputs.
Gentopia.AI: A Collaborative Platform for Tool-Augmented LLMs
Authors: Binfeng Xu; Xukun Liu; Hua Shen; Zeyu Han; Yuhan Li; Murong Yue; Zhiyuan Peng; Yuchen Liu; Ziyu Yao; Dongkuan Xu
Venue: EMNLP 2023 (Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: Augmented Language Models (ALMs) empower large language models with the ability to use tools, transforming them into intelligent agents for real-world interactions. However, most existing frameworks for ALMs, to varying degrees, are deficient in the following critical features: flexible customization, collaborative democratization, and holistic evaluation. This paper proposes Gentopia, a lightweight and extensible framework for ALMs. Gentopia allows the flexible customization of agents through simple configurations, seamlessly integrating various language models, task formats, prompting modules, and plugins into a unified paradigm. Furthermore, we establish Gentpool, a public platform enabling the registration and sharing of user-customized agents. Agents registered in Gentpool are composable such that they can be assembled together for agent collaboration, advancing the democratization of artificial intelligence. To ensure high-quality agents, Gentbench, an integral component of Gentpool, is designed to thoroughly evaluate user-customized agents across diverse aspects such as safety, robustness, efficiency, etc. We release Gentopia on Github and will continuously move forward.