GenAI4ELab/papercli-papers huggingface dataset:
SparkRA: A Retrieval-Augmented Knowledge Service System Based on Spark Large Language Model
Authors: Dayong Wu; Jiaqi Li; Baoxin Wang; Honghong Zhao; Siyuan Xue; Yanjie Yang; Zhijun Chang; Rui Zhang; Li Qian; Bo Wang; Shijin Wang; Zhixiong Zhang; Guoping Hu
Venue: EMNLP 2024 (Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: Large language models (LLMs) have shown remarkable achievements across various language tasks. To enhance the performance of LLMs in scientific literature services, we developed the scientific literature LLM (SciLit-LLM) through pre-training and supervised fine-tuning on scientific literature, building upon the iFLYTEK Spark LLM. Furthermore, we present a knowledge service system Spark Research Assistant (SparkRA) based on our SciLit-LLM. SparkRA is accessible online and provides three primary functions: literature investigation, paper reading, and academic writing. As of July 30, 2024, SparkRA has garnered over 50,000 registered users, with a total usage count exceeding 1.3 million.
Generative Dictionary: Improving Language Learner Understanding with Contextual Definitions
Authors: Kai-Wen Tuan; Hai-Lun Tu; Jason S. Chang
Venue: EMNLP 2024 (Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: We introduce GenerativeDictionary, a novel dictionary system that generates word sense interpretations based on the given context. Our approach involves transforming context sentences to highlight the meaning of target words within their specific context. The method involves automatically transforming context sentences into sequences of low-dimensional vector token representations, automatically processing the input embeddings through multiple layers of transformers, and automatically generate the word senses based on the latent representations derived from the context. At runtime, context sentences with target words are processed through a transformer model that outputs the relevant word senses.Blind evaluations on a combined set of dictionary example sentences and generated sentences based on given word senses demonstrate that our method is comparable to traditional word sense disambiguation (WSD) methods. By framing WSD as a generative problem, GenerativeDictionary delivers more precise and contextually appropriate word senses, enhancing the effectiveness of language learning tools.
WalledEval: A Comprehensive Safety Evaluation Toolkit for Large Language Models
Authors: Prannaya Gupta; Le Qi Yau; Hao Han Low; I-Shiang Lee; Hugo Maximus Lim; Yu Xin Teoh; Koh Jia Hng; Dar Win Liew; Rishabh Bhardwaj; Rajat Bhardwaj; Soujanya Poria
Venue: EMNLP 2024 (Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: WalledEval is a comprehensive AI safety testing toolkit designed to evaluate large language models (LLMs). It accommodates a diverse range of models, including both open-weight and API-based ones, and features over 35 safety benchmarks covering areas such as multilingual safety, exaggerated safety, and prompt injections. The framework supports both LLM and judge benchmarking, and incorporates custom mutators to test safety against various text-style mutations such as future tense and paraphrasing. Additionally, WalledEval introduces WalledGuard, a new, small and performant content moderation tool, and SGXSTest, a benchmark for assessing exaggerated safety in cultural contexts. We make WalledEval publicly available at https://github.com/walledai/walledeval with a demonstration video at https://youtu.be/50Zy97kj1MA.
RAGLAB: A Modular and Research-Oriented Unified Framework for Retrieval-Augmented Generation
Authors: Xuanwang Zhang; Yun-Ze Song; Yidong Wang; Shuyun Tang; Xinfeng Li; Zhengran Zeng; Zhen Wu; Wei Ye; Wenyuan Xu; Yue Zhang; Xinyu Dai; Shikun Zhang; Qingsong Wen
Venue: EMNLP 2024 (Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: Large Language Models (LLMs) demonstrate human-level capabilities in dialogue, reasoning, and knowledge retention. However, even the most advanced LLMs face challenges such as hallucinations and real-time updating of their knowledge. Current research addresses this bottleneck by equipping LLMs with external knowledge, a technique known as Retrieval Augmented Generation (RAG). However, two key issues constrained the development of RAG. First, there is a growing lack of comprehensive and fair comparisons between novel RAG algorithms. Second, open-source tools such as LlamaIndex and LangChain employ high-level abstractions, which results in a lack of transparency and limits the ability to develop novel algorithms and evaluation metrics. To close this gap, we introduce RAGLAB, a modular and research-oriented open-source library. RAGLAB reproduces 6 existing algorithms and provides a comprehensive ecosystem for investigating RAG algorithms. Leveraging RAGLAB, we conduct a fair comparison of 6 RAG algorithms across 10 benchmarks. With RAGLAB, researchers can efficiently compare the performance of various algorithms and develop novel algorithms.
AutoTrain: No-code training for state-of-the-art models
Authors: Abhishek Thakur
Venue: EMNLP 2024 (Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: With the advancements in open-source models, training(or finetuning) models on custom datasets has become a crucial part of developing solutions which are tailored to specific industrial or open-source applications. Yet, there is no single tool which simplifies the process of training across different types of modalities or tasks.We introduce AutoTrain(aka AutoTrain Advanced)—an open-source, no code tool/library which can be used to train (or finetune) models for different kinds of tasks such as: large language model (LLM) finetuning, text classification/regression, token classification, sequence-to-sequence task, finetuning of sentence transformers, visual language model (VLM) finetuning, image classification/regression and even classification and regression tasks on tabular data. AutoTrain Advanced is an open-source library providing best practices for training models on custom datasets. The library is available at https://github.com/huggingface/autotrain-advanced. AutoTrain can be used in fully local mode or on cloud machines and works with tens of thousands of models shared on Hugging Face Hub and their variations.
Sailor: Open Language Models for South-East Asia
Authors: Longxu Dou; Qian Liu; Guangtao Zeng; Jia Guo; Jiahui Zhou; Xin Mao; Ziqi Jin; Wei Lu; Min Lin
Venue: EMNLP 2024 (Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: We present Sailor, a family of open language models ranging from 0.5B to 14B parameters, tailored for South-East Asian (SEA) languages. From Qwen1.5, Sailor models accept 200B to 400B tokens during continual pre-training, primarily covering the languages of English, Chinese, Vietnamese, Thai, Indonesian, Malay, and Lao. The training leverages several techniques, including BPE dropout for improving the model robustness, aggressive data cleaning and deduplication, and small proxy models to optimize the data mixture. Experimental results on four typical tasks indicate that Sailor models demonstrate strong performance across different benchmarks, including commonsense reasoning, question answering, reading comprehension and examination. We share our insights to spark a wider interest in developing large language models for multilingual use cases.
RepoAgent: An LLM-Powered Open-Source Framework for Repository-level Code Documentation Generation
Authors: Qinyu Luo; Yining Ye; Shihao Liang; Zhong Zhang; Yujia Qin; Yaxi Lu; Yesai Wu; Xin Cong; Yankai Lin; Yingli Zhang; Xiaoyin Che; Zhiyuan Liu; Maosong Sun
Venue: EMNLP 2024 (Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: Generative models have demonstrated considerable potential in software engineering, particularly in tasks such as code generation and debugging. However, their utilization in the domain of code documentation generation remains underexplored. To this end, we introduce RepoAgent, a large language model powered open-source framework aimed at proactively generating, maintaining, and updating code documentation. Through both qualitative and quantitative evaluations, we have validated the effectiveness of our approach, showing that RepoAgent excels in generating high-quality repository-level documentation. The code and results are publicly accessible at https://github.com/OpenBMB/RepoAgent.
DeepPavlov 1.0: Your Gateway to Advanced NLP Models Backed by Transformers and Transfer Learning
Authors: Maksim Savkin; Anastasia Voznyuk; Fedor Ignatov; Anna Korzanova; Dmitry Karpov; Alexander Popov; Vasily Konovalov
Venue: EMNLP 2024 (Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: We present DeepPavlov 1.0, an open-source framework for using Natural Language Processing (NLP) models by leveraging transfer learning techniques. DeepPavlov 1.0 is created for modular and configuration-driven development of state-of-the-art NLP models and supports a wide range of NLP model applications. DeepPavlov 1.0 is designed for practitioners with limited knowledge of NLP/ML. DeepPavlov is based on PyTorch and supports HuggingFace transformers. DeepPavlov is publicly released under the Apache 2.0 license and provides access to an online demo.
Kandinsky 3: Text-to-Image Synthesis for Multifunctional Generative Framework
Authors: Vladimir Arkhipkin; Viacheslav Vasilev; Andrei Filatov; Igor Pavlov; Julia Agafonova; Nikolai Gerasimenko; Anna Averchenkova; Evelina Mironova; Anton Bukashkin; Konstantin Kulikov; Andrey Kuznetsov; Denis Dimitrov
Venue: EMNLP 2024 (Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: Text-to-image (T2I) diffusion models are popular for introducing image manipulation methods, such as editing, image fusion, inpainting, etc. At the same time, image-to-video (I2V) and text-to-video (T2V) models are also built on top of T2I models. We present Kandinsky 3, a novel T2I model based on latent diffusion, achieving a high level of quality and photorealism. The key feature of the new architecture is the simplicity and efficiency of its adaptation for many types of generation tasks. We extend the base T2I model for various applications and create a multifunctional generation system that includes text-guided inpainting/outpainting, image fusion, text-image fusion, image variations generation, I2V and T2V generation. We also present a distilled version of the T2I model, evaluating inference in 4 steps of the reverse process without reducing image quality and 3 times faster than the base model. We deployed a user-friendly demo system in which all the features can be tested in the public domain. Additionally, we released the source code and checkpoints for the Kandinsky 3 and extended models. Human evaluations show that Kandinsky 3 demonstrates one of the highest quality scores among open source generation systems.
MIMIR: A Customizable Agent Tuning Platform for Enhanced Scientific Applications
Authors: Xiangru Tang; Chunyuan Deng; Hanmin Wang; Haoran Wang; Yilun Zhao; Wenqi Shi; May Fung; Wangchunshu Zhou; Jiannan Cao; Heng Ji; Arman Cohan; Mark Gerstein
Venue: EMNLP 2024 (Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: Recently, large language models (LLMs) have evolved into interactive agents, proficient in planning, tool use, and task execution across various tasks. However, without agent-tuning, open-source models like LLaMA2 currently struggle to match the efficiency of larger models such as GPT-4 in scientific applications due to a lack of agent tuning datasets. In response, we introduce MIMIR, a streamlined platform that leverages large LLMs to generate agent-tuning data for fine-tuning smaller, specialized models. By employing a role-playing methodology, MIMIR enables larger models to simulate various roles and create interaction data, which can then be used to fine-tune open-source models like LLaMA2. This approach ensures that even smaller models can effectively serve as agents in scientific tasks. Integrating these features into an end-to-end platform, MIMIR facilitates everything from the uploading of scientific data to one-click agent fine-tuning. MIMIR is publicly released and actively maintained at https://github. com/gersteinlab/MIMIR, along with a demo video for quick-start, calling for broader development.
WildVis: Open Source Visualizer for Million-Scale Chat Logs in the Wild
Authors: Yuntian Deng; Wenting Zhao; Jack Hessel; Xiang Ren; Claire Cardie; Yejin Choi
Venue: EMNLP 2024 (Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: The increasing availability of real-world conversation data offers exciting opportunities for researchers to study user-chatbot interactions. However, the sheer volume of this data makes manually examining individual conversations impractical. To overcome this challenge, we introduce WildVis, an interactive tool that enables fast, versatile, and large-scale conversation analysis. WildVis provides search and visualization capabilities in the text and embedding spaces based on a list of criteria. To manage million-scale datasets, we implemented optimizations including search index construction, embedding precomputation and compression, and caching to ensure responsive user interactions within seconds. We demonstrate WildVis’ utility through three case studies: facilitating chatbot misuse research, visualizing and comparing topic distributions across datasets, and characterizing user-specific conversation patterns. WildVis is open-source and designed to be extendable, supporting additional datasets and customized search and visualization functionalities.
Instruction-Driven Game Engine: A Poker Case Study
Authors: Hongqiu Wu; Xingyuan Liu; Yan Wang; Hai Zhao
Venue: EMNLP 2024 (Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: The Instruction-Driven Game Engine (IDGE) project aims to democratize game development by enabling a large language model (LLM) to follow free-form game descriptions and generate game-play processes. The IDGE allows users to create games simply by natural language instructions, which significantly lowers the barrier for game development. We approach the learning process for IDGEs as a Next State Prediction task, wherein the model autoregressively predicts the game states given player actions. The computation of game states must be precise; otherwise, slight errors could corrupt the game-play experience. This is challenging because of the gap between stability and diversity. To address this, we train the IDGE in a curriculum manner that progressively increases its exposure to complex scenarios.Our initial progress lies in developing an IDGE for Poker, which not only supports a wide range of poker variants but also allows for highly individualized new poker games through natural language inputs. This work lays the groundwork for future advancements in transforming how games are created and played.
LM-Interview: An Easy-to-use Smart Interviewer System via Knowledge-guided Language Model Exploitation
Authors: Hanming Li; Jifan Yu; Ruimiao Li; Zhanxin Hao; Yan Xuan; Jiaxi Yuan; Bin Xu; Juanzi Li; Zhiyuan Liu
Venue: EMNLP 2024 (Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations)

Abstract: Semi-structured interviews are a crucial method of data acquisition in qualitative research. Typically controlled by the interviewer, the process progresses through a question-and-answer format, aimed at eliciting information from the interviewee. However, interviews are highly time-consuming and demand considerable experience of the interviewers, which greatly limits the efficiency and feasibility of data collection. Therefore, we introduce LM-Interview, a novel system designed to automate the process of preparing, conducting and analyzing semi-structured interviews. Experimental results demonstrate that LM-interview achieves performance comparable to that of skilled human interviewers.
Optimizing Entity Resolution in Voice Interfaces: An ASR-Aware Entity Reference Expansion Approach
Authors: Jiangning Chen; Ziyun Zhang; Qianli Hu
Venue: EMNLP 2024 (Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track)

Abstract: This paper tackles the challenges presented by Automatic Speech Recognition (ASR) errors in voice-based dialog systems, specifically, their adverse impact on Entity Resolution (ER) as a downstream task. Navigating the equilibrium between accuracy and online retrieval’s speed requirement proves challenging, particularly when limited data links the failed mentions to resolved entities. In this paper, we propose a entity reference expansion system, injecting pairs of failed mentions and resolved entity names into the knowledge graph, enhancing its awareness of unresolved mentions. To address data scarcity, we introduce a synthetic data generation approach aligned with noise patterns. This, combined with an ASR-Error-Aware Loss function, facilitates the training of a RoBERTa model, which filters failed mentions and extracts entity pairs for knowledge graph expansion. These designs confront obstacles related to ASR noise, data limitations, and online entity retrieval.
Two-tiered Encoder-based Hallucination Detection for Retrieval-Augmented Generation in the Wild
Authors: Ilana Zimmerman; Jadin Tredup; Ethan Selfridge; Joseph Bradley
Venue: EMNLP 2024 (Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track)

Abstract: Detecting hallucinations, where Large Language Models (LLMs) are not factually consistent with a Knowledge Base (KB), is a challenge for Retrieval-Augmented Generation (RAG) systems. Current solutions rely on public datasets to develop prompts or fine-tune a Natural Language Inference (NLI) model. However, these approaches are not focused on developing an enterprise RAG system; they do not consider latency, train or evaluate on production data, nor do they handle non-verifiable statements such as small talk or questions. To address this, we leverage the customer service conversation data of four large brands to evaluate existing solutions and propose a set of small encoder models trained on a new dataset. We find the proposed models to outperform existing methods and highlight the value of combining a small amount of in-domain data with public datasets.
The Program Testing Ability of Large Language Models for Code
Authors: Weimin Xiong; Yiwen Guo; Hao Chen
Venue: EMNLP 2024 (Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track)

Abstract: Recent development of large language models (LLMs) for code like CodeX and CodeT5+ shows promise in achieving code intelligence. Their ability of synthesizing program targeting a pre-defined algorithmic coding task has been intensively tested and verified on datasets including HumanEval and MBPP. Yet, evaluation of these LLMs from more perspectives (than just program synthesis) is also anticipated, considering their broad scope of applications. In this paper, we explore their ability of automatic test cases generation. We show intriguing observations and reveal how the quality of their generated test cases can be improved. Following recent work which uses generated test cases to enhance program synthesis, we further leverage our findings in improving the quality of the synthesized programs and show +11.77% and +4.22% higher code pass rates on HumanEval+ comparing with the GPT-3.5-turbo baseline and the recent state-of-the-art, respectively. Our code is publicly available at https://github.com/asdasxzxcq/TestCaseGen.
Salient Information Prompting to Steer Content in Prompt-based Abstractive Summarization
Authors: Lei Xu; Mohammed Asad Karim; Saket Dingliwal; Aparna Elangovan
Venue: EMNLP 2024 (Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track)

Abstract: Large language models (LLMs) can generate fluent summaries across domains using prompting techniques, reducing the effort required for summarization applications. However, crafting effective prompts that guide LLMs to generate summaries with the appropriate level of detail and writing style remains a challenge. In this paper, we explore the use of salient information extracted from the source document to enhance summarization prompts. We show that adding keyphrases in prompts can improve ROUGE F1 and recall, making the generated summaries more similar to the reference and more complete. The number of keyphrases can control the precision-recall trade-off. Furthermore, our analysis reveals that incorporating phrase-level salient information is superior to word- or sentence-level. However, the impact on summary faithfulness is not universally positive across LLMs. To enable this approach, we introduce Keyphrase Signal Extractor (SigExt), a lightweight model that can be finetuned to extract salient keyphrases. By using SigExt, we achieve consistent ROUGE improvements across datasets and LLMs without any LLM customization. Our findings provide insights into leveraging salient information in building prompt-based summarization systems.
Predicting Entity Salience in Extremely Short Documents
Authors: Benjamin Bullough; Harrison Lundberg; Chen Hu; Weihang Xiao
Venue: EMNLP 2024 (Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track)

Abstract: A frequent challenge in applications that use entities extracted from text documents is selecting the most salient entities when only a small number can be used by the application (e.g., displayed to a user). Solving this challenge is particularly difficult in the setting of extremely short documents, such as the response from a digital assistant, where traditional signals of salience such as position and frequency are less likely to be useful. In this paper, we propose a lightweight and data-efficient approach for entity salience detection on short text documents. Our experiments show that our approach achieves competitive performance with respect to complex state-of-the-art models, such as GPT-4, at a significant advantage in latency and cost. In limited data settings, we show that a semi-supervised fine-tuning process can improve performance further. Furthermore, we introduce a novel human-labeled dataset for evaluating entity salience on short question-answer pair documents.