GenAI4ELab/papercli-papers huggingface dataset:
A Comparative Study of Prompting Strategies for Legal Text Classification
Authors: Ali Hakimi Parizi; Yuyang Liu; Prudhvi Nokku; Sina Gholamian; David Emerson
Venue: EMNLP 2023 (Proceedings of the Natural Legal Language Processing Workshop 2023)

Abstract: In this study, we explore the performance oflarge language models (LLMs) using differ-ent prompt engineering approaches in the con-text of legal text classification. Prior researchhas demonstrated that various prompting tech-niques can improve the performance of a di-verse array of tasks done by LLMs. However,in this research, we observe that professionaldocuments, and in particular legal documents,pose unique challenges for LLMs. We experi-ment with several LLMs and various promptingtechniques, including zero/few-shot prompting,prompt ensembling, chain-of-thought, and ac-tivation fine-tuning and compare the perfor-mance on legal datasets. Although the newgeneration of LLMs and prompt optimizationtechniques have been shown to improve gener-ation and understanding of generic tasks, ourfindings suggest that such improvements maynot readily transfer to other domains. Specifi-cally, experiments indicate that not all prompt-ing approaches and models are well-suited forthe legal domain which involves complexitiessuch as long documents and domain-specificlanguage.
Tracing Influence at Scale: A Contrastive Learning Approach to Linking Public Comments and Regulator Responses
Authors: Linzi Xing; Brad Hackinen; Giuseppe Carenini
Venue: EMNLP 2023 (Proceedings of the Natural Legal Language Processing Workshop 2023)

Abstract: U.S. Federal Regulators receive over one million comment letters each year from businesses, interest groups, and members of the public, all advocating for changes to proposed regulations. These comments are believed to have wide-ranging impacts on public policy. However, measuring the impact of specific comments is challenging because regulators are required to respond to comments but they do not have to specify which comments they are addressing. In this paper, we propose a simple yet effective solution to this problem by using an iterative contrastive method to train a neural model aiming for matching text from public comments to responses written by regulators. We demonstrate that our proposal substantially outperforms a set of selected text-matching baselines on a human-annotated test set. Furthermore, it delivers performance comparable to the most advanced gigantic language model (i.e., GPT-4), and is more cost-effective when handling comments and regulator responses matching in larger scale.
calamanCy: A Tagalog Natural Language Processing Toolkit
Authors: Lester James V. Miranda
Venue: EMNLP 2023 (Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023))

Abstract: We introduce calamanCy, an open-source toolkit for constructing natural language processing (NLP) pipelines for Tagalog. It is built on top of spaCy, enabling easy experimentation and integration with other frameworks. calamanCy addresses the development gap by providing a consistent API for building NLP applications and offering general-purpose multitask models with out-of-the-box support for dependency parsing, parts-of-speech (POS) tagging, and named entity recognition (NER). calamanCy aims to accelerate the progress of Tagalog NLP by consolidating disjointed resources in a unified framework.The calamanCy toolkit is available on GitHub: https://github.com/ljvmiranda921/calamanCy.
Jina Embeddings: A Novel Set of High-Performance Sentence Embedding Models
Authors: Michael Günther; Louis Milliken; Jonathan Geuter; Georgios Mastrapas; Bo Wang; Han Xiao
Venue: EMNLP 2023 (Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023))

Abstract: Jina Embeddings constitutes a set of high-performance sentence embedding models adept at translating textual inputs into numerical representations, capturing the semantics of the text. These models excel in applications like dense retrieval and semantic textual similarity. This paper details the development of Jina Embeddings, starting with the creation of high-quality pairwise and triplet datasets.It underlines the crucial role of data cleaning in dataset preparation, offers in-depth insights into the model training process, and concludes with a comprehensive performance evaluation using the Massive Text Embedding Benchmark (MTEB). Furthermore, to increase the model’s awareness of grammatical negation, we construct a novel training and evaluation dataset of negated and non-negated statements, which we make publicly available to the community.
Deepparse : An Extendable, and Fine-Tunable State-Of-The-Art Library for Parsing Multinational Street Addresses
Authors: David Beauchemin; Marouane Yassine
Venue: EMNLP 2023 (Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023))

Abstract: Segmenting an address into meaningful components, also known as address parsing, is an essential step in many applications from record linkage to geocoding and package delivery. Consequently, a lot of work has been dedicated to develop accurate address parsing techniques, with machine learning and neural network methods leading the state-of-the-art scoreboard. However, most of the work on address parsing has been confined to academic endeavours with little availability of free and easy-to-use open-source solutions.This paper presents Deepparse, a Python open-source, extendable, fine-tunable address parsing solution under LGPL-3.0 licence to parse multinational addresses using state-of-the-art deep learning algorithms and evaluated on over 60 countries. It can parse addresses written in any language and use any address standard. The pre-trained model achieves average 99% parsing accuracies on the countries used for training with no pre-processing nor post-processing needed. Moreover, the library supports fine-tuning with new data to generate a custom address parser.
PyThaiNLP: Thai Natural Language Processing in Python
Authors: Wannaphong Phatthiyaphaibun; Korakot Chaovavanich; Charin Polpanumas; Arthit Suriyawongkul; Lalita Lowphansirikul; Pattarawat Chormai; Peerat Limkonchotiwat; Thanathip Suntorntip; Can Udomcharoenchaikit
Venue: EMNLP 2023 (Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023))

Abstract: We present PyThaiNLP, a free and open-source natural language processing (NLP) library for Thai language implemented in Python. It provides a wide range of software, models, and datasets for Thai language. We first provide a brief historical context of tools for Thai language prior to the development of PyThaiNLP. We then outline the functionalities it provided as well as datasets and pre-trained language models. We later summarize its development milestones and discuss our experience during its development. We conclude by demonstrating how industrial and research communities utilize PyThaiNLP in their work. The library is freely available at https://github.com/pythainlp/pythainlp.
Empowering Knowledge Discovery from Scientific Literature: A novel approach to Research Artifact Analysis
Authors: Petros Stavropoulos; Ioannis Lyris; Natalia Manola; Ioanna Grypari; Haris Papageorgiou
Venue: EMNLP 2023 (Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023))

Abstract: Knowledge extraction from scientific literature is a major issue, crucial to promoting transparency, reproducibility, and innovation in the research community. In this work, we present a novel approach towards the identification, extraction and analysis of dataset and code/software mentions within scientific literature. We introduce a comprehensive dataset, synthetically generated by ChatGPT and meticulously curated, augmented, and expanded with real snippets of scientific text from full-text publications in Computer Science using a human-in-the-loop process. The dataset contains snippets highlighting mentions of the two research artifact (RA) types: dataset and code/software, along with insightful metadata including their Name, Version, License, URL as well as the intended Usage and Provenance. We also fine-tune a simple Large Language Model (LLM) using Low-Rank Adaptation (LoRA) to transform the Research Artifact Analysis (RAA) into an instruction-based Question Answering (QA) task. Ultimately, we report the improvements in performance on the test set of our dataset when compared to other base LLM models. Our method provides a significant step towards facilitating accurate, effective, and efficient extraction of datasets and software from scientific papers, contributing to the challenges of reproducibility and reusability in scientific research.
Zelda Rose: a tool for hassle-free training of transformer models
Authors: Loïc Grobol
Venue: EMNLP 2023 (Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023))

Abstract: Zelda Rose is a command line interface for pretraining transformer-based models. Its purpose is to enable an easy start for users interested in training these ubiquitous models, but unable or unwilling to engage with more comprehensive — but more complex — frameworks and the complex interactions between libraries for managing models, datasets and computations. Training a model requires no code on the user’s part and produce models directly compatible with the HuggingFace ecosystem, allowing quick and easy distribution and reuse. A particular care is given to lowering the cost of maintainability and future-proofing, by making the code as modular as possible and taking advantage of third-party libraries to limit ad-hoc code to the strict minimum.
GPT4All: An Ecosystem of Open Source Compressed Language Models
Authors: Yuvanesh Anand; Zach Nussbaum; Adam Treat; Aaron Miller; Richard Guo; Benjamin Schmidt; Brandon Duderstadt; Andriy Mulyar
Venue: EMNLP 2023 (Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023))

Abstract: Large language models (LLMs) have recently achieved human-level performance on a range of professional and academic benchmarks.The accessibility of these models has lagged behind their performance.State-of-the-art LLMs require costly infrastructure; are only accessible via rate-limited, geo-locked, and censored web interfaces; and lack publicly available code and technical reports.In this paper, we tell the story of GPT4All, a popular open source repository that aims to democratize access to LLMs.We outline the technical details of the original GPT4All model family, as well as the evolution of the GPT4All project from a single model into a fully fledged open source ecosystem.It is our hope that this paper acts as both a technical overview of the original GPT4All models as well as a case study on the subsequent growth of the GPT4All open source ecosystem.
Kani: A Lightweight and Highly Hackable Framework for Building Language Model Applications
Authors: Andrew Zhu; Liam Dugan; Alyssa Hwang; Chris Callison-Burch
Venue: EMNLP 2023 (Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023))

Abstract: Language model applications are becoming increasingly popular and complex, often including features like tool usage and retrieval augmentation. However, existing frameworks for such applications are often opinionated, deciding for developers how their prompts ought to be formatted and imposing limitations on customizability and reproducibility. To solve this we present Kani: a lightweight, flexible, and model-agnostic open-source framework for building language model applications. Kani helps developers implement a variety of complex features by supporting the core building blocks of chat interaction: model interfacing, chat management, and robust function calling. All Kani core functions are easily overridable and well documented to empower developers to customize functionality for their own needs. Kani thus serves as a useful tool for researchers, hobbyists, and industry professionals alike to accelerate their development while retaining interoperability and fine-grained control.
Beyond the Repo: A Case Study on Open Source Integration with GECToR
Authors: Sanjna Kashyap; Zhaoyang Xie; Kenneth Steimel; Nitin Madnani
Venue: EMNLP 2023 (Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023))

Abstract: We present a case study describing our efforts to integrate the open source GECToR code and models into our production NLP pipeline that powers many of Educational Testing Service’s products and prototypes. The paper’s contributions includes a discussion of the issues we encountered during integration and our solutions, the overarching lessons we learned about integrating open source projects, and, last but not least, the open source contributions we made as part of the journey.
Two Decades of the ACL Anthology: Development, Impact, and Open Challenges
Authors: Marcel Bollmann; Nathan Schneider; Arne Köhn; Matt Post
Venue: EMNLP 2023 (Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023))

Abstract: The ACL Anthology is a prime resource for research papers within computational linguistics and natural language processing, while continuing to be an open-source and community-driven project. Since Gildea et al. (2018) reported on its state and planned directions, the Anthology has seen major technical changes. We discuss what led to these changes and how they impact long-term maintainability and community engagement, describe which open-source data and software tools the Anthology currently provides, and provide a survey of literature that has used the Anthology as a main data source.
nanoT5: Fast & Simple Pre-training and Fine-tuning of T5 Models with Limited Resources
Authors: Piotr Nawrot
Venue: EMNLP 2023 (Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023))

Abstract: State-of-the-art language models like T5 have revolutionized the NLP landscape, but their computational demands hinder a large portion of the research community. To address this challenge, we present nanoT5, a specially-optimized PyTorch framework for efficient pre-training and fine-tuning of T5 models. Drawing on insights from optimizer differences and prioritizing efficiency, nanoT5 allows a T5-Base model to be pre-trained on a single GPU in just 16 hours, without any loss in performance. With the introduction of this open-source framework, we hope to widen the accessibility to language modelling research and cater to the community’s demand for more user-friendly T5 (Encoder-Decoder) implementations. We make our contributions, including configurations, codebase, pre-training insights, and pre-trained models, available to the public.
AWARE-TEXT: An Android Package for Mobile Phone Based Text Collection and On-Device Processing
Authors: Salvatore Giorgi; Garrick Sherman; Douglas Bellew; Sharath Chandra Guntuku; Lyle Ungar; Brenda Curtis
Venue: EMNLP 2023 (Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023))

Abstract: We present the AWARE-text package, an open-source software package for collecting textual data on Android mobile devices. This package allows for collecting short message service (SMS or text messages) and character-level keystrokes. In addition to collecting this raw data, AWARE-text is designed for on device lexicon processing, which allows one to collect standard textual-based measures (e.g., sentiment, emotions, and topics) without collecting the underlying raw textual data. This is especially important in the case of mobile phones, which can contain sensitive and identifying information. Thus, the AWARE-text package allows for privacy protection while simultaneously collecting textual information at multiple levels of granularity: person (lifetime history of SMS), conversation (both sides of SMS conversations and group chats), message (single SMS), and character (individual keystrokes entered across applications). Finally, the unique processing environment of mobile devices opens up several methodological and privacy issues, which we discuss.
SOTASTREAM: A Streaming Approach to Machine Translation Training
Authors: Matt Post; Thamme Gowda; Roman Grundkiewicz; Huda Khayrallah; Rohit Jain; Marcin Junczys-Dowmunt
Venue: EMNLP 2023 (Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023))

Abstract: Many machine translation toolkits make use of a data preparation step wherein raw data is transformed into a tensor format that can be used directly by the trainer. This preparation step is increasingly at odds with modern research and development practices because this process produces a static, unchangeable version of the training data, making common training-time needs difficult (e.g., subword sampling), time-consuming (preprocessing with large data can take days), expensive (e.g., disk space), and cumbersome (managing experiment combinatorics). We propose an alternative approach that separates the generation of data from the consumption of that data. In this approach, there is no separate pre-processing step; data generation produces an infinite stream of permutations of the raw training data, which the trainer tensorizes and batches as it is consumed. Additionally, this data stream can be manipulated by a set of user-definable operators that provide on-the-fly modifications, such as data normalization, augmentation or filtering. We release an open-source toolkit, SOTASTREAM, that implements this approach: https://github.com/marian-nmt/sotastream. We show that it cuts training time, adds flexibility, reduces experiment management complexity, and reduces disk space, all without affecting the accuracy of the trained models.
An Open-source Web-based Application for Development of Resources and Technologies in Underresourced Languages
Authors: Siddharth Singh; Shyam Ratan; Neerav Mathur; Ritesh Kumar
Venue: EMNLP 2023 (Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023))

Abstract: The paper discusses the Linguistic Field Data Management and Analysis System (LiFE), a new open-source, web-based software that systematises storage, management, annotation, analysis and sharing of linguistic data gathered from the field as well as that crawled from various sources on the web such as YouTube, Twitter, Facebook, Instagram, Blog, Newspaper, Wikipedia, etc. The app supports two broad workflows - (a) the field linguists’ workflow in which data is collected directly from the speakers in the field and analysed further to produce grammatical descriptions, lexicons, educational materials and possibly language technologies; (b) the computational linguists’ workflow in which data collected from the web using automated crawlers or digitised using manual or semi-automatic means, annotated for various tasks and then used for developing different kinds of language technologies. In addition to supporting these workflows, the app provides some additional features as well - (a) it allows multiple users to collaboratively work on the same project via its granular access control and sharing option; (b) it allows the data to be exported to various formats including CSV, TSV, JSON, XLSX, , PDF, Textgrid, RDF (different serialisation formats) etc as appropriate; (c) it allows data import from various formats viz. LIFT XML, XLSX, JSON, CSV, TSV, Textgrid, etc; (d) it allows users to start working in the app at any stage of their work by giving the option to either create a new project from scratch or derive a new project from an existing project in the app.The app is currently available for use and testing on our server (http://life.unreal-tece.co.in/) and its source code has been released under AGPL license on our GitHub repository (https://github.com/unrealtecellp/life). It is licensed under separate, specific conditions for commercial usage.
Rumour Detection in the Wild: A Browser Extension for Twitter
Authors: Andrej Jovanovic; Björn Ross
Venue: EMNLP 2023 (Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023))

Abstract: Rumour detection, particularly on social media, has gained popularity in recent years. The machine learning community has made significant contributions in investigating automatic methods to detect rumours on such platforms. However, these state-of-the-art (SoTA) models are often deployed by social media companies; ordinary end-users cannot leverage the solutions in the literature for their own rumour detection. To address this issue, we put forward a novel browser extension that allows these users to perform rumour detection on Twitter. Particularly, we leverage the performance from SoTA architectures, which has not been done previously. Initial results from a user study confirm that this browser extension provides benefit. Additionally, we examine the performance of our browser extension’s rumour detection model in a simulated deployment environment. Our results show that additional infrastructure for the browser extension is required to ensure its usability when deployed as a live service for Twitter users at scale.
DeepZensols: A Deep Learning Natural Language Processing Framework for Experimentation and Reproducibility
Authors: Paul Landes; Barbara Di Eugenio; Cornelia Caragea
Venue: EMNLP 2023 (Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023))

Abstract: Given the criticality and difficulty of reproducing machine learning experiments, there have been significant efforts in reducing the variance of these results. The ability to consistently reproduce results effectively strengthens the underlying hypothesis of the work and should be regarded as important as the novel aspect of the research itself. The contribution of this work is an open source framework that has the following characteristics: a) facilitates reproducing consistent results, b) allows hot-swapping features and embeddings without further processing and re-vectorizing the dataset, c) provides a means of easily creating, training and evaluating natural language processing deep learning models with little to no code changes, and d) is freely available to the community.
Improving NER Research Workflows with SeqScore
Authors: Constantine Lignos; Maya Kruse; Andrew Rueda
Venue: EMNLP 2023 (Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023))

Abstract: We describe the features of SeqScore, an MIT-licensed Python toolkit for working with named entity recognition (NER) data.While SeqScore began as a tool for NER scoring, it has been expanded to help with the full lifecycle of working with NER data: validating annotation, providing at-a-glance and detailed summaries of the data, modifying annotation to support experiments, scoring system output, and aiding with error analysis.SeqScore is released via PyPI (https://pypi.org/project/seqscore/) and development occurs on GitHub (https://github.com/bltlab/seqscore).