GenAI4ELab/papercli-papers huggingface dataset:
AutoRef: Generating Refinements of Reviews Given Guidelines
Authors: Soham Chitnis; Manasi Patwardhan; Ashwin Srinivasan; Tanmay Tulsidas Verlekar; Lovekesh Vig; Gautam Shroff
Venue: ACL 2024 (Proceedings of the Fourth Workshop on Scholarly Document Processing (SDP 2024))

Abstract: When examining reviews of research papers, we can distinguish between two hypothetical referees: the maximally lenient referee who accepts any paper with a vacuous review and the maximally strict one who rejects any paper with an overly pedantic review. Clearly, both are of no practical value. Our interest is in a referee who makes a balanced judgement and provides a review abiding by the guidelines. In this paper, we present a case study of automatic correction of an existing machine-generated or human review. The AutoRef\ system implements an iterative approach that progressively “refines” a review by attempting to make it more compliant with pre-defined requirements of a “good” review. It implements the following steps: (1) Translate the review requirements into a specification in natural language, of “yes/no” questions; (2) Given a (paper,review) pair, extract answers to the questions; (3) Use the results in (2) to generate a new review; and (4) Return to Step (2) with the paper and the new review. Here, (2) and (3) are implemented by large language model (LLM) based agents. We present a case study using papers and reviews made available for the International Conference on Learning Representations (ICLR). Our initial empirical results suggest that AutoRef\ progressively improves the compliance of the generated reviews to the specification. Currently designed specification makes AutoRef\ progressively generate reviews which are stricter, making the decisions more inclined towards “rejections”. This demonstrates the applicability of $AutoRef $ for: (1) The progressive correction of overly lenient reviews, being useful for referees and meta-reviewers; and (2) The generation of progressively stricter reviews for a paper, starting from a vacuous review (“Great paper. Accept.”), facilitating authors when trying to assess weaknesses in their papers.
Artificial Intuition: Efficient Classification of Scientific Abstracts
Authors: Harsh Sakhrani; Naseela Pervez; Anirudh Ravikumar; Fred Morstatter; Alexandra Graddy-Reed; Andrea Belz
Venue: ACL 2024 (Proceedings of the Fourth Workshop on Scholarly Document Processing (SDP 2024))

Abstract: It is desirable to coarsely classify short scientific texts, such as grant or publication abstracts, for strategic insight or research portfolio management. These texts efficiently transmit dense information to experts possessing a rich body of knowledge to aid interpretation. Yet this task is remarkably difficult to automate because of brevity and the absence of context. To address this gap, we have developed a novel approach to generate and appropriately assign coarse domain-specific labels. We show that a Large Language Model (LLM) can provide metadata essential to the task, in a process akin to the augmentation of supplemental knowledge representing human intuition, and propose a workflow. As a pilot study, we use a corpus of award abstracts from the National Aeronautics and Space Administration (NASA). We develop new assessment tools in concert with established performance metrics.
Synthetic Context with LLM for Entity Linking from Scientific Tables
Authors: Yuji Oshima; Hiroyuki Shindo; Hiroki Teranishi; Hiroki Ouchi; Taro Watanabe
Venue: ACL 2024 (Proceedings of the Fourth Workshop on Scholarly Document Processing (SDP 2024))

Abstract: Tables in scientific papers contain crucial information, such as experimental results.Entity Linking (EL) is a promising technology that analyses tables and associates them with a knowledge base.EL for table cells requires identifying the referent concept of each cell while understanding the context relevant to each cell in the paper. However, extracting the relevant context from the paper is challenging because the relevant parts are scattered in the main text and captions.This study defines a rule-based method for extracting broad context from the main text, including table captions and sentences that mention the table.Furthermore, we propose synthetic context as a more refined context generated by large language models (LLMs).In a synthetic context, contexts from the entire paper are refined by summarizing, injecting supplemental knowledge, and clarifying the referent concept.We observe this approach improves accuracy for EL by more than 10 points on the S2abEL dataset, and our qualitative analysis suggests potential future works.
Papilusion at DAGPap24: Paper or Illusion? Detecting AI-generated Scientific Papers
Authors: Nikita Andreev; Alexander Shirnin; Vladislav Mikhailov; Ekaterina Artemova
Venue: ACL 2024 (Proceedings of the Fourth Workshop on Scholarly Document Processing (SDP 2024))

Abstract: This paper presents Papilusion, an AI-generated scientific text detector developed within the DAGPap24 shared task on detecting automatically generated scientific papers. We propose an ensemble-based approach and conduct ablation studies to analyze the effect of the detector configurations on the performance. Papilusion is ranked 6th on the leaderboard, and we improve our performance after the competition ended, achieving 99.46 (+9.63) of the F1-score on the official test set.
Multi-head Span-based Detector for AI-generated Fragments in Scientific Papers
Authors: German Gritsai; Ildar Khabutdinov; Andrey Grabovoy
Venue: ACL 2024 (Proceedings of the Fourth Workshop on Scholarly Document Processing (SDP 2024))

Abstract: This paper describes a system designed to distinguish between AI-generated and human-written scientific excerpts in the DAGPap24 competition hosted within the Fourth Workshop on Scientific Document Processing. In this competition the task is to find artificially generated token-level text fragments in documents of a scientific domain. Our work focuses on the use of a multi-task learning architecture with two heads. The application of this approach is justified by the specificity of the task, where class spans are continuous over several hundred characters. We considered different encoder variations to obtain a state vector for each token in the sequence, as well as a variation in splitting fragments into tokens to further feed into the input of a transform-based encoder. This approach allows us to achieve a 9% quality improvement relative to the baseline solution score on the development set (from 0.86 to 0.95) using the average macro F1-score, as well as a score of 0.96 on a closed test part of the dataset from the competition.
Guiding Large Language Models via External Attention Prompting for Scientific Extreme Summarization
Authors: Yuan Chang; Ziyue Li; Xiaoqiu Le
Venue: ACL 2024 (Proceedings of the Fourth Workshop on Scholarly Document Processing (SDP 2024))

Abstract: Scientific extreme summarization, the task of generating concise one-sentence summaries (TLDRs) for scientific papers, presents significant challenges due to the need for deep domain-specific understanding and the ability to distill salient information. This study identifies the critical role of titles and keywords in enhancing TLDR generation through quantitative analysis. We propose a novel method, External Attention Prompting (EAP), which leverages LLMs by guiding them to focus on the most critical parts of the source text through varying degrees of attention signals. Our method employs Markdown emphasis syntax to annotate attention levels, enabling LLMs to prioritize salient information effectively. Extensive experiments demonstrate that EAP significantly outperforms baseline methods across various LLMs and metrics in both zero-shot and few-shot settings. Further evaluations by GPT-4 demonstrate that EAP can enable LLMs to generate TLDRs of higher human-aligned quality.
Simulating Expert Discussions with Multi-agent for Enhanced Scientific Problem Solving
Authors: Ziyue Li; Yuan Chang; Xiaoqiu Le
Venue: ACL 2024 (Proceedings of the Fourth Workshop on Scholarly Document Processing (SDP 2024))

Abstract: Large Language Models (LLMs) have shown remarkable potential across various domains, yet their application in addressing complex scientific problems remains a formidable challenge. This paper presents a novel methodology to augment the problem-solving capabilities of LLMs by assigning them roles as domain-specific experts. By simulating a panel of experts, each LLM is tasked with delivering professional and cautious responses to scientific inquiries. Our approach involves querying multiple LLMs and assessing the consistency of their responses. High agreement among the LLMs suggests greater confidence in the proposed solution, whereas discrepancies prompt a collaborative discussion among the LLMs to reach a consensus. This method emulates real-world scientific problem-solving processes, fostering a more reliable and robust mechanism for LLMs to tackle scientific questions. Our experimental results show that assigning roles to multiple LLMs as domain-specific experts significantly improves their accuracy and reliability in solving scientific problems. This framework has the potential to advance the application of AI in scientific research, enhancing its effectiveness and trustworthiness.
An Analysis of Tasks and Datasets in Peer Reviewing
Authors: Moritz Staudinger; Wojciech Kusa; Florina Piroi; Allan Hanbury
Venue: ACL 2024 (Proceedings of the Fourth Workshop on Scholarly Document Processing (SDP 2024))

Abstract: Taking note of the current challenges of the peer review system, this paper inventories the research tasks for analysing and possibly automating parts of the reviewing process, like matching submissions with a reviewer’s domain of expertise. For each of these tasks we list their associated datasets, analysing their quality in terms of available documentation of creation and use. Building up on this, we give a set of recommendations to take into account when collecting and releasing data.
Zero-shot Scientific Claim Verification Using LLMs and Citation Text
Authors: Carlos Alvarez; Maxwell Bennett; Lucy Wang
Venue: ACL 2024 (Proceedings of the Fourth Workshop on Scholarly Document Processing (SDP 2024))

Abstract: Due to rapidly changing and advancing science, it is important to check the veracity of scientific claims and whether they are supported by research evidence. Previous versions of this task depended on supervised training, where labeled datasets were constructed through manual claim writing and evidence identification, sometimes coupled with mining citation relationships in papers. In this work, we investigate whether zero-shot scientific claim verification could be enabled using large language models (LLMs) and distant supervision examples taken directly from citation texts. We derive an in-context learning (ICL) dataset, SCitance, consisting of citation sentences (“citances”), LLM-generated negations, evidence documents, and veracity labels, and find that prompting GPT-4 with ICL examples from this dataset yields comparable performance (within 1 point F1) to previous finetuned models trained on manually curated claim-evidence pairs. Our results suggest that prompting LLMs with citance-evidence pairs directly poses a viable alternative to finetuning scientific claim verification models with manually-curated data.
Researcher Representations Based on Aggregating Embeddings of Publication Titles: A Case Study in a Japanese Academic Database
Authors: Hiroyoshi Nagao; Marie Katsurai
Venue: ACL 2024 (Proceedings of the Fourth Workshop on Scholarly Document Processing (SDP 2024))

Abstract: Constructing researcher representations is crucial for search and recommendation in academic databases. While recent studies presented methods based on knowledge graph embeddings, obtaining a complete graph of academic entities might be sometimes challenging due to the lack of linked data.By contrast, the textual list of publications of each researcher, which represents their research interests and expertise, is usually easy to obtain.Therefore, this study focuses on creating researcher representations based on textual embeddings of their publication titles and assesses their practicality. We aggregate embeddings of each researcher’s multiple publications into a single vector and apply it to research field classification and similar researcher search tasks. We experimented with multiple language models and embedding aggregation methods to compare their performance.From the model perspective, we confirmed the effectiveness of using sentence embedding models and a simple averaging approach.
CoSAEmb: Contrastive Section-aware Aspect Embeddings for Scientific Articles
Authors: Shruti Singh; Mayank Singh
Venue: ACL 2024 (Proceedings of the Fourth Workshop on Scholarly Document Processing (SDP 2024))

Abstract: Research papers are long documents that contain information about various aspects such as background, prior work, methodology, and results. Existing works on scientific document representation learning only leverage the title and abstract of the paper. We present CoSAEmb, a model that learns representations from the full text of 97402 scientific papers from the S2ORC dataset. We present a novel supervised contrastive training framework for long documents using triplet loss and margin gradation. Our framework can be used to learn representations of long documents with any existing encoder-only transformer model without retraining it from scratch. CoSAEmb shows improved performance on information retrieval from the paper’s full text in comparison to models trained only on paper titles and abstracts. We also evaluate CoSAEmb on SciRepEval and CSFCube benchmarks, showing comparable performance with existing state-of-the-art models.
Integrating Table Representations into Large Language Models for Improved Scholarly Document Comprehension
Authors: Buse Sibel Korkmaz; Antonio Del Rio Chanona
Venue: ACL 2024 (Proceedings of the Fourth Workshop on Scholarly Document Processing (SDP 2024))

Abstract: We address the challenge of interpreting and reasoning over scientific tables with Large Language Models (LLMs), a crucial aspect of scholarly documents. Despite significant progress in natural language processing, the integration of tabular data into scientific LLMs remains limited. We propose an innovative approach leveraging intermediate task pre-training on table question-answering datasets, followed by model adaptation to comprehend tables in computer science literature. Our findings reveal that incorporating table understanding substantially improves the performance of LLMs on scientific literature understanding tasks, which we showcase in peer-review score prediction. This improvement underscores the importance of utilizing tabular data in the training of scientific language models. The code and models are publicly available at [this link](https://github.com/buseskorkmaz/Integrating-Table-Representations-into-LLMs).
Harnessing CLIP for Evidence Identification in Scientific Literature: A Multimodal Approach to Context24 Shared Task
Authors: Anukriti Kumar; Lucy Wang
Venue: ACL 2024 (Proceedings of the Fourth Workshop on Scholarly Document Processing (SDP 2024))

Abstract: Knowing whether scientific claims are supported by evidence is fundamental to scholarly communication and evidence-based decision-making. We present our approach to Task 1 of the Context24 Shared Task—Contextualizing Scientific Figures and Tables (SDP@ACL2024), which focuses on identifying multimodal evidence from scientific publications that support claims. We finetune CLIP, a state-of-the-art model for image-text similarity tasks, to identify and rank figures and tables in papers that substantiate specific claims. Our methods focus on text and image preprocessing techniques and augmenting the organizer-provided training data with labeled examples from the SciMMIR and MedICaT datasets. Our best-performing model achieved NDCG@5 and NDCG@10 values of 0.26 and 0.30, respectively, on the Context24 test split. Our findings underscore the effectiveness of data augmentation and preprocessing in improving the model’s ability in evidence matching.
CSIRO at Context24: Contextualising Scientific Figures and Tables in Scientific Literature
Authors: Necva Bölücü; Vincent Nguyen; Roelien C. Timmer; Huichen Yang; Maciej Rybinski; Stephen Wan; Sarvnaz Karimi
Venue: ACL 2024 (Proceedings of the Fourth Workshop on Scholarly Document Processing (SDP 2024))

Abstract: Finding evidence for claims from content presented in experimental results of scientific articles is difficult. The evidence is often presented in the form of tables and figures, and correctly matching it to scientific claims presents automation challenges. The Context24 shared task is launched to support the development of systems able to verify claims by extracting supporting evidence from articles. We explore different facets of this shared task modelled as a search problem and as an information extraction task. We experiment with a range of methods in each of these categories for the two sub-tasks of evidence identification and grounding context identification in the Context24 shared task.
OSX at Context24: How Well Can GPT Tackle Contexualizing Scientific Figures and Tables
Authors: Tosho Hirasawa
Venue: ACL 2024 (Proceedings of the Fourth Workshop on Scholarly Document Processing (SDP 2024))

Abstract: Identifying the alignment between different parts of a scientific paper is fundamental to scholarly document processing.In the Context24 shared task, participants are given a scientific claim and asked to identify (1) key figures or tables that support the claim and (2) methodological details.While employing a supervised approach to train models on task-specific data is a prevailing strategy for both subtasks, such an approach is not feasible for low-resource domains.Therefore, this paper introduces data-free systems supported by Large Language Models.We propose systems based on GPT-4o and GPT-4-turbo for each task.The experimental results reveal the zero-shot capabilities of GPT-4* in both tasks.
Automatic Quote Attribution in Chinese Literary Works
Authors: Xingxing Yang; Yu Wang
Venue: ACL 2024 (Proceedings of the 10th SIGHAN Workshop on Chinese Language Processing (SIGHAN-10))

Abstract: Quote attribution in fiction refers to the extraction of dialogues and speaker identification of dialogues, which can be divided into 2 steps: quotation annotation and speaker annotation. We use a pipeline for quote attribution, which involves classification, extractive QA, multi-choice QA, and coreference resolution. We also had an evaluation of our model performance by predicting explicit and implicit speakers using a combination of different models.
TeleChat: An Open-source Billingual Large Language Model
Authors: Zihan Wang; XinZhang Liu; Shixuan Liu; Yitong Yao; Yunyao Huang; Mengxiang Li; Zhongjiang He; Yongxian Li; Luwen Pu; Huinan Xu; Chao Wang; Shuangyong Song
Venue: ACL 2024 (Proceedings of the 10th SIGHAN Workshop on Chinese Language Processing (SIGHAN-10))

Abstract: In this paper, we present TeleChat, a collection of large language models (LLMs) with parameters of 7 billion and 12 billion. TeleChat is initially pretrained on an extensive corpus containing a diverse collection of texts from both English and Chinese languages, encompassing trillions of tokens. Subsequently, the model undergoes fine-tuning to align with human preferences, following a detailed methodology that we describe. We evaluate the performance of TeleChat on various tasks, including general dialogue generation, language understanding, mathematics, reasoning, code generation, and knowledge-based question answering. Our findings indicate that TeleChat achieves state-of-the-art performance to other open-source models of similar size across a wide range of public benchmarks. To support future research and applications utilizing LLMs, we release the fine-tuned model checkpoints of TeleChat-7B and TeleChat-12B, along with code and a portion of our filtered high-quality pretraining data, to the public community.
Few-shot Question Generation for Reading Comprehension
Authors: Yin Poon; John Sie Yuen Lee; Yu Yan Lam; Wing Lam Suen; Elsie Li Chen Ong; Samuel Kai Wah Chu
Venue: ACL 2024 (Proceedings of the 10th SIGHAN Workshop on Chinese Language Processing (SIGHAN-10))

Abstract: According to the internationally recognized PIRLS (Progress in International Reading Literacy Study) assessment standards, reading comprehension questions should require not only information retrieval, but also higher-order processes such as inferencing, interpreting and evaluation. However, these kinds of questions are often not available in large quantities for training question generation models. This paper investigates whether pre-trained Large Language Models (LLMs) can produce higher-order questions. Human assessment on a Chinese dataset shows that few-shot LLM prompting generates more usable and higher-order questions than two competitive neural baselines.
Adversarial Learning for Multi-Lingual Entity Linking
Authors: Bingbing Wang; Bin Liang; Zhixin Bai; Yongzhuo Ma
Venue: ACL 2024 (Proceedings of the 10th SIGHAN Workshop on Chinese Language Processing (SIGHAN-10))

Abstract: Entity linking aims to identify mentions from the text and link them to a knowledge base. Further, Multi-lingual Entity Linking (MEL) is a more challenging task, where the language-specific mentions need to be linked to a multi-lingual knowledge base. To tackle the MEL task, we propose a novel model that employs the merit of adversarial learning and few-shot learning to generalize the learning ability across languages. Specifically, we first randomly select a fraction of language-agnostic unlabeled data as the language signal to construct the language discriminator. Based on it, we devise a simple and effective adversarial learning framework with two characteristic branches, including an entity classifier and a language discriminator with adversarial training. Experimental results on two benchmark datasets indicate the excellent performance in few-shot learning and the effectiveness of the proposed adversarial learning framework.
Incremental pre-training from smaller language models
Authors: Han Zhang; Hui Wang; Ruifeng Xu
Venue: ACL 2024 (Proceedings of the 10th SIGHAN Workshop on Chinese Language Processing (SIGHAN-10))

Abstract: Large language models have recently become a new learning paradigm and led to state-of-the-art performance across a range of tasks. As explosive open-source pre-trained models are available, it is worth investigating how to better utilize existing models. We propose a simple yet effective method, Incr-Pretrain, for incrementally pre-training language models from smaller well-trained source models. Different layer-wise transfer strategies were introduced for model augmentation including parameter copying, initial value padding, and model distillation. Experiments on multiple zero-shot learning tasks demonstrate satisfying inference performance upon transferring and promising training efficiency during continuing pre-training. Compared to training from scratch, Incr-Pretrain can save up to half the training time to get a similar testing loss.