GenAI4ELab/papercli-papers huggingface dataset:
Improving Chain-of-Thought Reasoning via Quasi-Symbolic Abstractions
Authors: Leonardo Ranaldi; Marco Valentino; Andre Freitas
Venue: ACL 2025 (Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers))

Abstract: Chain-of-Though (CoT) represents a common strategy for reasoning in Large Language Models (LLMs) by decomposing complex tasks into intermediate inference steps. However, explanations generated via CoT are susceptible to content biases that negatively affect their robustness and faithfulness. To mitigate existing limitations, recent work has proposed using logical formalisms coupled with external symbolic solvers. However, fully symbolic approaches possess the bottleneck of requiring a complete translation from natural language to formal languages, a process that affects efficiency and flexibility. To achieve a trade-off, this paper investigates methods to disentangle content from logical reasoning without a complete formalisation. In particular, we present QuaSAR (for Quasi-Symbolic Abstract Reasoning), a variation of CoT that guides LLMs to operate at a higher level of abstraction via quasi-symbolic explanations. Our framework leverages the capability of LLMs to formalise only relevant variables and predicates, enabling the coexistence of symbolic elements with natural language. We show the impact of QuaSAR for in-context learning and for constructing demonstrations to improve the reasoning capabilities of smaller models. Our experiments show that quasi-symbolic abstractions can improve CoT-based methods by up to 8% accuracy, enhancing robustness and consistency on challenging adversarial variations on both natural language (i.e. MMLU-Redux) and symbolic reasoning tasks (i.e., GSM-Symbolic).
Information Extraction from Visually Rich Documents using LLM-based Organization of Documents into Independent Textual Segments
Authors: Aniket Bhattacharyya; Anurag Tripathi; Ujjal Das; Archan Karmakar; Amit Pathak; Maneesh Gupta
Venue: ACL 2025 (Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers))

Abstract: Information extraction (IE) from Visually Rich Documents (VRDs) containing layout features along with text is a critical and well-studied task. Specialized non-LLM NLP-based solutions typically involve training models using both textual and geometric information to label sequences/tokens as named entities or answers to specific questions. However, these approaches lack reasoning, are not able to infer values not explicitly present in documents, and do not generalize well to new formats. Generative LLMs-based approaches proposed recently are capable of reasoning, but struggle to comprehend clues from document layout especially in previously unseen document formats, and do not show competitive performance in heterogeneous VRD benchmark datasets. In this paper, we propose BLOCKIE, a novel LLM-based approach that organizes VRDs into localized, reusable semantic textual segments called semantic blocks, which are processed independently. Through focused and more generalizable reasoning,our approach outperforms the state-of-the-art on public VRD benchmarks by 1-3% in F1 scores, is resilient to document formats previously not encountered and shows abilities to correctly extract information not explicitly present in documents.
Enhancing Open-Domain Task-Solving Capability of LLMs via Autonomous Tool Integration from GitHub
Authors: Bohan Lyu; Xin Cong; Heyang Yu; Pan Yang; Cheng Qian; Zihe Wang; Yujia Qin; Yining Ye; Yaxi Lu; Chen Qian; Zhong Zhang; Yukun Yan; Yankai Lin; Zhiyuan Liu; Maosong Sun
Venue: ACL 2025 (Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers))

Abstract: Large Language Models (LLMs) excel in traditional natural language processing tasks but struggle with problems that require complex domain-specific calculations or simulations. While equipping LLMs with external tools to build LLM-based agents can enhance their capabilities, existing approaches lack the flexibility to address diverse and ever-evolving user queries in open domains. Currently, there is also no existing dataset that evaluates LLMs on open-domain knowledge that requires tools to solve. To this end, we introduce OpenAct benchmark to evaluate the open-domain task-solving capability, which is built on human expert consultation and repositories in GitHub. It comprises 339 questions spanning 7 diverse domains that need to be solved with domain-specific methods. In our experiments, even state-of-the-art LLMs and LLM-based agents demonstrate unsatisfactory success rates, underscoring the need for a novel approach.Furthermore, we present OpenAgent, a novel LLM-based agent system that can tackle evolving queries in open domains through autonomously integrating specialized tools from GitHub. OpenAgent employs 1) a hierarchical framework where specialized agents handle specific tasks and can assign tasks to inferior agents, 2) a bi-level experience learning mechanism to learn from both humans’ and its own experiences to tackle tool flaws. Experiments demonstrate its superior effectiveness and efficiency, which significantly outperforms baselines. Our data and code are open-source at https://github.com/OpenBMB/OpenAct.
LLMs Can Simulate Standardized Patients via Agent Coevolution
Authors: Zhuoyun Du; Lujie Zheng; Renjun Hu; Yuyang Xu; Xiawei Li; Ying Sun; Wei Chen; Jian Wu; Haolei Cai; Haochao Ying
Venue: ACL 2025 (Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers))

Abstract: Training medical personnel using standardized patients (SPs) remains a complex challenge, requiring extensive domain expertise and role-specific practice. Most research on Large Language Model (LLM)-based simulated patients focuses on improving data retrieval accuracy or adjusting prompts through human feedback. However, this focus has overlooked the critical need for patient agents to learn a standardized presentation pattern that transforms data into human-like patient responses through unsupervised simulations. To address this gap, we propose EvoPatient, a novel simulated patient framework in which a patient agent and doctor agents simulate the diagnostic process through multi-turn dialogues, simultaneously gathering experience to improve the quality of both questions and answers, ultimately enabling human doctor training. Extensive experiments on various cases demonstrate that, by providing only overall SP requirements, our framework improves over existing reasoning methods by more than 10% in requirement alignment and better human preference, while achieving an optimal balance of resource consumption after evolving over 200 cases for 10 hours, with excellent generalizability. Our system will be available at https://github.com/ZJUMAI/EvoPatient
Donate or Create? Comparing Data Collection Strategies for Emotion-labeled Multimodal Social Media Posts
Authors: Christopher Bagdon; Aidan Combs; Carina Silberer; Roman Klinger
Venue: ACL 2025 (Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers))

Abstract: Accurate modeling of subjective phenomena such as emotion expression requires data annotated with authors’ intentions. Commonly such data is collected by asking study participants to donate and label genuine content produced in the real world, or create content fitting particu- lar labels during the study. Asking participants to create content is often simpler to implement and presents fewer risks to participant privacy than data donation. However, it is unclear if and how study-created content may differ from genuine content, and how differences may impact models. We collect study-created and genuine multimodal social media posts labeled for emotion and compare them on several dimen- sions, including model performance. We find that compared to genuine posts, study-created posts are longer, rely more on their text and less on their images for emotion expression, and focus more on emotion-prototypical events. The samples of participants willing to donate versus create posts are demographically different. Study-created data is valuable to train models that generalize well to genuine data, but realistic effectiveness estimates require genuine data.
Which Demographics do LLMs Default to During Annotation?
Authors: Johannes Schäfer; Aidan Combs; Christopher Bagdon; Jiahui Li; Nadine Probol; Lynn Greschner; Sean Papay; Yarik Menchaca Resendiz; Aswathy Velutharambath; Amelie Wuehrl; Sabine Weber; Roman Klinger
Venue: ACL 2025 (Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers))

Abstract: Demographics and cultural background of annotators influence the labels they assign in text annotation – for instance, an elderly woman might find it offensive to read a message addressed to a “bro”, but a male teenager might find it appropriate. It is therefore important to acknowledge label variations to not under-represent members of a society. Two research directions developed out of this observation in the context of using large language models (LLM) for data annotations, namely (1) studying biases and inherent knowledge of LLMs and (2) injecting diversity in the output by manipulating the prompt with demographic information. We combine these two strands of research and ask the question to which demographics an LLM resorts to when no demographics is given. To answer this question, we evaluate which attributes of human annotators LLMs inherently mimic. Furthermore, we compare non-demographic conditioned prompts and placebo-conditioned prompts (e.g., “you are an annotator who lives in house number 5”) to demographics-conditioned prompts (“You are a 45 year old man and an expert on politeness annotation. How do you rate instance”). We study these questions for politeness and offensiveness annotations on the POPQUORN data set, a corpus created in a controlled manner to investigate human label variations based on demographics which has not been used for LLM-based analyses so far. We observe notable influences related to gender, race, and age in demographic prompting, which contrasts with previous studies that found no such effects.
Can You Really Trust Code Copilot? Evaluating Large Language Models from a Code Security Perspective
Authors: Yutao Mou; Xiao Deng; Yuxiao Luo; Shikun Zhang; Wei Ye
Venue: ACL 2025 (Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers))

Abstract: Code security and usability are both essential for various coding assistant applications driven by large language models (LLMs). Current code security benchmarks focus solely on single evaluation task and paradigm, such as code completion and generation, lacking comprehensive assessment across dimensions like secure code generation, vulnerability repair and discrimination. In this paper, we first propose CoV-Eval, a multi-task benchmark covering various tasks such as code completion, vulnerability repair, vulnerability detection and classification, for comprehensive evaluation of LLM code security. Besides, we developed VC-Judge, an improved judgment model that aligns closely with human experts and can review LLM-generated programs for vulnerabilities in a more efficient and reliable way. We conduct a comprehensive evaluation of 20 proprietary and open-source LLMs. Overall, while most LLMs identify vulnerable codes well, they still tend to generate insecure codes and struggle with recognizing specific vulnerability types and performing repairs. Extensive experiments and qualitative analyses reveal key challenges and optimization directions, offering insights for future research in LLM code security.
From Sub-Ability Diagnosis to Human-Aligned Generation: Bridging the Gap for Text Length Control via MarkerGen
Authors: Peiwen Yuan; Chuyi Tan; Shaoxiong Feng; Yiwei Li; Xinglin Wang; Yueqi Zhang; Jiayi Shi; Boyuan Pan; Yao Hu; Kan Li
Venue: ACL 2025 (Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers))

Abstract: Despite the rapid progress of large language models (LLMs), their length-controllable text generation (LCTG) ability remains below expectations, posing a major limitation for practical applications. Existing methods mainly focus on end-to-end training to reinforce adherence to length constraints. However, the lack of decomposition and targeted enhancement of LCTG sub-abilities restricts further progress. To bridge this gap, we conduct a bottom-up decomposition of LCTG sub-abilities with human patterns as reference and perform a detailed error analysis. On this basis, we propose MarkerGen, a simple-yet-effective plug-and-play approach that: (1) mitigates LLM fundamental deficiencies via external tool integration; (2) conducts explicit length modeling with dynamically inserted markers; (3) employs a three-stage generation scheme to better align length constraints while maintaining content quality. Comprehensive experiments demonstrate that MarkerGen significantly improves LCTG across various settings, exhibiting outstanding effectiveness and generalizability.
AGD: Adversarial Game Defense Against Jailbreak Attacks in Large Language Models
Authors: Shilong Pan; Zhiliang Tian; Zhen Huang; Wanlong Yu; Zhihua Wen; Xinwang Liu; Kai Lu; Minlie Huang; Dongsheng Li
Venue: ACL 2025 (Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers))

Abstract: LLMs demonstrate remarkable utility but remain vulnerable to jailbreak attacks that aim to elicit harmful responses. Existing defenses, including post-training alignment and prompt engineering, rely on training on safety-annotated datasets and safe prompt templates, struggling with adaptability to out-of-distribution (OOD) attacks. Steering internal representations of LLMs provides real-time adjustments to defend against OOD attacks. However, it struggles with maintaining model utility, since modifying the representation disrupts the forward pass of inference. It barely considers the competitive objectives of helpfulness and harmlessness in LLMs. We argue that adversarial game-based approaches promise a solution for conflicts between the two objectives. In this paper, we propose **A**dversarial **G**ame **D**efense (AGD), an adversarial game-based defense method that dynamically adjusts LLMs’ internal representations to achieve a balanced trade-off between helpfulness and harmlessness. AGD first proposes an interquartile range (IQR) method to detect abnormal attention weights and correct the abnormal weights via adversarial training. AGD adopts a bi-level optimization to play a two-player variable-sum game to approach Nash Equilibrium (NE), where the two players adversarially refine head activations for helpfulness and harmlessness respectively. Furthermore, AGD applies an expert model to next-token sampling to generate safer responses. Experiments show that AGD significantly improves LLMs’ safety over all baselines.
SCOP: Evaluating the Comprehension Process of Large Language Models from a Cognitive View
Authors: Yongjie Xiao; Hongru Liang; Peixin Qin; Yao Zhang; Wenqiang Lei
Venue: ACL 2025 (Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers))

Abstract: Despite the great potential of large language models (LLMs) in machine comprehension, it is still disturbing to fully count on them in real-world scenarios. This is probably because there is no rational explanation for whether the comprehension process of LLMs is aligned with that of experts. In this paper, we propose SCOP to carefully examine how LLMs perform during the comprehension process from a cognitive view. Specifically, it is equipped with a systematical definition of five requisite skills during the comprehension process, a strict framework to construct testing data for these skills, and a detailed analysis of advanced open-sourced and closed-sourced LLMs using the testing data. With SCOP, we find that it is still challenging for LLMs to perform an expert-level comprehension process. Even so, we notice that LLMs share some similarities with experts, e.g., performing better at comprehending local information than global information. Further analysis reveals that LLMs can be somewhat unreliable — they might reach correct answers through flawed comprehension processes. Based on SCOP, we suggest that one direction for improving LLMs is to focus more on the comprehension process, ensuring all comprehension skills are thoroughly developed during training.
Table-Critic: A Multi-Agent Framework for Collaborative Criticism and Refinement in Table Reasoning
Authors: Peiying Yu; Guoxin Chen; Jingjing Wang
Venue: ACL 2025 (Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers))

Abstract: Despite the remarkable capabilities of large language models (LLMs) in various reasoning tasks, they still struggle with table reasoning tasks, particularly in maintaining consistency throughout multi-step reasoning processes. While existing approaches have explored various decomposition strategies, they often lack effective mechanisms to identify and correct errors in intermediate reasoning steps, leading to cascading error propagation. To address these issues, we propose Table-Critic, a novel multi-agent framework that facilitates collaborative criticism and iterative refinement of the reasoning process until convergence to correct solutions. Our framework consists of four specialized agents: a Judge for error identification, a Critic for comprehensive critiques, a Refiner for process improvement, and a Curator for pattern distillation. To effectively deal with diverse and unpredictable error types, we introduce a self-evolving template tree that systematically accumulates critique knowledge through experience-driven learning and guides future reflections. Extensive experiments have demonstrated that Table-Critic achieves substantial improvements over existing methods, achieving superior accuracy and error correction rates while maintaining computational efficiency and lower solution degradation rate.
An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT)
Authors: Laurie Burchell; Ona de Gibert; Nikolay Arefyev; Mikko Aulamo; Marta Bañón; Pinzhen Chen; Mariia Fedorova; Liane Guillou; Barry Haddow; Jan Hajič; Jindřich Helcl; Erik Henriksson; Mateusz Klimaszewski; Ville Komulainen; Andrey Kutuzov; Joona Kytöniemi; Veronika Laippala; Petter Mæhlum; Bhavitvya Malik; Farrokh Mehryary; Vladislav Mikhailov; Nikita Moghe; Amanda Myntti; Dayyán O’Brien; Stephan Oepen; Proyag Pal; Jousia Piha; Sampo Pyysalo; Gema Ramírez-Sánchez; David Samuel; Pavel Stepachev; Jörg Tiedemann; Dušan Variš; Tereza Vojtěchová; Jaume Zaragoza-Bernabeu
Venue: ACL 2025 (Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers))

Abstract: Training state-of-the-art large language models requires vast amounts of clean and diverse textual data. However, building suitable multilingual datasets remains a challenge. In this work, we present HPLT v2, a collection of high-quality multilingual monolingual and parallel corpora, extending prior work of the HPLT project. The monolingual portion of the data contains 8T tokens covering 193 languages, while the parallel data contains 380M sentence pairs covering 51 languages. We document the entire data pipeline and release the code to reproduce it. We provide extensive analysis of the quality and characteristics of our data. Finally, we evaluate the performance of language models and machine translation systems trained on HPLT v2, demonstrating its value.
Scaling Text-Rich Image Understanding via Code-Guided Synthetic Multimodal Data Generation
Authors: Yue Yang; Ajay Patel; Matt Deitke; Tanmay Gupta; Luca Weihs; Andrew Head; Mark Yatskar; Chris Callison-Burch; Ranjay Krishna; Aniruddha Kembhavi; Christopher Clark
Venue: ACL 2025 (Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers))

Abstract: Reasoning about images with rich text, such as charts and documents, is a critical application of vision-language models (VLMs). However, VLMs often struggle in these domains due to the scarcity of diverse text-rich vision-language data. To address this challenge, we present CoSyn, a framework that leverages the coding capabilities of text-only large language models (LLMs) to automatically create synthetic text-rich multimodal data. Given input text describing a target domain (e.g., “nutrition fact labels”), CoSyn prompts an LLM to generate code (Python, HTML, LaTeX, etc.) for rendering synthetic images. With the underlying code as textual representations of the synthetic images, CoSyn can generate high-quality instruction-tuning data, again relying on a text-only LLM. Using CoSyn, we constructed a dataset comprising 400K images and 2.7M rows of vision-language instruction-tuning data. Comprehensive experiments on seven benchmarks demonstrate that models trained on our synthetic data achieve state-of-the-art performance among competitive open-source models, including Llama 3.2, and surpass proprietary models such as GPT-4V and Gemini 1.5 Flash. Furthermore, CoSyn can produce synthetic pointing data, enabling VLMs to ground information within input images, showcasing its potential for developing multimodal agents capable of acting in real-world environments.
Hierarchical Attention Generates Better Proofs
Authors: Jianlong Chen; Chao Li; Yang Yuan; Andrew C Yao
Venue: ACL 2025 (Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers))

Abstract: Large language models (LLMs) have shown promise in formal theorem proving, but their token-level processing often fails to capture the inherent hierarchical nature of mathematical proofs. We introduce Hierarchical Attention, a regularization method that aligns LLMs’ attention mechanisms with mathematical reasoning structures. Our approach establishes a five-level hierarchy from foundational elements to high-level concepts, ensuring structured information flow in proof generation. Experiments demonstrate that our method improves proof success rates by 2.05% on miniF2F and 1.69% on ProofNet while reducing proof complexity by 23.81% and 16.50% respectively. The code and models will be available.