GenAI4ELab/papercli-papers huggingface dataset:
MultiMind at SemEval-2025 Task 7: Crosslingual Fact-Checked Claim Retrieval via Multi-Source Alignment
Authors: Mohammad Mahdi Abootorabi; Alireza Ghahramani Kure; Mohammadali Mohammadkhani; Sina Elahimanesh; Mohammad Ali Ali Panah
Venue: ACL 2025 (Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025))

Abstract: This paper presents our system for SemEval-2025 Task 7: Multilingual and Crosslingual Fact-Checked Claim Retrieval. In an era where misinformation spreads rapidly, effective fact-checking is increasingly critical. We introduce {textbf{TriAligner}}, a novel approach that leverages a dual-encoder architecture with contrastive learning and incorporates both native and English translations across different modalities. Our method effectively retrieves claims across multiple languages by learning the relative importance of different sources in alignment. To enhance robustness, we employ efficient data preprocessing and augmentation using large language models while incorporating hard negative sampling to improve representation learning. We evaluate our approach on monolingual and crosslingual benchmarks, demonstrating significant improvements in retrieval accuracy and fact-checking performance over baselines.
NLPART at SemEval-2025 Task 4: Forgetting is harder than Learning
Authors: Hoorieh Sabzevari; Milad Molazadeh Oskuee; Tohid Abedini; Ghazal Zamaninejad; Sara Baruni; Zahra Amirmahani; Amirmohammad Salehoof
Venue: ACL 2025 (Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025))

Abstract: Unlearning is a critical capability for ensuring privacy, security, and compliance in AI systems, enabling models to forget specific data while retaining overall performance. In this work, we participated in Task 4 of SemEval 2025, which focused on unlearning across three sub-tasks: (1) long-form synthetic creative documents, (2) short-form synthetic biographies containing personally identifiable information, and (3) real documents sampled from the target model’s training dataset. We conducted four experiments, employing Supervised Fine-Tuning (SFT) and Negative Preference Optimization (NPO). Despite achieving good performance on the retain set—data that the model was supposed to remember—our findings demonstrate that these techniques did not perform well on the forget set, where unlearning was required.
RAGthoven at SemEval 2025 - Task 2: Enhancing Entity-Aware Machine Translation with Large Language Models, Retrieval Augmented Generation and Function Calling
Authors: Demetris Skottis; Gregor Karetka; Marek Suppa
Venue: ACL 2025 (Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025))

Abstract: This paper presents a system for SemEval 2025 Task 2 on entity-aware machine translation, integrating GPT-4o with Wikidata-based translations, retrieval augmented generation (RAG), and function calling. Implemented in RAGthoven, a lightweight yet powerful toolkit, our approach enriches source sentences with real-time external knowledge to address challenging or culturally specific named entities. Experiments on English-to-ten target languages show notable gains in translation quality, illustrating how LLM-based translation pipelines can leverage knowledge sources with minimal overhead. Its simplicity makes it a strong baseline for future research in entity-focused machine translation.
fact check AI at SemEval-2025 Task 7: Multilingual and Crosslingual Fact-checked Claim Retrieval
Authors: Pranshu Rastogi
Venue: ACL 2025 (Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025))

Abstract: The SemEval-2025 Task 7 on Multilingualand Crosslingual Fact-checked Claim Retrievalfocuses on retrieving relevant Fact-checkedclaims for social media Posts across multiplelanguages. This task is particularly challengingdue to linguistic barriers and the vast numberof languages Fact-checkers must consider.In this work, I approach the problem as aLearning-to-Rank task and solve it using abi-encoder-based model, fine-tuned on a pre-trained transformer optimized for sentence sim-ilarity. For the monolingual task, training wasperformed in both the source languages andtheir English translations. For cross-lingualretrieval, the training relied on English transla-tions.Most fine-tuned models have fewer than 500Mparameters, and the training was carried outefficiently using kaggle T4 GPUs with paral-lelization. Despite this lightweight setup, ourapproach achieved 92% Success@10 for mul-tilingual retrieval and 80% Success@10 forcross-lingual retrieval, securing 5th place inthe cross-lingual track and 10th place in themultilingual setting.
AlphaPro at SemEval-2025 Task 8: A Code Generation Approach for Question-Answering over Tabular Data
Authors: Anshuman Aryan; Laukik Wadhwa; Kalki Eshwar; Aakarsh Sinha; Durgesh Kumar
Venue: ACL 2025 (Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025))

Abstract: This work outlines the AlphaPro team’s solution to SemEval-2025 Task 8: Question Answering on Tabular Data. Our system utilizes a three-stage pipeline that uses natural language questions along with the table’s structural information to generate executable Python code, which is subsequently used to query the table and produce answers. The method achieves up to 67% accuracy in task data, demonstrating the feasibility of code generation for tabular question answering. The strengths and limitations of the approach are outlined and suggestions for further research are provided. The code has been made available in a public code repository to promote reproducibility and research in this area.
SHA256 at SemEval-2025 Task 4: Selective Amnesia – Constrained Unlearning for Large Language Models via Knowledge Isolation
Authors: Saransh Agrawal; Kuan - Hao Huang
Venue: ACL 2025 (Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025))

Abstract: Large language models (LLMs) frequently memorize sensitive information during training, posing risks when deploying publicly accessible models. Current machine unlearning methods struggle to selectively remove specific data associations without degrading overall model capabilities. This paper presents our solution to SemEval-2025 Task 4 on targeted unlearning, which introduces a two-stage methodology that combines causal mediation analysis with layer-specific optimization. Through systematic causal tracing experiments on OLMo architectures (1B and 7B parameters), we identify the critical role of the first few transformer layers (layers 0-5) in storing subject-attribute associations within MLP modules. Building on this insight, we develop a constrained optimization approach that freezes upper layers while applying a novel joint loss function to lower layers—simultaneously maximizing forget set loss via output token cross-entropy penalties and minimizing retain set deviation through adaptive regularization. Our method achieves 2nd place in the 1B model track, demonstrating strong task performance while maintaining 88% of baseline MMLU accuracy. These results establish causal-informed layer optimization as a promising paradigm for efficient, precise unlearning in LLMs, offering a significant step forward in addressing data privacy concerns in AI systems.
Team ACK at SemEval-2025 Task 2: Beyond Word-for-Word Machine Translation for English-Korean Pairs
Authors: Daniel Lee; Harsh Sharma; Jieun Han; Sunny Jeong; Alice Oh; Vered Shwartz
Venue: ACL 2025 (Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025))

Abstract: Translating knowledge-intensive and entity-rich text between English and Korean requires transcreation to preserve language-specific and cultural nuances beyond literal, phonetic or word-for-word conversion. We evaluate 13 models (LLMs and MT systems) using automatic metrics and human assessment by bilingual annotators. Our findings show LLMs outperform traditional MT systems but struggle with entity translation requiring cultural adaptation. By constructing an error taxonomy, we identify incorrect responses and entity name errors as key issues, with performance varying by entity type and popularity level. This work exposes gaps in automatic evaluation metrics and hope to enable future work in completing culturally-nuanced machine translation.
silp_nlp at SemEval-2025 Task 2: An Effect of Entity Awareness in Machine Translation Using LLM
Authors: Sumit Singh; Pankaj Goyal; Uma Tiwary
Venue: ACL 2025 (Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025))

Abstract: In this study, we investigated the effect of entity awareness on machine translation (MT) using large language models (LLMs). Our approach utilized GPT-4o and NLLB-200, integrating named entity recognition (NER) to improve translation quality. The results indicated that incorporating entity information enhanced translation accuracy, especially when dealing with named entities. However, performance was highly dependent on the effectiveness of the NER model.
clujteam at SemEval-2025 Task 10: Finetuning SmolLM2 with Taxonomy-based Prompting for Explaining the Dominant Narrative in Propaganda Textt
Authors: Anca Marginean
Venue: ACL 2025 (Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025))

Abstract: XAI has been a long-standing goal of AI. Explaining why a text can be considered to have a dominant narrative, where the narrative is known, is of great importance for dealing with propaganda in news. This paper reports on the participation of the system clujteam in Subtask 3 of Task 10 of Semveal 2025. The system obtained 7th place with a value of 0.72464 for F1macro, at 0.026 distance from the 1st place. The key components of the solution are the011given taxonomy and supervised fine-tuning of SmolLM2.013
Homa at SemEval-2025 Task 5: Aligning Librarian Records with OntoAligner for Subject Tagging
Authors: Hadi Bayrami Asl Tekanlou; Jafar Razmara; Mahsa Sanaei; Mostafa Rahgouy; Hamed Babaei Giglou
Venue: ACL 2025 (Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025))

Abstract: This paper presents our system, Homa, for SemEval-2025 Task 5: Subject Tagging, which focuses on automatically assigning subject labels to technical records from TIBKAT using the Gemeinsame Normdatei (GND) taxonomy. We leverage OntoAligner, a modular ontology alignment toolkit, to address this task by integrating retrieval-augmented generation (RAG) techniques. Our approach formulates the subject tagging problem as an alignment task, where records are matched to GND categories based on semantic similarity. We evaluate OntoAligner’s adaptability for subject indexing and analyze its effectiveness in handling multilingual records. Experimental results demonstrate the strengths and limitations of this method, highlighting the potential of alignment techniques for improving subject tagging in digital libraries.
Jim at SemEval-2025 Task 5: Multilingual BERT Ensemble
Authors: Jim Hahn
Venue: ACL 2025 (Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025))

Abstract: The SemEval-2025 Task 5 calls for the utilization of LLM capabilities to apply controlled subject labels to record descriptions in the multilingual library collection of the German National Library of Science and Technology. The multilingual BERT ensemble system described herein produces subject labels for various record types, including articles, books, conference papers, reports, and theses. Results indicate that for English language article records, bidirectional encoder-only LLMs can achieve high recall in automated subject assignment.
LA²I²F at SemEval-2025 Task 5: Reasoning in Embedding Space – Fusing Analogical and Ontology-based Reasoning for Document Subject Tagging
Authors: Andrea Salfinger; Luca Zaccagna; Francesca Incitti; Gianluca De Nardi; Lorenzo Dal Fabbro; Lauro Snidaro
Venue: ACL 2025 (Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025))

Abstract: The LLMs4Subjects shared task invited system contributions that leverage a technical library’s tagged document corpus to learn document subject tagging, i.e., proposing adequate subjects given a document’s title and abstract. To address the imbalance of this training corpus, team LA²I²F devised a semantic retrieval-based system fusing the results of ontological and analogical reasoning in embedding vector space. Our results outperformed a naive baseline of prompting a llama 3.1-based model, whilst being computationally more efficient and competitive with the state of the art.
Annif at SemEval-2025 Task 5: Traditional XMTC augmented by LLMs
Authors: Osma Suominen; Juho Inkinen; Mona Lehtinen
Venue: ACL 2025 (Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025))

Abstract: This paper presents the Annif system in SemEval-2025 Task 5 (LLMs4Subjects), which focussed on subject indexing using large language models (LLMs). The task required creating subject predictions for bibliographic records from the bilingual TIBKAT database using the GND subject vocabulary. Our approach combines traditional natural language processing and machine learning techniques implemented in the Annif toolkit with innovative LLM-based methods for translation and synthetic data generation, and merging predictions from monolingual models. The system ranked first in the all-subjects category and second in the tib-core-subjects category in the quantitative evaluation, and fourth in qualitative evaluations. These findings demonstrate the potential of combining traditional XMTC algorithms with modern LLM techniques to improve the accuracy and efficiency of subject indexing in multilingual contexts.
Last Minute at SemEval-2025 Task 5: RAG System for Subject Tagging
Authors: Zahra Sarlak; Ebrahim Ansari
Venue: ACL 2025 (Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025))

Abstract: Last Minute at SemEval-2025 Task 5: RAG System for Subject TaggingZahra Sarlak, Ebrahim AnsariIn this study, we explore the LLMs4Subjects shared task, which focuses on leveraging retrieval-augmented generation (RAG) to enhance subject classification in technical records from the Leibniz University’s Technical Library (TIBKAT). The challenge requires participants to recommend appropriate subject headings from the GND taxonomy while processing bibliographic data in both German and English.
RUC Team at SemEval-2025 Task 5: Fast Automated Subject Indexing: A Method Based on Similar Records Matching and Related Subject Ranking
Authors: Xia Tian; Yang Xin; Wu Jing; Xiu Heng; Zhang Xin; Li Yu; Gao Tong; Tan Xi; Hu Dong; Chen Tao; Jia Zhi
Venue: ACL 2025 (Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025))

Abstract: This paper presents MaRSI, an automatic subject indexing method designed to address the limitations of traditional manual indexing and emerging GenAI technologies. Focusing on improving indexing accuracy in cross-lingual contexts and balancing efficiency and accuracy in large-scale datasets, MaRSI mimics human reference learning behavior by constructing semantic indexes from pre-indexed document. It calculates similarity to retrieve relevant references, merges, and reorders their topics to generate index results. Experiments demonstrate that MaRSI outperforms supervised fine-tuning of LLMs on the same dataset, offering advantages in speed, effectiveness, and interpretability.
YNU-HPCC at SemEval-2025 Task 5: Contrastive Learning for GND Subject Tagging with Multilingual Sentence-BERT
Authors: Hong Jiang; Jin Wang; Xuejie Zhang
Venue: ACL 2025 (Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025))

Abstract: This paper describes YNU-HPCC(Alias JH) team’s participation in the sub-task 2 of the SemEval-2025 Task 5, which requires fine-tuning language models to align subject tags with the TIBKAT collection. The task presents three key challenges: cross-disciplinary document coverage, bilingual (English-German) processing requirements, and extreme classification over 200,000 GND Subjects. To address these challenges, we apply a contrastive learning framework using multilingual Sentence-BERT models, implementing two innovative training strategies: mixed-negative multi-label sampling, and single-label sampling with random negative selection. Our best-performing model achieves significant improvements of 28.6% in average recall, reaching 0.2252 on the core-test set and 0.1677 on the all-test set. Notably, we reveal model architecture-dependent response patterns: MiniLM-series models benefit from multi-label training (+33.5% zero-shot recall), while mpnet variants excel with single-label approaches (+230.3% zero-shot recall). The study further demonstrates the effectiveness of contrastive learning for multilingual semantic alignment in low-resource scenarios, providing insights for extreme classification tasks.
TartuNLP at SemEval-2025 Task 5: Subject Tagging as Two-Stage Information Retrieval
Authors: Aleksei Dorkin; Kairit Sirts
Venue: ACL 2025 (Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025))

Abstract: We present our submission to the Task 5 of SemEval-2025. We frame the task as an information retrieval problem, where the document content is used to retrieve subject tags from a large subject taxonomy. We leverage two types of encoder models to build a two-stage information retrieval system—a bi-encoder for coarse-grained candidate extraction at the first stage, and a cross-encoder for fine-grained re-ranking at the second stage.
silp_nlp at SemEval-2025 Task 5: Subject Recommendation With Sentence Transformer
Authors: Sumit Singh; Pankaj Goyal; Uma Tiwary
Venue: ACL 2025 (Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025))

Abstract: This work explored subject recommendation using sentence transformers within the SemEval-2025 Task 5 (LLMs4Subjects) challenge. Our approach leveraged embedding-based cosine similarity and hierarchical clustering to predict relevant GND subjects for TIB technical records in English and German. By experimenting with different models, including JinaAi, Distiluse-base-multilingual, and TF-IDF, we found that the JinaAi sentence transformer consistently outperformed other methods in terms of precision, recall, and F1-score.Our results highlight the effectiveness of transformer-based embeddings in semantic similarity tasks for subject classification. Additionally, hierarchical clustering helped reduce computational complexity by narrowing down candidate subjects efficiently. Despite the improvements, future work can focus on fine-tuning domain-specific embeddings, exploring knowledge graph integration, and enhancing multilingual capabilities for better generalization.
SemEval-2025 Task 6: Multinational, Multilingual, Multi-Industry Promise Verification
Authors: Chung-Chi Chen; Yohei Seki; Hakusen Shu; Anaïs Lhuissier; Juyeon Kang; Hanwool Lee; Min-Yuh Day; Hiroya Takamura
Venue: ACL 2025 (Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025))

Abstract: While extensive research exists on misinformation and disinformation, there is limited focus on future-oriented commitments, such as corporate ESG promises, which are often difficult to verify yet significantly impact public trust and market stability. To address this gap, we introduce the task of promise verification, leveraging natural language processing (NLP) techniques to automatically detect ESG commitments, identify supporting evidence, and evaluate the consistency between promises and evidence, while also inferring potential verification time points. This paper presents the dataset used in SemEval-2025 PromiseEval, outlines participant solutions, and discusses key findings. The goal is to enhance transparency in corporate discourse, strengthen investor trust, and support regulators in monitoring the fulfillment of corporate commitments.
SemEval-2025 Task 3: Mu-SHROOM, the Multilingual Shared-task on Hallucinations and Related Observable Overgeneration Mistakes
Authors: Raul Vazquez; Timothee Mickus; Elaine Zosa; Teemu Vahtola; Jörg Tiedemann; Aman Sinha; Vincent Segonne; Fernando Sanchez - Vega; Alessandro Raganato; Jindřich Libovický; Jussi Karlgren; Shaoxiong Ji; Jindřich Helcl; Liane Guillou; Ona De Gibert; Jaione Bengoetxea; Joseph Attieh; Marianna Apidianaki
Venue: ACL 2025 (Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025))

Abstract: We present the Mu-SHROOM shared task which is focused on detecting hallucinations and other overgeneration mistakes in the output of instruction-tuned large language models (LLMs).Mu-SHROOM addresses general-purpose LLMs in 14 languages, and frames the hallucination detection problem as a span-labeling task. We received 2,618 submissions from 43 participating teams employing diverse methodologies. The very high number of submissions highlights the interest of the community in hallucination detection. We present the results of the participating systems and provide an empirical analysis in order to better understand the factors that can lead to strong performance in this task. We also underscore current challenges, notably the varying degree of hallucinations across languages and the high annotator disagreement when labeling hallucination spans.
SemEval-2025 Task 7: Multilingual and Crosslingual Fact-Checked Claim Retrieval
Authors: Qiwei Peng; Robert Moro; Michal Gregor; Ivan Srba; Simon Ostermann; Marian Simko; Juraj Podrouzek; Matúš Mesarčík; Jaroslav Kopčan; Anders Søgaard
Venue: ACL 2025 (Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025))

Abstract: The rapid spread of online disinformation presents a global challenge, and machine learning has been widely explored as a potential solution. However, multilingual settings and low-resource languages are often neglected in this field. To address this gap, we conducted a shared task on multilingual claim retrieval at SemEval 2025, aimed at identifying fact-checked claims that match newly encountered claims expressed in social media posts across different languages. The task includes two subtracks: 1) a monolingual track, where social posts and claims are in the same language 2) a crosslingual track, where social posts and claims might be in different languages. A total of 179 participants registered for the task contributing to 52 test submissions. 23 out of 31 teams have submitted their system papers. In this paper, we report the best-performing systems as well as the most common and the most effective approaches across both subtracks. This shared task, along with its dataset and participating systems, provides valuable insights into multilingual claim retrieval and automated fact-checking, supporting future research in this field.
SemEval-2025 Task 8: Question Answering over Tabular Data
Authors: Jorge Osés Grijalba; L. Alfonso Ureñ - López; Eugenio Martínez Cámara; Jose Camacho - Collados
Venue: ACL 2025 (Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025))

Abstract: We introduce the findings and results of SemEval-2025 Task 8: Question Answering over Tabular Data. We featured two subtasks, DataBench and DataBench Lite. DataBench consists on question answering over tabular data, and DataBench Lite small comprising small datasets that might be easier to manage by current models by for example fitting them into a prompt. The task was open for any approach, but their answer has to conform to a required typing format. In this paper we present the task, analyze a number of system submissions and discuss the results. The results show how approaches leveraging LLMs dominated the task, with larger models exhibiting a considerably superior performance compared to small models.