GenAI4ELab/papercli-papers huggingface dataset:
Many-Objective Multi-Solution Transport
Authors: Ziyue Li; Tian Li; Virginia Smith; Jeff Bilmes; Tianyi Zhou
Venue: ICLR 2025 (ICLR 2025 Poster)

Abstract: Optimizing the performance of many objectives (instantiated by tasks or clients) jointly with a few Pareto stationary solutions (models) is critical in machine learning. However, previous multi-objective optimization methods often focus on a few objectives and cannot scale to many objectives that outnumber the solutions, leading to either subpar performance or ignored objectives. We introduce ''Many-objective multi-solution Transport (MosT)'', a framework that finds multiple diverse solutions in the Pareto front of many objectives. Our insight is to seek multiple solutions, each performing as a domain expert and focusing on a specific subset of objectives while collectively covering all of them. MosT formulates the problem as a bi-level optimization of weighted objectives for each solution, where the weights are defined by an optimal transport between objectives and solutions. Our algorithm ensures convergence to Pareto stationary solutions for complementary subsets of objectives. On a range of applications in federated learning, multi-task learning, and mixture-of-prompt learning for LLMs, MosT distinctly outperforms strong baselines, delivering high-quality, diverse solutions that profile the entire Pareto frontier, thus ensuring balanced trade-offs across many objectives.
An Auditing Test to Detect Behavioral Shift in Language Models
Authors: Leo Richter; Xuanli He; Pasquale Minervini; Matt Kusner
Venue: ICLR 2025 (ICLR 2025 Poster)

Abstract: As language models (LMs) approach human-level performance, a comprehensive understanding of their behavior becomes crucial. 
This includes evaluating capabilities, biases, task performance, and alignment with societal values. Extensive initial evaluations, including red teaming and diverse benchmarking, can establish a model’s behavioral profile. However, subsequent fine-tuning or deployment modifications may alter these behaviors in unintended ways. We present an efficient statistical test to tackle Behavioral Shift Auditing (BSA) in LMs, which we define as detecting distribution shifts in qualitative properties of the output distributions of LMs. Our test compares model generations from a baseline model to those of the model under scrutiny and provides theoretical guarantees for change detection while controlling false positives. The test features a configurable tolerance parameter that adjusts sensitivity to behavioral changes for different use cases. We evaluate our approach using two case studies: monitoring changes in (a) toxicity and (b) translation performance. We find that the test is able to detect meaningful changes in behavior distributions using just hundreds of examples.
Radar: Fast Long-Context Decoding for Any Transformer
Authors: Yongchang Hao; Mengyao Zhai; Hossein Hajimirsadeghi; Sepidehsadat Hosseini; Frederick Tung
Venue: ICLR 2025 (ICLR 2025 Poster)

Abstract: Transformer models have demonstrated exceptional performance across a wide range of applications. Though forming the foundation of Transformer models, the dot-product attention does not scale well to long-context data since its time requirement grows quadratically with context length. In this work, we propose Radar, a training-free approach that accelerates inference by dynamically searching for the most important context tokens. For any pre-trained Transformer, Radar can reduce the decoding time complexity without training or heuristically evicting tokens. Moreover, we provide theoretical justification for our approach, demonstrating that Radar can reliably identify the most important tokens with high probability. We conduct extensive comparisons with the previous methods on a wide range of tasks. The results demonstrate that Radar achieves the state-of-the-art performance across different architectures with reduced time complexity, offering a practical solution for efficient long-context processing of Transformers. The code is publicly available at https://github.com/BorealisAI/radar-decoding.
Can Knowledge Editing Really Correct Hallucinations?
Authors: Baixiang Huang; Canyu Chen; Xiongxiao Xu; Ali Payani; Kai Shu
Venue: ICLR 2025 (ICLR 2025 Poster)

Abstract: Large Language Models (LLMs) suffer from hallucinations, referring to the non-factual information in generated content, despite their superior capacities across tasks. Meanwhile, knowledge editing has been developed as a new popular paradigm to correct  erroneous factual knowledge encoded in LLMs with the advantage of avoiding retraining from scratch. However, a common issue of existing evaluation datasets for knowledge editing is that they do not ensure that LLMs actually generate hallucinated answers to the evaluation questions before editing. When LLMs are evaluated on such datasets after being edited by different techniques, it is hard to directly adopt the performance to assess the effectiveness of different knowledge editing methods in correcting hallucinations. Thus, the fundamental question remains insufficiently validated: Can knowledge editing really correct hallucinations in LLMs? We proposed HalluEditBench to holistically benchmark knowledge editing methods in correcting real-world hallucinations. First, we rigorously construct a massive hallucination dataset with 9 domains, 26 topics and more than 6,000 hallucinations. Then, we assess the performance of knowledge editing methods in a holistic way on five dimensions including Efficacy, Generalization, Portability, Locality, and Robustness. Through HalluEditBench, we have provided new insights into the potentials and limitations of different knowledge editing methods in correcting hallucinations, which could inspire future improvements and facilitate progress in the field of knowledge editing.
INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge
Authors: Angelika Romanou; Negar Foroutan; Anna Sotnikova; Sree Harsha Nelaturu; Shivalika Singh; Rishabh Maheshwary; Micol Altomare; Zeming Chen; Mohamed A. Haggag; Snegha A; Alfonso Amayuelas; Azril Hafizi Amirudin; Danylo Boiko; Michael Chang; Jenny Chim; Gal Cohen; Aditya Kumar Dalmia; Abraham Diress; Sharad Duwal; Daniil Dzenhaliou; Daniel Fernando Erazo Florez; Fabian Farestam; Joseph Marvin Imperial; Shayekh Bin Islam; Perttu Isotalo; Maral Jabbarishiviari; Börje F. Karlsson; Eldar Khalilov; Christopher Klamm; Fajri Koto; Dominik Krzemiński; Gabriel Adriano de Melo; Syrielle Montariol; Yiyang Nan; Joel Niklaus; Jekaterina Novikova; Johan Samir Obando Ceron; Debjit Paul; Esther Ploeger; Jebish Purbey; Swati Rajwal; Selvan Sunitha Ravi; Sara Rydell; Roshan Santhosh; Drishti Sharma; Marjana Prifti Skenduli; Arshia Soltani Moakhar; Bardia soltani moakhar; Ayush Kumar Tarun; Azmine Toushik Wasi; Thenuka Ovin Weerasinghe; Serhan Yilmaz; Mike Zhang; Imanol Schlag; Marzieh Fadaee; Sara Hooker; Antoine Bosselut
Venue: ICLR 2025 (ICLR 2025 Spotlight)

Abstract: The performance differential of large language models (LLM) between languages hinders their effective deployment in many regions, inhibiting the potential economic and societal value of generative AI tools in many communities. However, the development of functional LLMs in many languages (i.e., multilingual LLMs) is bottlenecked by the lack of high-quality evaluation resources in languages other than English. Moreover, current practices in multilingual benchmark construction often translate English resources, ignoring the regional and cultural knowledge of the environments in which multilingual systems would be used. In this work, we construct an evaluation suite of 197,243 QA pairs from local exam sources to measure the capabilities of multilingual LLMs in a variety of regional contexts.
Our novel resource, INCLUDE, is a comprehensive knowledge- and reasoning-centric benchmark across 44 written languages that evaluates multilingual LLMs for performance in the actual language environments where they would be deployed.
SysCaps: Language Interfaces for Simulation Surrogates of Complex Systems
Authors: Patrick Emami; Zhaonan Li; Saumya Sinha; Truc Nguyen
Venue: ICLR 2025 (ICLR 2025 Poster)

Abstract: Surrogate models are used to predict the behavior of complex energy systems that are too expensive to simulate with traditional numerical methods. 
Our work introduces the use of language descriptions, which we call "system captions" or SysCaps, to interface with such surrogates. 
We argue that interacting with surrogates through text, particularly natural language, makes these models more accessible for both experts and non-experts.
We introduce a lightweight multimodal text and timeseries regression model and a training pipeline that uses large language models (LLMs) to synthesize high-quality captions from simulation metadata. 
Our experiments on two real-world simulators of buildings and wind farms show that our SysCaps-augmented surrogates have better accuracy on held-out systems than traditional methods while enjoying new generalization abilities, such as handling semantically related descriptions of the same test system.
Additional experiments also highlight the potential of SysCaps to unlock language-driven design space exploration and to regularize training through prompt augmentation.
Generative Classifiers Avoid Shortcut Solutions
Authors: Alexander Cong Li; Ananya Kumar; Deepak Pathak
Venue: ICLR 2025 (ICLR 2025 Poster)

Abstract: Discriminative approaches to classification often learn shortcuts that hold in-distribution but fail even under minor distribution shift. This failure mode stems from an overreliance on features that are spuriously correlated with the label. We show that generative classifiers, which use class-conditional generative models, can avoid this issue by modeling all features, both core and spurious, instead of mainly spurious ones. These generative classifiers are simple to train, avoiding the need for specialized augmentations, strong regularization, extra hyperparameters, or knowledge of the specific spurious correlations to avoid. We find that diffusion-based and autoregressive generative classifiers achieve state-of-the-art performance on five standard image and text distribution shift benchmarks and reduce the impact of spurious correlations in realistic applications, such as medical or satellite datasets. Finally, we carefully analyze a Gaussian toy setting to understand the inductive biases of generative classifiers, as well as the data properties that determine when generative classifiers outperform discriminative ones.
Compute-Constrained Data Selection
Authors: Junjie Yin; Alexander M Rush
Venue: ICLR 2025 (ICLR 2025 Poster)

Abstract: Data selection can reduce the amount of training data needed to finetune LLMs; however, the efficacy of data selection scales directly with its compute. Motivated by the practical challenge of compute-constrained finetuning, we consider the setting in which both the cost of selecting data and training are budgeted for. We first formalize the problem of data selection with a cost-aware utility function, 
and model the data selection problem as trading off initial-selection cost for training gain. We run a comprehensive sweep of experiments across multiple tasks, varying compute budget by scaling finetuning tokens, model sizes, and data selection compute. Interestingly we find that many powerful data selection methods are almost never compute-optimal, and that cheaper data selection alternatives dominate both from a theoretical and empirical perspective. For compute-optimal training, we find that perplexity and gradient data selection require training-to-selection model size ratios of 5x and 10x, respectively.
Generalization through variance: how noise shapes inductive biases in diffusion models
Authors: John Vastola
Venue: ICLR 2025 (ICLR 2025 Poster)

Abstract: How diffusion models generalize beyond their training set is not known, and is somewhat mysterious given two facts: the optimum of the denoising score matching (DSM) objective usually used to train diffusion models is the score function of the training distribution; and the networks usually used to learn the score function are expressive enough to learn this score to high accuracy. We claim that a certain feature of the DSM objective—the fact that its target is not the training distribution's score, but a noisy quantity only equal to it in expectation—strongly impacts whether and to what extent diffusion models generalize. In this paper, we develop a mathematical theory that partly explains this 'generalization through variance' phenomenon. Our theoretical analysis exploits a physics-inspired path integral approach to compute the distributions typically learned by a few paradigmatic under- and overparameterized diffusion models. We find that the distributions diffusion models effectively learn to sample from resemble their training distributions, but with `gaps' filled in, and that this inductive bias is due to the covariance structure of the noisy target used during training. We also characterize how this inductive bias interacts with feature-related inductive biases.
Learning to Explore and Exploit with GNNs for Unsupervised Combinatorial Optimization
Authors: Utku Umur ACIKALIN; Aaron M Ferber; Carla P Gomes
Venue: ICLR 2025 (ICLR 2025 Poster)

Abstract: Combinatorial optimization (CO) problems are pervasive
across various domains, but their NP-hard nature often necessitates problem-specific
heuristic algorithms. Recent advancements in deep learning have led to the development of learning-based heuristics, yet these approaches often struggle with limited search capabilities.
We introduce  Explore-and-Exploit GNN ($X^2$GNN, pronounced x-squared GNN), 
a novel unsupervised neural framework that combines exploration and exploitation for combinatorial search optimization:
i) Exploration - $X^2$GNN generates multiple  solutions simultaneously, promoting diversity in the search space; 
(ii) Exploitation - $X^2$GNN  employs neural stochastic iterative refinement to exploit partial existing solutions, guiding the search toward promising regions and helping escape local optima.
By balancing exploration and exploitation, $X^2$GNN achieves superior performance and generalization on several graph CO problems including Max Cut, Max Independent Set, and Max Clique. Notably, for large Max Clique problems, $X^2$GNN consistently generates solutions within 1.2\% of optimality, while other state-of-the-art learning-based approaches struggle to reach within 22\% of optimal. Moreover, $X^2$GNN consistently generates better solutions than Gurobi on large graphs for all three problems under reasonable time budgets. Furthermore, $X^2$GNN exhibits exceptional generalization capabilities. For the Maximum Independent Set problem, $X^2$GNN outperforms state-of-the-art methods even when trained on smaller or out-of-distribution graphs compared to the test set.  Our framework offers a more effective and flexible approach to neural combinatorial optimization, addressing a key challenge in the field and providing a promising direction for future research in learning-based heuristics for combinatorial optimization.
Fitting Networks with a Cancellation Trick
Authors: Jiashun Jin; Jingming Wang
Venue: ICLR 2025 (ICLR 2025 Poster)

Abstract: The degree-corrected block model (DCBM), latent space model (LSM), and $\beta$-model are all popular network models. We combine their modeling ideas and propose the logit-DCBM as a new model. Similar as the $\beta$-model and LSM, the logit-DCBM contains nonlinear factors, where fitting the parameters is a challenging open problem. We resolve this problem by introducing a cancellation trick. We also propose R-SCORE as a recursive community detection algorithm, where in each iteration, we first use the idea above to update our parameter estimation, and then use the results to remove the nonlinear factors in the logit-DCBM so the renormalized model approximately satisfies a low-rank model, just like the DCBM. Our numerical study suggests that R-SCORE significantly improves over existing spectral approaches in many cases. Also, theoretically, we show that  the Hamming error rate of R-SCORE is faster than that of SCORE in a specific sparse region, and is at least as fast outside this region.
Do LLM Agents  Have Regret? A Case Study in Online Learning and Games
Authors: Chanwoo Park; Xiangyu Liu; Asuman E. Ozdaglar; Kaiqing Zhang
Venue: ICLR 2025 (ICLR 2025 Poster)

Abstract: Large language models (LLMs) have been increasingly employed for (interactive) decision-making, via the development of LLM-based autonomous agents. Despite their emerging successes, the performance of LLM agents in decision-making has not been fully investigated through quantitative metrics, especially in the multi-agent setting when they interact with each other, a typical scenario in real-world LLM-agent applications. To better understand the limits of LLM agents in these interactive environments, we propose to study their interactions in benchmark decision-making settings in online learning and game theory, through the performance metric of regret. We first empirically study the no-regret behaviors of LLMs in canonical non-stochastic online learning problems, as well as the emergence of equilibria when LLM agents interact through playing repeated games. We then provide some theoretical insights into the no-regret behaviors of LLM agents, under certain assumptions on the supervised pre-training and the rationality model of human decision-makers who generate the data. Notably, we also identify (simple) cases where advanced LLMs such as GPT-4 fail to be no-regret. To further promote the no-regret behaviors, we propose a novel unsupervised training loss of regret-loss, which, in contrast to the supervised pre-training loss, does not require the labels of (optimal) actions. Finally, we establish the statistical guarantee of generalization bound for regret-loss minimization, and more importantly, the optimization guarantee that minimizing such a loss may automatically lead to known no-regret learning algorithms, when single-layer self-attention models are used. Our further experiments demonstrate the effectiveness of our regret-loss, especially in addressing the above “regrettable” cases.
Seq-VCR: Preventing  Collapse in Intermediate Transformer Representations for Enhanced Reasoning
Authors: Md Rifat Arefin; Gopeshh Subbaraj; Nicolas Gontier; Yann LeCun; Irina Rish; Ravid Shwartz-Ziv; Christopher Pal
Venue: ICLR 2025 (ICLR 2025 Poster)

Abstract: Decoder-only Transformers often struggle with complex reasoning tasks, particularly arithmetic reasoning requiring multiple sequential operations. In this work, we identify representation collapse in the model’s intermediate layers as a key factor limiting their reasoning capabilities. To address this, we propose Sequential Variance-Covariance Regularization (Seq-VCR), which enhances the entropy of intermediate representations and prevents collapse. Combined with dummy pause tokens as substitutes for chain-of-thought (CoT) tokens, our method significantly improves performance in arithmetic reasoning problems. In the challenging 5 × 5 integer multiplication task, our approach achieves 99.5% exact match accuracy, outperforming models of the same size (which yield 0% accuracy) and GPT-4 with five-shot CoT prompting (44%). We also demonstrate superior results on arithmetic expression and longest increasing subsequence (LIS) datasets. Our findings highlight the importance of preventing intermediate layer representation collapse to enhance the reasoning capabilities of Transformers and show that Seq-VCR offers an effective solution without requiring explicit CoT supervision.
Underdamped Diffusion Bridges with Applications to Sampling
Authors: Denis Blessing; Julius Berner; Lorenz Richter; Gerhard Neumann
Venue: ICLR 2025 (ICLR 2025 Poster)

Abstract: We provide a general framework for learning diffusion bridges that transport prior to target distributions. It includes existing diffusion models for generative modeling, but also underdamped versions with degenerate diffusion matrices, where the noise only acts in certain dimensions. Extending previous findings, our framework allows to rigorously show that score-matching in the underdamped case is indeed equivalent to maximizing a lower bound on the likelihood. Motivated by superior convergence properties and compatibility with sophisticated numerical integration schemes of underdamped stochastic processes, we propose *underdamped diffusion bridges*, where a general density evolution is learned rather than prescribed by a fixed noising process. We apply our method to the challenging task of sampling from unnormalized densities without access to samples from the target distribution. Across a diverse range of sampling problems, our approach demonstrates state-of-the-art performance, notably outperforming alternative methods, while requiring significantly fewer discretization steps and almost no hyperparameter tuning.
BoneMet: An Open Large-Scale Multi-Modal Murine Dataset for Breast Cancer Bone Metastasis Diagnosis and Prognosis
Authors: Tiankuo Chu; Fudong Lin; Shubo Wang; Jason Jiang; Wiley Jia-Wei Gong; Xu Yuan; Liyun Wang
Venue: ICLR 2025 (ICLR 2025 Poster)

Abstract: Breast cancer bone metastasis (BCBM) affects women’s health globally, calling
 for the development of effective diagnosis and prognosis solutions. While deep
 learning has exhibited impressive capacities across various healthcare domains, its
 applicability in BCBM diseases is consistently hindered by the lack of an open,
 large-scale, deep learning-ready dataset. As such, we introduce the Bone Metastasis
 (BoneMet) dataset, the first large-scale, publicly available, high-resolution medical
 resource, which is derived from a well-accepted murine BCBM model. The unique
 advantage of BoneMet over existing human datasets is repeated sequential scans
 per subject over the entire disease development phases. The dataset consists of
 over 67 terabytes of multi-modal medical data, including 2D X-ray images, 3D
 CT scans, and detailed biological data (e.g., medical records and bone quantitative
 analysis), collected from more than five hundreds mice spanning from 2019 to
 2024. Our BoneMet dataset is well-organized into six components, i.e., Rotation
X-Ray, Recon-CT, Seg-CT, Regist-CT, RoI-CT, and MiceMediRec. We further
 show that BoneMet can be readily adopted to build versatile, large-scale AI models
 for managing BCBM diseases in terms of diagnosis using 2D or 3D images, prognosis of bone deterioration, and sparse-angle 3D reconstruction for safe long-term
 disease monitoring. Our preliminary results demonstrate that BoneMet has the
 potentials to jump-start the development and fine-tuning of AI-driven solutions
 prior to their applications to human patients. To facilitate its easy access and
 wide dissemination, we have created the BoneMet package, providing three APIs
 that enable researchers to (i) flexibly process and download the BoneMet data
 filtered by specific time frames; and (ii) develop and train large-scale AI models for
 precise BCBM diagnosis and prognosis. The BoneMet dataset is officially available on Hugging Face Datasets at https://huggingface.co/datasets/BoneMet/BoneMet. The BoneMet package is available on the Python Package Index (PyPI) at https://pypi.org/project/BoneMet. Code and tutorials are available at https://github.com/Tiankuo528/BoneMet.