dataset-review/dataset-review huggingface dataset:
LensLLM: Unveiling Fine-Tuning Dynamics for LLM Selection
ICML 2025

LENSLLM: Unveiling Fine-Tuning Dynamics for LLM Selection
Xinyue Zeng 1 Haohui Wang 1 Junhong Lin 2 Jun Wu 3 Tyler Cody 4 Dawei Zhou 1
Abstract
The proliferation of open-sourced Large Lan-
guage Models (LLMs) and diverse downstream
tasks necessitates efficient model selection, given
the impracticality of fine-tuning all candidates due
to computational constraints. Despite the recent
advances in LLM selection, a fundamental re-
search question largely remains nascent: how can
we model the dynamic behaviors of LLMs during
fine-tuning, thereby enhancing our understand-
ing of their generalization performance across
diverse downstream tasks? In this work, we pro-
pose a novel theoretical framework that provides
a proper lens to assess the generalization capabili-
ties of LLMs, thereby enabling accurate and effi-
cient LLM selection for downstream applications.
In particular, we first derive a PAC-Bayesian Gen-
eralization Bound that unveils fine-tuning dynam-
ics of LLMs and then introduce LENSLLM, a
Neural Tangent Kernel (NTK)-based Rectified
Scaling Model that enables accurate performance
predictions across diverse tasks while maintain-
ing computational efficiency. Extensive empirical
results on 3 large-scale benchmarks demonstrate
that our model achieves up to 91.1% accuracy and
reduces up to 88.5% computational cost in LLM
selection, outperforming 5 state-of-the-art meth-
ods. We open-source our proposed LENSLLM
model and corresponding results at LensLLM.io.
1. Introduction
The rise of large language models (LLMs) has revolu-
tionized natural language processing, driving remarkable
1Department of Computer Science, Virginia Tech, Blacksburg,
VA, USA. 2Department of Electrical Engineering and Computer
Science, Massachusetts Institute of Technology, Cambridge, MA,
USA. 3Department of Computer Science and Engineering, Michi-
gan State University, East Lansing, MI, USA. 4Department of In-
telligent Systems Division, Virginia Tech, Blacksburg, VA, USA..
Correspondence to...
Wyckoff Transformer: Generation of Symmetric Crystals
ICML 2025

Wyckoff Transformer: Generation of Symmetric Crystals
Nikita Kazeev 1 Wei Nong 2 Ignat Romanov 3 Ruiming Zhu 2 Andrey Ustyuzhanin 4 5 1 Shuya Yamazaki 2
Kedar Hippalgaonkar 1 2 6
Abstract
Crystal symmetry plays a fundamental role in
determining its physical, chemical, and elec-
tronic properties such as electrical and thermal
conductivity, optical and polarization behavior,
and mechanical strength.
Almost all known
crystalline materials have internal symmetry.
However, this is often inadequately addressed by
existing generative models, making the consistent
generation of stable and symmetrically valid
crystal structures a significant challenge.
We
introduce WyFormer, a generative model that
directly tackles this by formally conditioning
on space group symmetry. It achieves this by
using Wyckoff positions as the basis for an
elegant,
compressed,
and discrete structure
representation. To model the distribution, we
develop a permutation-invariant autoregressive
model based on the Transformer encoder and
an absence of positional encoding. Extensive
experimentation
demonstrates
WyFormer’s
compelling combination of attributes: it achieves
best-in-class symmetry-conditioned generation,
incorporates a physics-motivated inductive bias,
produces structures with competitive stability,
predicts material properties with competitive
accuracy even without atomic coordinates, and
exhibits unparalleled inference speed. https:
//github.com/SymmetryAdvantage/
WyckoffTransformer
1Institute for Functional Intelligent Materials University of Sin-
gapore, Block S9, Level 9, 4 Science Drive 2, Singapore 117544
2School of Materials Science and Engineering, Nanyang Techno-
logical University, Singapore 639798 3HSE University, Myasnit-
skaya Ulitsa, 20, Moscow, Russia, 101000 4Constructor University,
Bremen, Campus Ring 1, 28759, Germany 5Constructor Knowl-
edge Labs, Bremen, Campus Ring 1, 28759, Germany 6Institute of
Materials Research and Engineering, Agency for Science Technol-
ogy and Re...
Multilayer Matrix Factorization via Dimension-Reducing Diffusion Variational Inference
ICML 2025

Multilayer Matrix Factorization via Dimension-Reducing
Diffusion Variational Inference
Junbin Liu 1 Farzan Farnia 2 Wing-Kin Ma 1
Abstract
Multilayer matrix factorization (MMF) has re-
cently emerged as a generalized model of, and
potentially a more expressive approach than, the
classic matrix factorization. This paper considers
MMF under a probabilistic formulation, and our
focus is on inference methods under variational
inference. The challenge in this context lies in
determining a variational process that leads to a
computationally efficient and accurate approxima-
tion of the maximum likelihood inference. One
well-known example is the variational autoen-
coder (VAE), which uses neural networks for the
variational process. In this work, we take insight
from variational diffusion models in the context
of generative models to develop variational infer-
ence for MMF. We propose a dimension-reducing
diffusion process that results in a new way to
interact with the layered structures of the MMF
model. Experimental results demonstrate that the
proposed diffusion variational inference method
leads to improved performance scores compared
to several existing methods, including the VAE.
1. Introduction
Over decades, matrix factorization (MF) methods have
played a crucial role in a wide variety of problems such
as dimensionality reduction, low-dimension representation
learning, blind source separation (Hyv¨arinen et al., 2023),
hyperspectral unmixing (Ma et al., 2013), topic model-
ing (Arora et al., 2012), community detection (Yang &
Leskovec, 2013), to name a few. The broad interest of
1Department of Electronic Engineering, the Chinese Univer-
sity of Hong Kong, Hong Kong SAR of China 2Department
of Computer Science and Engineering, the Chinese Univer-
sity of Hong Kong, Hong Kong SAR of China.
Cor-
respondence to:
Junbin Liu <liujunbin@link.cuhk.edu.hk>,
Farzan
Farnia
<farnia@cse.cuhk.edu.hk>,
Wing-Kin
Ma
<wkma@ee.cuhk.edu.hk>.
Proceedings of the 42 nd International Confe...
Training a Generally Curious Agent
ICML 2025

Training a Generally Curious Agent
Fahim Tajwar * 1 Yiding Jiang * 1 Abitha Thankaraj 1 Sumaita Sadia Rahman 2 J Zico Kolter 1 Jeff Schneider 1
Russ Salakhutdinov 1
Abstract
Efficient exploration is essential for intelligent
systems interacting with their environment, but
existing language models often fall short in sce-
narios that require strategic information gathering.
In this paper, we present PAPRIKA, a fine-tuning
approach that enables language models to develop
general decision-making capabilities that are not
confined to particular environments. By training
on synthetic interaction data from different tasks
that require diverse strategies, PAPRIKA teaches
models to explore and adapt their behavior on
a new task based on environment feedback in-
context without more gradient updates. Experi-
mental results show that models fine-tuned with
PAPRIKA can effectively transfer their learned
decision-making capabilities to entirely unseen
tasks without additional training. Unlike tradi-
tional training, our approach’s primary bottleneck
lies in sampling useful interaction data instead
of model updates. To improve sample efficiency,
we propose a curriculum learning strategy that
prioritizes sampling trajectories from tasks with
high learning potential. These results suggest
a promising path towards AI systems that can
autonomously solve novel sequential decision-
making problems that require interactions with
the external world.
1. Introduction
Large language models (LLMs) are considered to be a
promising foundation for autonomous agents, systems capa-
ble of achieving goals independently with minimal human
supervision or intervention. A crucial requirement for such
systems is the ability to interact effectively with external en-
vironments and gather the information necessary to achieve
their objectives. This capability can be formalized as solving
*Equal contribution
1CMU 2North Carolina State University.
Correspondence to: Fahim Tajwar <ftajwar@cs.cmu.edu>.
Proce...
Statistical Test for Feature Selection Pipelines by Selective Inference
ICML 2025

Statistical Test for Feature Selection Pipelines by Selective Inference
Tomohiro Shiraishi * 1 2 Tatsuya Matsukawa * 1 Shuichi Nishino 1 2 Ichiro Takeuchi 1 2
Abstract
A data analysis pipeline is a structured sequence
of steps that transforms raw data into meaning-
ful insights by integrating various analysis algo-
rithms. In this paper, we propose a novel statis-
tical test to assess the significance of data analy-
sis pipelines. Our approach enables the system-
atic development of valid statistical tests applica-
ble to any feature selection pipeline composed of
predefined components. We develop this frame-
work based on selective inference, a statistical
technique that has recently gained attention for
data-driven hypotheses. As a proof of concept,
we focus on feature selection pipelines for linear
models, composed of three missing value imputa-
tion algorithms, three outlier detection algorithms,
and three feature selection algorithms. We theoret-
ically prove that our statistical test can control the
probability of false positive feature selection at
any desired level, and demonstrate its validity and
effectiveness through experiments on synthetic
and real data. Additionally, we present an im-
plementation framework that facilitates testing
across any configuration of these feature selection
pipelines without extra implementation costs.
1. Introduction
In practical data-driven decision-making tasks, integrating
various types of data analysis steps is crucial for address-
ing diverse challenges. For instance, in genetic research
aimed at identifying genes linked to a specific disease, the
process often begins with preprocessing tasks such as filling
in missing values and detecting outliers. This is followed
by screening for potentially related genes using simple de-
scriptive statistics and then applying more complex machine
learning-based feature selection algorithms. Such a system-
atic sequence of steps designed to analyze data and derive
*Equal contribution
...
RAPID: Retrieval Augmented Training of Differentially Private Diffusion Models
ICLR 2025

Published as a conference paper at ICLR 2025
RAPID: RETRIEVAL AUGMENTED TRAINING OF DIF-
FERENTIALLY PRIVATE DIFFUSION MODELS
Tanqiu Jiang†
Changjiang Li†
Fenglong Ma⋆
Ting Wang†
†Stony Brook University
⋆Pennsylvania State University
ABSTRACT
Differentially private diffusion models (DPDMs) harness the remarkable gener-
ative capabilities of diffusion models while enforcing differential privacy (DP)
for sensitive data.
However, existing DPDM training approaches often suffer
from significant utility loss, large memory footprint, and expensive inference
cost, impeding their practical uses. To overcome such limitations, we present
RAPID1, a novel approach that integrates retrieval augmented generation (RAG)
into DPDM training. Specifically, RAPID leverages available public data to build
a knowledge base of sample trajectories; when training the diffusion model on
private data, RAPID computes the early sampling steps as queries, retrieves sim-
ilar trajectories from the knowledge base as surrogates, and focuses on training
the later sampling steps in a differentially private manner. Extensive evaluation
using benchmark datasets and models demonstrates that, with the same privacy
guarantee, RAPID significantly outperforms state-of-the-art approaches by large
margins in generative quality, memory footprint, and inference cost, suggest-
ing that retrieval-augmented DP training represents a promising direction for de-
veloping future privacy-preserving generative models. The code is available at:
https://github.com/TanqiuJiang/RAPID.
1
INTRODUCTION
The recent advances in diffusion models have led to unprecedented capabilities of generating high-
quality, multi-modal data (Ho et al., 2020; Kong et al., 2020; Bar-Tal et al., 2024). However, training
performant diffusion models often requires massive amounts of training data, raising severe privacy
concerns in domains wherein data is sensitive. For instance, Carlini et al. (2023) show that compared
with other generative models,...
CursorCore: Assist Programming through Aligning Anything
ICML 2025

CursorCore: Assist Programming through Aligning Anything
Hao Jiang 1 Qi Liu 1 2 Rui Li 1 Shengyu Ye 1 Shijin Wang 3
Abstract
Large language models have been successfully
applied to programming assistance tasks, such
as code completion, code insertion, and instruc-
tional code editing. However, these applications
remain insufficiently automated and struggle to
effectively integrate various types of information
during the programming process, including cod-
ing history, code context, and user instructions. In
this work, we propose a new framework that com-
prehensively integrates these information sources,
and collect data to train models and evaluate their
performance. Firstly, to thoroughly evaluate how
well models align with different types of informa-
tion and the quality of their outputs, we introduce
a new benchmark, APEval (Assist Programming
Eval), to comprehensively assess the performance
of models in programming assistance tasks. Then,
for data collection, we develop a data generation
pipeline, Programming-Instruct, which synthe-
sizes training data from diverse sources, such as
GitHub and online judge platforms. This pipeline
can automatically generate various types of mes-
sages throughout the programming process. Fi-
nally, using this pipeline, we generate 219K sam-
ples, fine-tune multiple models, and develop the
CursorCore series. We show that CursorCore out-
performs other models of comparable size. This
framework unifies applications such as inline chat
and automated editing, contributes to the advance-
ment of coding assistants.
1. Introduction
Since the rise of large language models (LLMs), AI-assisted
programming technology has developed rapidly, with many
powerful LLMs being applied in this field (Zan et al., 2022;
1State Key Laboratory of Cognitive Intelligence, University of
Science and Technology of China 2Institute of Artificial Intelli-
gence, Hefei Comprehensive National Science Center 3iFLYTEK
Co., Ltd. Correspondence to: Qi Liu <qiliuql@us...
Censor Dependent Variational Inference
ICML 2025

Censor Dependent Variational Inference
Chuanhui Liu 1 Xiao Wang 1
Abstract
This paper provides a comprehensive analysis
of variational inference in latent variable models
for survival analysis, emphasizing the distinctive
challenges associated with applying variational
methods to survival data. We identify a critical
weakness in the existing methodology, demon-
strating how a poorly designed variational distri-
bution may hinder the objective of survival anal-
ysis tasks—modeling time-to-event distributions.
We prove that the optimal variational distribu-
tion, which perfectly bounds the log-likelihood,
may depend on the censoring mechanism. To
address this issue, we propose censor-dependent
variational inference (CDVI), tailored for latent
variable models in survival analysis. More prac-
tically, we introduce CD-CVAE, a V-structure
Variational Autoencoder (VAE) designed for the
scalable implementation of CDVI. Further discus-
sion extends some existing theories and training
techniques to survival analysis. Extensive experi-
ments validate our analysis and demonstrate sig-
nificant improvements in the estimation of indi-
vidual survival distributions. Codes can be found
at https://github.com/ChuanhuiLiu/CDVI.
1. Introduction
Survival analysis, a fundamental topic in statistics, finds
wide-ranging applications across healthcare, insurance, qual-
ity management, and finance (Harrell et al., 2001; Nelson,
2005; Frees, 2009). It focuses on modeling the relation-
ship between time-to-event outcomes and individual demo-
graphic covariates, where the event of interest could be
death, disease progression, or similar occurrences. A key
challenge in survival analysis arises from censored obser-
vations, which provide only partial information about the
survival time, necessitating specialized methods to handle
1Department of Statistics, Purdue University, USA. Correspon-
dence to: Xiao Wang <wangxiao@purdue.edu>.
Proceedings of the 42 nd International Conference on Machine
Lea...
What can large language models do for sustainable food?
ICML 2025

What Can Large Language Models Do for Sustainable Food?
Anna T. Thomas 1 Adam Yee 2 Andrew Mayne 1 Maya B. Mathur 1 Dan Jurafsky 1 Kristina Gligori´c 1
Abstract
Food systems are responsible for a third of human-
caused greenhouse gas emissions. We investigate
what Large Language Models (LLMs) can con-
tribute to reducing the environmental impacts of
food production. We define a typology of design
and prediction tasks based on the sustainable food
literature and collaboration with domain experts,
and evaluate six LLMs on four tasks in our typol-
ogy. For example, for a sustainable protein design
task, food science experts estimated that collabo-
ration with an LLM can reduce time spent by 45%
on average, compared to 22% for collaboration
with another expert human food scientist. How-
ever, for a sustainable menu design task, LLMs
produce suboptimal solutions when instructed to
consider both human satisfaction and climate im-
pacts. We propose a general framework for in-
tegrating LLMs with combinatorial optimization
to improve reasoning capabilities. Our approach
decreases emissions of food choices by 79% in
a hypothetical restaurant while maintaining par-
ticipants’ satisfaction with their set of choices.
Our results demonstrate LLMs’ potential, sup-
ported by optimization techniques, to accelerate
sustainable food development and adoption. Our
code is available at https://github.com/
thomasat/llms-sustainable-food.
1. Introduction
Global food production practices are a leading contributor to
climate change, deforestation, water pollution, biodiversity
loss, antibiotic resistance, and zoonotic disease transmis-
sion (Allan et al., 2023; Hopwood et al., 2020; Scherer et al.,
2019). Given the disproportionate harms of animal agricul-
ture in particular, which accounts for 57% of food systems
emissions, a number of companies have formed to develop
1Stanford University 2Umai Works. Correspondence to: Anna
T. Thomas <thomasat@stanford.edu>, Kristina Gligori´c <glig-
oric...
Uni$^2$Det: Unified and Universal Framework for Prompt-Guided Multi-dataset 3D Detection
ICLR 2025

Published as a conference paper at ICLR 2025
UNI2DET: UNIFIED AND UNIVERSAL FRAMEWORK FOR
PROMPT-GUIDED MULTI-DATASET 3D DETECTION
Yubin Wang1∗, Zhikang Zou2∗, Xiaoqing Ye2, Xiao Tan2, Errui Ding2, Cairong Zhao1†
1School of Computer Science and Technology, Tongji University, 2Baidu Inc.
ABSTRACT
We present Uni2Det, a brand new framework for unified and universal multi-
dataset training on 3D detection, enabling robust performance across diverse
domains and generalization to unseen domains. Due to substantial disparities
in data distribution and variations in taxonomy across diverse domains, train-
ing such a detector by simply merging datasets poses a significant challenge.
Motivated by this observation, we introduce multi-stage prompting modules for
multi-dataset 3D detection, which leverages prompts based on the characteristics
of corresponding datasets to mitigate existing differences. This elegant design
facilitates seamless plug-and-play integration within various advanced 3D detection
frameworks in a unified manner, while also allowing straightforward adaptation
for universal applicability across datasets. Experiments are conducted across mul-
tiple dataset consolidation scenarios involving KITTI, Waymo, and nuScenes,
demonstrating that our Uni2Det outperforms existing methods by a large margin in
multi-dataset training. Notably, results on zero-shot cross-dataset transfer validate
the generalization capability of our proposed method. Our code is available at
https://github.com/ThomasWangY/Uni2Det.
1
INTRODUCTION
With the ability to capture precise geometric information of entire scenes, LiDAR has become
an essential sensor for most autonomous vehicles. Due to the rapid development of large-scale
annotated 3D LiDAR datasets such as Waymo (Sun et al., 2020), nuScenes (Caesar et al., 2020), and
KITTI (Geiger et al., 2012), LiDAR-based models play a significant role in various critical perception
tasks for autonomous vehicles, particularly in 3D object detection....
Subobject-level Image Tokenization
ICML 2025

Subobject-level Image Tokenization
Delong Chen 1 2 Samuel Cahyawijaya 2 Jianfeng Liu 3 Baoyuan Wang 4 Pascale Fung 1 2
Abstract
Patch-based image tokenization ignores the mor-
phology of the visual world, limiting effective
and efficient learning of image understanding. In-
spired by subword tokenization, we introduce sub-
object-level adaptive token segmentation and ex-
plore several approaches, including superpixel,
SAM, and a proposed Efficient and PanOptiC
(EPOC) image tokenizer. Our EPOC combines
boundary detection–a simple task that can be han-
dled well by a compact model–with watershed
segmentation, which inherently guarantees no pix-
els are left unsegmented. Intrinsic evaluations
across 5 datasets demonstrate that EPOC’s seg-
mentation aligns well with human annotations
of both object- and part-level visual morphol-
ogy, producing more monosemantic tokens and
offering substantial efficiency advantages. For
extrinsic evaluation, we designed a token embed-
ding that handles arbitrary-shaped tokens, and
trained VLMs with different tokenizers on 4
datasets of object recognition and detailed cap-
tioning.
The results reveal that subobject to-
kenization enables faster convergence and bet-
ter generalization while using fewer visual to-
kens.
Project website:
https://github.
com/ChenDelong1999/subobjects.
1. Introduction
Partitioning the raw observational data stream into a man-
ageable number of segments is often beneficial for learning,
particularly when nearby elements in the data stream exhibit
strong correlations and therefore high redundancy (Cover,
1999; Barlow et al., 1961; Biederman, 1987; Marr, 2010;
Del´etang et al., 2024). In Computer Vision (CV), nearby
pixels are often highly correlated (He et al., 2022; Simon-
1Meta FAIR Paris
2The Hong Kong University of Sci-
ence and Technology 3Alibaba Group 4Zillow. Correspondence
to: Pascale Fung <pascale@ece.ust.hk>, Delong Chen <de-
long.chen@connect.ust.hk>.
Proceedings of the 42 nd International Conferenc...