dataset-review/dataset-review huggingface dataset:
Resolving Lexical Bias in Model Editing
ICML 2025

Resolving Lexical Bias in Model Editing
Hammad Rizwan 1 Domenic Rosati 1 Ga Wu 1 Hassan Sajjad 1
Abstract
Model editing aims to modify the knowledge
of a pre-trained language model. Previous ap-
proaches have often involved direct alterations to
model weights, which can result in model degrada-
tion. Recent weight-preserving techniques avoid
making modifications to the model’s weights
by employing an adapter that implements edits
through auxiliary components. These rely heavily
on scoping mechanisms based on distance func-
tions on the model’s representation space to de-
termine when to trigger edits. We demonstrate
that current adapter methods are critically vul-
nerable to strong lexical biases, leading to is-
sues such as applying edits to irrelevant prompts
with overlapping words. This paper presents a
principled approach to learning a disentangled
representation space that facilitates precise lo-
calization of edits by maintaining distance be-
tween irrelevant prompts while preserving prox-
imity among paraphrases. In our empirical study,
we show that our method, Projector Editor Net-
works for Model Editing - PENME, achieves
state-of-the-art model editing results while be-
ing computationally efficient during inference
compared to previous methods and adaptable
across different architectures. We provide the
codebase of PENME here: https://github.
com/hammadrizwan/PENME.git
1. Introduction
Large Language Models (LLMs) are successful in solving a
diverse range of natural language processing tasks (Devlin
et al., 2019; Liu et al., 2019; Touvron et al., 2023b; Radford
et al., 2019). Despite their successes, LLMs are fallible in
large part due to the noisy and imperfect nature of the data
used for training (Zhu et al., 2020). Moreover, as the world
1Department of Computer Science, Dalhousie University, Hal-
ifax, Canada.
Correspondence to: Hammad Rizwan <ham-
mad.rizwan@dal.ca>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canad...
Leveraging Predictive Equivalence in Decision Trees
ICML 2025

Leveraging Predictive Equivalence in Decision Trees
Hayden McTavish * 1 Zachery Boner * 1 Jon Donnelly * 1 Margo Seltzer 2 Cynthia Rudin 1
Abstract
Decision trees are widely used for interpretable
machine learning due to their clearly structured
reasoning process. However, this structure belies
a challenge we refer to as predictive equivalence:
a given tree’s decision boundary can be repre-
sented by many different decision trees. The pres-
ence of models with identical decision boundaries
but different evaluation processes makes model
selection challenging. The models will have dif-
ferent variable importance and behave differently
in the presence of missing values, but most opti-
mization procedures will arbitrarily choose one
such model to return. We present a boolean logi-
cal representation of decision trees that does not
exhibit predictive equivalence and is faithful to the
underlying decision boundary. We apply our rep-
resentation to several downstream machine learn-
ing tasks. Using our representation, we show that
decision trees are surprisingly robust to test-time
missingness of feature values; we address predic-
tive equivalence’s impact on quantifying variable
importance; and we present an algorithm to opti-
mize the cost of reaching predictions.
1. Introduction
Decision trees are widely used for interpretable machine
learning (Rudin et al., 2022). Their structure of discrete
decisions has long been leveraged for difficult tasks such as
handling missing data (Therneau et al., 1997) and measur-
ing variable importance (Breiman, 1984). Recent advances
in decision tree optimization (Lin et al., 2020; Demirovi´c
*Equal contribution
1Department of Computer Science,
Duke University, Durham, North Carolina, USA 2Department
of Computer Science, University of British Columbia, Van-
couver,
BC, Canada.
Correspondence to:
Hayden Mc-
Tavish <hayden.mctavish@duke.edu>, Zachery Boner <zach-
ery.boner@duke.edu>.
Proceedings of the 42 nd International Conference on Mac...
PokéChamp: an Expert-level Minimax Language Agent
ICML 2025

PokéChamp: an Expert-level Minimax Language Agent
Seth Karten * 1 Andy Luu Nguyen 1 Chi Jin 1
Abstract
We introduce PokéChamp, a minimax agent
powered by Large Language Models (LLMs) for
Pokémon battles. Built on a general framework
for two-player competitive games, PokéChamp
leverages the generalist capabilities of LLMs to
enhance minimax tree search. Specifically, LLMs
replace three key modules: (1) player action sam-
pling, (2) opponent modeling, and (3) value func-
tion estimation, enabling the agent to effectively
utilize gameplay history and human knowledge to
reduce the search space and address partial observ-
ability. Notably, our framework requires no addi-
tional LLM training. We evaluate PokéChamp
in the popular Gen 9 OU format. When powered
by GPT-4o, it achieves a win rate of 76% against
the best existing LLM-based bot and 84% against
the strongest rule-based bot, demonstrating its su-
perior performance. Even with an open-source 8-
billion-parameter Llama 3.1 model, PokéChamp
consistently outperforms the previous best LLM-
based bot, Pokéllmon powered by GPT-4o, with
a 64% win rate. PokéChamp attains a projected
Elo of 1300-1500 on the Pokémon Showdown
online ladder, placing it among the top 30%-10%
of human players. In addition, this work compiles
the largest real-player Pokémon battle dataset, fea-
turing over 3 million games, including more than
500k high-Elo matches. Based on this dataset,
we establish a series of battle benchmarks and
puzzles to evaluate specific battling skills. We fur-
ther provide key updates to the local game engine.
This work establishes Pokémon as a benchmark to
integrate LLM technologies with game-theoretic
algorithms addressing general multi-agent prob-
lems. Videos, code, and dataset are available on-
line.
*Equal contribution 1Princeton University. Correspondence to:
Seth Karten <sethkarten@princeton.edu>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267, 2025. Copyright ...
What Limits Bidirectional Model's Generative Capabilities? A Uni-Bi-Directional Mixture-of-Expert Method For Bidirectional Fine-tuning
ICML 2025

What Limits Bidirectional Model’s Generative Capabilities?
A Uni-Bi-Directional Mixture-of-Expert Method For Bidirectional Fine-tuning
Zuchao Li 1 Yonghua Hei 2 Qiwei Li 3 Lefei Zhang 3 Ping Wang 4 Hai Zhao 5 Baoyuan Qi 6 Guoming Liu 6
Abstract
Large Language Models (LLMs) excel in genera-
tion tasks, yet their causal attention mechanisms
limit performance in embedding tasks. While
bidirectional modeling may enhance embeddings,
naively fine-tuning unidirectional models bidi-
rectionally severely degrades generative perfor-
mance. To investigate this trade-off, we analyze
attention weights as dependence indicators and
find that bidirectional fine-tuning increases subse-
quent dependence, impairing unidirectional gen-
eration. Through systematic Transformer module
evaluations, we discover the FFN layer is least
affected by such dependence. Leveraging this dis-
covery, we propose UBMoE-LLM, a novel Uni-
Bi-directional Mixture-of-Experts LLM, which
integrates the original unidirectional FFN with a
bidirectionally fine-tuned FFN via unsupervised
contrastive learning. This MoE-based approach
enhances embedding performance while preserv-
ing robust generation.
Extensive experiments
across diverse datasets and model scales vali-
date our attention dependence metric and demon-
strate UBMoE-LLM’s superior generative qual-
ity and reduced hallucination. Code is available
at: https://github.com/heiyonghua/
ubmoe_llm.
1. Introduction
Creating a model capable of both generation and text em-
bedding has been a long-term goal in the field of natural
1School of Artificial intelligence, Wuhan University 2This work
was conducted during Yonghua Hei’s internship at the School of
Artificial Intelligence, Wuhan University. 3School of Computer
Science, Wuhan University 4School of Information Management,
Wuhan University 5Department of Computer Science and Engineer-
ing, Shanghai Jiao Tong University 6Xiaomi, Beijing, China. Cor-
respondence to: Lefei Zhang <zhanglefei@whu.edu.cn>, Ping
Wan...
Adversarial Reasoning at Jailbreaking Time
ICML 2025

Adversarial Reasoning at Jailbreaking Time
Mahdi Sabbaghi 1 Paul Kassianik 2 George Pappas 1 Amin Karbasi 1 Hamed Hassani 1
Abstract
As large language models (LLMs) are becoming
more capable and widespread, the study of their
failure cases is becoming increasingly important.
Recent advances in standardizing, measuring, and
scaling test-time compute suggest new method-
ologies for optimizing models to achieve high per-
formance on hard tasks. In this paper, we apply
these advances to the task of “model jailbreaking”:
eliciting harmful responses from aligned LLMs.
We develop an adversarial reasoning approach to
automatic jailbreaking that leverages a loss signal
to guide the test-time compute, achieving SOTA
attack success rates against many aligned LLMs,
even those that aim to trade inference-time com-
pute for adversarial robustness. Our approach in-
troduces a new paradigm in understanding LLM
vulnerabilities, laying the foundation for the de-
velopment of more robust and trustworthy AI sys-
tems. Code is available at Github.
1. Introduction
Large language models (LLMs) are increasingly deployed
with various safety techniques to ensure alignment with hu-
man values. Common strategies include RLHF (Christiano
et al., 2023; Ouyang et al., 2022), DPO (Rafailov et al.,
2024), and the usage of dedicated guardrail models (Inan
et al., 2023; Rebedea et al., 2023). In nominal use cases,
alignment methods typically refuse to generate objection-
able content, but adversarially designed prompts can bypass
these guardrails. A challenge known as jailbreaking con-
sists of finding prompts that circumvent safety measures
and elicit undesirable behaviors.
Current jailbreaking methods fall into two categories: token-
space and semantic-space attacks.
Token-space attacks
(Shin et al., 2020; Wen et al., 2023; Zou et al., 2023; Hayase
et al., 2024; Andriushchenko et al., 2024a) focus on token-
1University of Pennsylvania 2Robust Intelligence @ Cisco.
Correspondence to: Mahdi Sabagghi ...
Revisiting Convergence: Shuffling Complexity Beyond Lipschitz Smoothness
ICML 2025

Revisiting Convergence: Shuffling Complexity Beyond Lipschitz Smoothness
Qi He 1 Peiran Yu 2 Ziyi Chen 1 Heng Huang 1
Abstract
Shuffling-type gradient methods are favored in
practice for their simplicity and rapid empirical
performance. Despite extensive development of
convergence guarantees under various assump-
tions in recent years, most require the Lipschitz
smoothness condition, which is often not met in
common machine learning models. We highlight
this issue with specific counterexamples. To ad-
dress this gap, we revisit the convergence rates of
shuffling-type gradient methods without assuming
Lipschitz smoothness. Using our stepsize strategy,
the shuffling-type gradient algorithm not only con-
verges under weaker assumptions but also match
the current best-known convergence rates, thereby
broadening its applicability. We prove the conver-
gence rates for nonconvex, strongly convex, and
non-strongly convex cases, each under both ran-
dom reshuffling and arbitrary shuffling schemes,
under a general bounded variance condition. Nu-
merical experiments further validate the perfor-
mance of our shuffling-type gradient algorithm,
underscoring its practical efficacy.
1. Introduction
Gradient-based optimization has always been a critical area
due to its extensive practical applications in machine learn-
ing, including reinforcement learning (Sutton and Barto,
2018), hyperparameter optimization (Feurer and Hutter,
2019), and large language models (Radford et al., 2018).
While numerous gradient-based algorithms have been devel-
oped for convex functions (Nemirovskij and Yudin, 1983;
Nesterov, 2013; d’Aspremont et al., 2021), research on non-
convex functions has become particularly active in recent
years, driven by advances in deep learning. Notably, with
unbiased stochastic gradients and bounded variance, SGD
1Department of Computer Science, University of Maryland,
College Park. 2Department of Computer Science and Engineer-
ing, University of Texas Arlington. Correspo...
Algorithms with Calibrated Machine Learning Predictions
ICML 2025

Algorithms with Calibrated Machine Learning Predictions
Judy Hanwen Shen 1 Ellen Vitercik 1 2 Anders Wikum 2
Abstract
The field of algorithms with predictions incorpo-
rates machine learning advice in the design of
online algorithms to improve real-world perfor-
mance. A central consideration is the extent to
which predictions can be trusted—while existing
approaches often require users to specify an aggre-
gate trust level, modern machine learning models
can provide estimates of prediction-level uncer-
tainty. In this paper, we propose calibration as
a principled and practical tool to bridge this gap,
demonstrating the benefits of calibrated advice
through two case studies: the ski rental and on-
line job scheduling problems. For ski rental, we
design an algorithm that achieves near-optimal
prediction-dependent performance and prove that,
in high-variance settings, calibrated advice offers
more effective guidance than alternative methods
for uncertainty quantification. For job scheduling,
we demonstrate that using a calibrated predictor
leads to significant performance improvements
over existing methods. Evaluations on real-world
data validate our theoretical findings, highlighting
the practical impact of calibration for algorithms
with predictions.
1. Introduction
In recent years, advances in machine learning (ML) mod-
els have inspired researchers to revisit the design of classic
online algorithms, incorporating insights from ML-based
advice to improve decision-making in real-world environ-
ments. This research area, termed algorithms with predic-
tions, seeks to design algorithms that are both robust to
worst-case inputs and achieve performance that improves
with prediction accuracy (a desideratum termed consistency)
(Lykouris & Vassilvitskii, 2018). Many learning-augmented
1Department of Computer Science, Stanford University, Stan-
ford, CA, USA 2Department of Management Science & Engineer-
ing, Stanford University, Stanford, CA, USA. Correspondence to:
Anders ...
Efficient Sparse PCA via Block-Diagonalization
ICLR 2025

Published as a conference paper at ICLR 2025
EFFICIENT SPARSE PCA VIA BLOCK-
DIAGONALIZATION
Alberto Del Pia & Dekun Zhou
University of Wisconsin-Madison
{delpia,dzhou44}@wisc.edu
Yinglun Zhu
University of California, Riverside
yzhu@ucr.edu
ABSTRACT
Sparse Principal Component Analysis (Sparse PCA) is a pivotal tool in data analy-
sis and dimensionality reduction. However, Sparse PCA is a challenging problem
in both theory and practice: it is known to be NP-hard and current exact methods
generally require exponential runtime. In this paper, we propose a novel framework
to efficiently approximate Sparse PCA by (i) approximating the general input
covariance matrix with a re-sorted block-diagonal matrix, (ii) solving the Sparse
PCA sub-problem in each block, and (iii) reconstructing the solution to the original
problem. Our framework is simple and powerful: it can leverage any off-the-shelf
Sparse PCA algorithm and achieve significant computational speedups, with a
minor additive error that is linear in the approximation error of the block-diagonal
matrix. Suppose g(k, d) is the runtime of an algorithm (approximately) solving
Sparse PCA in dimension d and with sparsity constant k. Our framework, when in-
tegrated with this algorithm, reduces the runtime to O
 d
d⋆· g(k, d⋆) + d2
, where
d⋆≤d is the largest block size of the block-diagonal matrix. For instance, integrat-
ing our framework with the Branch-and-Bound algorithm reduces the complexity
from g(k, d) = O(k3 · dk) to O(k3 · d · (d⋆)k−1), demonstrating exponential
speedups if d⋆is small. We perform large-scale evaluations on many real-world
datasets: for exact Sparse PCA algorithm, our method achieves an average speedup
factor of 100.50, while maintaining an average approximation error of 0.61%;
for approximate Sparse PCA algorithm, our method achieves an average speedup
factor of 6.00 and an average approximation error of -0.91%, meaning that our
method oftentimes finds better solutions.
1
INTRODUCTION
In this ...
Hierarchical Masked Autoregressive Models with Low-Resolution Token Pivots
ICML 2025

Hierarchical Masked Autoregressive Models with Low-Resolution Token Pivots
Guangting Zheng 1 † Yehao Li 2 Yingwei Pan 2 Jiajun Deng 3 Ting Yao 2 Yanyong Zhang 1 Tao Mei 2
Abstract
Autoregressive models have emerged as a pow-
erful generative paradigm for visual generation.
The current de-facto standard of next token pre-
diction commonly operates over a single-scale
sequence of dense image tokens, and is incapable
of utilizing global context especially for early
tokens prediction. In this paper, we introduce
a new autoregressive design to model a hierar-
chy from a few low-resolution image tokens to
the typical dense image tokens, and delve into a
thorough hierarchical dependency across multi-
scale image tokens. Technically, we present a
Hierarchical Masked Autoregressive models (Hi-
MAR) that pivot on low-resolution image tokens
to trigger hierarchical autoregressive modeling
in a multi-phase manner. Hi-MAR learns to pre-
dict a few image tokens in low resolution, func-
tioning as intermediary pivots to reflect global
structure, in the first phase. Such pivots act as
the additional guidance to strengthen the next au-
toregressive modeling phase by shaping global
structural awareness of typical dense image to-
kens. A new Diffusion Transformer head is fur-
ther devised to amplify the global context among
all tokens for mask token prediction. Extensive
evaluations on both class-conditional and text-to-
image generation tasks demonstrate that Hi-MAR
outperforms typical AR baselines, while requiring
fewer computational costs. Code is available at
https://github.com/HiDream-ai/himar.
† This work was performed when Guangting Zheng was visit-
ing HiDream.ai as a research intern. 1University of Science and
Technology of China, Anhui, China 2HiDream.ai Inc, Beijing,
China 3The University of Adelaide, Adelaide, Australia. Corre-
spondence to: Yingwei Pan <panyw.ustc@gmail.com>, Yanyong
Zhang <yanyongz@ustc.edu.cn>.
Proceedings of the 42 nd International Conference on Machin...
Closed-form Solutions: A New Perspective on Solving Differential Equations
ICML 2025

Closed-form Solutions: A New Perspective on Solving Differential Equations
Shu Wei 1 2 Yanjie Li 1 Lina Yu 1 2 Weijun Li 1 3 4 Min Wu 1 Linjun Sun 1 Jingyi Liu 1 Hong Qin 1 3
Yusong Deng 1 2 Jufeng Han 1 3 Yan Pang 1 4
Abstract
The quest for analytical solutions to differential
equations has traditionally been constrained by
the need for extensive mathematical expertise.
Machine learning methods like genetic algorithms
have shown promise in this domain, but are hin-
dered by significant computational time and the
complexity of their derived solutions. This paper
introduces SSDE (Symbolic Solver for Differen-
tial Equations), a novel reinforcement learning-
based approach that derives symbolic closed-form
solutions for various differential equations. Eval-
uations across a diverse set of ordinary and par-
tial differential equations demonstrate that SSDE
outperforms existing machine learning methods,
delivering superior accuracy and efficiency in ob-
taining analytical solutions.
1. Introduction
Differential equations (DEs) are foundational to mathemat-
ics, physics, and the natural sciences, providing abstract
models for diverse physical phenomena. For example, Pois-
son’s equation governs the distribution of electrostatic poten-
tial, while the heat equation describes temperature variations
within an object (Evans, 2022). Solving these DEs is critical
for understanding and predicting the dynamics of complex
physical systems (Alkhadhr & Almekkawy, 2021; Savovi´c
et al., 2023).
Deriving analytical solutions requires rigorous analysis of
their existence, uniqueness, stability, and behavioral proper-
ties, often leveraging advanced mathematical theories and
1AnnLab, Institute of Semiconductors, Chinese Academy of
Sciences, Beijing, China 2College of Materials Science and Opto-
Electronic Technology, University of Chinese Academy of Sci-
ences, Beijing, China 3School of Integrated Circuits, University
of Chinese Academy of Sciences, Beijing, China 4School of
Industry-ed...
Alpha-SQL: Zero-Shot Text-to-SQL using Monte Carlo Tree Search
ICML 2025

Alpha-SQL: Zero-Shot Text-to-SQL using Monte Carlo Tree Search
Boyan Li 1 Jiayi Zhang 1 Ju Fan 2 Yanwei Xu 3 Chong Chen 3 Nan Tang 1 Yuyu Luo 1
Abstract
Text-to-SQL, which enables natural language
interaction with databases, serves as a pivotal
method across diverse industries. With new, more
powerful large language models (LLMs) emerg-
ing every few months, fine-tuning has become
incredibly costly, labor-intensive, and error-prone.
As an alternative, zero-shot Text-to-SQL, which
leverages the growing knowledge and reason-
ing capabilities encoded in LLMs without task-
specific fine-tuning, presents a promising and
more challenging direction. To address this chal-
lenge, we propose Alpha-SQL, a novel approach
that leverages a Monte Carlo Tree Search (MCTS)
framework to iteratively infer SQL construction
actions based on partial reasoning states.
To
enhance the framework’s reasoning capabilities,
we introduce LLM-as-Action-Model to dynami-
cally generate SQL construction actions during
the MCTS process, steering the search toward
more promising SQL queries. Moreover, Alpha-
SQL employs a self-supervised reward function
to evaluate the quality of candidate SQL queries,
ensuring more accurate and efficient query gener-
ation. Experimental results show that Alpha-SQL
achieves 69.7% execution accuracy on the BIRD
development set, using a 32B open-source LLM
without fine-tuning. Alpha-SQL outperforms the
best previous zero-shot approach based on GPT-
4o by 2.5% on the BIRD development set. The
code is available at https://github.com/
HKUSTDial/Alpha-SQL.
1. Introduction
Text-to-SQL (a.k.a. NL2SQL) converts natural language
queries into SQL, simplifying access to relational databases
1The Hong Kong University of Science and Technology
(Guangzhou) 2Renmin University of China 3Huawei Technologies
Ltd. Correspondence to: Yuyu Luo <yuyuluo@hkust-gz.edu.cn>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025
...