dataset-review/dataset-review huggingface dataset:
FedMGP: Personalized Federated Learning with Multi-Group Text-Visual Prompts
NeurIPS 2025

FedMGP: Personalized Federated Learning with
Multi-Group Text-Visual Prompts
Weihao Bo1,Yanpeng Sun2∗, Yu Wang3, Xinyu Zhang4, Zechao Li1
1Nanjing University of Science and Technology
2National University of Singapore
3Baidu VIS
4University of Auckland
Abstract
In this paper, we introduce FedMGP, a new paradigm for personalized federated
prompt learning in vision-language models (VLMs). Existing federated prompt
learning (FPL) methods often rely on a single, text-only prompt representation,
which leads to client-speciﬁc overﬁtting and unstable aggregation under heteroge-
neous data distributions. Toward this end, FedMGP equips each client with mul-
tiple groups of paired textual and visual prompts, enabling the model to capture
diverse, ﬁne-grained semantic and instance-level cues. A diversity loss is intro-
duced to drive each prompt group to specialize in distinct and complementary se-
mantic aspects, ensuring that the groups collectively cover a broader range of local
characteristics. During communication, FedMGP employs a dynamic prompt ag-
gregation strategy based on similarity-guided probabilistic sampling: each client
computes the cosine similarity between its prompt groups and the global prompts
from the previous round, then samples s groups via a softmax-weighted distribu-
tion. This soft selection mechanism preferentially aggregates semantically aligned
knowledge while still enabling exploration of underrepresented patternseffectively
balancing the preservation of common knowledge with client-speciﬁc features.
Notably, FedMGP maintains parameter efﬁciency by redistributing a ﬁxed prompt
capacity across multiple groups, achieving state-of-the-art performance with the
lowest communication parameters (5.1k) among all federated prompt learning
methods. Theoretical analysis shows that our dynamic aggregation strategy pro-
motes robust global representation learning by reinforcing shared semantics while
suppressing client-speciﬁc noise. Extensive experiments dem...
Modulated Diffusion:  Accelerating Generative Modeling with Modulated Quantization
ICML 2025

Modulated Diffusion:
Accelerating Generative Modeling with Modulated Quantization
Weizhi Gao 1 Zhichao Hou 1 Junqi Yin 2 Feiyi Wang 2 Linyu Peng 3 Xiaorui Liu 1
Abstract
Diffusion models have emerged as powerful gen-
erative models, but their high computational cost
in iterative sampling remains a significant bot-
tleneck. In this work, we present an in-depth
and insightful study of state-of-the-art accelera-
tion techniques for diffusion models, including
caching and quantization, and reveal their limita-
tions in computation error and generation quality.
To break these limits, this work introduces Modu-
lated Diffusion (MoDiff), an innovative, rigorous,
and principled framework that accelerates gener-
ative modeling through modulated quantization
and error compensation. MoDiff not only inherits
the advantages of existing caching and quantiza-
tion methods but also serves as a general frame-
work to accelerate all diffusion models. The ad-
vantages of MoDiff are supported by solid theoret-
ical insight and analysis. In addition, extensive ex-
periments on CIFAR-10 and LSUN demonstrate
that MoDiff significantly reduces activation quan-
tization from 8 bits to 3 bits without performance
degradation in post-training quantization (PTQ).
Our code implementation is available at https:
//github.com/WeizhiGao/MoDiff.
1. Introduction
Diffusion models have emerged as powerful generative mod-
els for producing high-quality data samples, ranging from
images to audio and beyond (Ho et al., 2020; Song et al.,
2021a;b). These models work by iteratively transforming
a simple noise distribution into complex, structured out-
puts, guided by a learned reverse diffusion process. Despite
their effectiveness, diffusion models come with significant
1Department of Computer Science, North Carolina State Uni-
versity 2National Center for Computational Science, Oak Ridge
National Lab 3Department of Mechanical Engineering, Keio Uni-
versity. Correspondence to: Xiaorui Liu <xliu96@ncsu.edu>.
P...
Joint Localization and Activation Editing for Low-Resource Fine-Tuning
ICML 2025

Joint Localization and Activation Editing for Low-Resource Fine-Tuning
Wen Lai 1 2 Alexander Fraser 1 2 Ivan Titov 3 4
Abstract
Parameter-efficient fine-tuning (PEFT) methods,
such as LoRA, are commonly used to adapt LLMs.
However, the effectiveness of standard PEFT
methods is limited in low-resource scenarios with
only a few hundred examples. Recent advances
in interpretability research have inspired the emer-
gence of activation editing (or steering) tech-
niques, which modify the activations of specific
model components. Due to their extremely small
parameter counts, these methods show promise
for small datasets. However, their performance is
highly dependent on identifying the correct mod-
ules to edit and often lacks stability across differ-
ent datasets. In this paper, we propose Joint Local-
ization and Activation Editing (JOLA), a method
that jointly learns (1) which heads in the Trans-
former to edit (2) whether the intervention should
be additive, multiplicative, or both and (3) the
intervention parameters themselves - the vectors
applied as additive offsets or multiplicative scal-
ings to the head output. Through evaluations on
three benchmarks spanning commonsense reason-
ing, natural language understanding, and natural
language generation, we demonstrate that JOLA
consistently outperforms existing methods.1
1. Introduction
Parameter-efficient fine-tuning (PEFT; Han et al., 2024)
methods are widely used to adapt large language models
(LLMs). However, popular PEFT approaches like LoRA
(Hu et al., 2021) often struggle in low-resource settings with
*Work done during Wen Lai’s visit to The University of Edin-
burgh. 1Technical University of Munich 2Munich Center for Ma-
chine Learning 3ILLC, University of Edinburgh 4ILLC, University
of Amsterdam. Correspondence to: Wen Lai <wen.lai@tum.de>,
Ivan Titov <ititov@inf.ed.ac.uk>, Alexander Fraser <alexan-
der.fraser@tum.de>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. ...
Ditch the Denoiser: Emergence of Noise Robustness in Self-Supervised Learning from Data Curriculum
NeurIPS 2025

Ditch the Denoiser: Emergence of Noise Robustness in
Self-Supervised Learning from Data Curriculum
Wenquan Lu1
Jiaqi Zhang1
Hugues Van Assel2
Randall Balestriero1
1Brown University
2Genentech
{wenquan_lu, jiaqi_zhang6, randall_balestriero}@brown.edu
van_assel.hugues@gene.com
Abstract
Self-Supervised Learning (SSL) has become a powerful solution to extract rich
representations from unlabeled data. Yet, SSL research is mostly focused on
clean, curated and high-quality datasets. As a result, applying SSL on noisy data
remains a challenge, despite being crucial to applications such as astrophysics,
medical imaging, geophysics or finance. In this work, we present a fully self-
supervised framework that enables noise-robust representation learning without
requiring a denoiser at inference or downstream fine-tuning. Our method first
trains an SSL denoiser on noisy data, then uses it to construct a denoised-to-
noisy data curriculum (i.e., training first on denoised, then noisy samples) for
pretraining a SSL backbone (e.g., DINOv2), combined with a teacher-guided
regularization that anchors noisy embeddings to their denoised counterparts. This
process encourages the model to internalize noise robustness. Notably, the denoiser
can be discarded after pretraining, simplifying deployment. On ImageNet-1k with
ViT-B under extreme Gaussian noise (σ = 255, SNR = 0.72 dB), our method
improves linear probing accuracy by 4.8% over DINOv2, demonstrating that
denoiser-free robustness can emerge from noise-aware pretraining. The code is
available at https://github.com/wenquanlu/noisy_dinov2.
1
Introduction
Self-supervised methods like DINOv2 [31, 7, 16, 9] have demonstrated remarkable success by
leveraging unlabeled data to learn visual representations able to solve many tasks in zero or few
shot manners [3]. However, the performances of those methods heavily rely on the availability
of clean and high-quality datasets. As a result, some prior works have focused on building data
curation ...
Towards General Continuous Memory for Vision-Language Models
NeurIPS 2025

Towards General Continuous Memory for
Vision-Language Models
Wenyi Wu∗, Zixuan Song∗, Kun Zhou†, Yifei Shao, Zhiting Hu, Biwei Huang
University of California, San Diego.
kuzhou@ucsd.edu
Abstract
Language models (LMs) and their extension, vision-language models (VLMs),
have achieved remarkable performance across various tasks. However, they still
struggle with complex reasoning tasks that require multimodal or multilingual
real-world knowledge. To support such capabilities, an external memory system
that can efficiently provide relevant multimodal information is essential. Existing
approaches generally concatenate image and text tokens into a long sequence as
memory, which, however, may drastically increase context length and even de-
grade performance. In contrast, we propose using continuous memory-a compact
set of dense embeddings-to more effectively and efficiently represent multimodal
and multilingual knowledge. Our key insight is that a VLM can serve as its own
continuous memory encoder. We empirically show that this design improves per-
formance on complex multimodal reasoning tasks. Building on this, we introduce a
data-efficient and parameter-efficient method to fine-tune the VLM into a memory
encoder, requiring only 1.2% of the model’s parameters and a small corpus of
15.6K self-synthesized samples. Our approach CoMEM utilizes VLM’s origi-
nal capabilities to encode arbitrary multimodal and multilingual knowledge into
just 8 continuous embeddings. Since the inference-time VLM remains frozen,
our memory module is plug-and-play and can be flexibly integrated as needed.
Extensive experiments across eight multimodal reasoning benchmarks demon-
strate the effectiveness of our approach. Code and data is publicly released here
https://github.com/WenyiWU0111/CoMEM.
1
Introduction
Figure 1: CoMEM architecture in comparison
to the traditional RAG method.
Through
large-scale
training,
language
mod-
els (LMs) [1, 2] have demonstrated remarkable
performance across diver...
Bootstrapped Model Predictive Control
ICLR 2025

Published as a conference paper at ICLR 2025
BOOTSTRAPPED MODEL PREDICTIVE CONTROL
Yuhang Wang, Hanwei Guo, Sizhe Wang, Long Qian, Xuguang Lan∗
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence
National Engineering Research Center for Visual Information and Application
Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Xi’an, China
{wertyuilife,clannnagisa,wangsizhe,qianlongym}@stu.xjtu.edu.cn
xglan@mail.xjtu.edu.cn
ABSTRACT
Model Predictive Control (MPC) has been demonstrated to be effective in con-
tinuous control tasks. When a world model and a value function are available,
planning a sequence of actions ahead of time leads to a better policy. Existing
methods typically obtain the value function and the corresponding policy in a
model-free manner. However, we find that such an approach struggles with com-
plex tasks, resulting in poor policy learning and inaccurate value estimation. To
address this problem, we leverage the strengths of MPC itself. In this work, we
introduce Bootstrapped Model Predictive Control (BMPC), a novel algorithm that
performs policy learning in a bootstrapped manner. BMPC learns a network pol-
icy by imitating an MPC expert, and in turn, uses this policy to guide the MPC
process. Combined with model-based TD-learning, our policy learning yields bet-
ter value estimation and further boosts the efficiency of MPC. We also introduce
a lazy reanalyze mechanism, which enables computationally efficient imitation
learning. Our method achieves superior performance over prior works on diverse
continuous control tasks. In particular, on challenging high-dimensional loco-
motion tasks, BMPC significantly improves data efficiency while also enhancing
asymptotic performance and training stability, with comparable training time and
smaller network sizes. Code is available at https://github.com/wertyuilife2/bmpc.
1
INTRODUCTION
Model-based reinforcement learning (MBRL) algorithms that incorporate online pla...
Reducing Tool Hallucination via Reliability Alignment
ICML 2025

Reducing Tool Hallucination via Reliability Alignment
Hongshen Xu 1 2 3 Zichen Zhu 1 2 3 Lei Pan 4 Zihan Wang 1 2 3 Su Zhu 4
Da Ma 1 2 3 Ruisheng Cao 1 2 3 Lu Chen 1 2 3 5 Kai Yu 1 2 3 5
Abstract
Large Language Models (LLMs) have expanded
their capabilities beyond language generation to
interact with external tools, enabling automation
and real-world applications. However, tool hallu-
cinations—where models either select inappropri-
ate tools or misuse them—pose significant chal-
lenges, leading to erroneous task execution, in-
creased computational costs, and reduced system
reliability. To systematically address this issue,
we define and categorize tool hallucinations into
two main types: tool selection hallucination and
tool usage hallucination. To evaluate and mitigate
these issues, we introduce RelyToolBench, which
integrates specialized test cases and novel metrics
to assess hallucination-aware task success and ef-
ficiency. Finally, we propose Relign, a reliability
alignment framework that expands the tool-use
action space to include indecisive actions, allow-
ing LLMs to defer tool use, seek clarification, or
adjust tool selection dynamically. Through exten-
sive experiments, we demonstrate that Relign sig-
nificantly reduces tool hallucinations, improves
task reliability, and enhances the efficiency of
LLM tool interactions. The code and data are
publicly available at https://github.com/
X-LANCE/ToolHallucination.
1. Introduction
The primary goal of tool learning is to enable Large Lan-
guage Models (LLMs; Gemini Team, 2023; Achiam et al.,
2023; Dubey et al., 2024) to understand and effectively use
1X-LANCE Lab, School of Computer Science, Shanghai Jiao
Tong University, Shanghai, China. 2MoE Key Lab of Artificial
Intelligence, Shanghai, China.
3Jiangsu Key Lab of Language
Computing, Suzhou, China. 4AISpeech Co., Ltd., Suzhou, China.
5Suzhou Laboratory, Suzhou, China. xuhongshen@sjtu.edu.cn.
Correspondence to: Lu Chen <chenlusz@sjtu.edu.cn>, Kai Yu
<kai.yu@sj...
Revealing and Mitigating Over-Attention in Knowledge Editing
ICLR 2025

Published as a conference paper at ICLR 2025
REVEALING AND MITIGATING OVER-ATTENTION IN
KNOWLEDGE EDITING
Pinzheng Wang1,2
Zecheng Tang1,2
Keyan Zhou1,2
Juntao Li1,2∗
Qiaoming Zhu1,2
Min Zhang1,2
1School of Computer Science and Technology, Soochow University
2Key Laboratory of Data Intelligence and Advanced Computing, Soochow University
{pzwang1,zctang,kyzhou}@stu.sud.edu.cn
{ljt,qmzhu,minzhang}@suda.edu.cn
ABSTRACT
Large Language Models (LLMs) have demonstrated superior performance across
a wide range of tasks, but they still exhibit undesirable errors due to incorrect
knowledge learned from the training data. To avoid this, knowledge editing methods
emerged to precisely edit the specific model knowledge via efficiently modifying
a very small percentage of parameters. However, those methods can lead to the
problem of Specificity Failure, where the existing knowledge and capabilities are
severely degraded due to editing. Our preliminary indicates that Specificity Failure
primarily stems from the model’s attention heads assigning excessive attention
scores to entities related to the edited knowledge, thereby unduly focusing on
specific snippets within the context, which we denote as the Attention Drift
phenomenon. To mitigate such Attention Drift issue, we introduce a simple yet
effective method Selective Attention Drift Restriction (SADR), which introduces
an additional regularization term during the knowledge editing process to restrict
changes in the attention weight distribution, thereby preventing undue focus on the
edited entity. Experiments on five frequently used strong LLMs demonstrate the
effectiveness of our method, where SADR can significantly mitigate Specificity
Failure in the predominant knowledge editing tasks1.
1
INTRODUCTION
Large language models (LLMs) have demonstrated outstanding performance on various downstream
natural language processing tasks, e.g., dialogue generation (Ni et al., 2023; Yang et al., 2024a),
attributed to the numerous inherent...
Stacey: Promoting Stochastic Steepest Descent via Accelerated $\ell_p$-Smooth Nonconvex Optimization
ICML 2025

STACEY: Promoting Stochastic Steepest Descent via Accelerated
ℓp-Smooth Nonconvex Optimization
Xinyu Luo * 1 Cedar Site Bai * 1 Bolian Li * 1 Petros Drineas 1 Ruqi Zhang 1 Brian Bullins 1
Abstract
While popular optimization methods such as
SGD, AdamW, and Lion depend on steepest de-
scent updates in either ℓ2 or ℓ∞norms, there re-
mains a critical gap in handling the non-Euclidean
structure observed in modern deep networks train-
ing. In this work, we address this need by intro-
ducing a new accelerated ℓp steepest descent al-
gorithm, called STACEY, which uses interpolated
primal-dual iterate sequences to effectively nav-
igate non-Euclidean smooth optimization tasks.
In addition to providing novel theoretical guar-
antees for the foundations of our algorithm, we
empirically compare our approach against these
popular methods on tasks including image classi-
fication and language model (LLM) pretraining,
demonstrating both faster convergence and higher
final accuracy. We further evaluate different val-
ues of p across various models and datasets, un-
derscoring the importance and efficiency of non-
Euclidean approaches over standard Euclidean
methods.
Code can be found at https://
github.com/xinyuluo8561/Stacey.
1. Introduction
Stochastic first-order methods have proven essential for effi-
ciently training modern deep learning models. Beyond the
basic stochastic gradient descent (SGD) algorithm (Robbins
& Monro, 1951) and its momentum-based variants (Nes-
terov, 1983; Polyak, 1964), a variety of adaptive methods
have been developed, such as AdaGrad (Duchi et al., 2011a),
Adam (Kingma & Ba, 2015), and AdamW (Loshchilov &
Hutter, 2019), which incorporate second-moment gradient
information to provide per-coordinate scaling. Meanwhile,
*Equal contribution
1Department of Computer Science, Pur-
due University, Indiana, USA. Correspondence to: Xinyu Luo
<luo466@purdue.edu>, Cedar Site Bai <bai123@purdue.edu>,
Bolian Li <li4468@purdue.edu>.
Proceedings of the 42 nd Internat...
Riemannian Diffusion Adaptation for Distributed Optimization on Manifolds
ICML 2025

Riemannian Diffusion Adaptation for Distributed Optimization on Manifolds
Xiuheng Wang 1 Ricardo Borsoi 1 C´edric Richard 2 Ali H. Sayed 3
Abstract
Online distributed optimization is particularly use-
ful for solving optimization problems with stream-
ing data collected by multiple agents over a net-
work. When the solutions lie on a Riemannian
manifold, such problems become challenging to
solve, particularly when efficiency and continuous
adaptation are required. This work tackles these
challenges and devises a diffusion adaptation strat-
egy for decentralized optimization over general
manifolds. A theoretical analysis shows that the
proposed algorithm is able to approach network
agreement after sufficient iterations, which allows
a non-asymptotic convergence result to be derived.
We apply the algorithm to the online decentral-
ized principal component analysis problem and
Gaussian mixture model inference. Experimental
results with both synthetic and real data illustrate
its performance.
1. Introduction
In the decentralized setting, this work deals with the multi-
agent optimization problem seeking consensus on a Rieman-
nian manifold M:
min
w∈M
1
K
K
X
k=1
Jk(w) ,
(1)
where Jk : M →R is a local risk function defined for each
agent by Jk(w) = Exk

Q(w; xk)
	
in terms of the expec-
tation of some loss function Q(w; xk). The computation
of Jk(w) is over the unknown distribution of the data xk,
which makes it necessary to use a stochastic approximation
for the gradient vector based on a set of independent realiza-
tions xk,t, observed sequentially over time. A wide range
Part of this work was done while Xiuheng Wang was a PhD
student at Universit´e Cˆote d’Azur. 1Universit´e de Lorraine, CNRS,
CRAN, France 2Universit´e Cˆote d’Azur, CNRS, OCA, France
3 ´Ecole Polytechnique F´ed´erale de Lausanne, Switzerland. Corre-
spondence to: Xiuheng Wang <dr.xiuheng.wang@gmail.com>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 26...
Non-Stationary Predictions May Be More Informative: Exploring Pseudo-Labels with a Two-Phase Pattern of Training Dynamics
ICML 2025

Non-Stationary Predictions May Be More Informative:
Exploring Pseudo-Labels with a Two-Phase Pattern of Training Dynamics
Hongbin Pei 1 Jingxin Hai 1 Yu Li 1 Huiqi Deng 1
Denghao Ma 2 Jie Ma 1 Pinghui Wang 1 Jing Tao 1 Xiaohong Guan 1
Abstract
Pseudo-labeling is a widely used strategy in semi-
supervised learning. Existing methods typically
select predicted labels with high confidence scores
and high training stationarity, as pseudo-labels to
augment training sets. In contrast, this paper ex-
plores the pseudo-labeling potential of predicted
labels that do not exhibit these characteristics. We
discover a new type of predicted labels suitable for
pseudo-labeling, termed two-phase labels, which
exhibit a two-phase pattern during training: they
are initially predicted as one category in early
training stages and switch to another category in
subsequent epochs. Case studies show the two-
phase labels are informative for decision bound-
aries. To effectively identify the two-phase labels,
we design a 2-phasic metric that mathematically
characterizes their spatial and temporal patterns.
Furthermore, we propose a loss function tailored
for two-phase pseudo-labeling learning, allowing
models not only to learn correct correlations but
also to eliminate false ones. Extensive experi-
ments on eight datasets show that our proposed
2-phasic metric acts as a powerful booster for
existing pseudo-labeling methods by additionally
incorporating the two-phase labels, achieving an
average classification accuracy gain of 1.73% on
image datasets and 1.92% on graph datasets.
1. Introduction
Pseudo-labeling (Lee et al., 2013) is a widely employed strat-
egy in many semi-supervised learning methods, e.g., self-
training (Amini et al., 2025), co-training (Blum & Mitchell,
1998), consistency regularization (Sohn et al., 2020), and
1MOE KLINNS Lab, Xi’an Jiaotong University, China;
2Beijing Information Science and Technology University, China.
Correspondence to: Jing Tao <jtao@mail.xjtu.edu.cn...