ICML 2025

Does Low Rank Adaptation Lead to Lower Robustness
against Training-Time Attacks?
Zi Liang 1 Haibo Hu 1 Qingqing Ye 1 Yaxin Xiao 1 Ronghua Li 1
Abstract
Low rank adaptation (LoRA) has emerged as
a prominent technique for ﬁne-tuning large lan-
guage models (LLMs) thanks to its superb efﬁ-
ciency gains over previous methods. While ex-
tensive studies have examined the performance
and structural properties of LoRA, its behavior
upon training-time attacks remain underexplored,
posing signiﬁcant security risks. In this paper, we
theoretically investigate the security implications
of LoRA’s low-rank structure during ﬁne-tuning,
in the context of its robustness against data poi-
soning and backdoor attacks. We propose an ana-
lytical framework that models LoRA’s training dy-
namics, employs the neural tangent kernel to sim-
plify the analysis of the training process, and ap-
plies information theory to establish connections
between LoRA’s low rank structure and its vul-
nerability against training-time attacks. Our anal-
ysis indicates that LoRA exhibits better robust-
ness to backdoor attacks than full ﬁne-tuning,
while becomes more vulnerable to untargeted
data poisoning due to its over-simpliﬁed infor-
mation geometry. Extensive experimental evalua-
tions have corroborated our theoretical ﬁndings.
1. Introduction
With the rapid growth in the parameter size of large language
models (LLMs), parameter-efﬁcient ﬁne-tuning (PEFT) (Xu
et al., 2023; Han et al., 2024) has gained increasing attention
in both research and industry communities. Among various
PEFT strategies, low-rank adaptation (LoRA) (Hu et al.,
2021) has emerged as the de facto standard for ﬁne-tuning
LLMs thanks to its computational efﬁciency and minimal
performance degradation.
1The Hong Kong Polytechnic University, Hong Kong, China.
Correspondence to: Haibo Hu <haibo.hu@polyu.edu.hk>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025
by the...