DiaBlo: Diagonal Blocks Are Sufficient For Finetuning
Authors: Selcuk Gurses
Aozhong Zhang
Yanxia Deng
Xun Dong
Xin Li
Naigang Wang
Penghang Yin
Zi Yang
Venue: ICLR 2026 Poster (2026)
Keywords: Finetuning
Parameter-Efficient
LLM
Diagonal Block
Primary area: foundation or frontier models, including LLMs
TL;DR: Fine-tuning only the diagonal blocks of weights yields superior performance

Abstract: Fine-tuning is a critical step for adapting large language models (LLMs) to domain-specific downstream tasks. To mitigate the substantial computational and memory costs of full-model fine-tuning, Parameter-Efficient Fine-Tuning (PEFT) methods have been proposed to update only a small subset of model parameters. However, performance gaps between PEFT approaches and full-model fine-tuning still exist. In this work, we present *DiaBlo*, a simple yet effective PEFT approach that updates only the diagonal blocks of selected model weight matrices. Unlike Low-Rank Adaptation (LoRA) and its variants, DiaBlo eliminates the need for low-rank matrix products, thereby avoiding the reliance on auxiliary initialization schemes or customized optimization strategies to improve convergence. This design leads to stable and robust convergence while maintaining comparable memory efficiency and training speed to LoRA. Moreover, we provide theoretical guarantees showing that, under mild low-rank conditions, DiaBlo is more expressive than LoRA in the linear problem and converges to a stationary point of the general nonlinear full fine-tuning. Through extensive experiments across a range of tasks—including commonsense reasoning, arithmetic reasoning, code generation, and safety alignment—we show that fine-tuning only diagonal blocks is sufficient for strong and consistent performance. DiaBlo not only achieves competitive accuracy but also preserves high memory efficiency and fast fine-tuning speed. Codes are available at https://github.com/ziyangjoy/DiaBlo.

Published as a conference paper at ICLR 2026
DIABLO: DIAGONAL BLOCKS ARE SUFFICIENT FOR FINETUNING
Selcuk Gurses Aozhong Zhang
Yanxia Deng'
Xun Dong'
Xin Li'
Naigang Wang Penghang Yin Zi Yangh
Universiit at Albany, SUNY 'IBM Watson Research Center
(sgurses, azhang3, ydeng5, xdong5, x1i48, pyin, zyang8 @albany edu nwangeus. 1bm. com
ABSTRACT
Fine-tuning is critical step for adapting large language models (LLLMs) to domain- specific downstream tasks. To mitigate the substantial computational and memory costs of full-model fine-tuning, Parameter-Efffciee Fine- Sine-Tuning (PEFT) methods have been proposed to update only small subset of model parameters. However, performance gaps between PEFT approaches and full-model fine-tuning still exist. In this work, we present DiaBlo, simple yet effective PEFT approach that updates only the diagonal blocks of selected model weight matrices. Unlike Low-Rank Adaptation (LoRA) and its variants, DiaBlo eliminates the need for low-rank ma- trix products, thereby avoiding the reliance on auxiliary initialization schemes customized optimization strategies to improve convergence. This design leads to stable and robust convergence while maintaining comparable memory efficiency and training speed to LoRA. Moreover, we provide theoretical guarantees showing that, under mild low-rank conditions, DiaBlo more expressive than LoRA in the linear problem and converges stationary point of the general nonlinear full fine-tuning. Through extensive experiments across a range of tasks including commonsense reasoning, arithmetic reasoning, code generation, and safety align- me show that fine-tuning only diagonal blocks is sufficient for strong and consistent performance. DiaBlo not only achieves competitive accuracy but also preserves high memory efficiency and fast fine-tuning speed. Codes are available at ltttps://githitii wwwp:hhw..ooooooeeaeeeeeeeee com http./////..aaaayoooiaaaa...
INTRODUCTION
Large language models (LLMs) (Achiam al. 2023 Brown et al., 2020 Touvron al. 2023a) have achieved remarkable success across wide range of natural language processing tasks, including reasoning, generation, and alignment. These models, often consisting billions of parameters, typically pre-trained on broad, general-purposs corpora. However, to adapt them to specific downstream tasks or domains, fine-tuning is essential. Full fine-tuning, which updates all parameters of the model, has been shown yield strong performance but is prohibitively expensive in terms of computational cost, memory usage, and storage Saaegge-eppccillly on resource- consurcienn devices.
To address these challenges, Grrameter-Efficieen Fine-Tuning (PEFT) methods (Ding et al. 2023 Han
et al. 2024) have emerged as promising alternative. These methods aim to retain the performance benefits of full fine-tuning while updating only small fraction of the model's parameters. Early approaches, such as Prompt Tuning (Lester et al.) (221I) and Prefix Tuning (Li & Liang] 2021) introduce task-specific embeddings or continuous prompts that steer model behavior without modify- ing its original weights. These methods are lightweight and straightforward to implement, making them suitable for low-resource adaptation scenarios. However, they often struggle with scalability and limited expressiveness, especially on complex tasks. To improve performance, more structured methods like Low-Rank Adaptation (LoRA) (Hayou et al. 2024 Hu al., 202T) have been proposed. LoRA adds trainable low-rank matrices to the pretrained weight, significantly reducing the number of trainable parameters. has demonstrated strong performance across a variety of applications.
* Equal contribution
Corresponding author

---

Published as a conference paper at ICLR 2026
Tminable
- Frozen
Pretrained Matrix
Pretrained Matrix
R
W k2
E RM K0
A- o
(4) Fine-tuning
(b) LoRA
(c) DiaBlo
Figure 1: Comparison between full fine-tuning, LoRA, and proposed DiaBlo.
Despite its efficiency, LoRA and its variants often encounter issues such as unstable convergence and performance degradation. growing number of work has proposed various extensions LoRA, in- cluding tailored initialization schemes (Meng et al., 2024 Wang et al. 2024a and structure-specinns
optimization strategies (Liu et al., 2024] Wang & Liang 2024). Nonetheless, such improvements may lead to greater algorithmic complexity, potentially impacting implementation and efficiency. Sparsity-based methods, in contrast, aim to fine-tune only subset of weight entries. Yet, most existing approaches rely unstructured sparsity-thouug
either random masking (Deng et al.) 2024; Rios et al. 2025 Xu & Zhang, 2024) or importance-basse selection (He et al. 2024) not only increases time complexity but also results in patterns that are difficult to exploit efficiently on modern hardware. S²FT Yang et al. 2024a) is an efficient LLM adaptation framework that selectively updates sparse components while computing
dense submatrices. SparseLoRA (Khaki) jet al.) (025) employs a lightweight SVD- based estimator to identify sparse weights for loss and gradient computation, thereby accelerating L.M fine-tuning.
In this work, we propose DiaBlo, a simple yet effective PEFT framework that fine-tunes only the diagonal blocks of the model's weight matrices. Figure [ illustrates the difference between full fine-tuning, LoRA, A, and DiaBlo. Unlike LoRA, which introduces low-rank adaptation through the product of trainable matrices that often requires careful initialization and customized optimization to ensure stable training, DiaBlo directly updates structured subset of the model's 's original weight parameters. By avoiding the use of matrix products, DiaBlo eliminates the inherent optimization difficulties associated with low-rank decomposition. This leads to greater training stability and more reliable convergence without the need for special tricks or tuning. Unlike existing fine-tuning approaches based on unstructured sparsity. DiaBlo's 's structured diagonal blocks preserve memory and computational efficiency while providing hardware-friende patterns and enhanced representational flexibility. It integrates easily into standard training pipelines and consistently outperforms existing PEFT methods across broad range of tasks, demonstrating that selectively updating diagonal blocks is powerful and practical alternative for parameter-efficient one-tuning.
As shown in Figure DiaBlo clearly outperforms other PEFT baselines in both commonsense and arithmetic reasoning with the full-precision I.LaMA2-BB model, and it maintains this advantage under quantized settings. Notably, DiaBlo demonstrates higher stability in 4-bit and 2-bit arithmetic reasoning tasks, where competing methods show significant performance degradation. These results highlight not only the expressive power of diagonal block updates but also the method's 's robustness across diverse domains. DiaBlo demonstrates strong convergence behavior, efficient resource usage, and broad applicability, making it compelling alternative to existing sarameter-efficient methods.
Our contributions are summarized follows:
Sufficiency of Diagonal Block: We propose DiaBlo, which fine-tunes only the diago- nal blocks of model weight matrices. Through extensive experiments on tasks including commonsense reasoning, arithmetic reasoning, code generation, and safety alignment, we demonstrate that DiaBlo consistently achieves strong downstream performance, often surpassing LoRA and its variants in accuracy.
We provide theoretical justifications for DiaBlo. We show that, under mild low-rank assumptions on activations and gradients that often approximately hold in practice, DiaBlo converges to a stationary solution of full fine tuning (FT), provided the
Theoretical Guarantees:
2

---

Published as a conference paper at ICLR 2026
60A
150
154
120
(a) Finetuning full-precision LLaMA2-7B
(h) Finetuning quantized LLaMAA-7B
Figure 2:
Comparison of DiaBlo with other PEFT methods on finetuning LLaMA2-7B.
diagonal block size is sufficiently large. In the linear least squares setting, we further prove that DiaBlo converges to a global minimizer of the full FT objective and is strictly more expressive than LoRA under the parameter budget.
Simple Optimization: DiaBlo directly updates diagonal blocks using standard training pipelines, without relying on matrix product structures or specialized initialization, resulting in more stable and reliable fine-tuning.
High computation efficiency: DiaBlo retains the low memory footprint and fast training of vanilla LoRA and other PEFT methods, ensuring practical usage for large fine-tuning.
BACKGROUND AND RELATED WORKS
Early PEFT techniques, such as Prompt Tuning (Han et al. 2021 Lester et al. 2021) and Prefix Tuning Li & Liang 2021). introduce Icarnable embeddings or token sequences that steer the model's behavior without modifying the core parameters. These methods are lightweight and casy to integrate, offering low computational overhead. While effecttve in low-resource few-shot scenarios, prompt-based methods often struggle to match the performance of full fine-tuning on more complex tasks due to their limited expressiveness and adaptability. Adapter-based approaches (Houlsby et al. 2019; Rücklé et al. 2020 Wang et al. 2020) insert small, trainable bottleneck layers into the transformer architecture. These modules enable task-specific learning while keeping most of the model's 's parameters frozen. Variants of adapter tuning further improve performance through mechanisms such as attention fusion or parameter sharing. Although adapters may reach a good performance, they often require changes to model architecture and introduce additional inference-time latency. In addition, there exist decomposition- based approaches (Lingam al., 2024 Liu al..
2025
Wang et al. 2024g] Yang et al. 2024b 2023
to reduce training or finetuning costs.
Low-Rank Adaptation (LoRA) (Hu et al. 2021) has emerged as one of the most popular PEFT methods. LoRA injects low-rank matrices into the update paths of existing model weights, signif- icantly reducing the number of trainable parameters. Its effectiveness and simplicity have led to wide adoption. but its core reliance the product of two low-rank matrices introduces optimization challenges. This structure can restrict representational flexibility and make training sensitive to initialization and optimization algorithms.
Several recent variants have been proposed to enhance the performance LoRA. DoRA (Lu et al. 2024) introduces novel optimization strategy that decouples the magnitude and direction of low-rank updates, leading to more stable training dynamics and better performance. Pissa (Meng et al. 2024) and MiLoRA /Wang et al. 2024a) focus different initialization techniques for low-rank adapters. Pissa initializes the low-rank matrices using the largest singular values and singular vectors of weights, thereby preserving the most significant components of the weight matrix for faster convergence. On the other hand, MiLoRA takes different approach by utilizing the smallest singular values during initialization. This strategy captures the finer details of the weight matrix. Additionally, LoRA-GA (Wag et al.) (004c)) proposes to initialize low-rank matrices to align with gradients of full fine-tuning at the first step. While these LoRA-based methods have achieved notable improvements, they often introduce additional complexity in design and implementation. Furthermore, the dependence mntrix-product based updates continues to present challenges for stable and efficient optimization. There exist many other variants of LoRA. We refer readers to (Kalajdzievski 2023 Ponkshe et al.) 2024] Valipour et al. 2022 Wang et al. (0244)/Zhng et al.
et al. al.] 2024). QLoRA
(Dettmers et al.] 2023) reduces the memory footprint of fine-tuning by applying low-rank adaptation
3

---

Published as a conference paper at ICLR 2026
quantized models. Building this idea, subsequent works (GPT 2023 Deng et al. 2025
Li et al.
2023| Liao et al. 2024) design customized quantization algorithms and develop effective initialization strategies for the low-rank adaptation components, aiming to improve fine-tuning performance.
METHODOLOGY
3.1 DIABLO: FINETUNING DIAGONAL BLOCKS
Motivation Despite the success of LoRA-based fine-tuning methods, low-rank factorizations remain difficult optimize in practice. In computational mathematics, problems such as low-rank approximation and low-rank matrix completion
well known be non-convex and challenging (Ma et al. 2019 Tong et al. 2020, Tu et al. 2015). These difficulties are reflected LoRA-based approaches, where the product of two trainable low-rank matrices lead to unstable gradient flow and affect convergence. As a result, much recent work has focused on designing better initialization schemes and custom optimization strategies to stabilize training. The development of DiaBlo is motivated by these limitations. To this end, DiaBlo directly fine-tunes only the diagonal blocks of model weight matrices, avoiding matrix-produm parameterizations entirely. This design simplifies the optimization landscape, improves training stability, and enables effective adaptation without introducing additional complexity.
Algorithm The state-of-ther seff-heeart art LLMs are constructed based on linear layers. Hence, it is sufficient to present our proposed approach on a simple linear layer. Consider a linear layer with input features my and output features m2,
Y YXW.
where Y E R ggxmm,, X E Rbxms W E 1 and b is the batch size. For given integer N that a factor of both my, m2, rewrite the weight W in the form of block matrices
WI
11
W 12
WIN INN
W
W22
2N
W
:
WN1
W. N2
WNN
m2 N During finetuning, only the diagonal blocks W, i.e., W W.,...., W NN, are trainable while all other blocks {W Wiyy frozen. DiaBlo also works when N is not common factor of m1, m2. In this case. set dj we pad X with zeros so that its dimension becomes x NN1, perform the blockwise computation, and then truncate the resulting output back to the size b m2- This situation is uncommon in practice because modern LLM architectures purposely choose hidden sizes and intermediate dimensions that highly composite, often powers of two products of small primes with powers of two, ensuring many convenient common factors. For instance, the Llama3-8B model has a hidden dimension of 4096 12 and an FFN intermediate dimension of 14336
where Wi E Rdi xd2 20 ,2
N
de
At inference time,
2 This allows us to choose N = 2 for k = I..... 11. In practice, N = 32, 64, 128 are commonly used, as these choices are always common factors and align well with the trainable- paraaeee budgets of other PEFT methods.
Efficient Implementation For convenience of implementation, like LoRA and its variants. We introduce a block diagonal adaptation matrix D to the linear layer:
similar adaptation method
= XW XWo XD.
(1)
The adaptation matrix D is block diagonal
D
0
0
0
D2
0
D
DN
where D; E dd xd2, During fine-tuning, only D,'s are trainable. In real implementations, the
adaptation matrix D can be saved tensor D R Nxd, xd2 where D D;. During forward-
4

---

Published as a conference paper at ICLR 2026
and back- propagation, it is not necessary to reconstruct the matrix D well. The matrix product XD NNDN). where X; E Raxd1, Therefore, XD is mathematically equivalent to batched matrix multiplications, which can be computed efficiently GPUs. In Py Torch, the operation can be easily implemented using torch. einsum. After reshaping X into tensor X of size x N x d1, XD can be computed as
can be written as (X
torch. e insumbbdd, Nd11d2 + bNd2, X, D).
Let gy be the gradient of loss to the output Y, then the gradient to D; is
gD. X gy.
(2)
where gY, ( +1,-))1dd+..d +1.id. Hence, the backpropagation of DiaBlo can be efficiently imple-
mented using batched matrix multiplications as well.
Initialization of DiaBlo LoRA and its variants typically represent adaptation a low-rank product AB. Standard initialization schemes set using Kaiming uniform and B as zero, ensuring AB while the gradients A, B do not vanish. However, such initialization often leads suboptimal performance, as observed recent studies (Meng et al.) 2024 Wang et al. 2024a|c), Consequently, variants like Pissa (Meng al., 2024), MiLoRA (aan al. 2024a), and LoRA-GA (Wang et al. 2024c) propose alternative initialization techniques to improve optimization behavior-e.g. Pissa uses large singular values, while MiLoRA leverages small singular values for initialization. In contrast, DiaBlo does not involve matrix products and can be initialized more straightforwardly. We simply initialize ID an all-zero tensor. This avoids issues related to vanishing gradients or entangled parameter updates, enabling stable and efficient training without the need for specialized initialization and optimization techniques.
3.2 THEORETICAL ANALYSIS
Comparison with LoRA
The gradient with respect to the full weight matrix W is given by:
X gY
X gyn
gw X gy
(gy,
gyx)
X NNYY
XN
This implies that the gradient of the DiaBlo adaptation block D, is exactly the gradient of the corresponding diagonal block W.. i.c., ED. = gw.. Therefore, optimizing DiaBlo's diagonal block adaptation is mathematically equivalent fine-tuning the diagonal blocks of W, which closely mirrors the behavior of full-model fine-tuning in those subspaces.
In contrast, for LoRA, where the adaptation takes the form Y XW + XAB, the gradients with respect to the low-rank matrices are computed as:
gA BT, EB A gg.
These gradients depend on the values of both A and B, and can suffer from vanishing or unstable updates. As result, LoRA's performance is sensitive to initialization strategies and often requires carefully designed optimization techniques, In contrast, DiaBlo directly applies gradients to the full-rank diagonal blocks, making it inherently more stable and effective for fine-tuning. The stability of DiaBlo is demonstrated by gradient norm comparisons with LoRA in Appendix [A.3.4] where DiaBlo consistently exhibits lower variance, except for the vanishing A A-matrix gradients of LoRA in the early stage of fine-tuning.
DiaBlo Converges to A Solution of Full FT In the linear least squares problem (LSQ) setting, DiaBlo is strictly more expressive than LoRA when the input matrix X has low rank. Consider the fine-tuning objective
WER'n Xia 2 I - XW, -
min
X
33)
where X b Y m2xxa and Wo E my Xm2 For DiaBlo, we instead optimize over
block-diagonal updates:
min D=ding(D, Dx)
Y XW, - X
(4)
5

---

Published as a conference paper at ICLR 2026
where D; E Rdixda d1 and d2 N. The following theorem shows that, under mild assumptions, any minimizer of the DiaBlo-LSQ (4) is also a minimizer of the full-LSQ (3).
Theorem 1. Suppose that X is generic rank-r matrix. If the number of diagonal blocks N < MA1 a common factor of mi m2, then any solution to the DiaBlo-LSQ 4 also solves the full-LSQ
The proof of Theorem D is provided in Appendix A.I.I full FT performance with as few as Nd1d2-
According to the theorem, DiaBlo matches MyT trainable parameters. In contrast, LoRA must use a rank of at least r to solve the full-LSQ, necessitating at least (m1 + m2)r parameters. Consequently, DiaBlo remains strictly more expressive than LoRA under the same parameter constraints.
VISTILC N.
Remarkably, DiaBlo's effectiveness not restricted to the linear setting. general neural network fine-tuning, updating only diagonal blocks is often sufficient to match full FT performance. Our theoretical result substantiates this: DiaBlo converges to stationary point of the full FT objective whenever the activation matrix X and output gradient gY exhibit low-rank-a property empirically
observed in recent literature (Chee et al. 2023 Zhang et al. 2024 Zhao et al. 2024). Formally:
Theorem 2. Suppose the activation matrix X E Raxmm and the linear output gradient gy E bRbxm2
are generic with ranks 1ı and r2. respectively. If N is a common factor of m, m2 and satisfies
N 2 [ IraN 1. then any adaptation D diag ID..... DN) E IR Xma produced by DiaBlo that satisfies the stationarity conditions gD; = Od, xd2 for all 1 S i S N yields adapted weights W Wo + D that is also a stationary point of the full FT objective, i.e. gw 0m xm22
The proof is provided in Appendix A.1.2 Ignoring the ceiling operators in the assumption N 2 [r.. 12N gives an approximate condition N < Mi: m2,
Fi"2 Consequently, lower ranks for X and gy permit a larger numbers of blocks N in Diablo. This reduces trainable parameters while maintaining the guarantee that the finetuned model is stationary point of the full FT objective. An empirical validation of both the low-rank assumptions and the convergence of the full-weight gradient is presented in Appendix A.2
EXPERIMENTS
In this section, we evaluate the performance of DiaBlo variety of benchmark datasets. Specifi- cally, we assess its effectiveness on commonsense reasoning tasks, arithmetic reasoning tasks, code generation tasks, and safety alignment tasks. Our experiments are conducted using LLaMA2-7B
(Touvron et al. 2023b). LLaMA3-8B (Dubey et al., 2024), and Mistral-7B-vO. (Jiang et al. 2023) models. All fine-tuning and evaluation are performed on a single NVIDIA A100 GPU with 80GB of memory. Additional experiments are shown in Appendix A.3 and the implementation details are provided in Appendix A.4
4.1 COMMONSENSE REASONING
Models and Datasets Commonsense reasoning 170k includes eight benchmark datasets: BoolQ (Clark et al. 2019), PIQA BBsk et al. 2020). SocialIQA (Sap et al. 2019), HellaSwag (Zellers) et al. 2019). WinoGrande (Sakaguchi et al. 2021), ARC-Easy, ARC-Challenge (Clark et al. 2018) and OpenBookQA MMhayyov et al.) 2018). Each task is framed as a multiple-choice problem. We fine-tune the Llama2-7B. Llama3-8B 8B, and Llama- 13B models on the combined training data from all eight tasks, and report accuracy on the test set of each task individually.
Results The results on Commonsense reasoning tasks are shown Table We compare DiaBlo with existing methods, including DoRA (Liu et al. 2024), Pissa (Meng et al. 2024). MiLoRA (Wang et al. 2024a). and SMT (He et al. 2024). DiaBlo consistently outperforms existing PEFT basclines on commonsense reasoning tasks using Llama2-7B, Llama3-8B. and Llama- 13B models in FP1c precision. For Llama2-7B, DiaBlo achieves the highest average score of 83.5% while using only 0.52% trainable parameters. For Llamma-8B., DiaBlo with N 64 achieves the best overall average score of 87.3%. For the larger Llama-133 model, DiaBlo again demonstrates superior performance with score of 84.9% significantly outperforming strong basclines. The findings demonstrate DiaBlo is an efficient, scalable, and effective fine-tuning strategy.
Comparison with SMT SMT (He et al., 2024) proposes to select most important sub-matrices in pretrained weights based magnitudes of gradients and then only fine-tune these sub-matrices.

---

Published as a conference paper at ICLR 2026
Table Commonsense reasoning results of fine-tuning Llama2-7B, Llama3-8B, and Llama-13B in
FP 1 6. Results marked with T, are taken from Dora (Liu et al. 2024), MiLoRA (Wang et al.
2024a), and SMT (He al. 2024), respectively.
Llama2-7B
PEFT
r/N
#Purams
BoolQ
PIQA
SIQA
HellaS
WinoG
ARC-e
ARC-c
OBQA
AVG
88.6
85.2
Zero-shot
NAA
0%
61.9
47.4
2.2
12.3
0.0
12.5
10.0
23.4
21.2
LoRA
=32
0.83%
69.8
79.9
79.5
83.6
82.6
79.8
64.7
81.0
77.6
DORA
=32
0.83%
71,8
83.7
76.0
89.1
82.6
83.7
68.2
82.4
79.7
DoRA
=
0.42%
72.0
83.1
79.9
89.1
83.0
84.5
71.0
81.2
80.5
Pissa
32
0.83%
67.6
78.1
78.4
76.6
78.0
75.8
60.2
75.6
73.8
MiLoRAF
67.6
SMT"
N/A
0.84%
72.0
83.8
80.8
93.3
82.8
86.7
74.0
81.0
81.8
SMT(Beat)*
N/A
4.91%
72.6
85.2
82.0
94.4
85.7
87.8
74.5
85.0
83.4
DiaBlo
N =64
1.04%
73.9
84.8
81.7
90.0
85.0
87.9
76.8
86.8
83.4
DiaBlo
LJama3-88
PEFT
r/N
#Params
BoolQ
PIQA
SIQA
Hella8
WinoG
ARC-e
ARC-c
OBQA
AVG
Full FT
N/A
100%
76.4
89.7
82.5
95.5
89.6
92.9
84.3
89.2
87.5
Zero-shot
N/A
0%
56.3
66.3
32.2
25.2
26.6
24.1
21.5
24.8
34.6
LoRA
=32
0.78%
70.8
85.2
79.9
91.7
84.3
84.2
71.2
79.0
80.8
DoRAT
=32
0.78%
74.6
89.3
79.9
95.5
85.6
90.5
80.4
85.8
85.2
16
74.5
$8.8
84.7
79.
Pissa
74.6
MiL.RAF
32
0.78%
68.8
86.7
77.2
92.9
85.6
86.8
75.5
81.8
81.9
SMT*
N/A
0.71%
75.7
88.4
81.4
96.2
88.2
92.7
83.2
88.6
86.8
SMT(hest))
82.4
DiaBlo
=64
1.04%
75.8
90.8
80.7
95.6
89.9
93.4
83.0
89.2
87.3
DiaBlo
N 128
0.52%
76.1
81.5
95.7
89.7
93.4
83.4
87.2
87.2
Llama-
PEFT
r/N
#Params
BoolQ
PIQA
SIQA
HellaS
WinoG
ARC-e
ARC-c
OBQA
AVG
Full FT
N/A
100%
74.8
86.2
80.5
90.5
87.1
88.6
75.4
88.0
83.9
Zero-shot
59.5
16.0
22.9
0.3
15.8
24.0
LoRA
0.67%
72.1
83.5
80.5
90.5
83.7
82.8
68.3
82.4
80.5
DoRA
== 32
0.68%
72.5
85.3
79.9
90.1
82.9
82.7
69.7
83.6
80.8
DiaBlo
N 6-1
1.06%
74.7
86.9
81.2
91.5
87.4
89.8
78.2
90.2
84.9
DiaBlo
N 128
0.53%
75.7
84.8
82.0
90.6
86.8
90.1
77.1
89.2
84.5
In contrast, DiaBlo directly selects diagonal blocks from the pretrained weights, offering simpler and more implementaaionnfrienddy approach. The experiments further demonstrate that fine-tuning diagonal blocks provide even better results. As shown in Table T DiaBlo achieves average score of 83.5% on Llama2-7B, significantly outperforming SMT's 81.8% while using a less number of trainable parameters. Even when SMT increases its trainable parameter 4.91% to achieve its best result, DiaBlo, with only 0.52% trainable parameters, still attains better performance. A similar trend holds for Llama3-8B, where DiaBlo outperforms SMT even when SMT uses considerably more trainable parameters. These results demonstrate that fine-tuning only the diagonal blocks of pretrained weights is not only highly efficient but also sufficient to achieve strong performance.
4.2 ARITHMETIC REASONING
Models and Datasets We evaluate DiaBlo arithmetic reasoning using the MetaMathQA Yu al. 2023) dataset, which consists of 395K samples augmented from the training sets of GSM8K (Cobbe et al. 2021 and MATH (Hendrycks et al. 2021). We fine-tune the LLaMA2-7B model on this dataset and evaluate performance on the test sets GSM8K and MATH. The training follows the setting in MiLoRA and report results from the final checkpoint and use the Exact Match (EM) ratio to measure accuracy.
As shown in Table 2 on arithmetic reasoning tasks, DiaBlo demonstrates strong perfor- mance, achieving results that match exceed full fine-tuning while maintaining parameter efficiency. With N 32 blocks and only 2.09% trainable parameters, DiaBlo attains an average accuracy of 43.4%, slightly outperforming full fine-tuning with 43.2% and significantly surpassing LoRA,
Results
7

---

Published as a conference paper at ICLR 2026
Table Performance of fine-tuning Llama2-7B on arithmetic reasoning tasks. Results of baseline methods are taken from MiLoRA (Wang et al. 2024a).
Method
r/N
GSM8K
MATH
AVG
Full
N/A
100%
66.3
19.8
43.2
Zero-shot
N/A
0%
2.2
0.0
1.1
LoRA
7= 61
167%%
60.6
16.9
38.7
PISSA
64
MiLoRA
64
1.67%
63.5
17.8
40.7
DiaBlo N
2.09%
66.3
20.4
43.4
DiaBlo
N 64
1.04%
66.5
17.6
42.1
Table 3: Performance on code generation and safety alignment tasks. Results of baseline methods are taken from LoRI (Zhang et al. 2025a).
HumanEyal P
Model
Method
r/N
#Params
HEx-PHI
Pass@1
Pass
LoRA
32
1.12%
34.7
46.4
50.8
91.6
Llama3-8B
DoRA
r ==
1.12%
33.1
44.0
48.6
93.6
LoRI
0.56%
43.2
57.6
63.2
92.8
1.51%
DiaBlo
N 128
0.76%
39.4
55.0
61.9
96.3
Zero-shot
N/A
0%
28.6
38.2
41.9
80.5
LoRA
32
1.25%
33.8
42.4
45.3
91.9
Mistral-7B
DoRA
1.25%
33.7
42.6
46.8
95.3
LoRI
r= 32
0.63%
33.8
42.0
45.1
94.7
DiaBlo
N 64
I 1.68%
34.4
44.8
48.7
98.8
DiaBlo
128
0.84%
34.0
45.7
98.8
PiSSA, and MiLoRA by 4.7 %, 6.4%, and 2.7% 7,,, respectively. Even with N = 64 blocks and 1.04% trainable parameters, DiaBlo with fewer trainable parameters maintains high performance with an average accuracy of 42. 1,, demonstrating that it remains effective under restrictive parameter budgets. Notably, DiaBlo achieves the highest score of 20.4% on the MATH dataset among all methods, including full fine-tuning.
4.3 CODE GENERATION AND SAFETY ALIGNMENT
Models and Datasets We further evaluate the performance of DiaBlo code generation and safety alignment tasks, following the training setting in LoRI (Zhang et al., 2025a).. For code generation, we fine-tune on the CodeAlpaca dataset (Chaudhary 2023 and evaluate using the HumanEval benchmark
(Chen et al. 2021 with standard metrics: Pass @ Pass @5, and Pass @ 10. For safety alignment, fine-tune tune the SaferPaca dataset (Bianchi al. 2023), , an extension of Alpaca- Cleanee (Taori
et al. 2023), and assess performance based on the refusal rate to harmful prompts from the HEX -HI
dataset (Qi al. 2023). We conduct experiments both the LLaMA3-8B and Mistral-7B models.
Results The experimental results are summarized in Table 3 DiaBlo achieves strong performance in the code generation task HumanEval on Llama3-8B. With 1.51% % trainable parameters, DiaBlo reaches Pass @1 I score of 43.2% and Pass @ 10 score of 63.5%, outperforming all other methods. Its Pass @ score is also very cloes to the best result obtained by LORL. With a reduced parameter budget of 0.76%, DiaBlo still maintains high performance, achieving Pass 10 score of 61.9% On Mistral- 7B, DiaBlo with N = 128 achieves the best Pass@ 5 and Pass @ 10 scores and the second-best Pass @ score, while DiaBlo with N 64 achieves the best Pass @ score and the second-best Pass @5 and Pass 10 scores, outperforming LoRA, DoRA, and LoRI. In terms of HEX-PHI, DiaBlo outperforms all other methods with scores of 97. 6..%% on Llama3-88 and 98 8% on Mistral-7B. These findings highlight DiaBlo's 's effectiveness in both code generation and safety alignment, while maintaining high level of parameter efficiency.
4.4 FINE-TUNING WITH QUANTIZED MODELS
To further reduce the memory footprint fine-tuning, QLoRA (Guo et al. 2024) and its variants store the frozen pretrained weights in low-precision while training small low-rank adaptation modules in
8

---

Published as a conference paper at ICLR 2026
Table 4: Performance of fine-tuning quantized Llama2-7B and Llama2-13B models on arithmetic reasoning tasks. Results of baseline methods are taken from ApiQ (Liao et al. 2024
Llama2-7B
Method
r/N
#Params
GSM8K
SVAMP
MAWPS
AQuA
AVG
QLoRA
64
112M
42.7
58.7
87.3
26.4
53.7
GPTQ-LoRA
r=
112M
43.0
52.9
4-bit
LonQ
64
112M
41.7
56.0
86.3
25.3
52.3
ApiQ bw
r= 64
112M
43.2
59.0
85.7
26.0
53.5
MagR Diablo
N 64
70M
44.1
59.0
89.5
26.4
54.8
QLaRA
64
112M
0.9
1.5
2.1
GPTO-LaRA
mm 64
112M
21.7
39.0
76.6
22.
39.9
2-bit
MagR-LoRA
112M
30.9
86.6
20.5
46.2
LoftQ
r= 61
112M
29.5
43.8
83.6
23.2
45.6
Apii-bw
r=
GPTQ-DiaBlo
70M
33.3
50.9
84.0
22.8
47.8
MagR-DiaBlo
64
70M
32.1
51.5
87.0
24.0
48.7
Hlmma--13B
QLeRA
59.5
4-bit
GPTO-LORA
64
234M
53.2
67.5
85,3
25.6
57.9
LonQ
r= 64
239M
54.9
66.5
87.7
23.9
58.3
ApiQ-bw
64
239M
55.3
67.4
87.8
25.6
59.0
MayR DiaBlo
59.6
QLoRA
r= 64
234M
0.5
0.7
0.9
0.6
2-bit
GPTQ-LoRA
31.9
49.6
23.6
46.9
MagR-LoRA
T= 64
239M
38.6
60.5
81.1
24.8
51.2
LonQ
37.0
ApiQ-bw
64
239M
43.1
59.2
85.1
23.4
52.7
GPTQ DiaBlo
N
64
189M
41.7
58.0
87.4
23.2
52.6
MagR DiaBlo
64
189M
44.5
59.9
88.2
27.6
55.1
higher precision. In this section, we explore the compatibility the proposed DiaBlo with quantized models and evaluate its performance.
Models and Datasets We evaluate DiaBlo on quantized models. We fine-tune the quantized
LLaMA2-7B and ILLMM22-3BB models, which are quantized by GPTQ (Frantar et al. 2022) and
MagR (Zhang al., 2024). All models are fine-tuned on Math10K (Hu (023) and then
evaluated on the test sets of AQuA (Ling et al., 2017). GSMSK (Cobbe et al. 2021) MAWPS (Koncel-Kedziosst al. 2016), and SVAMP (Patel et 2021).
Results We evaluate DiaBlo on 4-bit and 2-bit quantized Llama2-7B and Llama2-132 models. The results are presented in Table For the Llama2-7B model, MagR-Diea RiaaBo achieves the highest average accuracy of 54.8% in the 4-bit setting. In the more challenging 2-bit setting, again outperforms existing baselines with average accuracy of 48.7%. The results the larger Llama2-133 model further demonstrate DiaBlo's scalability and robustness. Its advantage particularly stark in the of 55.1%, substantially outperforming
ultra-low 2-bit setting, where it achieves average baselines. The LoftQ and ApiQ methods jointly determine quantized weights and LoRA initialization enhance fine-tuning performance. contrast, DiaBlo directly fine-tunes quantized models without any customized quantization procedure or special initialization, highlighting its practical simplicity while still achieving superior or comparable results. To isolate the impact of quantization methods, we also evaluate GPTQ-DiaBlo and MagR-LoRA under the 2-bit setting. Under both GPTQ quantization and MagR quantization, simply replacing LoRA with DiaBlo leads to substantial performance improvements. This indicates that the observed gains come primarily from the fine-tuning method DiaBlo rather than from changes the quantization method.
4.5 MEMORY AND COMPUTATION EFFICIENCY OF DIABLO
The simple structure of DiaBlo allows the use of efficient batched operations, thus reducing training time and memory usage. In this subsection, we evaluate the memory consumption and training speed of DiaBlo and compare them with other PEFT training methods.
Consider linear layer = XW, where W E mm X E gm Let be the number of blocks in DiaBlo. Each diagonal block is of size d d, where d = 1n and the total number of trainable

---

Published as a conference paper at ICLR 2026
Table Comparison of DiaBlo with random sparse patterns on GSM8K. Left part reports accuracy from retaining sparse updates, and right part reports accuracy and training time from fine-tuning
Relained Sparse Update Accuracy (%)
Fine-tuning (Sparsity 1/64)
Method
Sparsity 1/32
Sparsity 1/64
Sparsity 1/128
Accuracy (%)
Time (min)
Diatãlo
67,30
65.47
63.11
67.68
17.26
Random Entries
65.98
64.05
62.19
65.35
26.51
Random Block
65. 96
63:75
62.
64.86
29.76
Random Column
65.73
62,93
61.66
65.19
17.01
Random Row
49.14
24.08
2,85
61.71
17.76
parameters NdP. The forward pass using the block matrix structure requires operations bNd?. In comparison, LoRA with rank r has 2mr trainable parameters and requires 2bmrr operations during the forward pass, When DiaBlo and LoRA are configured to have the same number of trainable parameters, that is, N = 2mr, it is straightforward to show that the computational cost of the forward pass is also the same. A similar argument applies to the backward pass. Therefore, DiaBlo has the same theoretical computational complexity and memory footprint LoRA when they share the same number of trainable parameters. This indicates that DiaBlo is as efficient as vanilla LoRA in terms of resource usage and computation complexity.
To further validate efficiency, we benchmark the fine-tuning speed of DiaBlo on the 170k common- sense reasoning dataset, comparing it against LoRA and DoRA. All experiments are conducted on a single A 100 GPU with 80GB memory, using batch size of 8 and a gradient accumulation step of 4. DiaBlo with N 64 and N 128 matches the training speed of LoRA with rank r 32, requiring 170 minutes per epoch, whereas DoRA is substantially slower at 480 minutes per epoch. These results confirm the practical efficiency of the proposed DiaBlo method.
4.6 SUFFICIENCY OF DIAGONAL BLOCKS
We investigate whether diagonal blocks provide a more effective subset for fine-tuning compared to alternative sparsity patterns. Following the methodology of Deng et al. (2024), we first analyze the performance retained from a fully one-tuned model. Specifically, we fully fine-tune LLaMA3-8B
on GSM8K and compute the update matrix A- W finetuned W pretrained We then measure accuracy after reapplying only selected parts of A, scaled by a factor 8, to the pre-trained weights. We compare DiaBlo's diagonal block selection with random entries, random blocks of the same size as DiaBlo, random columns, and random rows, As shown in Table 5 retaining diagonal blocks preserves accuracy substantially better than all other sparsity patterns, and degrades more gracefully as sparsity increases. We further evaluate these sparse patterns in direct fine-tuning Table shows that DiaBlo not only achieves the highest accuracy but also runs more efficiently than unstructured random methods, while avoiding the severe accuracy loss of row- or block-based updates. Together, these results demonstrate that diagonal blocks form a principled and hardware-friee friinddy structure for sparse fine-tuning, sufficient to capture task-relevant updates.
5 CONCLUSION
In this work, we introduced DiaBlo, simple yet effective parameter-efffcient fine-tuning (PEFT) framework that directly updates diagonal blocks of model weight matrices. Unlike low-rank ap- proaches such LoRA and its many variants, DiaBlo avoids matrix-product product-based adaptations, eliminating the need for special initialization or customized optimization techniques. Our theoretical guarantees show that, under mild low-rank conditions, DiaBlo converges stationary point of the general full fine-tuning and is more expressive than LoRA in the linear least squares problem. Through extensive experiments across a diverse set of tasks, we demonstrated that DiaBlo consistently achieves strong or state-of th-the- performance while maintaining a comparable or smaller number of trainable parameters. Moreover, DiaBlo generalizes well to low-precision settings: in both 4-bit and 2-bit quantized models, it outperforms existing PEFT methods for quantized models without relying on carefully designed quantization or initialization strategies. This robustness and simplicity make DiaBlo appealing choice for scalable and reliable fine-tuning of large language models. In conclusion, DiaBlo offers a practical and effective approach for efficient model fine-tuning. As Al models continue to scale, the demand for robust and straightforward fine-tuning techniques be- comes increasingly important. DiaBlo addresses this demand by achieving a strong balance between performance, efficiency, and implementation simplicity.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY
All conducted experiments are reproducible. All models and datasets used are publicly available, as stated in Section 4 and the finetuning hyperparametters are detailed in Appendix A.4 The codes available at https://github. com ctt.coo//iiiiii. cotp:///iyaagggyyo//a......
ACKNOWLEDGMENT
This work was in part supported by NSF grants DMS-2205 AMS-2208126, DMS-2110836, IIS-2110546, SUNY- IBM AI Research Alliance grant, UAlbany-IBM CEAIS seed grant, and start-up grant from UAlbany.
REFERENCES
Gptqlora: Efficient finetuning of quantized Ilms with gptq. GitHub repository, 2023, URL Available at https:/ /tops://gii /ttt:://gtttt//// thub
https://github. wttp:/hhw.ooo.ocooooee.eeeeeeeee
wwtpp//ppw/////ttooooooooo Com/ wwwpp//wp22//ggg.....000
OpenAl Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, and et al. Gpt-4 technical report. 2023. URL https://app www.seeeineneiieeeeee
257532815
Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Röttger, Dan Jurafsky, Tatsunori
Hashimoto, and James Zou. Safety-tuned Ilamas: Lessons from improving the safety of
large language models that follow instructions. ArXiv, abs/2309.07875, 2023. URL https:
//api. semant icacholar. wttpa//a.........a..oooooooaaaaa
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical
commonsense in natural language. In Proceedings of the AAAI conference artificial intelligence,
volume 34, pp. 7432-7439, 2020.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Ma teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020. htR.:tttts:////.. api semant www.eaanticcchho.ooo wwchcccchccccc.cccccccccccccc www..ooooooooo.oooo.......
Sahil Chaudhary, Code alpaca: An instractionn-folloiing llama model for code generation. https
2023.
Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher M De Sa. Quip: 2-bit quantization of large language models with guarantees. In Advances in Neural Information Processing Systems, 2023.
Mark Chen, Jerry Tworek, Hcewoo Jun, Qiming Yuan, Henrique Pondé, Jared Kaplan, and et al. Evaluating large language models trained on code. ArXiv, abs/2107 003374, 2021. URL https //api semant www.holacccccccaccccccccm wwtp:aar.rrrooooooooooooo
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski. Michael Collins, and Kristina Toutanova, Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint /rXiv:/905.10044, 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. 2018.
arXiv preprint
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reñchiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14168, 2021.
11

---

Published as a conference paper at ICLR 2026
David Cox, John Little, Donal O'shea, and Moss Sweedler. Ideals, varieties, and algorithms. Springer, 1997.
Wenlong Deng, Yize Zhao, Vala Vakilian, Minghui Chen, Xiaoxiao Li, and Christos Thrampoulidis. Dare the extreme: Revisiting delta-parameter pruning for fine-tuned models. arXiv preprini arXiv:2410.09344, 2024.
Yanxia Deng, Aozhong Zhang, Naigang Wang, Selcuk Gurses, Yang, and Penghang Yin. Cloq: Enhancing fine-tuning of quantized Ilms via calibrated lora initialization. arXiv preprint (Xii:25011.187555 2025.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305. 2023.
Ning Ding, Yujia Qin, Guang Yang, Fu Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, Jing Yi, Weilin Zhao, Xiaozhi Wang. Zhiyuan Liu, Haitao Zheng, Jianfei Chen, Y. Liu, Jie Tang, Juanzi Li, and Maosong Sun. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence. 5:220-235, 2023. URL
https://api. sema nt. cscho ar www.oor//ooooooooooo rpus ID: 25731 6425
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, and et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024. URL htt.:http:///tt //atps::/ap..... http::///app.sma.iicc...... www.nticscclllar.ooomo
CorpusID:27157101
Elias Frantar, Salch Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210. 17323, 2022.
Han Guo, Philip Greengard, Eric Xing, and Yoon Kim. LQ-IoRA: Low-rank plus quantized matrix decomposition for efficient language model finetuming. In The Tweifth International Confer- ence on Learning Representations, 2024. URL https: htttps::/operrevieee http///pppeeeeieeeeeeeeeeeee net/ forum? di
xw29Vv0MmL]
Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, and Maosong Sun. Ptr: Prompt tuning with rules for text classification. ArXiv, abs/2105. 1259, 2021. URL http:::/tp:/ppp semant icschol ar
oxc//oorpusD::2511655
Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-efficient fine- tuning for large models: A comprehensive survey, arXiv preprint arXiv:2403. 14608, 2024. URL
semant
Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large models.
arXiv preprint arXiv:2402.122: 12354, 2024. URL ptttps://ap..semaaiicooo
org/
CorpusID:2677500
Haoze He, Juncheng B. Li, Xuan Jiang, and Heather Miller. Sparse matrix large language model fine-tuning. ArXiv, abs/2405.15525, 2024. URL httpt:tt////at www.ntmmscmomlllat....eom org/
CorpusID: 27004
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Xi- aodong Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint 103::::033308744 2021. URL BRR.h:tps://agg htt.:pppp//ppp..saattcccccooooo swwanticscoooo.o....
Dtse/CoopuuSD::2213333
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Pfffmeter-efffcient transfer learning for nlp. ArXiv, 1bs/1902.00751. 2019. URL httpt:ttt/// /aapi semant wcccool ar. Orgg/CrrpuSDD
59599816
J. Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation large language models. arXiv preprint arXiv:2106 09685, 2021. URL httth:t////aat semant cachol a otg.ehoooaa.orooooooooos/2255530
12

---

Published as a conference paper at ICLR 2026
Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Lee. LLM -aM-aaaeerrs: An adapter family for parameter- language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5254-5276, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 1/633////223.mmmlllp/ 319. URL
fine-ture ane-tuning of large
emn main. 319/
Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L'elio Renard Lavaud, Marie-Ann Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. ArXiv, 2023. URL (tRpsttss:// http://///.pp.seaanttiiiiiooo.
Damjan Kalajdzievski. A rank stabilization scaling factor for fine-tuning with lora. ArXiv, abs/2312.037332, 2023.
URL https://api. htt.://pppesemmiiiiiim..mm www.emmaaiiss..... cssho ar. org/ /Co rpus ( DD;
266053836
Samir Khaki, Xiuyu Li, Junxian Guo, Ligeng Zhu, Chenfeng Xu, Konstantinos N Plataniotis, Amir Yazdanbakhsh, Kurt Keutzer, Song Han, and Zhijian Liu. Sparselora: Accelerating Ilm fine-tuning with contextual sparsity. arXiv preprint arXiv:2506. 16500, 2025.
Rik Soncel-Kedziorske Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. Mawps A math word problem repository. In Proceedings of the 2016 conference of the north american chapter of the association for computational linguistics: human language technologies, pp. 1152- 1157, 2016.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Conference on Empirical Methods in Natural Language Processing, 2021. URL https://apt www.:eeseaaatscchoo.oee
Bttg/Corpussiii : 233296808
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joinu Conference Natural Language Processing (Volume 1: Long Papers), pp. 4582-4597, 2021. URLhttps:://at 230433941
Yixiao Li, Yifan Yu, Chen Liang, Pengcheng He, Nikos Karampatziakis, Weizhu Chen, and Tuo Zhao. Loftq: traafine-tuning-wware quantization for large language models. arXiv preprint arXiv:2310.08659, 2023.
Baohao Liao, Christian Herold, Shahram Khadivi, and Christof Monz. Apiq: Finetuning of 2-bit quantized large language model. arXiv preprint arX1v:2402.05147, 2024.
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale genera- tion: Learning to solve and explain algebraic word problems. arXiv preprint 1rXX::1705.0414:: 2017.
Vijay Lingam, Atula Tejaswi, Aditya Vavre, Aneesh Shetty, Gautham Krishna Gudur, Joydeep Ghosh, Alexandros G. Dimakis, Funsol Choi, Aleksandar Bojchevski, and Sujay Sanghavi. Svft: Parameter-efficient fine-tuning with singular vectors. ArXiv, abs/2405. 19597, 2024. URL https sewanteenntttii icscholaa
Shih-Yang Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang. Kwang- Ting Cheng, and Min-Hung Chen. Dora: weight-decomposed low-rank adaptation. In Proceedings of the 41st International Conference on Machine Learning, (CM.'24. JMLR.org, 2024.
Zi Qian Liu, Ruijie Zhang, Zhengyang Wang. Zi Yang, Paul Hovland, Bogdan Nicolae, Franck Cappello, and Zheng Zhang. Cola: Compute-efficient pre-training of lims via low-rank activation. ArXiv, abs/2502. 10940, 2025. URL https: /pii semanticscholar rrg/CorpusID; 277409063
13

---

Published as a conference paper at ICLR 2026
Cong Ma, Yuanxin Li, and Yuejie Chi. Beyond procrustes: Balancing-free free gradient descent for asym- metric low-rank matrix sensing. 2019 53rd Asilomar Conference on Signals. Systems, and Com- puters, pp. 721-725, 2019.
URL https:/ /tpppapppppp//// semanticscholar httpaa/ii..ccoaa...ooooooaaa.pa. WttgCooppss/CD.C.
209467547
Fanxu Meng. Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models. arXiv preprint arXiv:2404.02943 2024. URL
attps::/api.ssmennt www.nt http://ccccc..a...ororrrrrrpsaaa
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? a new dataset for open book question answering, arXiv preprint arXiv: 1009.027997 2018.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nip models really able to solve simple math word problems? arXiv preprint 1rX.1:2103.07191, 2021.
Kaustubh Ponkshe, Raghav Singhal, Eduard Gorbunov, Alexey Tumanov, Samuel Horvath, and Praneeth Vepakomma. Initialization using update approximation a silver bullet for extremely efficient low-rank fine-tuning. arXiv preprini 2rXiv:2411.195577 2024.
Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin- Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! ArXiv, abs/2310.03693, 2023. URI.tttps: ponttps://aptie stepe////t...ssiiiieaeee.aeaaeee www.nti cschoo ar. org/Corpus ID: 263671 523
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140): 2020.
Jesus Rios, Pierre Dognin, Ronny Luss, and Karthikeyan N Ramamurthy, Sparsity may be all you need: Sparse random parameter adaptation. arXiv preprint arXiv:2502.15975 2025.
Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. Adapterdrop: On the efficiency of adapters transformers. In Conference on Empirical Methods in Natural Language Processing, 2020. htRh:ttp::// //api semanticscholar
Etcp/CorpusID:225050
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106,
2021.
Maarten Sap. Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXiv:1904.09728, 2019.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: fostluttonnfollowinng llama model. https://git thub. www.tto.//...oo.. stt su- lab/ htts/sood_aaaaaaaccco 2023.
Tian Tong, Cong Ma, and Yuejic Chi. Accelcrating ill-conditioned low-rank matrix estima- tion via scaled gradient descent. ArXiv, http/2005.08999 2020. URL https://api
Gemant.cscholar.ccmmmo wtw.ehhooo....og/ooooossaaaa:a:a
Hugo Touvron, Thibaut Lavril, Gautier Izacard. Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aur' elien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.139977 2023a. URL/https: //api. semant icschol a orre/GorpussD::257119900
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307. 092888 2023b.
14

---

Published as a conference paper at ICLR 2026
Stephen Tu, Ross Boczar, Max Simchowitz, Mahdi Soltanolkotabi, and Benjamin Recht. Low-rank solutions linear matrix equations via procrustes flow. In International Conference on Machine Learning, 2015. htthttts: secaa.ccccs... http://wwss....aaoooooooroooooaa :ttp/oorpss1:::1557333
Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. Dylora: Parameter- efficient tuning of pre-trained models using dynamic search-free low-rank adaptation ArXiv, abs/2210.00558, 2022. URL
252907428
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP. pp. 353-355,44 2018.
Hanqing Wang, Zeguan Xiao, Yixia Li, Shuo Wang, Guanhua Chen, and Yun Chen. Milora: Harnessing minor singular components for parameter-efficient I1m finetuning. arXiv preprint arXiv:2406.09044 2024a. URL httpps://apt http:////ap.smmaisissiiiieee 27C440848
Corg/or
Haoyu Wang, Tianci Liu, Tuo Zhao, and Jing Gao. Roselora: Row and column-wise sparse low-rank adaptation of pre-trained language model for knowledge editing and fine-tuning, In Conference on Empirical Methods in Natural Language Processing, 2024b. URL https:
7/api. www.ssseeaaac...oceeeeeee www.eoorrgooooooeoeeeee Co:p:ooouu
org/
Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang, and Ming Zhou. K-adapter: Infusing knowledge into pre-trained models with adapters. In Findings, 2020. URL http:: /ppi. www.nt wwtp/orppaaaaaaoaoooooooooo ID: 211031933
Shaowen Wang. Linxi Yu, and Jian Li. Lora-ga: Low-rank adaptation with gradient approximation. arXiv preprint arXiv:2407.05000 2024c. URL httpt://app....... http:///ppp..eeaaatcccc........o org/
CorpusID:2210500
Shaowen Wang. Linxi Yu, and Jian Li. Lora-ga: Low-rank adaptation with gradient approximation.
arXiv preprint arXiv:2407.05000, 2024d.
Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. Svd-Ilm: Truncation- -waree singular value decomposition for large language model compression. ArXiv, abs/2403.07377 2024e. URL
wwwp:o/ooo/ooouooooo sttp::////pp..sseattttmeem http:ppp...aaaaoooooooooooooooor 26836664228
Zhengbo Wang and Jian Liang. Lora-pro: Are low-rank adapters properly optimized? arXiv preprint 17224 2024.
URL https::/app.. htt.:apii..mmtiiicccccoo......o. wtt.:eccch....a.oooorrruuauaaaaa
271432381
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304. 12244, 2023.
Jing Xu and Jingzhao Zhang. Random masking finds winning tickets for parameter efficient fine- tuning, arXiv preprint arXiv:2405.02550 2024.
Xinyu Yang, Jixuan Leng, Geyang Guo, Jiawei Zhao, Ryumei Nakada, Linjun Zhang, Huaxiu Yao, and Beidi Chen. S2ft: Efficient, scalable and generalizable I1m fine-tuning by structured sparsity. Advances in Neural Information Processing Systems, 37:59912-59947, 2024a.
Yifan Yang, Jiajun Zhou, Ngai Wong, and Zheng Zhang. Loretta: Low-rank economic tensor-train adaptation for totta--lo--prramete fine- tuning of large language models. ArXiv, abs/2402. 11417, 2024b. URL httthttts: www.app...mmattt.mmmmmmmmm cscho ar httpooool....orooooosss::233:000
Zi Yang, Samridhi Choudhary, Xinfeng Xie, Cao Gao, Siegfried Kunzmann, and Zheng Zhang. Comera: Computing- and meforr-efficient training via rank-adaptive tensor optimization. ArXiv, abs/2405.14377, 2024c. URL/https: //apii semanticscholar. org/CorpusID
2699833788
15

---

Published as a conference paper at ICLR 2026
Ziao Yang, Samridhi Choudhary, Siegfried Kunzmann, and Zheng Zhang. Quantization- aware and tensor-compressed training of transformers for natural language understanding. ArXiv, abs/2306.01076, 2023. URL htthht:t//eeet http:/pppp..aaiiiiii.iimmemm wwe.mnntiicch.... org/CorpusiD;
259064180)
Long Long Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zheng Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309 arX.2:2309.12284, 12284, 2023. URL https
//api. sema nt nticscholar.ar...ommommmmoooor.e hoc.ccoooo....ooororouussss wwtpooooroosstt22000000
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.078830 2019.
Aozhong Zhang, Naigang Wang, Yanxia Deng. Xin Li, Zi Yang, and Penghang Yin. Magr. Weight magnitude reduction for enhancing post-training quantization. In Advances in neural information processing systems, 2024.
Juzheng Zhang, Jiacheng You, Ashwinee Panda, and Tom Goldstein. Lori: Reducing cross-task interference in multi-task low-rank adaptation. ArXiv, abs/2504.07444, 2025a. URL https
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter-efficient fine- tuning. ArXiv, abs/2303. 10512, 2023. URL http::://aal.smmmt..ccc........
CorpusID:264435233
Yuanhe Zhang, Fanghui Liu, and Yudong Chen. Lora-one: One-step full gradient could suffice for fine-tuning large language models, provably and efficiently. In Forty-second International Conference on Machine Learning. 2025b.
Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient I1m training by gradient low-rank projection. arXiv preprint arXiv:2403.03507, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46395-46623, 2023.
Yibo Zhong, Haoxiang Jiang, Lincan Li, Ryumei Nakada, Tianci Liu, Linjun Zhang, Huaxiu Yao, and Haoyu Wang. Neat: Nonlinear parameter-efficiet adaptation of pre-trained models. ArXiv, abs/2410.01870, 2024. URL https: / / /api. remanticschoon arg/CorpusID) 273098513
16

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 PROOFS
A.1.1 PROOF THEOREM! 1
Theorem 1. Suppose that X is generic rank-r matrix. If the number of diagonal blocks N < a common factor of mi; Myz, then any solution to the DiaBlo-LSQ 4 also solves the full-LSQ
101
Proof. Let d1 = m1 /N, = -m2/N. We write the partitioning X (X) Xx) and Zv), respectively, with X; Rbxdd and Ze E 18xx,,, Then the DiaBlo-LSQ problem is equivalent to solving
== Y XWo = [2
N
min
Z2 - X,D.IP i= N,
each which is a standard linear least squares problem, and hence has the minimizer
D = = i ,,
(5)
where denotes the pseudoinverse. Collectively, D' diag Dj -D)) solution to the original DiaBlo-LSQ problem 4
Next, we show that D is also minimizer of the full-LSQ problem
To this end, we rewrite D
in the block matrix form
Di
Din
D
Divi
D
where D == D., and D it holds that
Odd xd2 for + j. Then for full-LSQ, since gw -2X (Z XW).
Dig
gD; = -2X 2, -
-x2 (2, x,,ū;, =0x))xxxxxxDD X,;; 2X, Zj.
Dij
Since the full-LSQ problem (3) is convex, it suffices show that gD; Od, xd2 for all i.j-
Suppose there holds the low-rank decomposition X = RP, where R E Rxr has orthonormal columns and E Rrxmm, Alternatively, we assume the partitioning X; = RP, E Roxds, where
P: E gr Since X is generic rank matrix, if we choose the number of diagonal blocks N my or equivalently, r S m dj. then the matrix P, has full row rank of r, and thus N
P,P
(6)
Moreover, we have
X = RPP
(7)
Using 5 6 and that R R = I,,, we have
x Dj X x, x Z, = (P. T (P,, )P = =. Zj.
Therefore, gD; 2X X 2X. Od, xd, for all i.j. It proves the block diagonal matrix D' is minimizer of full-LSQ Therefore, the DiaBlo-LSQ and full-LSQ have the same minimum value. As a result, every solution to the DiaBlo-LSQ problem is a solution to the full-LSQ problem.
17

---

Published as a conference paper at ICLR 2026
A.1.2 PROOF OF THEOREM
We first prove the following auxiliary lemma.
Lemma 1. Suppose that N 2 12 For generic matrices (P, Rrixdi J and (Q:
Rrax AN 1i=1, the following matrix
Q P
F
E R Ndi22xr11
QN 8
PN
has full column rank.
Proof. The conclusion is equivalent to det(F' F) + for generic P;'s and Q ss. Since det F) is a polynomial in terms of entries in P,'s and Q 's it suffices to show there exist some P 's and Q such that F has full column rank (Cox al. 1997). We construct such matrices in the following.
For convenience, we denote kn = - 1, 12 = 7 Since the N > k 22 case can be reduced to N k2 case, we only consider N k2. Let u1 ur, be a basis of R and Vi...., Vra be basis of R then we construct
A = (u ("G--))d+1>>.
(zd Rrixd 1 S S k1
Bj = (Vo-1)d=+1.-.. Vjd,) E Braxd
S 12
Ak (U(k)-1)d
Up
0) Rridd
B (V(h2- 1)d2 2r 0, 0) 222dd
The above construction must exist because b [ 02 + + kidj 2 ri, rr, k2d2 2 72. Let P: As/k21 Q; (((--1) mod k2)+1 for i ,,..., N, then the rows of matrix contain vectors {v] & u, hisss,, which span the whole RRur space. Therefore, F has rank r172 and hence it has full column rank.
Theorem Suppose the activation matrix X Rbxm and the linear output gradient gy bx are generic with ranks and r2, respectively. If N is common factor of m2 and satisfies diag(D, D....... ,D.) (x) mh xma produced by DiaBlo
ma 1 [xx1 then any adaptation that satisfies the stationarity conditions gD. Od, xd, for all
N [
S N yields adapted weights W = Wo + D that is also a stationary point of the full FT objective, i.e., gw 0 Xm2
Proof. DiaBlo introduces a block-diagonal adaptation D = diag diag(D),.. Dr) E m xma with each D; E Rd xda (here, d1 m1/N and d2 M2/N), and the adapted linear output is given by
Y = XW = XW XD.
Suppose the activations X E gbxm and output Y E R mbx have the corresponding partitioning X (X, XN) and Y (Y, Recall from 2 that the gradient of loss with respect to each tunable block D; is
Y N) respectively, with X, E Rixdd and Y, E bx
gD, gY, == N,
where gy, E h are the i-th output gradient block.
When converged, the DiaBlo adaptation D satisfies the stationarity conditions:
(8) Suppose there holds the low-rank decompositions for the activations X UP with gab and P E In Xm; and that for the output gradient gy VQ with V E ixbers and Q E graxma Alternatively, we assume the partitioning:
ED. = X gy, Od
i= I,....,.
X, UP; gy. VQ. 1, I,..., N.
18

---

Published as a conference paper at ICLR 2026
where U and V span the column spaces of X and gy. respectively, and P. Roxd, Q2 &Raxd Then the stationarity conditions in gives
x x gY. U = Od,xda x i= i=
Using the celebrated Kronecker-vec identity, we have veof So the above stationarity conditions become
UV. (Q) @ P.) veo VVC(UVV)
(Q) + P veo(U"V) Odida: i= i=1,..., N.
Stacking all the N equations above leads to the linear system:
F vee(UTv V) Odd,d
(9)
where the coefficient matrix
Q & Pi :
E Nd Nd1dax1112
Q < P
Since the assumption N > full column rank for generic Pl.....
N is equivalent to N
1 using Lemma 1 F has
mi
Me
PN.
Q
QN- So the only solution to is vec
V)
Optrz i.e., U X Consequently,
gw X gy = P Omg Xing.
That is, the full-weight gradient vanishes, including those components with respect to off-diagonal entries of W. Therefore, the stationary point of DiaBlo permits a stationary point of full finetuning
when N > [ mg
A.2
EMPIRICAL DEMONSTRATION OF THEORETICAL GUARANTEES
A.2.1 VERIFICATION OF LOW-RANK ASSUMPTION
Theorem 2 relies on the low-rankness of the activation matrix X and output gradient gy. We empirically verify the low-rank assumptions usually hold in large language models.
For matrix U E Rmxn of rank r, let 01 2 02 2 2 d, be singular values of U. We define the p effective rank rp of U as the smallest number such that Ei o
2 p. The p effective ratio is Op mi] We randomly select decoder from Llama3-8B model, which is finetuned on the GSMSK dataset, and compute the activation matrices and output gradients. The 95% effective ratios reported in Figure 3 It is clear that all activation matrices and output gradients have low ranks.
Σi= ..
A.2.2 EMPIRICAL DEMONSTRATION
We use DiaBlo with N = 64 to finetune the Llama3-8B model on the GSM8K dataset and record the average gradient norm at the end of each epoch. The average norm of matrix U defined |Ulr
as - which normalizes by the number of parameters and prevents the full-weight gradients from dominating purely due to scale. This allows for fair comparison between the gradient signal received by DiaBlo's diagonal blocks and that of the full weight matrix.
We fine-tune only the diagonal blocks while monitoring the gradients of both the full weight matrix and the diagonal blocks. Figure shows the average gradient norms the two during Llama3-8B fine-tuning GSM8K. Across all epochs, the curves nearly overlap, indicating that the diagonal blocks receive gradient signals of almost identical magnitude to those of the full model. After the first few epochs, both gradients decay at the same rate and remain tightly aligned. This empirical observation confirms that updating only diagonal blocks is sufficient to drive the full-mode gradient toward zero, consistent with the conclusion of theorem.
19

---

Published as a conference paper at ICLR 2026
Activation
0:30
Output Gradient
0.7%
5214
0.244
0.20
0.200
0.15
0.10
0.004
0.55
2.024
DOWN
GATE
Figure 3: 95% effective rank ratios of modules in finetuned Llama3-8B
DiagBlack
10"4.
Average Gradient
107
10
11
12
13
14
16
37
18
20
Figure 4: Comparison average gradient norms for full weights and diagonal-bloc -lockk....b.........
20

---

Published as a conference paper at ICLR 2026
A.3 ADDITIONAL EXPERIMENTS
A.3.1 COMMONSENSE REASONING IN BFI6
The experiments in existing literature, e.g., DoRA (Liu et al. 2024 Pissa (Meng et al. 2024), and MiLoRA (Wang et al. 2024a). all follow the fine-tuning setting in (Hu et al 2023), which uses FP for model weights, FP32 for low-rank adaptation, and FP16 sixed-precision training. Our results reported in Table Gabllefoolla follow the same fine-tuning setting, However, we notice that LoRA finetuning in precision BF15 may achieve better results. For comparison, we also report results of finetuning on commonsense reasoning tasks trained in BF 16 precision following the setting in LoRI Zhang al., 2025a). The results are reported in Table 6 On LLaMA3-SB, DiaBlo achieves the highest average score of 88.3% with N = 64 blocks and 1.51% % of trainable parameters, surpassing LoRA, DoRA, and LoRI by at least 1.0% %. Even when reducing the number of trainable parameters 0.76% with N = 128 blocks, DiaBlo maintains strong performance, achieving 88.0%, again outperforming the baselines. On Mistral-7B, DiaBlo achieves an average accuracy of 88.0% with N 64 blocks, exceeding LoRA. DoRA, and LoRL. At N 128 blocks with less trainable parameters, it continues to perform competitively, reaching 87.9%. These results confirm that DiaBlo is highly effective under BF1 precision, offering both superior accuracy and parameter efficiency compared to other state-of-the-art PEFT methods.
Table 6: Commonsense reasoning results of fine-tuning Llama3-88 and Mistral-7B in RF1 following LoRI, Results of baseline methods are taken from LoRI (Zhang et al. 2025a
Mistral-7B
PEFT
r/N
#Params
BoolQ
PIQA
SIQA
HellaS
WinoG
ARC-e
ARC-e
OBQA
AVG
Full FT
NA
100%
76.6
90.6
83,5
97.0
90.0
93.4
82.9
91.5
88.2
LoRA
== 32
1.25%
75.2
90.1
82.9
95.1
88.1
92.0
82.9
88.7
86.9
DoRA
75.8
82,9
92.6
90.6
LoRI
== 32
0.63%
75.9
90.6
83.0
95.9
87.4
91.9
83.6
88.4
87.1
DiaBlo
N 64
1.68%
76.6
90.9
83.7
96.9
88.6
93.8
83.2
90.5
88.0
DiaBlo
N 128
0.84%
76.7
91.3
83.9
96.9
88.2
93.1
83.4
89.7
87.9
Llama3-8n
PEFT
r/N
#Params
BoolQ
PIQA
SIQA
HellaS
WinoG
ARC-e
ARC-c
OBQA
AVG
Full
84.0
88.3
93.3
90.5
LORA
32
1.12%
76.3
89.8
82.7
95.8
88.7
91.7
83.4
88.4
87.1
DoRA
== 32
1.12%
75.9
89.8
82.7
95.3
88.2
93.2
83.5
87.9
87.1
93.6
88.5
DiaBlo
1.51%
76.6
90.2
83.1
96.9
88.8
93.9
85.1
91.5
DiaBlo
N 128
0.76%
76.2
90.5
83.7
97.0
89.0
94.0
84.7
88.9
88.0
A.3.2 RESULTS ON MT-BENCH
We follow the training setting of LoRA-GA (Wang al. 2024d) to finetune Llama2-7B 52k subset the Wizard M dataset (Xu et al.) 2023) and evaluate the finetuned model on the MT-Bench dataset (Zheng al. 2023). turn scores.
The quality of the responses is judged by GPT-4 and report the first
As shown in Table 7 DiaBlo achieves the strongest performance MT-Bench despite using the same parameter budget. With only 0.3% trainable parameters, DiaBlo reaches a score of 6.26, outperforming LoRA, PiSSA, DoRA, and LoRA-GA. Notably, DiaBlo even surpasses LoRA-GA with a much more parameters (4.7%). These results highlight the effectiveness and parameter efficiency of DiaBlo for tasks with long onput- sequences.
A.3.3 RESULTS ON GLUE BENCHMARK
We finetune TS-base (Rffel et al. 2020 model on GLUE benchmark (Wang et al. 2018) and evaluate on development sets.
As shown in Table 8 DiaBlo demonstrates consistently strong performance across the GLUE benchmark. Using the same parameter budget LoRA, DoRA, and PiSSA, DiaBlo achieves the
21

---

Published as a conference paper at ICLR 2026
Table Test results of fine-tuned Llama2-7B on MT-Bench. Results of baseline methods are taken from LoRA-GA (Wang et al. 2024d).
Method
MTT-Bench
LoRA
8
0.3%
5.61
PiSSA
8
0.3%
5.30
Dora
r=
0.3%
5.97
LoRA-GA
LoRA-GA
128
4.7%
6.13
DiaBlo
N 320
0.3%
6.26
Table 8: Results of finetuning T5-bbse GLUE.
Method
r/N
#Params
MRPC
CoLA
SST-2
QNLI
MNLI
Avg
FT
100%
84.6
80.7
94.8
93.2
86.3
87.9
LoRA
8=8
1.1%
84.3
79.6
94,1
93.0
85.7
87.3
DoRA
r=8
1.1%
83.9
79.1
94.4
93.1
85.3
87.2
DiaBlo
N4
1.1%
86.0
80.5
94.3
93.0
85.4
87.8
highest average of 87.8, outperforming all competing PEFT methods and coming close to full fine-tuning. DiaBlo delivers robust and balanced improvements, indicating that diagonal-block updates offer efficient and effective alternative traditional low-rank adaptations for natural language understanding.
A.3.4 OPTIMIZATION STABILITY
DiaBlo is inherently more stable by avoiding LoRA's matrix-product parameterization. To support this, we measure the gradient norm variance for DiaBlo with N 64 and LoRA with 32 on LLaMA3-8B over three epochs of commonsense reasoning finetuning. We record the gradient norm every 100 steps and calculate the variance every epoch. As shown in Table 9 DiaBlo's o variance consistently decreases, reaching very low value in the last epoch. demonstrates that the convergence of DiaBlo is stable and smooth. In contrast, LoRA's variance remains high throughout training: its A-matrix variance is artificially low in the first epoch only because its B-matrix is initialized zero. This result suggests that DiaBlo achieves a more stable gradient flow and more reliable optimization process without special initializations and customized optimizers.
Table 9: Variance of gradient norm for LoRA and DiaBlo on LLaMA3-SB.
Method
Epoch
Epoch 2
Epoch
LoRA-A
6.13
4.97
4.48
LoRA-B
22.34
6.75
5.63
DiaBlo 10.17
3.10
0.75
A.3.5 IMPACT OF HYPERPARAMETER N AND LEARNING RATES
The number of blocks, N, controls the trade-off between parameter count and model capacity. We analyze its impact by fine-tuning LLaMA3.2-3B with varying N. As shown in Table performance steadily improves as N decreases (more parameters), but the gains diminish and plateau around N 64. This indicates that N 64 provides strong balance of high performance and parameter efficiency for this model scale.
We further evaluate the performance of DiaBlo under different numbers of diagonal blocks and learning rates by fine-tuning Llama3-8B GSMSK and reporting the test accuracy. The results in Figure show that DiaBlo achieves strong accuracy across a wide range of learning rates for all configurations, demonstrating robust optimization behavior. Overall, the performance improves as the number of trainable parameters increases and exhibits plateau around N=64, matching the trend observed in commonsense reasoning experiments.
22

---

Published as a conference paper at ICLR 2026
Table 10: Ablation on the number of blocks (N) for LLaMA3.2-3B on commonsense reasoning. Performance saturates around N=64.
BoolQ
SIQA
HellaS
WinoG
ARC-e
OBQA
Full-FFT FT
72.42
86.18
81.27
91.96
85.24
89.44
78.24
82.80
83 .4
8
220.50M
73.88
87.00
81.27
93.62
87.21
89.90
80.29
50
84.85
16
110.25M
74.10
86.13
80.96
93.17
86.58
89.98
78.50
85. 60
84.38
73.94
64
27.56M
74.62
86.83
81.32
93.22
85.71
89.94
78.24
85.60
84.44
128
13.78M
73.94
86.07
81.53
92.79
86.11
89.86
78.24
84,80
84.17
256
6.89M
73.00
86.02
81.06
92.45
85.65
89.06
78.50
85, 85.60
83.92
512
345M
72.35
85.36
80.60
91.58
84.85
88.85
74.32
82.54
1024
71.87
89.75
84.61
86,95
74.06
81.60
81.60
2048
1.31M
68.90
82.43
79.58
87.82
82,95
84.85
67.66
78.40
79.07
Accurasy va Louriin Radla for Difiirert Slock Siece (1)
03.5
ACCUO0
02.5
2.1300
Figure 5: Accuracy of DiaBlo on GSM8K across different learning rates and block N.
A.3.6 FINETUNING SELECTED MODULES
To further evaluate DiaBlo under more restrictive parameter-update settings, we fine-tume Llama3-88 time on GSM8K while updating only module time-QK. QK, VUG, OD follolloing the module selections used in S2FT (Yang et al. 2024a). The results are shown in Figure For each setting, we choose N and such that DiaBlo and LoRA comparable number of trainable parameters. Across all modules, DiaBlo consistently outperforms LoRA, demonstrating its advantage even when the update space is highly constrained. Moreover, updating the VUG and OD modules yields substantially better performance than updating QK alone, matching the observation reported in S²FT (Yang et al. 2024a).
913-
60.0
37.
26.
Figure 6: Performance of DiaBlo and LoRA when fine-tuning only selected modules
23

---

Published as a conference paper at ICLR 2026
23.22
0-7
B-15
16-23
24-31
Figure 7: Accuracy on GSM8K when fine-tuning only a contiguous block of layers Llama3-8B
Table 11: Model Details
Model name
HuggingFace load name
Llama- Rama-13B
yahma//lama-1 33b-hf
Llama2-7B
meta-llama/lllmaal
2lmh---77-hf
Llama2-13B
Mita-llama/lllmma2--11111
Llama3-3B
meta-llama/LLlaaa-32--B
Llama3-8B
meta-llamaMMett--LLaa--BB-
Mistral-7B
mistralai/Miistra-7BB-0011
T5-base
15-base
A.3.7 LAYER-WISE SENSITIVITY
To better understand how different layers contribute to DiaBlo's effectiveness, we conduct a layer- wise analysis by fine-tuning only a contiguous block of layers in Llama3-8B and evaluating accuracy on GSM8K. The results are presented in Figure
We observe clear pattern: fine-tuning early and middle layers yields substantially higher performance than fine-tuning deeper layers alone. Updating layers 0-7 8-15 achieves accuracies of 60.67% and 58.92%, respectively, while restricting updates layers 16-23 leads to large drop to 40.47%, and layers 24-31 31 perform the worst only 31.63%. This trend highlights a strong layer-wise sensitivity in reasoning tasks: early and mid-level representations LLMs contain far more adaptable structure, whereas the deepest layers significantly less effective when updated in isolation.
A.4 IMPLEMENTATION DETAILS
A.4.1 MODELS
Details of the evaluated models provided in Table II All models used in experiments non-instructtio-ttnee
A.4.2
COMMONSENSE REASONING
The results in Table on commonsense reasoning in FP16 follow the standard setting in (Hu et al.) 2023 Liu et al.) 2024). The results in Table 6 on commonsense reasoning in BF 16 follow the setting in (Zhang al. (025a).. The hyper-parameteres for FP1 are provided in Table and the hyper-parameteres for BF are provided in Table 13
A.4.3 ARITHMETIC REASONING
We follow the training setting of MiLoRA (Wang et al. 2024a) for the arithmetic reasoning task. The hyper-parameter setting for fine-tuning Llama2-7B MetaMathQA dataset shown in Table 14
24

---

Published as a conference paper at ICLR 2026
Table 12: Hyper-parameters for the finetuning of Llama2-7B and Llama3-8B on commonsense reasoning in FP16.
Hyper-parameter
Liamu2-7B
Liamm3388
number blocks
64
128
64
128
Optimizer
AdamW
Learning rate
2o-4
3c-4
5o-5
lc-4
Weight decay
LR scheduler
linear
Warmup steps
100
Epochs
3
Batch size
16
Max sequence length Modules
256
Q K. V D
Table 13: Hyper-parameters for the finetuning of Mistral-7B and Llama3-8B on commonsense reasoning in BF16.
Hyper-parameler
Mistral-7B
Lluma3-88
number of blocks
64
128
64
128
Optimizer
Adamm
Learning rate
5e-5
5e-5
le.4
2c-4
Weight decay
D
I.R schedulerr
linear
Warmup steps
100
Epochs
Batch size
32
sequence length
Modules
K V O UD
G
A.4.4 CODE GENERATION AND SAFETY ALIGNMENT
We follow the training setting of LoRI (Zhang et al. 2025a for the code generation and safety alignment tasks. The hyper-parameter setting for code generation task is shown in Table 15 and the setting for the safety alignment show in Table
A.4.5 QUANTIZED MODELS
We finetune with quantized models on MATH-10K dataset. The fine-tuning paper-parameters shown in Table
A.4.6 MT-BENCH
We finetune Llama2-7B on 52k subset of WizardLM and evaluate on MT-Bench. The hyper- parameters are shown in Table 18
A.4.7 GLUE
We finetune the T5-base model on training datasets of GLUE tasks and evaluate on the development sets. The hyper-parameters are shown in Table
A.4.8 RETAINING AND FINE-TUNING SPARSE UPDATES
For the results in Table 5 retaining sparse updates uses scales 0.75, 0.75,0.75, 0.75, 0.7 for sparsity 1/32, 1/64, 1/128, respectively. The finetuning follows the setting in Table
A.5 THE USE OF LARGE LANGUAGE MODELS
Parts of the manuscript were refined using large language model tools to improve clarity, grammar, and readability.
25

---

Published as a conference paper at ICLR 2026
Table 14: Hyper-parametee parameters for the finetuning of Llama2-7B on MetaMathOA.
Llama2-7B
number of blocks
32
64
Optimizer
AdamW
Learning rate
2c-4
Weight decay
U
LR scheduler
linear
Warmup steps
Epochs
3
Batch size
16
sequence longth
2048
Modulcs
Q V UD
Table 15: Hyper-parameters for the finetuning of Mistral-7B and Llama3-8B on Code dataset.
Hyper-parannn prper-parreeter
Mistral-7B
Llama3-88
number of blocks
64
128
64
128
Optimizer
Adam
Learning
le-4
le-4
20-5
3e-5
Weight docay
0
scheduler
linear
constant
Warmup steps
100
Fpachs
Batch size
32
Max sequence length
512
Modules
V O G UD
Table 16: Hyper-parameters for the finetuning Mistral-7B and Llama3-8B on Saferpaca dataset.
Hyper-parameter
Mistral-7B
Llama3-8B
blocks
Optimizer
AdamW lc-4
Learning rase
Weight decay
U
LR scheduler
linear
Warmup steps
100
Epochs
I
Ratch
32
Max sequence length
512
Modules
Q GUD
Table 17: Hyper-parametert for the finetuning of Llama2-7B with quantized models.
paraae-r
Llama2-7B
number of blocks
64 Adam
Optimizer
Learning rate
7e-4
Weight decay
LR scheduler
linear
Warmup ratio
10%
Epochs
3
Batch size
16
sequence length
Modules
Q VVVGUU
Table 18: Hyper-parametery for the finetuning of Llama2-7B for MT-Bende evaluation.
Hyper-parameter
Llama2-7B
number of blocks
320
Optimizer
Adam
Learning rate
1.5c-3
Weight decay
D
scheduler
linear
Warmup steps
100
Epochs
1
Batch size
32
Max sequence length
1024
Modules
26

---

Published as a conference paper at ICLR 2026
Table 19: Hyper-parameters for the finetuning of T5-base GLUE.
Hyper-parameter
MRPC
COLA
SST-2
QNLI
MNLI
number of blocks
64
Optimizer
AdamW
I Leaming
1.4e-2
le-2
5e-3
4o.3
2e-3
Weight docay
0
LR scheduler
cosine
Warmup ratio
3%
Fpochs
Batch size
32
Max sequence length
128
Modules
All lincar except embeddings and LM head
Table 20: for the finetuning of Llama3-88 8B on GSM8K.
parmmee
Llama3-8B
Optimizer
AdamW
Learning rate
4e-4
Weight decay
LR scheduler
lincar
Warmup stops
20
Epochs
2
Batch size
32
sequence length
Modules
Q V O
G UD
27


Reviews:
[{"review_id": "ohOoInPxUd", "reviewer": "Reviewer_uDXN", "summary": "This paper introduces a new parameter-efficient finetuning method which targets trains a block-diagonal sparse matrix on top of linear layers in model weights. This approach seems to outperform various other PEFTs with around the same number of parameters on a variety of tasks, as well as when compared to other baselines which involve training sparse weight updates.", "questions": "- In your experience how difficult was it to hyperparameter tune this method?\n- What model components seem most important for good performance when finetuning? Is there a layer-wise effect?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The idea is quite elegant, relatively simple to implement and efficient to train -- there isn't much adaptation required to existing finetuning libraries to get this working.\n- The results are broad (covering standard PEFT benchmarks) and thus convincing.\n- Ablations cover the first questions I had regarding whether block-diagonal is better than other ways of selecting entries to tune in the weight matrix; it does seem like it is broadly a better strategy than other ideas.", "weaknesses": "- Since we use a standard suite of benchmarks to evaluate PEFTs, it's possible that our literature is engaging in test-set overfitting (compare how the ImageNet challenge or LMSYS arena were overfit by organisations submitting many models repeatedly). It would thus be nice to show how the technique performs under varying learning rates and block sizes (e.g. as done for LoRA in [Schulman et al. (2025)](https://thinkingmachines.ai/blog/lora/)). It is nice though that there are not as many hyperparameters as other PEFTs!", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IEJIbZ9iIf", "reviewer": "Reviewer_qumS", "summary": "This work presents a Parameter-Efficient Fine-Tuning method called DiaBlo that updates only the diagonal blocks of selected model weight matrices. The work argues that this method enables axis-aligned, per-dimension scaling that LoRA cannot capture and verifies the method on a large variety of benchmarks with Llama 7B/8B/13B models.", "questions": "- The reviewer suggests to add Full Finetuning results in Table 1. \n\n- The paper mentions when N is not a common factor of m1,m2, it needs to expand and pad the weight into a proper size and then select the corresponding diagonal blocks. This can be common case. But the paper doesn’t touch this point clearly in later experiments.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The proposed work eliminates the inherent optimization difficulties associated with low-rank decomposition by avoiding the use of matrix products.\n\n- DiaBlo demonstrates higher stability in 4-bit and 2-bit arithmetic reasoning tasks.", "weaknesses": "- Compared to strong baselines like SMT with similar trainable parameter amount, the proposed method does not show significantly better performance. In other words, the paper argues the memory and computation efficiency of the proposed method, but the model doesn’t achieve significant improved performance compared to baselines when they share the same amount of trainable parameters.\n\n- In table 1, it shows DiaBlo N =128 doesn’t get better performance compared to DiaBlo N =64 although doubled trainable parameters. This raises concerns of the scaling ability of the proposed Diablo.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dMDAQP03Zo", "reviewer": "Reviewer_hx7w", "summary": "This paper presents DiaBlo, a param. efficient fine-tuning (PEFT) method for LLMs using the diagonal blocks of weight matrices. One of the key novelties in this work is that DiaBlo does not use extra low-rank matrices multiplied together (like LoRA’s A x B structure) to adapt model weights. Instead, it directly updates selected diagonal blocks within the model’s existing weight matrices, which eliminates the need for initialization or custom optimization strategies. The work covers tasks such as commonsense reasoning, arithmetic reasoning, and code generation, showing that DiaBlo can match or outperform LoRA with comparable memory/efficiency results. Furthermore, the method shows robustness under quantized architectures (4/2-bit).", "questions": "1. Most evaluated tasks focus on short output sequences, such as commonsense and reasoning benchmarks. Including long-form, multi-turn dialogue datasets like MT-Bench would better demonstrate DiaBlo’s scalability and effectiveness in extended conversational contexts.\n\n2. The Appendix shows that trainable modules (QKVO/GUD) are fixed across methods, which is reasonable, but it remains unclear whether DiaBlo’s robustness comes from having a larger set of modules trainable on all methods (for example, for LLaMA3-8B, all the QKVOGUD modules have some trainable parameters). An ablation over different subsets of trainable components would clarify if the observed gains persist under more constrained settings. For example, see Figure 4.0 in S2FT on adding trainable parameters to only a subset of modules, such as QK, or GUD, etc.\n\n3. Adding results on the GLUE benchmark would help assess general language understanding and show how well DiaBlo transfers to broader NLP tasks.\n\n4. Reporting zero-shot baseline performance would contextualize fine-tuning improvements. Clarifying whether the LLaMA3-8B variant used is “instruct” or base would also make the evaluation setup more transparent.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. By removing the complexity of low-rank structures, this work presents a clear alternative to LoRA-style PEFT. The results show that DiaBlo attains comparable performance without the added overhead of extra trainable matrices, simplifying both tuning and optimization.\n\n2. The evaluation spans diverse supervised fine-tuning tasks -- including code generation, arithmetic reasoning, and commonsense reasoning -- covering a balanced range of short to moderate sequence lengths.\n\n3. The results in Table 5 are particularly convincing, showing that random sparse update patterns (and SMT) underperform compared to DiaBlo. This supports the claim that the structured diagonal-block design is the key driver of its performance advantage.", "weaknesses": "1. Most evaluated benchmarks involve short output sequences, except for code generation. Testing DiaBlo on tasks with longer input–output contexts would better demonstrate its scalability and performance stability under extended sequence conditions (see q1).\n\n2. The discussion of sparsity-based PEFT methods misses some recent relevant work, such as S2FT (NeurIPS 2025)[1] and SparseLoRA (ICML 2025)[2]. Including these would strengthen the discussion on sparsity in the introduction and would provide a better picture of the current limitations in state-of-the-art PEFT methods.\n\nReferences:\n\n[1] Xinyu Yang, Jixuan Leng, Geyang Guo, Jiawei Zhao, Ryumei Nakada, Linjun Zhang, Huaxiu Yao, Beidi Chen, \"S2FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsity\", NeurIPS 2025\n\n[2] Samir Khaki, Xiuyu Li, Junxian Guo, Ligeng Zhu, Chenfeng Xu, Konstantinos N. Plataniotis, Amir Yazdanbakhsh, Kurt Keutzer, Song Han, Zhijian Liu, \"SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsity\", ICML 2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EVXx51y4gM", "reviewer": "Reviewer_yEpk", "summary": "- The paper proposes a new PEFT method called DiaBlo, which fine-tunes the weights as $\\hat{W} = W + D$, where $W$ are the pretrained weights and $D$ are the residual fine-tuning weights as a block-diagonal matrix.\n- $D$ is initialized with all zeros\n- The method is evaluated on commonsense reasoning, arithmetic reasoning, code generation tasks and safety alignment tasks.", "questions": "1. What is the GPU memory consumed by DiaBlo compared to LoRA/DoRA?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The use of block diagonal matrix for the residual fine-tuning weights is a experimentally motivated.\n- The method is easy to implement and computationally efficient.", "weaknesses": "1. The experimental comparisons on the quantization experiments are not correct. \n    - The papers experiments use a base model with a different quantization method than the baselines, and the baseline results are taken directly from Table 7 in [1].\n    - [1] proposes a new quantization method called AiQ, fine-tunes it using LoRA, and then compares it with other quantization methods fine-tuned using LoRA.\n    - However, the experiments in the paper initialize the quantized base model from MagR [2], fine-tunes using DiaBlo, and then compares it with results that use multiple different quantization methods fine-tuned using LoRA.\n    - To elaborate, baseline results in Table 4 (taken from Table 7 in [1]) are of the nature \"Quantization Method x/y/z + LoRA\", ment to compare quantization methods. However, DiaBlo results are of the form \"Quantization Method MagR + DiaBlo\". Hence, there is no way to tell if the performance difference is due to the quantization method or the fine-tuning method.\n    - Hence, the quantization method is a confounder that makes the comparisons invalid.\n2. The comparison with baselines in all the tables are taken from multiple sources. Even if the high level settings like precision and hypeparameters are matched, the results are not directly comparable as subtle details in the implementation can lead to different results. For a proper comparison, the baseline methods should be run with the same setup as the proposed method.\n    - For example, the baseline results in Table x are taken from 3 sources.\n    - When the the baseline methods and DiaBlo are evaluated on the same benchmark but with bf16 precision in Table 6, DiaBlo shows minor improvements over LoRA. As the baseline results are taken from a different work, the improvements could be due to a different training setup.\n3. Given that LoRA has had immense practical impact, the contribution of a new PEFT method does not have much impact without a significant advantage other than performance alone. In that sense, the contribution of the paper is limited.\n\n---\n\n## References\n[1] \"ApiQ: Finetuning of 2-Bit Quantized Large Language Model\", Liao et al., EMNLP 2024\n\n[2] \"MagR: Weight Magnitude Reduction for Enhancing Post-Training Quantization\", Zhang et al., NeurIPS 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)