Kron-LoRA: Hybrid Kronecker-LoRA Adapters for Scalable, Sustainable Fine-tuning

## Abstract

Abstract Fine-tuning massive pre-trained language models across many tasks demands adapters that are both parameter-efficient and expressive. We introduce Kron-LoRA , a hybrid adapter that combines Kronecker-structured factorization with low-rank LoRA compression—an integration that, to our knowledge, has not been explored in parameter-efficient fine-tuning or in matrix approximation literature. Kron-LoRA achieves up to 4× fewer parameters than standard LoRA while retaining similar expressivity. Experiments on DistilBERT, Mistral-7B, LLaMA-2-7B, and LLaMA-3-8B across eight benchmarks show that Kron-LoRA matches or exceeds LoRA baselines with modest memory savings and only a 5–8% speed overhead. In sequential fine-tuning, it also delivers competitive cross-task transfer despite using only one-quarter of the adapter parameters. Kron-LoRA thus offers a scalable, sustainable solution for multi-task adaptation of large language models.

## 1 Introduction

Large pre-trained language models (PLMs) such as BERT and GPT have set new benchmarks across a wide array of natural language processing tasks. However, fine-tuning these models independently for each downstream application is increasingly impractical: naively storing a full copy of model weights per task incurs prohibitive storage costs, and backpropagating through hundreds of millions or billions of parameters strains both GPU memory and training time.

Parameter-efficient fine-tuning (PEFT) methods address these challenges by freezing the bulk of the pre-trained network and learning only a small number of task-specific parameters. Adapter layers insert lightweight modules between transformer sublayers (Houlsby et al., 2019), prefix-tuning prepends trainable tokens to the input sequence (Li and Liang, 2021), and LoRA directly learns low-rank updates to weight matrices (Hu et al., 2022). While LoRA reduces the adapter footprint to $O(r(d_{\text{in}}+d_{\text{out}}))$ parameters per layer, storing and swapping even rank-8 adapters becomes costly when supporting hundreds of tasks.

More recent work extends LoRA with architectural and optimization variations. *DyLoRA* (Valipour et al., 2022) dynamically allocates ranks across training steps without search; *AdaLoRA* (Zhang et al., 2023) adaptively assigns parameter budgets across layers; *DoRA* (Liu et al., 2024) decomposes magnitude and direction to improve stability; *PiSSA* (Meng et al., 2024) tunes dominant singular components for faster convergence; and *MiLoRA* (Wang et al., 2025) leverages minor singular components to enhance instruction tuning. These variants improve robustness, convergence, or transfer, but they generally preserve LoRA’s parameter footprint rather than reducing it by large multiplicative factors.

In parallel, recent work has explored Kronecker-product structure to further compress adapter modules. (Tahaei et al., 2023) introduce *KronA*, which replaces LoRA’s low-rank projections with a pure Kronecker product and achieves improved accuracy on GLUE without added inference latency. (Braga and Li, 2024) propose *AdaKron*, combining outputs of two small networks via the Kronecker product. More recently, (Li et al., 2025) develop *MoKA*, a mixture-of-Kronecker-product adapter that dynamically interpolates multiple Kronecker factors to boost parameter efficiency on RoBERTa. These methods demonstrate the promise of Kronecker decompositions, but they either forgo rank-$r$ expressivity or incur additional computational overhead.

Although both Kronecker factorization and low-rank approximation are long-standing tools in matrix approximation and linear algebra, prior work has largely applied them separately, whether in PEFT or in classical settings. To the best of our knowledge, the combination of Kronecker structure with low-rank LoRA has not previously been formulated, either in the context of language model adaptation or in the broader theory of matrix decompositions. Moreover, Kron-LoRA’s formulation is complementary to many of the above LoRA or Kronecker variants: it can directly replace the LoRA or KronA component within DyLoRA, AdaLoRA, DoRA, PiSSA, MiLoRA, or MoKA, inheriting their benefits while delivering stronger parameter reduction.

In this work, we introduce *Kron-LoRA*, a hybrid two-stage adapter that augments LoRA with Kronecker structure. For each frozen linear layer with weight

$$ $W\in\mathbb{R}^{d_{\mathrm{out}}\times d_{\mathrm{in}}},$ $$

we model its task-specific update as

$$ $\Delta W=A\otimes B,$ $$

where

$$ $A\in\mathbb{R}^{d_{A2}\times d_{A1}}\quad\text{(with }d_{\mathrm{out}}/d_{A2}\approx 200\text{)}\quad\text{and}\quad B\in\mathbb{R}^{(d_{\mathrm{out}}/d_{A2})\times(d_{\mathrm{in}}/d_{A1})}.$ $$

We then apply an r-rank LoRA decomposition

$$ $B\approx B_{1}B_{2}\,.$ $$

By the identity

$$ $\mathrm{rank}(A\otimes B)=\mathrm{rank}(A)\,\mathrm{rank}(B),$ $$

and the fact that the Kronecker structure imposes repeated, structured patterns on the columns of $\Delta W$, Kron-LoRA can preserve expressivity while using up to $4\times$ fewer parameters than a standard rank-r LoRA adapter.

#### Contributions.

We summarize our main contributions as follows:

- •
We introduce Kron-LoRA, to our knowledge the first method that integrates Kronecker factorization with low-rank LoRA decomposition—an original combination not previously explored in parameter-efficient fine-tuning or in the broader matrix approximation literature.
- •
We propose Kron-LoRA, a drop-in adapter that combines Kronecker structure with low-rank LoRA compression for extreme parameter efficiency.
- •
We extensively evaluate KronA, Kron-LoRA, and LoRA on DistilBERT, Mistral-7B, LLaMA-2-7B, and LLaMA-3-8B across eight benchmarks. While KronA is extremely parameter-efficient but less accurate, Kron-LoRA matches or exceeds LoRA’s performance with far fewer parameters—for example, 840K parameters on DistilBERT and 5.8M on LLaMA-3-8B—incurring only 5–8% speed overhead.
- •
We analyze sequential fine-tuning, showing that Kron-LoRA slightly outperforms LoRA-8 in retention and positive transfer for similar tasks, and experiences much smaller drops than LoRA-8 on heterogeneous task sequences.

#### Paper organization

The remainder of this paper is structured as follows. Section [2](https://arxiv.org/html/2508.01961v2#S2) reviews related PEFT and tensor decomposition methods. Section [3](https://arxiv.org/html/2508.01961v2#S3) describes the Kron-LoRA factorization and implementation details. Section [4](https://arxiv.org/html/2508.01961v2#S4) and [5](https://arxiv.org/html/2508.01961v2#S5) presents our experimental setup and empirical results. Section [6](https://arxiv.org/html/2508.01961v2#S6) discusses future work, and Section [7](https://arxiv.org/html/2508.01961v2#S7) concludes.

## 2 Related work

#### Parameter-efficient fine-tuning

Adapter modules insert small bottleneck layers between transformer sublayers to learn task-specific transformations (Houlsby et al., 2019). Prefix-tuning prepends a sequence of trainable prompt tokens to the input, leaving the base model frozen (Li and Liang, 2021). LoRA directly learns a low-rank update to each weight matrix, reducing per-layer storage to $O(r(d_{\text{in}}+d_{\text{out}}))$ parameters (Hu et al., 2022). A number of LoRA variants have since been proposed: DyLoRA dynamically allocates ranks without search (Valipour et al., 2022); AdaLoRA adaptively distributes parameter budgets across layers (Zhang et al., 2023); DoRA decomposes magnitude and direction for stable training (Liu et al., 2024); PiSSA tunes dominant singular components (Meng et al., 2024); and MiLoRA leverages minor singular components to enhance transfer (Wang et al., 2025). These methods improve flexibility and robustness but generally do not target multiplicative parameter reduction.

#### Kronecker-structured adapters

To further compress adapter updates, recent work exploits Kronecker-product structure. (Tahaei et al., 2023) propose *KronA*, replacing LoRA’s low-rank projections with a pure Kronecker product. *AdaKron* (Braga and Li, 2024) combines outputs of two small subnetworks via the Kronecker product. *MoKA* (Li et al., 2025) extends this idea to a mixture-of-Kronecker-product adapter that dynamically interpolates multiple Kronecker factors. These approaches demonstrate the potential of Kronecker decompositions, but they either sacrifice rank-$r$ expressivity, incur additional computational overhead, or do not explicitly target extreme parameter reduction, leaving room for more efficient hybrid designs such as Kron-LoRA.

#### Continual learning and forgetting

Prior work has examined catastrophic forgetting in sequential fine-tuning of PLMs (Chen et al., 2020; Pfeiffer et al., 2021), but primarily for full-model or unstructured adapter updates.

#### Positioning of Kron-LoRA

Kron-LoRA is complementary to both LoRA variants and Kronecker-based adapters: it can directly replace the LoRA or KronA component within DyLoRA, AdaLoRA, DoRA, PiSSA, MiLoRA, or MoKA, inheriting their strengths while delivering stronger parameter reduction. For this reason, we do not include these variants in our experiments.

## 3 Kron-LoRA method

We now describe Kron-LoRA, a hybrid adapter that combines Kronecker-product structure with low-rank LoRA compression.

#### Kronecker factorization.

Given a frozen linear layer with weight

$$ $W\in\mathbb{R}^{d_{\mathrm{out}}\times d_{\mathrm{in}}},$ $$

we model the task-specific update as a Kronecker product

$$ $\Delta W\;=\;A\,\otimes\,B,$ $$

where

$$ $A\in\mathbb{R}^{d_{A2}\times d_{A1}},\quad B\in\mathbb{R}^{d_{B2}\times d_{B1}}.$ $$

For all layers except the vocabulary projection, we set $d_{A1}=2$ and pick $d_{A2}$ such that $d_{\mathrm{out}}/d_{A2}\!\approx\!200$, denoted as $d_{A2_{\text{others}}}$. For the vocabulary projection, we instead use $d_{A1}=1$ and halve this value, i.e., $d_{A2_{\text{voc}}}=\tfrac{1}{2}d_{A2_{\text{others}}}$. Writing $d_{B2}=d_{\mathrm{out}}/d_{A2}$ and $d_{B1}=d_{\mathrm{in}}/d_{A1}$ partitions the original matrix into Kronecker “slices.”

#### LoRA decomposition of B .

To further compress $B$, we apply the rank-$r$ LoRA factorization of Hu et al. (2022):

$$ $B\;\approx\;B_{1}\,B_{2},\quad B_{1}\in\mathbb{R}^{d_{B2}\times r},\quad B_{2}\in\mathbb{R}^{r\times d_{B1}}.$ $$

Thus the full adapter update is

$$ $\Delta W=A\otimes(B_{1}B_{2}).$ $$

For $\mathtt{x}$ as the input, we will use the fact that $\text{vec}(A\otimes(B_{1}B_{2})\mathtt{x})=\text{vec}((B_{1}B_{2})\mathtt{x_reshaped}(A^{T}))$ in the implementation part.

#### Expressivity and parameter efficiency.

By the Kronecker rank identity

$$ $\mathrm{rank}(A\otimes B)=\mathrm{rank}(A)\,\mathrm{rank}(B),$ $$

the composite update $\Delta W=A\otimes B$ has a high rank structured pattern than LoRA, and in practice a rank-8 Kron-LoRA decomposition attains the rank-8 or even rank-16 expressivity as a standard LoRA adapter, deponds on the model settings. At the same time, the total parameter count

$$ $\lvert A\rvert+\lvert B_{1}\rvert+\lvert B_{2}\rvert\;=\;d_{A1}\,d_{A2}\;+\;r\,(d_{B2}+d_{B1})$ $$

is then roughly $4\times$ smaller than that of a conventional rank-r LoRA adapter.

#### Implementation details.

In practice, we wrap each nn.Linear with a KronLoRALinear module that:

- 1.
Freezes the original weight $W$.
- 2.
Registers $B_{1},B_{2}$ as trainable $\mathtt{nn.Parameter}$ objects, and $A^{T}$ as trainable $\mathtt{nn.linear}$ object for faster computation purpose.
- 3.
In $\mathrm{forward}(x)$:
•
Reshapes $x\to(\,\ast,d_{B1},d_{A1}\,)$, denoted as $\mathtt{x_reshaped}$.
•
Computes the Kronecker-LoRA update in the following way:
$\displaystyle Y_{1}$
$\displaystyle=B_{2}\,x_{\mathrm{reshaped}}$
$\displaystyle\in\mathbb{R}^{8\times 2},$
$\displaystyle Y_{2}$
$\displaystyle=Y_{1}\,A^{T}$
$\displaystyle\in\mathbb{R}^{8\times d_{A2}},$
$\displaystyle Y_{3}$
$\displaystyle=B_{1}\,Y_{2}$
$\displaystyle\in\mathbb{R}^{d_{B2}\times d_{A2}}.$
In this way the dimensions of $Y_{1},Y_{2},Y_{3}$ are small, so can use slightly less CUDA memory than conventional LoRA.
•
Scales by $\alpha/r$, where $\alpha=32$, and applies dropout, with rate $0.1$, reshapes back, and adds to $Wx$.

Kron-LoRA integrates into existing LoRA codebases with only a few lines of change and its structure is amenable to custom CUDA kernels for $WX$ matmuls, which we leave as future work.

## 4 Experimental setup

#### Models and adapters

We evaluate Kron-LoRA on four transformer backbones: DistilBERT (uncased) (Sanh et al., 2019), Mistral-7B v0.1 (Mistral AI Team, 2024), LLaMA-2-7B (Touvron et al., 2023), and LLaMA-3-8B (Grattafiori et al., 2024). For each model, every nn.Linear layer is replaced with a Kron-LoRA adapter (using a slice size of $d_{A2_{\text{others}}}=4$ for DistilBERT and $d_{A2_{\text{others}}}=16$ for Mistral and LLaMA), and performance is compared against standard LoRA adapters of rank $r=8$. For DistilBERT, we additionally report results with $r=16$, since Kron-LoRA can achieve comparable accuracies on several tasks. For the Mistral model, we also include comparisons with KronA. In the case of DistilBERT, the weights of pre_classifier and classifier are excluded from the adapter scheme and thus trained fully, while all other model weights remain frozen.

#### Datasets and tasks

We fine-tune on eight commonsense and reasoning benchmarks used in Hu et al. (2023): PIQA (Bisk et al., 2020), HellaSwag (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2020), SiQA (Sap et al., 2019), OBQA (Mihaylov et al., 2018), BoolQ (Clark et al., 2019), ARC-Easy and ARC-Challenge (Clark et al., 2018). For each dataset, we train adapters for up to 30 epochs, monitor validation accuracy on the official validation set split, select the checkpoint with the highest performance, and report its test accuracy.

#### Training procedure

Adapters are trained with AdamW (Loshchilov and Hutter, 2019) (learning rate $3\times 10^{-4}$) with other default settings of Trainer. We use a micro-batch size of $8$ per GPU (no gradient accumulation) on NVIDIA A100s, mixed precision (FP16) via torch.cuda.amp, weight decay 0.01, a dropout rate of $0.1$, and the default linear scheduler. We found that introducing gradient accumulation to simulate larger effective batch sizes (e.g. accumulating $8$ steps to achieve an effective batch of $8$) can shift final accuracy by several percentage points, so all reported results use no accumulation. Checkpoint save/load overhead is excluded from our reported throughput; nevertheless, since Kron-LoRA’s adapters have a significantly smaller parameter footprint, its saving checkpoint time is substantially lower than that of standard LoRA by roughly 10% for each time of saving. All experiments can be performed within 24 hours, but we need to set epoch = 16 for Hellaswag.

#### Continual‐learning protocol

To assess forgetting, we fine-tune adapters sequentially on two tasks and then evaluate on the first task’s test set. We use the following schedules:

- •
ARC-Challenge$\leftrightarrow$ARC-Easy: 15 epochs on ARC-Challenge followed by 15 epochs on ARC-Easy, and vice versa.
- •
HellaSwag$\leftrightarrow$ARC-Easy: 10 epochs on HellaSwag followed by 15 epochs on ARC-Easy, and vice versa.

#### Evaluation metrics

We report (1) test accuracy, (2) training dynamics, (3) training throughput (examples/sec, excluding I/O), and (4) peak and intermediate GPU memory.

## 5 Results

### 5.1 Parameter efficiency vs. accuracy

To better understand the performance trade-offs between different adapter strategies, we evaluated KronA and LoRA-8 on the Mistral-7B model across eight standard multiple-choice reasoning benchmarks: PIQA, HellaSwag (HS), WinoGrande (WG), ARC-Easy (ARC-E), ARC-Challenge (ARC-C), OpenBookQA (OBQA), SocialIQA (SIQA), and BoolQ. Results are summarized in Table [1](https://arxiv.org/html/2508.01961v2#S5.T1).

LoRA-8 achieves a strong average accuracy of 74.39%, with competitive scores across all tasks. In contrast, KronA lags significantly behind, with an average accuracy of only 54.74% — roughly 20 percentage points lower. This trend is consistent across all benchmarks, with the gap being especially large on OBQA and SIQA, where KronA struggles to surpass 45%.

It is worth noting that KronA uses far fewer trainable parameters (2.28M) compared to LoRA-8 (21.26M), since the decomposition is constrained to factors that are the closest integers around the square root of the adapter dimension, as proposed in the original paper. This factorization yields the most parameter-efficient configuration possible under the KronA design, but at the cost of a substantial drop in task accuracy.

We also observed that similar performance patterns hold across other backbone models (DistilBERT, LLaMA-2-7B, and LLaMA-3-7B), where KronA consistently underperforms LoRA by a large margin, suggesting that the observed gap is not specific to Mistral-7B.

**Table 1: Mistral-7B test accuracy (%) comparing KronA and LoRA-8. KronA performs 20% below LoRA-8. HS = HellaSwag, WG = WinoGrande. Avg. = average over all 8 benchmarks.**
| Adapter | #Params | Avg. | PIQA | HS | WG | ARC-E | ARC-C | OBQA | SIQA | BoolQ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LoRA-8 | 21.26 M | 74.39 | 85.09 | 83.85 | 80.58 | 74.74 | 54.52 | 70.60 | 64.59 | 81.13 |
| KronA | 2.28 M | 54.74 | 72.36 | 70.96 | 67.56 | 51.40 | 34.78 | 43.60 | 45.04 | 54.22 |

Since KronA by itself does not perform well—achieving extreme parameter efficiency but at the cost of substantial accuracy loss—we next investigate whether combining Kronecker factorization with LoRA can yield a more favorable balance. This leads us to Kron-LoRA, which integrates the parameter savings of Kronecker decomposition with the representational capacity of LoRA.

We compared Kron-LoRA against LoRA across four backbone models—DistilBERT, Mistral-7B, LLaMA-2-7B, and LLaMA-3-8B—on the same eight multiple-choice reasoning benchmarks: PIQA, HellaSwag (HS), WinoGrande (WG), ARC-Easy (ARC-E), ARC-Challenge (ARC-C), OpenBookQA (OBQA), SocialIQA (SIQA), and BoolQ. Results are presented in Tables [2](https://arxiv.org/html/2508.01961v2#S5.T2) and [3](https://arxiv.org/html/2508.01961v2#S5.T3).

Across all four backbones, Kron-LoRA achieves performance comparable to—or in several cases better than—LoRA, while requiring only 25–30% of the parameters. On DistilBERT, Kron-LoRA (0.84M parameters) reaches 57.57% average accuracy, marginally surpassing LoRA-16 (57.06%) despite using less than half the parameters.

On Mistral-7B, Kron-LoRA (5.74M parameters) achieves 75.22% average accuracy, outperforming LoRA-8’s 74.39% with only 27% of the parameters. A task-level breakdown shows notable improvements on HellaSwag (+2.30 pp) and ARC-Easy (+1.93 pp), while performance on the remaining benchmarks remains comparable.

On LLaMA-2-7B, Kron-LoRA reaches 73.78% average accuracy, slightly higher than LoRA-8 (73.62%), with gains on PIQA, HellaSwag, and ARC-Easy. Minor drops are observed on WinoGrande and SIQA, likely due to the datasets’ sensitivity to subtle context and social reasoning cues that may be harder for the compressed Kronecker factorization to capture at this rank.

Finally, on LLaMA-3-8B, Kron-LoRA achieves 73.48% average accuracy versus LoRA-8’s 72.71%, with consistent gains on PIQA, HellaSwag, ARC-Easy, and BoolQ. A small decrease is observed on OBQA, which may reflect dataset-specific variance rather than a systematic limitation of Kron-LoRA.

**Table 2: Test accuracy (%) on reasoning and ARC benchmarks (Part 1). HS = HellaSwag, WG = WinoGrande.**
| Model | Adapter | #Params | PIQA | HS | WG | ARC-E | ARC-C |
| --- | --- | --- | --- | --- | --- | --- | --- |
| DistilBERT | LoRA-8 | 1.25 M | 65.56 | 25.84 | 50.20 | 50.53 | 34.78 |
|  | LoRA-16 | 1.92 M | 65.40 | 36.33 | 51.46 | 53.86 | 35.79 |
|  | Kron-LoRA | 0.84 M | 65.83 | 36.09 | 52.01 | 52.46 | 39.13 |
| Mistral-7B | LoRA-8 | 21.26 M | 85.09 | 83.85 | 80.58 | 74.74 | 54.52 |
|  | Kron-LoRA | 5.74 M | 85.85 | 85.32 | 80.66 | 76.67 | 54.85 |
| LLaMA-2-7B | LoRA-8 | 20.28 M | 81.83 | 86.19 | 74.03 | 80.70 | 57.85 |
|  | Kron-LoRA | 5.31 M | 82.05 | 86.76 | 73.16 | 81.40 | 57.85 |
| LLaMA-3-8B | LoRA-8 | 22.03 M | 82.70 | 88.90 | 68.98 | 76.49 | 54.18 |
|  | Kron-LoRA | 5.84 M | 84.00 | 89.41 | 69.92 | 77.19 | 55.52 |

**Table 3: Test accuracy (%) on commonsense benchmarks (Part 2). Avg. = average over all eight benchmarks.**
| Model | Adapter | OBQA | SIQA | BoolQ | Avg. |
| --- | --- | --- | --- | --- | --- |
| DistilBERT | LoRA-8 | 52.00 | 33.78 | 99.79 | 51.56 |
|  | LoRA-16 | 54.20 | 59.52 | 99.91 | 57.06 |
|  | Kron-LoRA ((Ours) | 55.20 | 59.93 | 99.91 | 57.57 |
| Mistral-7B | LoRA-8 | 70.60 | 64.59 | 81.13 | 74.39 |
|  | Kron-LoRA | 71.80 | 65.20 | 81.41 | 75.22 |
| LLaMA-2-7B | LoRA-8 | 66.80 | 61.31 | 80.28 | 73.62 |
|  | Kron-LoRA | 67.40 | 61.11 | 80.52 | 73.78 |
| LLaMA-3-8B | LoRA-8 | 68.80 | 58.29 | 83.30 | 72.71 |
|  | Kron-LoRA | 68.20 | 59.26 | 84.31 | 73.48 |

While most of our experiments focused on rank = 8, we also conducted preliminary trials at higher ranks. They also expected to yield similar results, as Kron-LoRA continued to perform on par with LoRA.

### 5.2 Training dynamics

Figure [1](https://arxiv.org/html/2508.01961v2#S5.F1) shows validation accuracy on HellaSwag over 16 epochs for Kron-LoRA ($d_{A2}=16,r=8$) and LoRA adapters of ranks 4, 8, and 16 on Mistral-7B. Despite using only 27% of LoRA-8’s parameters—and half of LoRA-4’s—Kron-LoRA outperforms LoRA-8 after the first two epochs and maintains its lead for the remainder of training. Although LoRA-16 achieves the highest accuracy, it requires over six times more adapter parameters than Kron-LoRA. Furthermore, Kron-LoRA’s learning curve is smoother, suggesting that the Kronecker-structured factorization provides an implicit regularization effect that stabilizes optimization and accelerates convergence.

Figure: Figure 1: Validation accuracy on HellaSwag over 16 epochs for Kron-LoRA and LoRA adapters of ranks 4, 8, and 16 on Mistral-7B.
Refer to caption: Hellaswag.png

### 5.3 Speed and memory

Table [4](https://arxiv.org/html/2508.01961v2#S5.T4) compares training throughput and GPU memory usage for LoRA-8 and Kron-LoRA on PIQA with LLaMA-2-7B:

**Table 4: Throughput (examples/sec) and memory (MiB) on PIQA (LLaMA-2-7B).**
| Adapter | Throughput | Peak mem. | Intermediate mem. |
| --- | --- | --- | --- |
|  | (ex/s) | (MiB) | (MiB) |
| LoRA-8 | 29.28 | 28050.54 | 26594.06 |
| Kron-LoRA | 27.04 | 27822.68 | 26493.38 |

#### Throughput

Kron-LoRA processes 27.04 ex/s versus LoRA-8’s 29.28 ex/s (a 7.65% slowdown). The modest overhead stems from Kron-LoRA’s three matrix operations per forward pass (two torch.matmul and one nn.Linear) versus LoRA’s two nn.Linear calls.

#### Peak memory

Kron-LoRA peaks at 27822.68 MiB, saving 227.86 MiB (0.8%) compared to LoRA-8’s 28050.54 MiB due to its much smaller adapter tensors, despite the extra reshape and batched kernels.

#### Intermediate memory

Kron-LoRA requires 26493.38 MiB for activations and gradients—100.68 MiB (0.4%) less than LoRA-8’s 26594.06 MiB—reflecting that, despite the additional Kronecker reshaping and batched operations, the smaller adapter tensors yield a net intermediate‐memory saving.

Overall, Kron-LoRA incurs only a modest throughput overhead (5–8%) while reducing peak memory usage by approximately 0.8%, suggesting a reasonable trade-off between speed and memory efficiency for large-scale fine-tuning.

### 5.4 Continual learning (forgetting)

To measure forgetting, we sequentially fine-tune adapters on two tasks (A→B) and then evaluate accuracy on task A’s test set. Table [5](https://arxiv.org/html/2508.01961v2#S5.T5) shows results for the closely related ARC‐Challenge $\leftrightarrow$ ARC‐Easy sequence and the more heterogeneous ARC‐Easy $\leftrightarrow$ HellaSwag sequence. Interestingly, when tasks are highly similar, both Kron-LoRA and LoRA-8 can exhibit not only limited forgetting but even positive transfer, whereas under larger domain shifts their robustness diverges.

**Table 5: Accuracy for sequential fine-tuning tasks under different sequences on LLaMA-3-8B. T1 and T2 denote the first and second tasks in the sequence, respectively. Columns show performance on the first task after training on T1 (T1→T1), on the second task after training on T2 (T2→T2), and on the first task after training on T2 (T2→T1). The fourth column shows the accuracy drop on the first task ($\Delta\text{T1}=\text{T2\textrightarrow T1}-\text{T1\textrightarrow T1}$).**
|  | Kron-LoRA (%) | LoRA-8 (%) |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Sequence | T1→T1 | T2→T2 | T2→T1 | $\Delta$ T1 | T1→T1 | T2→T2 | T2→T1 | $\Delta$ T1 |
| ARC-C→ARC-E | 48.83 | 74.74 | 52.51 | +3.68 | 48.16 | 75.96 | 51.84 | +3.68 |
| ARC-E→ARC-C | 77.02 | 56.19 | 72.98 | -4.04 | 76.18 | 55.52 | 71.96 | -4.22 |
| ARC-E→HS | 73.51 | 87.13 | 58.77 | -14.74 | 72.28 | 87.32 | 3.68 | -68.60 |
| HS→ARC-E | 87.23 | 72.81 | 83.08 | -4.15 | 87.00 | 71.40 | 82.63 | -4.37 |

#### Robust retention under task similarity

In the ARC-Challenge→ARC-Easy ordering, both Kron-LoRA and LoRA-8 slightly *improve* performance on the first task after training on the second ($+3.68$ pp), indicating beneficial transfer between the two datasets. In the reverse ARC-Easy→ARC-Challenge sequence, both methods experience a modest drop, with Kron-LoRA losing slightly less (–4.04 pp) than LoRA-8 (–4.22 pp). These results suggest that, under high task similarity, both approaches preserve and leverage shared representations effectively, with Kron-LoRA providing a small but consistent advantage in retention.

#### Increased interference under domain shift

For the heterogeneous ARC-Easy→HellaSwag sequence, Kron-LoRA retains 58.77% (–14.74 pp), while LoRA-8 collapses almost entirely (–68.60 pp). In the reverse HellaSwag→ARC-Easy direction, both methods suffer modest forgetting (–4.15 pp vs. –4.37 pp). These results highlight that Kron-LoRA is substantially more robust under large domain shifts, mitigating catastrophic interference that severely affects LoRA-8.

#### Regularization and optimization effects

We found that Kron-LoRA benefits noticeably from weight decay, which acts as a regularizer and helps stabilize retention across both similar and heterogeneous tasks. Moreover, Kron-LoRA’s performance indicates that its structured adaptation provides a better inductive bias for cross-domain generalization, though additional strategies such as task-specific adapter reinitialization or merging could further reduce forgetting.

## 6 Broader applicability and future work

Below, we outline several cross-disciplinary applications of Kron-LoRA that demonstrate its broader impact beyond NLP:

- 1.
Edge-scale medical imaging.
Radiology models must often be fine-tuned per hospital or modality (MRI, CT, ultrasound), yet storing dozens of adapters on a PACS server or embedded probe is impractical. Kron-LoRA’s $4\times$ smaller factors allow shipping and switching task- or device-specific adapters in seconds—even on low-power ARM or FPGA hardware.
- 2.
On-device mobile models.
Smartphones and wearables increasingly host large language or vision models for tasks like translation, summarization, or AR guidance. Yet frequent task-specific updates (per app, per user) are infeasible when adapters span tens of MB. Kron-LoRA’s compact Kronecker factors ($\sim$1–2 MB) can be shipped as lightweight app updates, cached per-user, or swapped across applications with negligible storage overhead.
- 3.
Multi-physics surrogate modeling.
High-fidelity simulators in climate science, fluid dynamics, or materials design frequently require fine-tuning to new boundary conditions, but storing a full model per scenario is prohibitively expensive. With Kron-LoRA, a single base network can host dozens of tiny adapters—one per regime—so that thousands can reside in GPU memory for real-time interpolation across parameters.
- 4.
Robotics & control.
A robot arm may need distinct control policies for assembly, painting, and inspection, yet continual fine-tuning risks catastrophic forgetting and swapping large networks adds latency. By attaching separate Kron-LoRA adapters ($\sim$1 MB each) per skill, the robot can load any policy in milliseconds.
- 5.
Neuromorphic & photonic accelerators.
Emerging hardware (e.g. spiking neural nets, photonic matrix multiplies) favors low-rank, structured updates to minimize on-chip memory and routing complexity. Kron-LoRA’s factorization maps naturally to 2D crossbar arrays (for the Kronecker factor) and digital peripheral logic (for the low-rank factors), squeezing adapters into tiny on-chip buffers.
- 6.
Federated & privacy-preserving learning.
In federated settings (e.g. personal keyboard models, health data), uploading full LoRA adapters (tens of MB) strains bandwidth and raises privacy concerns. Kron-LoRA requires only a few-MB factors per client—minimizing communication and simplifying secure server-side aggregation.

These vignettes show that Kron-LoRA is more than an NLP trick; it is a broadly applicable, structured, multi-task adapter framework poised to transform fine-tuning across disciplines.

## 7 Conclusion

We have introduced *Kron-LoRA*, a simple drop-in adapter that combines Kronecker‐structured updates with low-rank LoRA compression to achieve up to $\!4\times$ fewer parameters than standard LoRA while preserving similar accuracy. To the best of our knowledge, Kron-LoRA is the first method that explicitly integrates Kronecker factorization with LoRA—an original combination not previously explored in parameter-efficient fine-tuning or in the broader matrix approximation literature. Our empirical results demonstrate that Kron-LoRA matches or exceeds LoRA baselines in multi-task accuracy, incurs only a 5–8% training-speed overhead, reduces memory usage, and maintains competitive or slightly improved performance on similar tasks, while experiencing much smaller drops than LoRA-8 on heterogeneous task sequences.

Key takeaway: As the first integration of Kronecker structure with LoRA, Kron-LoRA offers a plug-and-play, scalable, and sustainable adaptation layer for large pre-trained transformers—empowering parameter-efficient and continual-learning-capable fine-tuning for similar datasets.

### 7.1 Broader Impact and Ethics

#### Potential benefits

Kron-LoRA’s extreme parameter efficiency makes it feasible to *deploy* fine-tuned adapters on resource-constrained hardware (e.g. mobile devices, edge nodes, or IoT sensors). This can democratize access to state-of-the-art language capabilities in low-resource settings—such as small clinics, field research stations, or educational programs without dedicated GPU infrastructure—and reduce the carbon footprint of continual updates by minimizing computation and memory requirements. Moreover, in privacy-sensitive applications (e.g. personalized keyboard suggestions, on-device medical note classification, or federated learning), Kron-LoRA’s tiny adapters can be exchanged instead of full model weights, lowering communication overhead and limiting the surface area for data leakage.

#### Responsible usage considerations

As with all adapter-based methods, Kron-LoRA inherits biases and limitations from its underlying pre-trained model, so practitioners should evaluate downstream fairness and robustness across demographic groups and application domains. Deploying compact adapters on edge or federated platforms also raises security concerns—malicious actors could craft poisoned updates or invert small adapters to extract private signals—so secure update channels, differential privacy, and audit logs are recommended. Finally, while parameter-lean adapters lower the barrier to customization, developers must still adhere to ethical standards, ensure transparency about model capabilities and limitations, and obtain informed consent when processing sensitive data.

## Acknowledgments

We would like to thank Prof. Yang Ning, Prof. Robert Kleinberg and Christian Belardi for insightful discussions on Kronecker factorization and low‐rank compression.
We are grateful to the Mistral AI team for open-sourcing Mistral-7B, to Meta AI for releasing LLaMA‑2 and LLaMA‑3, and to the Hugging Face community for maintaining the DistilBERT codebase.

## References

- Bisk et al. [2020]
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, and Yejin Choi.
Piqa: Reasoning about physical commonsense in natural language.
In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 174–184. Association for Computational Linguistics, 2020.
- Braga and Li [2024]
Elena Braga and Shuo Li.
Adakron: Adaptive kronecker adapters for parameter-efficient fine-tuning.
In *International Conference on Learning Representations*, ICLR, 2024.
- Chen et al. [2020]
Sanyuan Chen, Yutai Hou, Yiming Cui, Wanxiang Che, Ting Liu, and Xiangzhan Yu.
Recall and learn: Fine‐tuning deep pretrained language models with less forgetting.
In *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing*, pages 7870–7881. Association for Computational Linguistics, 2020.
- Clark et al. [2019]
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova.
Boolq: Exploring the surprising difficulty of natural yes/no questions.
In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)*, pages 2924–2936. Association for Computational Linguistics, 2019.
URL [https://aclanthology.org/N19-1300/](https://aclanthology.org/N19-1300/).
- Clark et al. [2018]
Peter Clark, Omer Tafjord, Matthew Richardson, and Luke Zettlemoyer.
Think you have solved question answering? try arc, the ai2 reasoning challenge.
In *Proceedings of the 32nd AAAI Conference on Artificial Intelligence*, pages 6381–6384. AAAI Press, 2018.
- Grattafiori et al. [2024]
Andrea Grattafiori et al.
The llama 3 herd of models.
*arXiv preprint*, 2024.
URL [https://arxiv.org/abs/2407.21783](https://arxiv.org/abs/2407.21783).
- Houlsby et al. [2019]
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Ben Morrone, Quentin de Laroussilhe, Andrés Gesmundo, Maria Attariyan, and Sylvain Gelly.
Parameter-efficient transfer learning for nlp.
In *Proceedings of the 36th International Conference on Machine Learning*, ICML, pages 2790–2799. PMLR, 2019.
- Hu et al. [2022]
Edward J. Hu, Yelong Shen, Philip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.
Lora: Low-rank adaptation of large language models.
In *International Conference on Learning Representations*, ICLR, 2022.
- Hu et al. [2023]
Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka-Wei Lee.
Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models.
In *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023)*, pages 5254–5276. Association for Computational Linguistics, 2023.
doi: 10.18653/v1/2023.emnlp-main.319.
URL [https://aclanthology.org/2023.emnlp-main.319/](https://aclanthology.org/2023.emnlp-main.319/).
- Li and Liang [2021]
Xinya Li and Percy Liang.
Prefix-tuning: Optimizing continuous prompts for generation.
In *Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics*, ACL, pages 4582–4597. ACL, 2021.
- Li et al. [2025]
Yifan Li, Wei Chen, and Aman Patel.
Moka: Mixture-of-kronecker-product adapters for large-scale language models.
In *Proceedings of the 2025 International Conference on Machine Learning*, ICML. PMLR, 2025.
- Liu et al. [2024]
Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen.
Dora: Weight-decomposed low-rank adaptation.
In *Proceedings of ICML 2024*, 2024.
URL [https://arxiv.org/abs/2402.09353](https://arxiv.org/abs/2402.09353).
- Loshchilov and Hutter [2019]
Ilya Loshchilov and Frank Hutter.
Decoupled weight decay regularization.
In *International Conference on Learning Representations*, ICLR, 2019.
- Meng et al. [2024]
Fanxu Meng, Zhaohui Wang, et al.
Pissa: Principal singular values and singular vectors adaptation of large language models.
*arXiv preprint*, 2024.
URL [https://arxiv.org/abs/2404.02948](https://arxiv.org/abs/2404.02948).
- Mihaylov et al. [2018]
Todor Mihaylov, Anish Das, and Hannaneh Hajishirzi.
Can a suit of armor conduct electricity? a new dataset for open book question answering.
*arXiv preprint arXiv:1809.02789*, 2018.
URL [https://arxiv.org/abs/1809.02789](https://arxiv.org/abs/1809.02789).
- Mistral AI Team [2024]
Mistral AI Team.
Mistral-7b v0.1.
[https://mistral.ai/models/Mistral-7B-v0.1](https://mistral.ai/models/Mistral-7B-v0.1), 2024.
- Pfeiffer et al. [2021]
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych.
Adapterfusion: Non-destructive task composition for transfer learning.
In *Findings of the 2021 Conference of the European Chapter of the Association for Computational Linguistics*, EACL Findings, pages 487–503. ACL, 2021.
- Sakaguchi et al. [2020]
Keisuke Sakaguchi, Siamak Emami, Chandra Bhagavatula, and Yejin Choi.
Winogrande: An adversarial winograd schema challenge at scale.
In *Proceedings of the 34th AAAI Conference on Artificial Intelligence*, pages 8732–8739. AAAI Press, 2020.
- Sanh et al. [2019]
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf.
Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.
In *NeurIPS Workshop on Energy Efficient Machine Learning and Cognitive Computing*, 2019.
- Sap et al. [2019]
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi.
Social iqa: Commonsense reasoning about social interactions.
*arXiv preprint arXiv:1904.09728*, 2019.
URL [https://arxiv.org/abs/1904.09728](https://arxiv.org/abs/1904.09728).
- Tahaei et al. [2023]
Alireza Tahaei, Ricardo Castro, and Zheng Yang.
Krona: Kronecker adapters for efficient fine-tuning.
In *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, EMNLP. ACL, 2023.
- Touvron et al. [2023]
Hugo Touvron, Louis Martin, Colin Raffel, Valentin Lostanlen, and other Meta AI authors.
Llama 2: Open foundation and fine-tuned chat models.
*arXiv preprint*, 2023.
URL [https://arxiv.org/abs/2307.09288](https://arxiv.org/abs/2307.09288).
- Valipour et al. [2022]
Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi.
Dylora: Parameter-efficient tuning of pre-trained models using dynamic search-free low-rank adaptation, 2022.
URL [https://arxiv.org/abs/2210.07558](https://arxiv.org/abs/2210.07558).
Accepted to EACL 2023.
- Wang et al. [2025]
Hanqing Wang, Yixia Li, Shuo Wang, Guanhua Chen, and Yun Chen.
Milora: Harnessing minor singular components for parameter-efficient llm finetuning.
In *Proceedings of NAACL 2025*, 2025.
URL [https://arxiv.org/abs/2406.09044](https://arxiv.org/abs/2406.09044).
Accepted at NAACL 2025.
- Zellers et al. [2019]
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi.
Hellaswag: Can a machine really finish your sentence?
In *Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*, pages 4795–4801. Association for Computational Linguistics, 2019.
- Zhang et al. [2023]
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Tuo Zhao, and Chen Weizhu.
Adalora: Adaptive budget allocation for parameter-efficient fine-tuning.
In *Proceedings of ICLR 2023*, 2023.
URL [https://arxiv.org/abs/2303.10512](https://arxiv.org/abs/2303.10512).

## Peer Reviews

confidence: 4
contribution: 1
initial_rating: 2
presentation: 3
questions: 1. Compare proposed method with both KronA and LoRA at *similar* parameter count. 
2. Provide an explanation for why further parameter count in KronA is necessary and why the proposed Kron-LoRA is a good way to achieve it. 
3. Provide discussion and comparisons with SOTA PEFT methods (e.g. DoRA (Liu eta al., 2024, MiLoRA (Wang et al., 2025), MOA [a], LoRA+ [b]) and methods that focus on significant parameter count reduction compared to LoRA (AdaLoRA (Zhang et al., 2023), DyLoRA (Valipour et al., 2022), VeRA [c], LoRA-XS [d], NOLA [e], Tiered-LoRA [f], BS-LoRA [g]). 
4. Provide details on the experimental setup and reasoning for differences with typical setup (see weakness point 3 above).

**References** \
[a] Cao, Jie, Tianwei Lin, Hongyang He, et al. “MoA: Heterogeneous Mixture of Adapters for Parameter-Efficient Fine-Tuning of Large Language Models.” arXiv preprint arXiv:2506.05928, 2025. \
[b] Hayou, Soufiane, Nikhil Ghosh, and Bin Yu. “LoRA+: Efficient Low Rank Adaptation of Large Models.” Proceedings of the 41st International Conference on Learning Representations (ICLR), 2024. PMLR 235: 17783-17806 \
[c] Kopiczko, Damian J., et al. “VeRA: Vector-Based Random Matrix Adaptation.” arXiv preprint arXiv:2310.11454, 2023. \
[d] Bałazy, Klaudia, et al. “LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters.” ECAI 2025 \
[e] Abbasi Koohpayegani, Soroush, et al. “NOLA: Compressing LoRA Using Linear Combination of Random Basis.” International Conference on Learning Representations (ICLR), 2024 \
[f] Renduchintala, Ashish, et al. “Enhancing Parameter Efficiency of LoRA with Weight Tying.” Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Long Papers (NAACL), 2024. ACL Anthology \
[g] Zhou, Yuhua, et al. “BSLoRA: Enhancing the Parameter Efficiency of LoRA with Intra-Layer and Inter-Layer Sharing.” International Conference on Machine Learning (ICML), 2025
soundness: 2
strengths: The proposed idea is simple. The paper is mostly clearly written and easy to understand. In the experiments on the NLU tasks, the proposed method performs comparably to LoRA while employing close to $4\times$ fewer parameters. The authors show that KronA alone uses significantly fewer parameters but also significantly underperforms LoRA.
summary: The work proposes a method for parameter-efficient fine-tuning (PEFT) of LLMs. Existing PEFT methods include LoRA which decomposes the update weight matrices into two low rank matrices and KronA which uses Kronecker product of matrices for the weight update. The proposed method (Kron-LoRA) combines the two by replacing one of the matrices in KronA with its LoRA equivalent (decomposing it into two low rank matrices). Authors experiment on NLU tasks and show that Kron-LoRA performs comparably to LoRA while using significantly fewer parameters.
weaknesses: 1. **Insufficient baselines:** The empirical analysis severely lacks important baselines and comparison with SOTA PEFT approaches and LoRA variants. The proposed method is a simple combination of KronA and LoRA and thus must include comparison with these approaches with a similar number of trainable parameters. However, comparison KronA with is provided on a single backbone model and with a setting that uses just 10% of parameters (2.3M for KronA vs 21.3M for LoRA). There are no comparisons except with LoRA on any other dataset / backbone LLM. It is necessary to additionally compare the work with SOTA approaches to help understand its benefits. Lack of any such comparisons makes it hard to evaluate the proposed method. 
2. **Lack of intuition/motivation:** There is no clear reasoning for why the matrix in KronA must be replaced with a low rank decomposition as in LoRA. Is the parameter count in KronA high enough that it requires low rank decomposition? In their work, authors of KronA show that it outperforms LoRA with similar parameter count. Is it not possible to adjust the dimensions of KronA to have a similar parameter count as Kron-LoRA here instead of using the low-rank decomposition? 
3. **Non-typical experimental setup:** The experimental set-up is not clear. Typically, for the commonsense reasoning benchmark (Hu et al., 2023), the training set of the 8 sub-tasks are combined and 1-3 epochs of fine-tuning is performed (e.g., refer Hu et al., 2023, Liu et al., 2024, Wang et al., 2025). However, in Kron-LoRA, it is not clear if fine-tuning is performed individually on each of the sub-tasks. Kron-LoRA also uses 30 epochs of fine-tuning, a lower batch size of 8 and no warmup. It is not clear why this deviation from typical evaluation is needed.

---

confidence: 5
contribution: 2
initial_rating: 2
presentation: 3
questions: Could you add ablations & additional experimentation of varying d_A1 and d_A2?
Could you add a comparison of the rank and #parameters of various PEFT methods (especially LoRA ones)?
Could you describe more in depth how do you tune each algorithm? Similarly, with such limited improvement, could add confidence intervals/stds?
See comments for additional questions.
soundness: 3
strengths: - Sound methodology
- Very well written paper
summary: This paper introduce Kron-LoRA, a hybrid adapter that combines Kronecker-structured factorization with low-rank LoRA. Kron-LoRA requires up to 4x fewer parameters than LoRA and performs similarly at the price of 5-8% overhead.

The proposed method can be summarized as 1) compute the task-specific update as a Kronecker product between A and B, and 2) compress B via rank-k LoRA factorization. I would appreciate if the authors could compare the rank and #parameters of various PEFT methods (especially LoRA ones).

In terms of experiments, multiple models are used from DistilBERT, Mistral, and LLaMA-2/3. Datasets are the ones used in the original LoRA paper. Since GLUE-base tasks are old, I would encourage the authors to conduct additional experiments. Moreover, since one of the main motivation of the work is that LoRA becomes costly when supporting hundreds of tasks, I would expect the authors to conduct analysis with O(10) tasks.

In the first experiment, why authors are comparing LoRA-8 and KronA while the latter has 10x fewer parameters? It is not a surprised that the performance drop is 20%. Please conduct fairer analysis with LoRA being smaller. Table 2 and 3 do not address the issue and sometimes the gain of Kron-LoRA is very small. I couldn't find details on how tuning each algorithm has been done, hence I'm little bit skeptical of the results.

Regarding the speed and memory experiments, please include all the models in the analysis, not only LLaMA-2-7B. I also suspect that larger models would indicate an overhead higher than 8%. Finally, the continual learning experiment is interesting. I would love to see all possible pairwise experiments with a given model instead of only ARC/HS.

Overall, the idea is sound and the paper well written. However, I do feel that multiple experiments are missing and would help to understand what is happening. I would encourage the authors to use another dataset than GLUE tasks and do more analysis/experiments with including mean/std or CIs.

Note:
- L053 - ... they either forgo rank-r expressiveness or incur additional computational overhead --> like this paper (5-8% overhead)
weaknesses: - Weak experimental results: more recent datasets, more analysis and ablations.

---

confidence: 3
contribution: 2
initial_rating: 2
presentation: 2
questions: See weaknesses
soundness: 2
strengths: 1. First integration of Kronecker structure and LoRA.
The combination of Kronecker factorization with LoRA is original and well-motivated. Prior work explored Kronecker adapters (KronA, AdaKron, MoKA) or LoRA variants separately, but not their hybridization.

2. Strong parameter-efficiency gains.
The method achieves up to 4× parameter reduction while maintaining similar or better accuracy compared to LoRA-8 across all backbones (Table 2–3). The analysis clearly shows that Kronecker structure provides multiplicative rank expansion, preserving expressivity even at low parameter budgets.
summary: This paper introduces Kron-LoRA, a novel adapter design that combines Kronecker-product factorization with low-rank LoRA decomposition to achieve high parameter efficiency in fine-tuning large language models (LLMs).
Unlike LoRA, which applies a pure low-rank update to frozen model weights, Kron-LoRA factorizes each task-specific update into a Kronecker structure, allowing structured compression and reuse of shared subspaces.
weaknesses: 1. Conceptual simplicity vs. depth.
While novel, the combination of Kronecker and LoRA is a natural hybrid rather than a deeply theoretical contribution. The paper’s strength lies in practicality and empirical rigor, not fundamental mathematical insight.

2. Limited ablation on Kronecker dimensions.
The choice of d seems heuristic. It’s unclear how sensitive results are to this partition, or whether learned Kronecker shapes could further improve results.

3. Minor computational overhead.
Kron-LoRA incurs a 5–8% slowdown and only marginal memory savings (~0.8%). While acceptable, it suggests the method trades compute for parameter compactness.

4. No comparison to other compression baselines.
The paper omits comparisons with AdaLoRA, DoRA, and PiSSA, which could clarify whether Kron-LoRA’s gains stem mainly from the Kronecker structure or rank allocation differences.


Decision: Reject