LoRAGen: Structure-Aware Weight Space Learning for LoRA Generation
Authors: Hao Huang
Jingtao Ding
Mengqi Liao
Xin Wang
Jinyang Ban
Yuan Yuan
Huaiyu Wan
Yong Li
Venue: ICLR 2026 Poster (2026)
Keywords: Weight space learning
hypernetworks
LoRA
latent diffusion
Primary area: generative models
TL;DR: LoRAGen is the first structure-aware method for generating LoRA parameters from natural language by addressing the unique geometric properties of low-rank adaptation spaces.

Abstract: The widespread adoption of Low-Rank Adaptation (LoRA) for efficient fine-tuning of large language models has created demand for scalable parameter generation methods that can synthesize adaptation weights directly from task descriptions, avoiding costly task-specific training. We present LoRAGen, a structure-aware method for generating LoRA parameters from natural language descriptions. Through empirical analysis of LoRA libraries, we identify two key structural properties of LoRA parameter spaces: non-uniqueness of low-rank decomposition and heterogeneous weight distributions across network modules. These properties necessitate specialized parameter generation methods rather than general weight space learning approaches. LoRAGen employs a latent diffusion model with two innovations: weight-space supervision on full adaptation matrices to handle decomposition non-uniqueness, and a module-aware Mix-of-Experts decoder that adapts to module-specific weight distributions. Experiments show LoRAGen achieves 96.0\% performance relative to task-specific LoRAs on FLAN-T5-large and 72.7\% on Gemma-2-2B-Instruct for in-distribution tasks, while obtaining 40.2\% on zero-shot generation across unseen tasks—surpassing baselines by nearly 5\%. Our work establishes the first structure-aware approach to LoRA generation with insights into adaptation weight space geometry.

Published as a conference paper at ICLR 2026
LORAGEN: STRUCTURE-AWARY WEIGHT SPACE LEARNING FOR LORA GENERATION
Hao Huang'
Jingtao Ding
Mengqi Liao'
Xin Wang'
Jinyang Ban'
Yuan Yuan
Huaiyu Wan
Yong Li
Beijjnn Jiaotong University
Tsinghua University
'New York University
hachuangeb juu. edu.
{ding]ingtan, (yyoongo]@ttinneeeee edu. on
ABSTRACT
The widespread adoption of Low-Rank Adaptation (LoRA) for efficient fine- tuning of large language models has created demand for scalable parameter gen- eration methods that can synthesize adaptation weights directly from task descrip- tions, avoiding costly task-specific training. We present LoRAGen, structure- aware method for generating LoRA parameters from natural language descrip- tions. Through empirical analysis of LoRA libraries, we identify two key struc- tural properties of LoRA parameter spaces: non-uniqueness of low-rank decom- position and heterogeneous weight distributions across network modules. These properties necessitate specialized parameter generation methods rather than gen- eral weight space learning approaches. LoRAGen employs latent diffusion model with two innovations: weight-space supervision on full adaptation ma- trices to handle decomposition non-uniqueness, and module-aware Mix-of- Experts decoder that adapts to modulc-specific weight distributions, Experiments show LoRAGen achieves 96.0% performance relative to task-specific LoRAs on FLAN-T5-large and 72.7% on Gemma-2-22 2-2-----ttttt for in-distribution tasks, while obtaining 40.2% on zero-shot generation across unseen tasks surpassing baselines by nearly 5%. Our work establishes the first structure- ap- proach to LoRA generation with insights into adaptation weight space geometry. The implementation of our approach is available: https:/ /gi thub. com/ t tsinghua- fii-lab//ll LLod/Ge.
INTRODUCTION
Exploring the weight space of neural networks, i.e., the sigh-dimensional parameter space spanned by populations of trained networks, has emerged as a powerful paradigm for understanding model mechanisms and enabling novel applications. Parameter generation, which trains models to produce weights for target networks, represents a particularly promising direction that has gained significant attention in recent years (Schürholt et al., bb; Schürholt al., 2022; Schürholt al., 2024: Wang et al., 2024). The rise of large language models (LLMs) has created opportunities to apply parameter generation techniques this domain, particularly through Low-Rank Adaptation (LoRA) (Hu al., 2022) generation, the direct synthesis of LoRA parameters for efficient fine-tuning. While traditional LoRA workflows require task-specific training with custom datasets and hyperparameters. creating engineering overhead and limiting reusability (He et al., 2022a; Lv et al., 2024), LoRA generation enables direct parameter synthesis from natural language task descriptions, improving scalability and unlocking adaptive model behavior without maintaining extensive adapter libraries.
Recent work generates LoRA parameters conditioned on task identifiers, datasets, or natural lan- guage task descriptions. A common design involves learning hypernetwork-a neural network that generates parameters for another base network (Ha et al., 2016). With advances in deep gen- erative models, one category of approaches learns a lower-dimensional representation directly from the weight space and decodes this latent representation into LoRA parameters (Shao et al., 2025). However, the underlying encoder-decoder model must encode the entire LoRA parameters once
1 Cooresponding author.

---

Published as a conference paper at ICLR 2026
0.000
look rame dwots gebs
Spectral entropy distribution per module
0.63
2.5
0.00
0.750
2.0
0.75
Spectral entropy
1.5
1.04
Goltee Sieiuity
1.0
0.44
Test Description Eweh Siantarity
Figure 1: Two empirical observations. Left: Task description embedding similarity versus LoRA parameter similarity (measured by full adaptation matrix and low-rank decomposition matrix, spectively). Right: Spectral-entropy distributions of LoRA parameters for FLAN-T5- -LAN-T5-large, grouped by module type: encoder self-attention, decoder self-attention, and cross-attention.
into the learned latent representation, which limits the size of LoRA that can be embedded. Another category learns conditional diffusion prior over the latent space to generate LoRA parameters from random noise based on specific task conditions at test time, but these approaches struggle to generate well-performing LoRA parameters across diverse architectures and datasets (Jin et al., 2024; Liao et al., 2024; Wu et al., 2024; Soro et al., 2025). A recent work, Text-to-LoRA (T2L) (Charakorn et al., 2025), employs hypernetwork trained to construct LoRA parameters forward pass, enabling zero-shot generation for entirely unseen tasks based on the hypothesis that different LoRAs share the same adaptation mechanism and can be optimized without explicit struc- ture. However, these methods treat LoRA parameter generation as an instantiation of general weight space learning approaches Schürholt et al. (2024); Wang et al. (2024), while lacking a tailored design specifically for LoRA characteristics.
single inexpensive
In this work, we begin with an empirical analysis of a LoRA library built on the Transformer- model FLAN-T5-12 -arr-geeeereeeee (Chung et al., 2024) (Figure 1), Our amalysis reveals the unn-uniqueness of low-rank decomposition as a key challenge. positive correlation between adapter similarity in the weight space and task description embedding similarity, but we find no correlation between the cosine similarity of low-rank decomposition matri- and task description embedding similarity. This suggests that supervision in the full adaptation matrix space should generalize better than element-wise reconstruction. However, most LoRA gen- erators reconstruct low-rank decomposition matrices directly (Ha et al., 2016; Jin al., 2024; Liao et al., 2024: Soro et al., 2025; Charakorn et al., 2025; Lv et al., 2024), which can make training sensitive arbitrary rescalings rotations of the low-rank decomposition matrices that produce the same full adaptation matrix (Gabrielsson et al., 2024). Additionally, in Figure (right), we identify significant heterogeneity in LoRA weight distributions across different modules. The spectral- entropy (Yunis et al., 2024; Roy & Vetterli, 2007) distributions differ systematically by module type within FLAN- T5 l rgge: encoder self-attention exhibits higher entropy, decoder self-attention shows the lowest entropy, and cross-attention lies in between, indicating heterogeneous weight dis- tributions across parts of the base model. However, existing methods typically use a single decoder across modules (L.v et al., 2024; Charakom et al., 2025), which overlooks the module structure (Ostapenko et al., 2024) and can mismatch the spectral-entropy distributions.
shown in Figure (left), we observe clear
Motivated by the two empirical observations that reveal distinct properties of the LoRA weight space, we introduce LoRAGen, a method that generates LoRA parameters from natural-language task descriptions for structure-aware LoRA weight space learning. Specifically, we LoRA weight autoencoder (LAE) to learn a latent space over LoRA parameters and a conditional latent diffusion model conditioned natural language task descriptions generate latent representations from random noise, followed by a decoder that processes these generated representations to produce new LoRA parameters. To address the non-uniqueness issue, we supervise the full adaptation matrix
2

---

Published as a conference paper at ICLR 2026
directly rather than the low-rank decomposition matrices, proposing two weight space loss terms: direction loss and a spectral loss on the full adaptation matrix. This avoids sensitivity to rescalings or rotations of low-rank decomposition matrices that yield the same full adaptation matrix, leading to more consistent and task-relevant weight space learning. To address the heterogeneity of weight distributions, introduce a module-aware Mix-of-Fxperts (MoE) decoder with routing that uses a structural embedding combining a latent variable with learnable module and layer embeddings. This allows experts specialize the observed module-specific weight distribution patterns while ensuring a controlled sharing mechanism via the chosen expert pool configuration, facilitating gen- eralization across different architectures of the base model.
Our key contributions are: (1) We propose the first structure- aware weight learning method tailored to the LoRA weight space, enabling effective LoRA parameter generation for diverse downstream tasks. (2) LoRAGen introduces weight-space losses on the full adaptation matrix to address non- uniqueness of low-rank decomposition and employs a module-aware MoE decoder to model het- erogeneous LoRA weight distributions.
(3) LoRAGen achieves strong in-distribution LoRA gener- ation performance close to sask-specific LoRAs across architectures: 96.0% on FLAN-TS rarlaage, 72.7% on Gemma-2 - Innsstustttttii ruct and further achieves 40.2% in zero-shot LoRA generation on seven unseen tasks, surpassing competitive baselines by nearly 5%.
2 RELATED WORK
Weight Space Learning for Parameter Generation. Research on weight space learning has fol- lowed two main directions: predicting model properties from trained weights (Schürholt et al., 2021a; Unterthiner et al., 2020; Zhao et al., 2025) and generating new parameters for neural net- works (Schürholt et al., 2022; Yuan et al., 2024; Schürholt et al., 2024; Li et al., 2025; Wang et al., 2025; Saragih et al., 2025; Gupta et al., 2026). Here, focus parameter generation. With ad- vances in deep generative models, one category of approaches learns a latent representation directly over populations of trained models and decodes this latent representation to generate parameters (Berardi et al., 2022). Another category employs conditional latent diffusion models to generate target parameters (Peebles al., 2022: Soro al., 2024). Recent studies introduce graph-based encoders that treat networks as graphs and enable parameter generation across architectures (Ko- finas et al., 2024). These approaches have been applied in areas such as meta-learning, transfer learning, and model compression (Finn et al.. 2017; Nava et al., 2022; Wang et al., 2023). Overall, prior work exploring parameter generation in weight space (Peebles et al., 2022; Berardi al., 2022; Schürholt et al., 2024) demonstrates that such methods can generalize across range of architectures and datasets.
Hypernetworks for LoRA Generation. A hypernetwork is neural network designed to gener- ate the parameters of another "base" network, providing an indirect encoding of the base model's weights (Ha et al., 2016; Zhang et al., 2018; Schug et al., 2025). With the development of deep generative models, one category of approaches (Shao et al., 2025) learns conditional latent repre- sentations of pretrained LoRA parameters for new LoRA parameter generation. Another category employs conditional latent diffusion priors over latent space (Wu et al., 2024; Jin et al., 2024; Soro et al., 2024), enabling the generation of task-specific LoRA parameters. Recently, hyperLoRA (Lv et al., 2024) employs instruction-tuned hypernetoorks with constrained loss and demo selection to produce stable and generalizable adapters. While these approaches have advanced multi-task LLM adaptation, they typically rely on learned task identifiers, limiting their capacity for zero-shot LoRA parameter generation unseen tasks (Ivison& & Peters, 2022; Mahabadi et al., 2021: He et al., 2022b; Schürholt et al.. 202 Ortiz- BBrajjs et al., 2024: Lv et al., 2024). Recent work explores natural language as conditioning signals for zero-shot generation (Xiao et al., 2023; Ivison al., 2023; Phang et al., 2023), with T2L. (Charakorn et al., 2025), DnD (Liang et al., 2025), and LoRA-Gen (Xiao et al., 2025) utilizing hypernetoorss to generate LoRA adapters from textual prompts. Futher- more, (Park et al., 2026; Wojcik et al., 2026; Liu et al., 2026; Charakorn et al., 2026) treat LoRA generation as a general weight space learning problem, overlooking the unique structural properties of LoRA parameter spaces. In contrast, LoRAGen is the first weight space learning approach that specifically accounts for the structural characteristics of LoRA spaces, leading to more effective parameter generation.
3

---

Published as a conference paper at ICLR 2026
3 PRELIMINARIES
Low-Rank Adaptation (Hu et al., 2022): Low-rank matrix AW serves as a adapter to a base model. For pretrained weight matrix Wo. the fine-tuned linear transformation is given by Wo + AW = Wo + BA, where A E ER and B E Roou Xr are low-rank decomposition matrices with r & min(din- dout): We ignore the module type and layer index of the LoRA parameters when referring (e.g., query projection) and a layer index E {l...., Lm.. A LoRA adapter at each location Bm, Am,t. We denote the low-rank adapter by
all LoRA parameters. Therefore we index them by module type E M
(m, E) specifies low-rank matrix AWm,t m, AW (AWm, MEM,, Note that the low-rank decomposition is not unique: for any invertible matrix Re 1tmm.tXmm.d (Bm. (P Am,)) produces the same low-rank matrix AWm,r-
Problem setting. We assume an LoRA library of pairs D = ((x11, AW( where x(0) is a natural language description of task {(0 and AW (Aw m.c} represents the fine-tuned low- rank adapter for task g(,). Our goal is to train a LoRA generator using D that produces new LoRA parameters AW given natural language task description x', where x' may be either in-distribution (from D) out-of-distribution (unseen tasks).
4 THE PROPOSED METHOD: LORAGEN
4.1 DESIGN PRINCIPLES FROM EMPIRICAL OBSERVATIONS
We first summarize two observations (Obs) over a library of low-rank adapters (two panels in Fig- 1), which motivates our tailored design in LoRAGen. The details of observations experiment implementation reported at the Appendix A.1.
Obs-1: Non-uniqueness of the low-rank decomposition. Here we focus on the non-uniqueness property of low-rank decompositions in LORA. Specifically, we examine a LoRA library trained on FLAN-TS-large. While AW is uniquely defined as a full adaptation matrix, its low-rank decomposition matrices (A, B) is not unique, illustrated in Sec. 3. This property motivates the following hypothesis: we supervise the low-rank decomposition matrices (A, B) directly, like element-wise reconstruction, training becomes sensitive to arbitrary rescalings and rotations that still yield the same full adaptation matrix AW. By supervising the full adaptation matrix itself, we avoid this ambiguity and directly align the generated adapters with the pretrained adapters in the full adaptation matrix space, which should lead to more consistent and task-relevant weight space learning.
We test this hypothesis by computing the pairwise similarity between its LoRA adapter and each of the other 111 adapters in the FLAN subset for representative task which is selected from the four major task categories in the FLAN subset so that reflects the overall distribution of the dataset rather than any specific task; further details are reported in Appendix A. To measure adapter simi- larity, we compute the cosine similarity of the concatenation of flattened low-rank A and B matrices of all layers and flattened AW, respectively. To avoid scale mismatch between the two similarity measures, they are plotted in separate subplots, each fitted with its own least-squares trend line. We observe a clear positive correlation between the task embedding similarity and the adapter similarity in the weight space, whereas the similarity measured on the low-rank decomposition matrices ex- hibits near-zero Spearman coefficients p. indicating the lack of correlation. This phenomenon aligns with the non-umiqueness property of low-rank decompositions in LoRA and suggests that supervi- sion in the full adaptation matrix space should generalize better than element-wiss reconstruction.
Motivated by Obs-1, we therefore introduce adapter-level supervision (Sec. 4.3), where losses are defined directly in the weight space of AW. Concretely, we introduce two weight space loss terms, combining direction loss Lang (Eq. 2) that aligns normalized LoRA directions, with a spectral loss Expor (Eq. 3) that matches leading singular values. These objectives enforee cosk-connistent LoRA generation while remaining robust to the inherent non-uniqueness of low-rank decompositions.
4

---

Published as a conference paper at ICLR 2026
Pretminned Parameters
welphr space
Router
Weighted
sum
Mol: Decoder
Downstream Evaluation
Repert
Train
I Latent Vector
Output
parmmettens
MoE Decoder
Ditfusion Trausformer
Medule Embhekklngg
Diffusion Transformer
paramiers
Encoder
Text Encoder
Moc Head
Layer Emmbedding
Encuder
t
Trik Deseriptions
Random Noise
Stugel. Training
HyperNetwork
LoRA
Uniseem Task
Descriitions
Stage2. Inference for Generation
Figure 2: Overall framework LoRAGen. Our approach consists of two stages: First, we train hypernetwork based on LoRA weight autoencoder to encode and reconstruct LoRA parameters, and diffusion process conditioned on natural language task descriptions to predict denoised latent. Second, random noise and un/seen natural language task descriptions are fed into LoR AGen to generate LoRA parameters, which can be incorporated with the LLM to evaluate downstream tasks.
Obs-2: Heterogeneity of LoRA weight distributions. Here we focus on the heterogeneity of LORA weight distributions across different module types in a transformer-based architecture, specifically the FLAN-T5-large model, which serves as the base model for LoRA adapters. For each adapter AWm.e at a modulc-layer location (), we analyze how its Frobenius en- engy distributed across singular directions. Let {az} denote the singular values of AWm,e and define the normalized spectrum p; of / oj. We then compute the spectral entropy Σ Pt log which quantifies the uniformity of energy over directions: low entropy indicates that the energy concentrates in few dominant directions, indicating effectively lower rank structure, whereas high entropy corresponds to a more even spread of energy (Roy & Vetterli, 2007; Yunis et al., 2024).
Hspee
For FLAN- T5 Large, we reports the spectral-entropy distributions for three module types in FLAN-22 T5 -5-1arree Encoder self-attention, Decode self-attention, and Cross-Atlention. We ob- serve systematic differences across modules: encoder adapters exhibit higher spectral entropy which means energy spread over more concentrated and effectively lower-rank structure, and cross-attention lies in between. This pattern implies that LoRA weight distributions not homogeneous across the network which means the way energy is distributed over singular directions differs consistently by module type. More results reported in Appendix A.3.
directions, decoder self-attention shows the lowest entropy with
Motivated by Obs-2, we employ a module-aware MoE decoder (Sec. 4.4)to match these module- specific patterns of energy distributions. Specifically, routing conditioned on structural embed- ding that combines latent variable with learnable module and layer embeddings. The decoder can operate with either (i) single shared expert pool for all locations or (ii) separate expert pools per module type (e.g., encoder attention). Per-module output heads H. map the gated expert outputs into LoRA parameters AW m,é- This architecture lets experts specialize to the observed module- specific energy distribution patterns (e.g., lower-entropyylow-ranl tendencies in Decoder-Self ver- higher-entropy Encoder), while the router and the chosen pool configuration ensures controlled sharing mechanism without losing modulc-specific specialization.
4.2 METHOО OVERVIEW
As shown in Figure 2, the overall framework of LoRAGen can be divided into stages:

---

Published as a conference paper at ICLR 2026
First, given pretrained LoRA parameters AW, the LAE (Eg. D3) encodes AW into per-location latents with a diagonal-Gaussinl posterior ( AW), and a module-aware MoE decoder Da (Sec. 4.4) decodes to full LoRA parameters ÁW. Unlike standard VAE that uses an construction likelihood, we train LAE using adapter-level structural supervision directly in the LoRA weight space (Sec. 4.3), i.e. direction and spectral losses defined on A W m,t- We then train a diffusion transformer to model the conditional prior (= | c) over LAE latents, where denote the embedding of a natural-language task description produced by the text encoder (Fig- ure 2). Specifically, we sample latent target 2g ~ 90(z AW), apply forward noising process ( 20) = 20, (1-āt)f), and train denoiser fg conditioned on e to predict the denoising
target:
Laitt (bb) = E Ben-ou()AAW:tt,-woo.. [lle- (z t, 11)]
(1)
where = Vārzo + - - ogC. Further diffusion details are reported in Appendix A.6.
Second, at inference, we feed the task description embedding c and Gaussian noise 2T -N(0 I) into the diffusion transformer to run the reverse process and obtain a denoised latent 2o P= c). We then pass 2g through the frozen MoE decoder Da from Stage- to generate full LoRA parameters for the LLM. Note that the LAE encoder is only used during Stage- 1 training to obtain latent targets; at inference we sample latents from the diffusion prior and decode them without using the encoder.
4.3 ADAPTER-LEVED SUPERVISION IN LORA WEIGHT SPACE
Distinet pairs of low-rank decomposition matrices (A, B) produce the same low-rank adapter AW. we choose to generate A and B separately, the LoRA generator must to commit to a specific decomposition of AW, even though many different low-rank decomposition matrices produce the identical adapter. To avoid this ambiguity induced by non-uniqueness property of the low-rank decomposition, we directly add supervision signal at the level of low-rank adapter
ÁW m. m,, and introduce two weight space loss terms: (i) a direction loss based on co- sine similarity that depends on the low-rank adapters after normalizing their Frobenius norm. (ii) spectral loss that matches the leading singular values of the predicted and pretrained low-rank adapters.
Direction loss. Because many pairs of low-rank decomposition matrices (A, B) produce the same low-rank adapter AW, supervising A and B individually can introduce arbitrary differences in the Frobenius norm of AW. To make supervision insensitive to it, normalize both the predicted and target low-rank adapters to unit Frobenius norm and compare their directions directly to capture the per-task direction. Thus we introduce direction loss that measures the angular mismatch between the predicted and target low-rank adapters:
(AW m,ey A
Lang (m, =1
(2)
J&W m.ell ||WWm,
where (r denotes the Frobenius inner product and ||
|| is the Frobenius norm.
Spectral loss. Direction supervision does not capture how the squared Frobenius norm is dis- tributed across the singular spectrum. Two low-rank adapters can have similar cosine similarity yet differ in their leading singular values, i.e., in the proportion of the squared Frobenius norm ex- plained by the top singular directions. To account for this, we introduce a spectral loss that matches the leading singular values of the two low-rank adapters:
o Likk (AW m,e) o 1.kkm. (AW ||
Lapec (m, E)
(3)
p.w...
where (i) (X) denotes the i-th singular value of X, and o (X) = (X) (X)) lists the top-k values in nonincreasing order; (ii) Tm, is the LoRA rank at location (m, {). and km, E {1...., Tm.e) is the smallest integer such that the top-k singular values of the target low-rank adapter explain at least fraction norm with p E {1,2); (iv) Wm.t = (Wm,f,1
(0, 1) of its squared Frobenius norm. (iii || is a weighted &p fmm.f are nonnegative normalized singular-
value weights.

---

Published as a conference paper at ICLR 2026
Thus, aggregate these two terms together as follows:
Ladapter (0, (6) E (Bgags()AAW))
Σ
Am,e CangLang E) Espechsspcc (m, (m, e))
(4)
mEE t=1
where (0(2 AW) is the encoder posterior; Orange Axper are hyperparameters balancing the two loss terms; and Am. 2 are location weights. Note that Lang, aligns direction, while Leper aligns the leading spectrum (i.e., the proportion of the squared Frobenius norm explained by the top singular directions). Thus Ladapter aligns the performance of the specific-task low-rank adapter yet remains robust to low-rank decompositions.
The overall objective of training LAE is
O .) = Ohadupter Ladapter (0, Φ) + DRI(g0(>) AW) [O ()) + Amoe Lmaee (0), (5)
where Oradapter: 9, Amoce > are scalar coefficients; Lmoe is the MoE load-balancing auxiliary loss(Sec. 4.4).
4.4 MODULE-AWARD MOE DECODER
To capture structural heterogeneity across the module types and layers, we further introduce a module-aware MoE decoder D, that or separate pools per module type (e.g., encoder attention). The notation below treats E' as the number of experts in the active pool used by the current location.
either single shared expert pool for all locations
Inputs and routing. For each location (m, f), we form structural embedding hm, E Cmi CE] IR",, where 2m, E Rd is the latent variable, 6m E IRd and E Rd are learnable module and layer embeddings. A router with parameters W, E REd,, outputs logits l,, W,hm, R. and applies top-K gating:
exp(lm,f,e/))
Ee'Esm. exp(lm.t.e/T) Il Sm,el.
where 7 0 is the temperature, S,n,t C .L...., E}} is the index set of the top-K experts by logit value, and I[-] is the indicator function (equal to if its argument is true and ( otherwise).
Experts and per-module heads. module head Hm:
Each expert & is small MLP. The gated output feeds per-
AW m.t Hm, eESm.
Σ
E
Note that Hm linear map into vector followed by reshape to the full LoRA parameters AW m. E (mddm(())))xddmmm.)) And its parameters are shared across all layers & for the same module m.
Load-balancing auxiliary loss. To discourage expert collapse we introduce load baa-nbalnncing aux- iliary loss:
Lmoc max E EE 0)
where Pe is the average gating probability, and Fo the expected fractional load under top-K K rout- ing. The details implementation of Lmos is reported at the Appendix A.1I.
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Dataset. In our main experiments, we consider two settings. First, we employ FLAN-TS-Large (Chung al., 2024), as the base model. We utilize subset of FLAN following (Lyv et al., 2024)

---

Published as a conference paper at ICLR 2026
Method
AP-Neg
AP-Rec
AP-Pos
QASC-1
QASC-2
WQA-T
WQA-A
Avg. (acc)
FLAN-T5 -Large
49.4
70.7
34.8
23.3
8.9
36.8
Average LoRA
96.8
96.5
96.9
99.4
87.3
96.7
97.0
95.8
D2NWG
59.5
87.3
65.9
47.0
31.1
33.4
84.8
58.4
T2L
90.5
94.1
92.7
87.9
76.8
85.5
93.3
88.7
Ours
96.8
96.6
97.1
99.5
87.3
97.1
97.3
96.0
Task misk-specific LoRAs
97.0
99.7
87.3
97.0
96.2
Table I: Benchmark performance of LoRAGen on FLAN subset 1PLAN-T5-LaS backbone). Bold numbers used when the performance is higher than the task-specific LoRAs.
Method
AreC
ArcE
BQ
GSM8K
OQA
PIQA
WG
Avg. (acc)
Gemma- 2imima---2B-nstrrue
74.0
89.9
81.0
55.9
55.1
71.2
71.2
51.8
68.8
Average LoRA
75.6
90.1
81.4
56.5
72.8
53.9
70.0
D2NWG
74.1
90.0
81.2
56.0
71.3
52.0
68.9
T2L
74.3
90.2
81.2
55.9
55.2
71.4
71.5
53.8
69.2
Ours
76.6
90.7
84.1
56.4
64.1
80.2
75.0
54.2
72.7
Task specispe LoRAs
76.7
90.6
84.7
55.9
75.4
80.2
78.0
54.6
74.5
Table 2: Benchmark performance of LoRAGen on 8 benchmark tasks (Cemma-2-2B-nne
backbone). Bold numbers are used when the performance is higher than the task-specific LoRAs.
for training and evaluation. Second, we use Gemma 2b-Instruct (Team et al., 2024) as the base model and evaluate on widely used benchmark tasks, including Are-challenge (AreC) and Arc-easy (ArcE) (Clark al., 2018), BoolQ (Clark et al., 2019), GSM8K (Cobbe et al., 2021), Hellaswag (HS) (Zellers et al., 2019), OpenBookQA (OQA) (Mihaylov et al., 2018), PIQA (Bisk al., 2020), and Winogrande (WG) (Sakaguchi al., 2021). More details about the datasets reported the Appendix. For both settings, we extract task emheddings from natural language task descriptions using the FLAN-T5- large encoder. Task descriptions for each dataset fully generated by LLM, as described in the Appendix A. 10. For each dataset, We sample and report the average performance of set of LoRA weights sampled with LoRAGen.
Baseline Setup. As baselines, we consider task LoRAs, reight-avveaged LoRA (Wortsman al., 2022). We compare D2NWG (Soro et al., 2024), which is latent diffusion conditioned on datasets for LoRAs generation. We also implement T2I. (Charakorn et al., 2025), which is a hypernetwork that generates LoRAs based on natural language task embedding. Reproduction details are provided in the Appendix A.
5.2 PERFORMANCE COMPARISON
In-distribution LoRA Generation Performance. First, we focus on whether LoRAGen can re- cover the performance of trained LoRAs (Charakorn al., 2025; Brüel-Gabrielsson et al., 2024), which enables low-rank adaptation with minimal compute requirements. Table reports results on seven FLAN tasks using natural-language task embeddings from the FLAN-T5 10a-TT5-Laree Large encoder. LoRAGen closely matches the oracle adapters and outperforms D2NWG and T2L on average perfor- mance. We think that the gain comes from the design of MoE decoder to capture the heterogeneity of weight distributions across different module types and layers within the structure-aware LoRA A weight space.
Moreover, we train separate LoRAGen model for decoder-only base model (Gemma 2- Instruct) to assess whether the same structure-aware design generalizes across architectures. As shown in Table 2, LoRAGen remains competitive superior to several baselines and close to task-specific LoRAs on average performance, indicating that our approach scales from T5-based adapters to decoder-only adapters. In several tasks. such as ArcE, GSM8K and OQA, our method even matches or surpasses task-specific adapters, suggesting that adapter-level supervision captures task-relevant structure rather than memorizing particular LoRA parameters.
8

---

Published as a conference paper at ICLR 2026
Method
AP-Rec
AP-Pos
QASC-I
QASC-2
WQA-T
WQA-A
Avg. (acc)
FLAN- T5 Large
70.7
34.8
23.3
8.9
8.5
62.0
34.7
D2NWG
71.8
34,4
23.7
9.1
8.3
62.4
35.0
T21.
71.6
34.6
23.3
11.1
8.5
62.0
35.2
Ours
75.1
42.2
28.1
14.5
14.3
67.2
40.2
Table Zero-shot generation performance of LoRAGen trained on the FLAN subset with
F FLAN-T5-Lad Ler as the base model. Bold numbers are used to represent the best performance.
Lang Lapec
D,
AP-Neg
AP-Rec
AP-Pos
QASC-1
QASC-2
WQA-T
WQA-A
Avg. (acc)
d
X
59.5
87.3
65.9
47.1
31.2
33.4
84.9
58.4
X
/
96.8
95.5
97.0
98.2
86.3
95.6
96.7
95.2
/
49.5
70.9
34.4
62.7
36.9
/
/
96.8
96.6
97.1
99.5
87.3
97.1
97.3
96.0
Table 4: Ablation study FLAN subset. Checkmarks indicate enabled components: direction loss faagg spectral loss Luper, and module-aware MoE decoder 'Dg. Bold numbers are used to represent the best performance.
LoRA Generation for unseen tasks. Furthermore, we explore whether LoRAGen can generate LoRA parameters for unseen tasks. We train LoRAGen on 136 tasks of FLAN subset, each with 20 task descriptions. For each task we sample three sets of LoRA weights and report the average accuracy. As shown in Table 3, LoRAGen achieves the best average accuracy 40.2, outperforning D2NWG and T2L. by +5.2 and +5.0 points, respectively. We observe that D2NWG and T2L recon- structs pre-tras pre-trained adapters and struggles to generalize to unseen tasks. This phenomenon is align with non-umine unnquiqqueess of the low-rank decomposition, which indicates that if supervise the low- rank decomposition matrices directly, training becomes sensitive to arbitrary rescalings and rotations that still yield the same full adaptation matrix, trending memorize task-specific LoRA parameters. Instead of this, our method supervise the full adaptation matrix directly to avoid this ambiguity and focus learn task-relevant weight space learning, resulting better performance unseen tasks. Details on computational cost and efficiency are reported in the Appendix A.4.
5.3 ABLATION STUDY
In this section, ablate the adapter-level supervision and the module-aware MoE decoder on the seven FLAN tasks (Table 4). Training with the two adapter-level losses Lang and Lepee with- out the decoder Dø achieves an average accuracy of 58.4. In contrast, enabling the decoder while removing both adapter-level losses and using only the reconstruction loss results in a significant improvement to an average accuracy of 95.2. This improvement is consistent with Obs. 2: the module-aware routing and per-module heads in D, effectively capture the heterogeneity in weight distributions across modules and layers, which substantially improves downstream performance.
counter-intuitive result occurs when Dg is combined with the spectral loss but the direction loss omitted: the average accuracy drops to 36.9. In this case, Lapec only enforces alignment of the magnitudes of the top-k singular values, without constraining the corresponding directions of the left and right singular vectors. As result, the decoder can match the singular-valut magnitudes correctly, but assign them to the wrong directions, which leads to an incorrect weight-space learning and consequently poor task performance. In the shot setting reported Appendix A.2, Table 5, the same combination yields much larger gain of +3.6 points in average accuracy (from 36.6 to 40.2), again with the largest improvements on QASC-1, QASC-2, and WQA-T, confirming that adapter-level supervision is particularly important for generalization to unseen tasks.
5.4 DETAILED ANALYSIS
Hyperparameter Analysis. We first analyze the effect of the spectral-energy threshold p in the spectral loss as shown in Figure 3(a). We conduct this experiment under the same setting as Table Across all tasks, accuracies remain stable as varies from 0.80 to 1.00, with only minor fluctuations.

---

Published as a conference paper at ICLR 2026
This shows that the adapter-level supervision is robust the choice of p. since the leading singular values already capture sufficient spectral information. In practice, we set = 0.85 as it provides a better performance while maintaining stability.
We then examine the hyperparameters of the MoE decoder as shown in Figure 3(b). Here we com- pare shared vs. unshared expert pools, different top- routing choices, and the number of experts. The results indicate that unshared pools consistently outperform shared ones, and increasing the number of active experts (top-2 vs. top-1) further improves performance, especially on challeng- ing multi-choice and QA tasks. The best configuration is unshared, top-1, E = 4, which strikes a balance between accuracy and computational efficiency.
Structural Embedding Analysis. In this section, we assess the contribution of the structural em- bedding in the MoE decoder shown in Figure 3(c). We compare three variants: (i) without structural embedding (routing only on latent variables), (ii) with structural embedding, and (iii) an oracle trained with task task-specific adapters. The results show that removing structural embeddings substantially reduces accuracy, particularly on tasks requiring fine-grained reasoning such as QASC- 1/2. Adding structural embeddings closes most of the gap to the oracle, confirming that encoding module-specific latent is critical for capturing the heterogeneous LoRA weight distributions.
1.00
2B2
100
Basafiddi
Check
Accurscy (h)
Accurai
60-
20
W014
gAC033
WOAY
(AJfpcctral Threshald
(b/7aska
IcTaako
Figure 3: Performance of LoRAGen with different hyperparameters and removing structural em- bedding. Here, for the middle figure, s / uns denote shared or unshared MoE experts; t1 / t2 denote top-1 or top-2 routing strategies; e4 e8 denote using 8 experts.
6 DISCUSSION AND CONCLUSION
We presented LoRAGen, a structure- aware approach to LoRA parameter generation grounded in two empirical observations of the LoRA weight space: non-uniqueness of low-rank decompositions and module-wise heterogeneity of weight distributions. Motivated by these observations, we su- pervise the full adaptation matrix using adapter-level direction and spectral losses, and decode with a module-aware MoE whose routing leverages structural embeddings with shared or per-module expert pools. Empirically, LoRAGen produces strong in-distribution adapters across architectures, closely matching task-specific adapters on FLAN-T5- 1 arge and (-emma2--222 Instruct. and attains competitive zero-shot performance unseen tasks from large LoRA library. Ablation studies show both adapter-level supervision and module-aware decoding are necessary, and sensitiv- ity studies indicate robustness the spectral-energy threshold and gains from unshared pools with top-K routing. Our results suggest that modeling adapters in the AW space with structure-aware supervision is a promising direction for parameter generation. Nevertheless, LoRAGen still exhibits a performance gap on unseen tasks, indicating that the current conditioning signal and prior may not fully capture the distribution shift implied by novel task descriptions. As future work, we will close this performance gap by improving the task- dask-desci encoder and incorporating richer supervi- sion to better align language descriptions with adapte structure and exploring the pool of training adapters and task families to effectively cover wider task manifold. In addition, diffusion provides a flexible conditional prior but can be computationally expensive when many denoising steps are required. We therefore plan to investigate whether rectified-flow consistency-basee priors can further reduce sampling time for large-scale adapter generation, while retaining the diversity and controllability benefits of LoRA generation.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China under U24B20180.
ETHICS STATEMENT
The authors have read and adhered to the ICLR Code of Ethics. The research presented in this paper focuses the generation of LoRA parameters, with primary applications in LLM domains.. All data used for training and evaluation is from publicly available, non-personal scientific datasets, ensuring privacy concerns. This work does not involve human subjects, and we do not foresee any direct negative societal impacts or risks of perpetuating social biases. Our aim is to advance the development of domain-specifie LLM applications.
REPRODUCIBILITY STATEMENT
The code associated with this paper is available at: https://gitt http//tttuu...m com/ ts Esinghua-fib-1 fitaa--iiiilaaaaoRGGeen
includes the necessary environment configurations and execution scripts. All datasets utilized in this work are publicly accessible. The task descriptions generated from LLM is provided in the Appendix.
REFERENCES
Gabriele Berardi, Luca De Luigi, Samuele Salti, and Luigi Di Stefano. Learning the space of deep models. arXiv preprint arXiv:2206.076800 2022.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical com- monsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7477 7439, 2020.
Rasmus Brbbil-Gari,elsson Jingzhao Zhu, Omkar Bhardwaj. Leshem Choshen, Kristjan Gree- newald, Mikhail Yurochkin, and Justin Solomon. Compress then serve: Serving thousands of lora adapters with little overhead. arXiv preprint arXiv:2407.00066 2024.
Rujikorn Charakorn, Edoardo Cetin, Yujin Tang, and Robert Tjarko Lange. Text-to-lora: Instant transformer adaption. In Proceedings of the 42nd International Conference Machine Learn- ing (ICML 2025), Poster Track, 2025. URL https: https:// wttpp//pwwwweeeeeee/ppeeeoeeeewp net/ forum?ids zWskCdu30A. Poster.
Rujikomn Charakorn, Edoardo Cetin, Shinnosuke Uesaka, and Robert Tjarko Lange. Doc-to-lora: Learning to instantly internalize contexts. arXiv preprint arXiv:2602. 15902, 2026.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, al. Scaling instruction-finetunee lan- guage models. Journal of Machine Learning Research, 25(70): 2024.
Christopher Clark, Kenton Lee, Ming-Was Wei Chang, Tom Kwiatkowski. Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. 2018.
arXiv preprint
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint 1rXi:22110.14168 2021.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learmin for fast adaptation of deep networks. Proceedings of the 34th International Conference on Machine Learning, 2017.
11

---

Published as a conference paper at ICLR 2026
Rickard Brüel Gabrielsson, Jiacheng Zhu, Onkar Bhardwaj. Leshem Choshen, Kristjan Gree- newald, Mikhail Yurochkin, and Justin Solomon. Compress then serve: Serving thousands of loRA adapters with little overhead, 2024. URL httpss:/ppperreiee...eet net www.:nnnne.ootiieooooooooom forum?id= hHNVn4hFPk.
Saumya Gupta, Scott Biggs, Moritz Laber, Zohair Shafi, Robin Walters, and Ayan Paul. Deep- weightflow: Re-basined flow matching for generating neural network weights. arXiv preprint arXiv:2601.05005. 2026.
David Ha, Andrew Dai, and Quoe V Le. Hypemnttorks.. arXiv preprint 1609.06609009000 2016.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- King-Kikkpatrick, and Graham Neubig. Towards unified view of seaametr--fffiienn transfer learning. In International Conference on Learning Representations, 2022a. URL https::////.
http:p:///pppreee /ttp////eeeeeeeeeee//eee/eeeeeee net/ Eorum wwwprmmd==ORRR Ld- wwwp:/nee//orurtiee/eeeee
Yelong He, Shuang Zheng, Yi Tay, Jatin Gupta, Yichong Du, Vamsi Aribandi, Zhen Zhao, Yichong Li, Zhiyu Chen, Donald Metzler, et al. Hyperprompt: Prompt-based task-conditioning of trans- formers. In International Conference on Machine Learning, pp. 8678-8699 PMLR, 2022b.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR 2022, April 2022. URL https://www mi wwwoosoottt......om..mooo com. cin-us/researe publication//e /oora-rra--- low rank adappatiapttinnnno of
modell//
Hamish Ivison and Matthew E. Peters. Hyperdecoders: Instance-specific decoders for multi-task nlp. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 1715-1730, 2022. URL https:t////mmm https///allaaaahoocoooeeo www.aaattttoooooooooooooooooee/e
org/ wttp:o//.....n....n... findings- emnlp. 1.nnnp..... 124.
Hamish Ivison, Akshita Bhagia, Yizhong Wang, Hannaneh Hajishirzi, and Matthew E Peters. Hint: Hypernetwork instruction tuning for efficient zero- and few-shot generalisation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11272-11288, 2023.
Zhihao Jin et al. Conditional lora parameter generation. arXiv preprint arXiv:2403. 2024.
Marios Kofinas, Boris Knyazev, Yulun Zhang, Yiming Chen, Gertjan Burghouts, Efstratios Gavves, Cees GM Snoek, and Dingwen Zhang. Graph neural networks for learning equivariant represen- tations of neural networks. In International Conference Learning Representations, 2024.
Ruikun Li, Huandong Wang, Jingtao Ding, Yuan Yuan, Qingmin Liao, and Yong Li. Predicting dynamical systems across environments via diffusive model weight generation. arXiv preprint arXiv:2505.13919, 2025.
Zhiyuan Liang, Dongwen Tang, Yuhao Zhou, Xuanlei Zhao, Mingjia Shi, Wangbo Zhao, Zekai Li, Peihao Wang, Konstantin Schürholt, Damian Borth, et al. Prag-and-drop llms: Zero-shot prompt-to-weights. arXiv preprint arXiv:2506. 2025.
Huanxuan Liao, Shizhu He, Yao Xu, Yuanzhe Zhang, Yanchao Hao, Shengping Liu, Kang Liu, and Jun Zhao. From instance training instruction learning: Task adapters generation from instructions. Advances in Neural Information Processing Systems, 37:45552-45577, 2024.
Yewei Liu, Xiyuan Wang, Yansheng Mao, Yoav Gelbery, Haggai Maron, and Muhan Zhang Shine: A scalable in-context hypernetwork for mapping context to lora in a single pass. arXiv preprint arXiv:2602.06358, 2026.
Chuancheng Lv, Lei Li, Shitou Zhang, Gang Chen, Fanchao Qi, Ningyu Zhang, and Hai-Tao Zheng. Hyperiora: Efficient cross-tass generalization via constrained low-rank adapters generation. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 16376-16393, 2024.
Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. Parameter- efficient multi-task fine-tuning for transformers via shared hypernetwors.. arXiv preprint arXiv:2/066 04489, 2021.
12

---

Published as a conference paper at ICLR 2026
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? a new dataset for open book question answering, arXiv preprint arXiv:1809.02789, 2018.
Elia Nava, Shogo Kobayashi, Yufei Yin, Raffaello Katzschmann, and Benjamin F Grewe. Meta- learning via classifier(-free diffusion guidance. arXiv preprint arXiv:2210.0566 2022.
nin-Gilberto Ortiz- Briiz-Brraja,, Helena Gomez Adomoo, and Thamar Solorio. Hyperloader: Integrating hypernetwork-baeed lora and adapter layers into multi-task transformers for sequence labelling. arXiv preprint arXiv:2407.014/1 2024.
Oleksiy Ostapenko, Zhan Su, Edoardo Ponti, Laurent Charlin, Nicolas Le Roux, Lucas Caccia, and Alessandro Sordoni. Towards modular ilms by building and reusing library of loras. In Proceedings of the stst International Conference Machine Learning (ICMI.), volume 235, pp- 38885-38904. PMLR, 2024.
Junho Park, Dohoon Kim, and Taesup Moon. Prisp: Privacy-safe few-shot personalization via lightweight adaptation. arXiv preprint arXiv:2601.06471, 2026.
William Peebles, Ilija Radosavovic, Tim Brooks, Alexei A Efros, and Jitendra Malik. Learning to learn with generative models of neural network checkpoints. arXiv preprint arXiv:2209. 2022.
Jason Phang, Yuning Mao, Pengcheng He, and Weizhu Chen. Hypertuning: Toward adapting large language models without backpropagation. In International Conference on Machine Learning, pp. 27854-27875. PMLR, 2023.
Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In 2007 15th European signal processing conference, pp. 606-610. IEEE, 2007.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Daniel Saragih, Deyu Cao, Tejas Balaji, and Ashwin Santhosh. Flow learn: Flow matching on neural network parameters. arXiv preprint arXiv:2503.19333 2025.
Simon Schug, Seijin Kobayashi, Yassir Akram, Joao Sacramento, and Razvan Pascanu. Attention as hypernetwork. In The Thirteenth International Conference on Learning Representations, 2025.
URL https: http:::://ppeeevviieee net// forum? iidiidd// 1dv44K9111x..
Konstantin Schürholt, Mostafa Dehghani, Neil Houlsby, and Damian Borth. Hyper-repressetaation:: Learning generative representations neural network parameters. In International Conference on Learning Representations (ICLR), 2021a. URL https: httpp::/openreviiii hwtp///ppnnreeeeeeeeeeeeeeoe..ep net forum?
HddHCSgyPUfeDj.
Konstantin Schürholt, Dimche Kostadinov, and Damian Borth. Hyperrseppeesentation::: Self- supervised representation learning on neural network weights for model characteristic prediction. 2021b. URL http:://aalllsemmniiiccooaaa org Cttp://orrssII.I...
240070334.
Konstantin Schürholt, Boris Knyazev, Xavier Giró-i Nieto, and Damian Borth. Hyper- representations generative models: sampling umseen neural network weights. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088.
Konstantin Schürholt, Boris Knyazev, Xavier Giró-i Nieto, and Damian Borth. Hyper- representations for pre-training and transfer learning. arXiv preprint arXiv:2207.1099 2022.
Konstantin Schürholt, Michael W. Mahoney, and Damian Borth. Towards scalable and versatile weight space learning. In International Conference Machine Learning (ICMI... PMLR, 2024. URL https:ttp///... /tap::://aree / /ttpxvivvvo////////////// orgo abs /22ss/2446..m 09997.
Jing Shao et al. In-context meta lora generation. arXiv preprint arXiv:2502.08976.00 2025.
13

---

Published as a conference paper at ICLR 2026
Bedionita Soro, Bruno Andreis, Hayeon Lee, Wonyong Jeong, Song Chong, Frank Hutter, and Sung Ju Hwang. Diffusion-based neural network weights generation. arXiv preprint arXiv:2402.18153, 18153, 2024.
Bedionita Soro, Bruno Andreis, Hayeon Lee, Wonyong Jeong, Song Chong, Frank Hutter, and Sung Ju Hwang. Diffusion-based neural network weights generation. In The Thirteenth Interna- lional Conference on Learning Representations, 2025. URL httt::::///penrevieeee
JM9
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, al. Gemma 2: Improving open language models at a practical size. arXiv preprint WxXi:2248.000/66 18, 2024.
Thomas Unterthiner, Daniel Keysers, Sylvain Gelly, Olivier Bousquet, and llya Tolstikhin. Predict- ing neural network accuracy from weights. arXiv preprint arXiv:2002. 1448, 2020.
Jun Wang, Yifan Chen, Stella Yu, Brian Cheung, and Yann LeCun. Compact and optimal deep learning with recurrent parameter generators. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2023.
Kai Wang, Dongwen Tang, Wangbo Zhao, Konstantin Schürholt, Zhangyang Wang, and Yang You. Recurrent diffusion for large-scale parameter generation. arXiv preprint 10XXiv:2501.118777 2025.
Kaitian Wang, Zhaopan Xu, Yukun Zhou, Zelin Zang, Trevor Darrell, Zhuang Liu, and Yang You. Neural network diffusion, 2024. URL https:/ /ttp://///peeeeeeeeoooo net formm ?id= 826UmFhhQS.
Piotr Wójcik, Maksym Petrenko, Wojeiech Gromski, Przemyslaw Spurek, and Maciej Zieba. Un- hype: Clip-guided hypernetworks for dynamic lora unlearning. arXiv preprint arXiv:2602.03410, 2026.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference machine learning, pp. 23965-23998, PMLR, 2022.
Yujia Wu, Yiming Shi, Jiwei Wei, Chengwei Sun, Yang Yang, and Heng Tao Shen. Difflora: Gener- ating personalized low-rank adaptation weights with diffusion. arXiv preprint FrXii:2408.06744 2024.
Yicheng Xiao, Lin Song, Rui Yang, Cheng Cheng, Yixiao Ge, Xiu Li, and Ying Shan. Lora-gen: Specializing large language model via online lora generation. arXiv preprint arXiv:2506. 1638, 2025.
Zhaofeng Xiao, William Held, Yutong Liu, and Diyi Yang. Task-agnostic low-rank adapters for unseen english dialects. In Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing, pp. 7857-7870. Association for Computational Linguistics, 2023. doi: 118855/v1/2023.com
Yuan Yuan, Chenyang Shao, Jingtao Ding, Depeng Jin, and Yong Li. Spatio-temppral few-shot learning via diffusive neural network generation. arXiv preprint arXiv:2402. 1192, 2024.
David Yunis, Kumar Kshitij Patel, Samuel Wheeler, Pedro Savarese, Gal Vardi, Karen Livescu, Michael Maire, and Matthew R Walter. Approaching deep learning through the spectral dynamics of weights. arXiv preprini arXiv:2408. 1804, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a ma- chine really finish your sentence? arXiv preprint 1xXv::9055007300 2019.
Chris Zhang, Mengye Ren, and Raquel Urtasun. Graph hypernetworks for neural architecture search. arXiv preprint arXiv: 2018.
Bo Zhao, Robin Walters, and Rose Yu. Symmetry in neural network parameter spaces. arXiv preprint arXiv:2506. XXX::2506.130/8. 2025.
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 DRTAILS FOR OBS-
As shown in Figure 4, provide more 9 representative tasks to compare with the remaining 111 tasks on the FLAN subset. These representative tasks are selected so that they cover the three major task categories present in the FLAN subset: Natural Language Understanding (NLU, 98 tasks), Natural Language Generation (NLG, 9 tasks), and Knowledge & Information Extraction (5 tasks). Specifically, we include five NLU tasks wrikiqa.tpqag wikiqa-jeop, fix-punct, t true.case, word.segment) three NLG tasks (duorc.tgen, duore-gqba, gem_e2e_nlg and two Knowledge & Information Extraction tasks (wiki wit.biiowwaaaa ensures that the visualization reflects the diversity of the dataset rather than any specific task.
miki.bio.comp) This
Concretely, for representative task and another task i, we define the task similarity as the cosine similarity between their text-description embeddings e(r) and e(t):
e(r) Te(t)
Stask (r, () []e(r)]l ||e(0]22
For adapter similarity, we consider two variants. Let AW, and AW, be the full adaptation matrices for tasks r and t, and let A,, B, and As, Be be their corresponding low-rank factors. We compute
()e)(AW,), vec(AW.))
(r = |vec(AW vec( (AW))
(10)
and
(Ur, vx)
vec(A.) (Ae) vecl B+)
(A.B))r, ||o=|l2 ||u:|22'
(11)
We then report the Spearman correlation coefficient p between {susk (r, t)} and each of (saw (r, t)} and (s(A))) (r, (r, } over all t.
We found that the full adaptation matrix similarity shows a consistent positive correlation with task description embedding similarity. This suggests that tasks with more semantically similar descrip- tions tend to produce more similar LoRAs, highlighting the presence structure the LoRA weight space. This observation also emphasizes the utility of the full adaptation space, it allows us to capture and exploit underlying similarities between tasks that may not immediately apparent from the low-rank decomposition matrices. By designing weight space loss based this similarity, we can better adapt models to generate new LoRA parameters. We note that the absolute cosine simi- larity between two concatenated full adapters can be close to zero due the high dimensionality of the concatenated vector; therefore, Figure 4 should be interpreted in terms of correlation rather than magnitude.
A.2 ABLATION STUDY ON ZERO-SHOT GENERATION
We further examine how the adapter-level losses affect zero-shot generalization on the seven FLAN tasks that are used in Table 1. As summarized in Table 5, enabling only the MoE decoder with reconstruction loss (i.e., using DD without Lang or Lapee) achieves an average zero-shot accuracy of 36.6. Adding both direction and spectral losses on top of the decoder increases the average accuracy to 40.2 (+3.6 points), with particularly gains on QASC-1 (+3.6), QASC-2 (+4.1), and WQA-T (+4.1). These tasks are among the most challenging reasoning benchmarks in our evaluation, so the improvements provide direct evidence that adapter-level supervision is especially beneficial for zero-shot generalization.
A.3 WEIGHT DISTRIBUTION ANALYSIS
To further investigate the effect of the module-aware MoE decoder, we analyze the weight distribu- tions of task-specific LoRAs and the adapters generated by LoRAGen on the same evaluation tasks as in the main tables. For the encoder-decoder base model FLAN-T5 -5-rge,, we use the seven - we
FLAN tasks reported in Table 1. For the decoder-only base model use the eight benchmark tasks reported in Table 2. In each case, compare three variants: (i) task- specific (oracle) LoRAs, (ii) LoRAGen (full model), and (iii LoRAGen without the MoE decoder (a single shared decoder).
15

---

Published as a conference paper at ICLR 2026
140
143 (-404))
24 (1
Worl 314
5463
1.3
24.0
212
Decrsdeen Borlerty
Taxt Seurrptin tebk Beelecty
6.01
0.2
0238
2-1444)
060
045
280
38
'nd fook Dedarit
Sab Brurraionn fans Eadurty
0.4
c300
2cm
100
cm
1:00
1:000
94 3.Bttdd 302
6200
1190
WWir- m9
-Wipel 622
6.00
6.00
besrippiin
Desoriation
Figure 4: Relation between LoRA similarity and task description embedding similarity. Each panel shows the similarity between representative task adapter and the other 11l adapters trained on a FLAN subset y-axis in the weight space against their similarity in the task embeddings space r--axis. LoRA cosine similarity is measured in two ways: (i) computing cosine on the low-rank decomposition matrices A and B separately (blue dots); (ii) computing cosine on the full adaptation matrix AW AB (orange triangles). Legends report the Spearman correlation coefficient p.
Lang
Lapec
Do
AP-Ree
AP-Pos
QASC-1
QASC-2
WQA-T
WQA-A
Avg. (acc)
X
/
72.7
35.9
24.5
10.4
10.2
65.9
36.6
1
/
75.1
42.2
28.1
14.5
14.3
67.2
40.2
Table 5: Zero-shot ablation study on the FLAN subset. Checkmarks indicate enabled components: direction loss Langs spectral loss Lapees and module-aware MoE decoder Dg. Bold numbers repre- sent the best performance.
Spectral entropy distribution analysis. For each adapter AWm.t compute its spectral en- tropy (Aee).Amm.t) For FLAN-TS-Large, Figure 5 shows the distributions of Hspec grouped by module type (Encoder, Decoder-Self, Cross-Attention) for the three variants above. Task-specific adapters exhibit clear heterogeneity: encoder modules have the highest spectral entropy, decoder self-attention the lowest, and cross-attention lies in between. The adapters generated by the full
16

---

Published as a conference paper at ICLR 2026
LoRAGen model faithfully capture this pattern, indicating that the MoE decoder has learned module- specific spectral profiles. In contrast, removing the MoE decoder collapses the three module types into almost identical high-entropy distributions, suggesting that single shared decoder fails to cap- ture the heterogeneous energy patterns across modules.
Figure 6 plots the spectral entropy distributions of decoder adapters for oracle LoRAs, LoRAGen, and LoRAGen without MoE. Task-specific LoRAs concen- trate around relatively high entropy values, and the full LoRAGen model shows very similar, slightly higher-entropy profile. In contrast, removing the MoE decoder shifts towards lower entropy and yields a much broader distribution, indicating that the shared decoder fails to match the spectral distribution of task- task-specific LoRAs.
For
Layer-wise cosine similarity analysis. Beyond spectral statistics, we also study the alignment between generated and task-specific adapters. Figures 7 and 8 report, for each layer and mod- ule type, the average cosine similarity between the generated adapter and the corresponding task- specific adapter, averaged over the seven FLAN tasks Table and the eight benchmark tasks in Table 2, respectively. With the full LoRAGen model, cosine similarities are consistently positive for FLAN-T5-Large and Gemma 2-----2---n- 1e----Innstucch indicating that the experts capture mean- ingful, layer-specific adaptation patters.. In contrast, the model without the MoE decoder yields similarities that remain close to zero across almost all layers, showing that fails to recover the fine-grained per-layer structure distribution tasks. Thus, the spectral-entroyy and cosine-similarity analyses confirm that the MoE decoder learns specialize to module types and their characteristic spectral patterns.
though it can still perform reasonable reconstruction
in-
speecpeecicc LoKAs
Onz
Ours (WL Moll
Figure 5: Spectral entropy distributions on FLAN-T5-Large Each group of violins corresponds Encoder, Decoder-Self, and Cross-Attention modules, respectively, for task-specifie LoRAs (left), LoRAGen (middle), and LoRAGen without the MoE decoder (right).
Spectral entropy distribution
Spectral entropy distribution
Spectral entropy distribution
1.75
1.8
1.50
1.-
1:55
Spectral entropy
3.25
Spectrol
1.50
2.00
1.0
1.25
0.8
030
DecaderSelf
DecadenSSlff Module Types
DecoderSelf Module Typea
Moduls Types
Task-specific LoRAs
Ours
Ours (w.o MoE)
Figure 6: Spectral entropy distributions on Gemma 2-2B Instruct for Decoder-Self module. Pask-specific LoRAs (left), LoRAGen (middle), and LoRAGen without the MoE decoder (right).
17

---

The cosine similarity between two adapters can be computed efficiently as Published as a conference paper at ICLR 2026
Ours
Ours (w.o MoE)
Figure 7: Layer-wise cosine similarity between generated and task-specific LoRA adapters on the FLAN subset with FLAN-T5- Larage as the base model. For each module type (rows) and layer index (X-axis), we plot the average cosine similarity over evaluation tasks. Left: LoRAGen (full model); right: LoRAGen without the MoE decoder.
Ours (w. MoE)
Figure 8: Layer-wise cosine similarity between generated and task-specific LoRA adapters on 8 benchmark tasks with Gemma 2---2-2-Tneee ruct as the base model. Left: LoRAGen (full model); right: LoRAGen without the MoF decoder.
A.4 EFFICIENCY OF ADAPTER-LEVES SUPERVISION
natural concern is that supervising adapters directly (i.e. losses defined AW ABT Rdxd) could be more expensive than reconstruction on A, E Rd We clarify that our implementation avoids any ((d2) complexity by using quadratic form, and thus remains comparable in cost to reconstruction loss.
Direction loss. quadratic form:
AWa) - B2A3) A
(A1, (B] B2)A4),
with | Ar B). This requires only r and r x r multiplications, yielding complexity (dr²) instead of (dd)). Hence the direction loss more expensive than element- wise reconstruction.
Spectral loss. For the spectral loss, we never compute an SVD the full adapter. Instead, we compute reduced QR decomposition. This reduces the problem to x core matrix K R R.. on which we perform SVD. The resulting complexity is O(dr + r²)) avoiding any (d)) cost.
Complexity comparison. Table 6 reports the per-layer complexities. Both direction and spectral losses are implemented in quadratic or QR decomposition form, avoiding explicit O(d?) cost. The overhead relative to element- reconstruction loss is bounded by a factor of r, which is small in practice.
18

---

Published as a conference paper at ICLR 2026
Loss term
Per-layer complexity
A, B reconstruction Direction (quadratic)
(dr)) Oldr²) (d + +
Spectral (QR+SVD core)
Table Per-layer complexity of different adapter-level supervision terms. Both direction and spec- tral losses avoid explicit (dd²) cost.
Table 7: Stage
LoRA autoencoder configuration for FLAN-T3-Large.
Item
Configuration
Backbone
FLAN-T5-Large (encoder-decodert)
Encoder hidden size
dmodel 1024
LoRA rank
r = 16 d, = 64 (per module)
Latent dimension
Encoder MLP hidden size
128
Decoder type
Module-aware MoE, 3 heads (enc, dec-self, cross)
Experts per head
==
Active experts
top-K 1 per location
Expert pools
Separate pool per module type (no sharing)
Expert MLP input
2d, 128 (latent structural embedding)
Expert MLP hidden size
256 (SiLU activation)
Expert MLP output size
r dmnddll = 16,384
Gating
Softmax, temperature 1.5
MoE aux loss
Load-balancing, weight 10-5
Encoder params
2.1M
Decoder (MoE) params
~ 50.9M
Total generator params
53M (Ne 7% of backbone)
A.5 COMPUTATIONAL COST AND RESOURCE USAGE
We will quantify the practical computational cost of LoRAGen and compare it with the baseline T2L. All experiments for FLAN-TA -----T----geeeee nad Gemma 22-2B- IBBInstruct are run on a single NVIDIA A40 GPU (40GB).
Training time. For the in on-disrribution experiments in Tables and 2, Stage- training for T2L and LoRAGen takes about 2 and 2.5 hours, respectively. For the zero-shot experiments in Table 3, the total training time of both methods is less than one day on the same A40 GPU.
Generator parameter counts. For MLAN-T5-Largee T2L and LoRAGen (Stage 1) have com- parable generator sizes (50M 53M), while LoRAGen already achieves better zero-shot perfor- mance. For Gemma 1omma-22 2----------- 1ee-2-IInstran Instruct, we intentionally use a larger generator for LoRAGen (69M) than for T2L (30M) that the MoE decoder has enough capacity to generate all four types of LoRA weights, namely qa, qb, va and vb. Notably, the activated parameters of LoRAGen are 15M and 22M per forward pass, respectively.
Memory usage. During LoRA weight autoencoder (LAE) training, enabling the module-aware MoE decoder introduces about 1000 MiB of additional GPU memory, and the adapter-level losses introduce about 300 MiB. In practice, the complete LoRAGen training requires roughly 2500 MiB of extra memory for these components.
A.6 IMPLEMENTATION DETAILS
In this section we summarize the main architectural and training configurations used in our experi- ments shown in Tables 7, 8, 9 and 10.
19

---

Published as a conference paper at ICLR 2026
Table 8: Stage 1: LoRA autoencoder configuration for Gemma-2-2B-Instmn
Item
Configuration
Backbone
2eemma---BBInssrruet -B--nstrtct (decoder-only)
Encoder hidden size
dmodel = 2304
LoRA rank
Latent dimension
d= = 64 (per module)
Encoder MLP hidden size
128
Decoder type
4-head MoE for qa Vaa
do output size
dm 8 2304 18,432
q output size
dg r 2048 x 8 16,384
va output size
18,432
output size
ou 1024 x 8 8192 E ==
Experts per head
Active experts
top-K 1 per location
Expert pools
Separate pool per head
Expert MLP input
2d.
128
Expert MLP hidden size
256 (SiLU activation)
Expert MLP output
Matching head-specifee output sizes above
Encoder params
RJ 5.6M
Decoder (MoE) params
2 63.7M
Total generator params
~ 69.3M (=s 3-4% of backbone)
Table 9: Stage 1: losses and training schedule for both backbones.
Item
Configuration
Adapter-level losses
Direction loss Lair and spectral loss Lspec on AW 1.0 for Ldir. 2.0 for Lypec
Loss weights
Optimizer
AdamW
Learning rate
1.115 10-3
Gradient accumulation
Factor 8 1000
Max epochs
Complexity details
See Appendix A.4
A.7 END-TO-END INFERENCE LATENCY WITH GENERATED ADAPTERS
Once a LoRA is generated, we simply load the weights into the base model; at inference time the runtime is the same as for any standard LoRA fine-tuned model. On an NVIDIA H800, measuring real evaluation runs, we observe the end-to-end latencies per forward pass shown in Table 11.
The small differences are within standard runtime variability; there is no additional per-token over- head induced by LoRAGen itself. All extra cost is incurred once per task when sampling LoRA adapter set, as discussed in Appendix A.4.
A.8 SCALABILITY ANALYSIS
In this section, we evaluate the scalability of LoRAGen with respect to both the number of LoRA locations and the LoRA rank. In the experiments, we define the total generation latency as the combined time taken for three parts: (1) Encoding (Enc.): The time taken to encode the input task description and map it to the latent space. (2) Diffusion (Diff.): The time taken to perform diffusion sampling over the latent space. (3) Decoding (Dec.): The time taken to decode the latent vector back into the LoRA parameters.
Scaling with the number of locations. We evaluate the performance of LoRAGen as the number of LoRA locations increases. We performed scaling tests Gemma-2-22 (rank r 8, T 500 diffusion steps), varying the number of LoRA locations L across values 24, 48, 72, 96 and measured the total latency.
20

---

Published as a conference paper at ICLR 2026
Table 10: Stage 2: latent diffusion adapter latents.
Item
Configuration
Latent sequence length (FLAN-T5)
L 288 (one token per LoRA location)
Latent sequence length (Gemma-2-22) 2-2B)
= 96 (one token per LoRA location)
Denoiser architecture
1D Transformer
Denoiser layers
Model dimension
256
Input/output dimension
d₂ = 64
Conditioning
Task embedding module/layer embeddings
Diffusion steps
T = 500
Noise schedule
Linear 3 schedule
Objective
v-prediction with l loss
Optimizer
Adam
Learning rate
10-5
Batch size
32
Training steps
100k
Table 11: End-to-end latency per forward pass with oracle LoRAGien-generated LoRA adapters on an NVIDIA H800.
Base model & adapter
Task
Mean latency (ms)
LLANeTS-Large, r = 16, oracle LoRA
AP-Rec
42.10
FLAN-TS- Large,, = 16, LoRAGen LoRA
AP-Rec
40.33
Gemma-2-25 r 8, oracle LoRA
GSM8K
206.55
Gemma-2-22 r = LoRAGen LoRA
GSMSK
130.85
From these results presented in Table 12, we observe that the total latency remains low even as the number of LoRA locations increases. Even for the configuration with 96 locations, the total latency stays below 0.6 seconds, which is negligible compared to training a task-specific LoRA from scratch. The performance remains stable with negligible fluctuations in accuracy across different L values, further confirming the efficiency of LoRAGen in scaling with the number of locations.
Scaling with LoRA ranks. We evaluate the scaling of LoRAGen with respect to LoRA rank r. We keep the latent dimension fixed at d2 64 and reuse the same decoder/denoiser widths. The scaling experiment was conducted on the Gemma-2-2222 model with diffusion steps for rank r E {8, 32}.
locations and T 500
As shown in Table 13, the total generation latency remains stable even as the LoRA rank in- creases. The MoE' decoder structure remains unchanged, with only the final linear projections scaling with r. The average accuracy shows negligible fluctuations, indicating that increasing the LoRA rank does not introduce significant performance degradation. These results suggest that our design is sufficiently flexible handle larger ranks without requiring a redesign the architecture.
In summary, the total generation latency remains low even we increase the number of LoRA lo- cations, and LoRA rank scaling does not necessitate any architectural changes. The small changes in latency are well within acceptable limits and do not hinder the performance of the model. These findings demonstrate the scalability and efficiency of LoRAGen real-world applications.
A.9 TRAINING AND EVALUATION DATASETS FOR ZERO-SHOT GENERATION
As shown in Figure 9 and Figure 10, conduct zero-shot generation experiment on 136 training tasks and 7 evaluating tasks from FLAN subset. These evaluated tasks are seperate from the training datasets.
21

---

Published as a conference paper at ICLR 2026
Table 12: Latency and accuracy scaling with LoRA locations. Avg. Acc. (%) reports the average
accuracy on benchmark tasks as in Table 2.
LoRA locations
Enc. Diff. Dec. latency (ms)
Total gen. latency (ms)
Avg. Acc. (%)
24
0.25 / 563.6 5.2
$69.0
72.5
48
0.25 566.9 5.5
572.6
72.4
72
0,40 / 562.7 5.6
568.7
72.6
96
0,40 / 569.6 5.7
575.7
72.7
Table 13:
Latency and accuracy scaling with LoRA rank. Avg. Acc. (%) reports the average
accuracy on benchmark tasks in Table 2.
Rank
Enc. Diff. Dec. (ms)
Total gen. latency (ms)
Avg, Acc. (%)
8
0.40 573.5 5.7
579.6
72.7
16
0.49 567.5 6.2
574.2
72.8
32
0.64 5 6.5
582.7
72.7
A.10
TASK DESCRIPTIONS GENERATED BY A LARGE LANGUAGE MODEL
We automate task description generation for each task by leveraging Deepseek*. We query its model with carefully constructed prompts that incentivize diversity to facilitate downstream generalization shown in Figure 11. In particular, we generate 20 descriptions per task. Figure 12 presents representative examples of task descriptions employed in our experiments.
A.11 DETAILS OF EXPERIMENT SETUP
More details about the diffusion architecture, baseline settings hyperparameter settings, training details and implementation of weight space loss and module-aware MoE decoder can be found in anonymous repository https: wwt.:/ttttoo.o....oooeoeeoeoo
fib- Teab-labbLoRaae LoRAGe..
A.12 LLM USAGE
We utilized CharGPT-40 to refine the content based original writing. All revised text was subsequently reviewed and verified by us. The natural language task descriptions we used are generated by DeepSeek All code has undergone comprehensive testing to ensure its reliability.
htttps/wwww.ddepsccc.coo
"httss//chhttgt..co
fhtps://www.deepscc.ccom
22

---

Published as a conference paper at ICLR 2026
Training Tasks
Torahut fon lage- oeveorrrrrr...oo oo, Sbdof based os",
"oatuo Fon. 15 bogegew hog org www.enteeeooooooooooo atyert
loge aderrsri trobero
guetton conted,
"oatue 37 bage 48 hotookee compler, cwtre"
Torshut For aag-aberrrri www.rtaabbbaaao. boser_ po".
Nilowing.".
bostuu Fan lage
"oatun, Fon, 5. wage whit k. the Frall s90p, of the, bilniing_pmooss
horanue Jun ange-aberaan ou steaal quesson cortast, arowee",
Twatte: Jun. 5 orge-opes prompe bobam Per bugarring
Toratu Jan wgg-aareraenn aroburaa
"watuu 1 lege-race made Select beat answe"
Toraue Jun vgg.aaberrrl.coo Sotul uroner Uhe Salowing
"outut: Fun 8, lege aocil Chock suninoo answer vall
Toutue Jan R. sage-ateennnd 00. abortatse anseer Tv. Jollwing
"wotut: Jun. 9, www.gg-oooe roper pwomprooooooe. mLC.
bostu: (wn begaoeteenent 19, atwe based ont
"patu Inc. 8. bege-cppe ghen Iacaggnund shookknn
bege-ecn
Tostu: Fwn begewmmens polwtyy
"saati: 5
chosse bast otject, atwmrraa
Tostee two lege-sp mideos oxeegrow, exting uxing medeer
"oostu:: Fun. 0. begessex micile Select th beet areaw, to inatuutie
1orahe Jon lagg apc rodoos comoort to rating
"wotuu, Fon. 92 lagg auoect Answer Friend Guossonn.
fon lage amazon polotty negathe pos.ooiieeeeee sone"
"wihhu: 32 loge
org
queston "witho Ton. 9, brge ropos p_ompoo beghting.
bootue For lagg- amazon polority would, you Buy"
1osshe fon H. lane amazon powortyy 16 thi__tttoo
"ootu Fon. 37 boge 100 Dind Oueston Closed Book
Toratu: Jan lege-yele pa anty poosswor
"xaatu: Jan brge-rac righ lakin
Taratu Jun lang-ggue 36a0"
"wratu: 9. lrge-cuorrr PanggrassfC sotract uronwer,
Toutu: Jun 1t. www.eggegg..oo .ooa
"wratu: Jun. 5. lrgsssi su. Decdda good. answer
Toatu: Fan jeggewwii_i
"patup, Fae. R bege-cuore PangpraseR 18 gereestont bege-qeeel
"vahe 5
(iww Cinolier
boste: Two 12. leige-uper _lan wic.
"oatu Iin. 0. legeedd Top. (igg chosss best otject jrlemggtiin
Tostu:: [wn leegeeiinn
"oatip 0 begeodki on ssarrton"
Torsuu: Fon luge-gaze ga2a
nomfrne for Jolowing
"echu:: 18 lagg ropes tockgroun.
snower
boroue for 0. dipedi. glron a crokk of categolics
"wohu: Ton 2, kggs cua hoving reod, sbove passage"
1ochhe__onn lang dipedi__1 gieie Syor_fa F.L oatogory doct, the ponggoggohhh..oooooooooooooooo bolong,tof
"octul, Fon, brg. ropo prompt botam 20, JIC.
Torate: (an lergeedi Me. whoo
"saatu Jan 5 begescull corlest queeias arever descrptirr Jasf
separitd,
bege-ecese
Toratt: Jan k.gg-ddpeess. grenn
shut coser the tie balong.or Dr.
wwgg--uurs Sal10 cusss woth
Tostu: Jun lege--pec with sepanded seca a".
"wahut Jun. 0. bege-ourzz use info, Foom pww.gruph questir""
Tootu: Fon /3_ange-goo0__0 with separated fom
"ootup Fon. 17 Aoge-ousl Lno promptt in
sepanted
11 bnge-uueel
Tostue Fmn bage--aritt_aa. ghwn the fwo mower Instuctions
Hi.. Select best
"watuo
high I.
"patur Tan. 0. begeouul Looteat queeon jnnwer descrppion Jr.,
Tontue: Twn lege-ant queesione whate the sroww"
"watup 11 brgenday effect with jatw poower"
Toratu: Jun luge-gucc"
"eatu: 1 lage-web sucatore poiottd comeet unswer.
1orhhe fun larg- dream boscine"
"wolu: fun 8, lags mas wh 3h. folowing
supocced petubatorn, "wotul: Jan. 8, kgg awo SUIRC move svector.
Tootue Jun J. lagg--o0 M.lppe Ghoce, Cloood Book
Tostu:: (wn lengesppore [iamen Assoser
"satui: Isn. 8 begendii on Tople Predecte Arawer Dny"
baskgmuntt
bege-oaga
prosee"
Toatus (en bege-eds Iourst
"satti:
the
Tostue Jwn it bege-qusi conteot queeson desception, srower it",
"wottu Tan. 9 begessii hep ong reg thoses best copect allmmtha 2",
Tontut: (an lengeedic Iec ungrnel penente sutgect and otject
"satti:: Jan 8 begescurrs Panggh move
Torshu
ducrt FarogrosoRR rosoRC anower
"wotu: 1 large duor BMMRR
gueddon
Fon
Sontevt pesorialion quesion
Torshb Fon lage rooe midd Select he best snower gentrate
"potu.. Ton. 7. brge cual descrppoon conter quediton snower M".
Torahe Fon 1f (ogge-ooo)) 10, Srow shokes and generate answer
"wote,, Fon 17 bege 120 Mu Choke"
Toratut Jun ange-guent Contect
Arowe.
"kratu: 5 lrge-cuond What The Arowar.
Tbratu: Jun lang-ropes bodigroun Sttttoo midoa."
"wratu:
lrge-cuorel AAsswy Cusstion Given Content
Torate Jun 1: lage-wecaa oteet with ping arawa".
"wratu: Jun 5, lrge-spps. sood background huttor"
Tootub: Jan Jeggedoor Pamppmeedf decis worth 1f
"pahu:: Exe. 8 bege-curro SwifC, estratl aroww?"
beg.e-eea..... high
"bostu: (wn begewal Trpec Predktion Quwedinn On-y.
"vate:: 5
soee"
Tostre: Twi I lega-dhore SwitoC Li.e geremaion."
"oahu Inc. 0. Iegermmmnnn"
Tostu:: [wn leegeepas Centeol quedtion descmpiin pennarr Ieaf"
1sati:: Isc, 6 bege-gem web rit
milt choice question Sptons giron. Toratut Fon lago-rooo bigh Contost
"ectu:: 38 lang powa
Questor'.
besting studenss
Torchu, for laro quall promppt Jesf"
"wotu: Fon 3, lngs cuarcl Jogk Jost
Torhwe for lage wil, 91 jo_parayryyy.....oo..ooooooo.... stple"
twotul, Fon, 32, br.. cuarol choose, between"
Toratu Jan lege-etze down supposed parturtedarn Jure, an, aflect
"aratu Jan. bege-cuurel do use""
Jege-rppe jlan backgourd
story
Figure 9: Training tasks from FLAN dataset used for training the LoRAGen model
Validation Tasks
"lorahub Baaihhuuffaan____ t5 _aaaa--amazzo_pplrrittyy Is tet_thhs_pppe
"lorahub flan t5 large-gasc_is_conee
1",
"lorahub flan t5 sogge-gaac_is_correee 2",
"lorahub flan 15 1aa______argewwiiiaa_aab Lackii__a_____ This True_".
Baaiuuuuffan_t_llrrgeeeee "lorahub_flan_t55 t5 http://wwkiiaaaaatttttcicsssssss
Figure 10: Validation tasks used during the training the LoRAGen model
23

---

Published as a conference paper at ICLR 2026
Prompt
#t Objective
every LoRA adapter directory, construct clean list of task descriptions and them into single sentence-lewe embedding.
The generator reads curated descriptions from YAMI, applies light normalization, matches them name, and averages T5 sentence embeddings.
## Inputs
*'--yaml_root' "omil__oo**"_omm directory with subfolder per task; each contains 'metadata.yaml with descriptions list.
"*.-logsrroot bogs_root*** * Stage-1 checkpoint tree (used to load the VAE encoder for latents; text generation file-driven).
## Options
"-strip_generic" remove leading bollerplate like "The task is involves/ requires
remove
--text_pooling (mean, fi sentence pooling for T5 (mean is mask-aware and default)..
## Procedure
1. **Scan YAML repositary**
For each subfolder under 'amml_ooo",, load 'metadata.yam! and read descriptions. Skip missing.
2. **Optional light normalization*
ff--stripeeeeerrt generic', strip only the leading boilerplate using regex and keep the substantive remainder.
**Build multi-key map**
Register the cleaned descriptions under:
"centry>"
'lorahub flauuub_fa_____ 15_large-<entry""
-lorahub L_________ _a__aare__eette
**Iterate LoRA adapters**
For each '..//fla_tt5__ Barge_lora/ttas__kee///te htrr//ddpter_mmoell.ii" resolve descriptions by probing variants (prefix
removal and -/" swaps).
""Fallbac**";ff
[[taskkkey]' the only description and log the miss.
5. *Encode with T5**
Tokenize each description, "TSEncodrModde".
Pool to sentence vector via:
mean pooling (default), ar
first taken pooling.
Average across all descriptions > one 1024-d text embedding per task.
6. **Normalize save key**
Drop Yoorhub faan_attaaaanaaaaa json
15_ ls_large/__ prefix to form the canonical task name and save:
"task, name": "<canonical, name>"
"text embedding": "e1024-d tensor>".
"latent": "<288 x latent, dim tensors"
Output
PyTorch per experiment, e.g.:
0.009966 with task name_aamaaaaaeee task 172, latent_ 288, embed 1024.pt Mapping canonical task names averaged embeddings (plus latents produced the same pass).
Figure 11: The prompt template used by pipeline for task descriptions.
24

---

Published as a conference paper at ICLR 2026
Task descriptions
adversarial
_e__aaaa_a d___ebert___ answer the followi
The task involves reading passage and answering a question based on the information provided in the text. The task requires identifying specific details from given passage to answer question.
The task is about locating exact information within a text to respond to a direct question.
adversarial qa dbidaf_ answer the following
The task is locate the answer question within provided passage.
The task requires finding the exact words phrases in passage that answer question.
The task is about answering questions by referring to specific details in a text.
adversarial qa droberta, droberta_question cttpl_ooottttteeeeeeeeeeeeee answer
The task requires matching questions with relevant facts from the given context.
The task about locating key details
answer direct question.
The task involves reasoning about passage derive the correct
adversarial _a______________ dberr_ based
The task involves finding specific information in a given text to answer a direct question.
The task requires identifying key details from passage that directly respond to a question.
The task about locating exact answers within provided context based on question.
Figure 12: Examples of task descriptions generated by our pipeline.
25


Reviews:
[{"review_id": "RfEl5SPF4Z", "reviewer": "Reviewer_miEg", "summary": "This paper introduces LoRAGen, a method for generating Low-Rank Adaptation (LoRA) parameters directly from natural language task descriptions. The authors argue that LoRA parameter spaces have unique structural properties that are ignored by general-purpose weight-space learning methods. They identify two key properties from an empirical analysis: (1) the **non-uniqueness of low-rank decomposition**, where task similarity correlates with the full adaptation matrix $\\Delta W = BA$ but not with the individual $A, B$ matrices, and (2) the **heterogeneity of weight distributions**, where different modules (e.g., encoder, decoder) exhibit different spectral properties. To address these, LoRAGen introduces two main innovations: 1. **Adapter-level supervision** and 2. **Module-aware MoE decoder**. The overall framework uses a LoRA Weight Autoencoder (LAE) and a conditional latent diffusion model to generate latents from task descriptions, which are then passed to the MoE decoder. Experiments on FLAN-T5 and Gemma-2 models show that LoRAGen achieves performance close to task-specific (oracle) LoRAs on in-distribution tasks and, more importantly, outperforms the T2L (Text-to-LoRA) baseline by nearly 5% on zero-shot generation for unseen tasks.", "questions": "1.   The ablation in Table 4 suggests your novel adapter-level losses ($L_{ang}$, $L_{spec}$) provide only a marginal (0.8%) benefit on in-distribution tasks over a standard reconstruction loss when the MoE decoder is used. You motivate these losses as being essential for zero-shot generalization. To support this claim, please provide the **zero-shot ablation results** for the seven unseen tasks (the same setup as Table 3). Specifically, what is the zero-shot performance of the model with *only* the MoE decoder and a reconstruction loss (the one that scored 95.2% in Table 4)? This is crucial to validate your central contribution.\n2.  Please correct the \"Average LoRA\" baseline results in Table 2. The current values are nonsensical, as they significantly outperform the task-specific oracle.\n3.  Have you analyzed the expert specialization in your MoE decoder? The routing is based on module and layer embeddings, motivated by Obs-2 (heterogeneity). Do the experts indeed learn to specialize on specific module types (e.g., encoder vs. decoder) or spectral-entropy profiles as hypothesized?\n4.  What is the training time and parameter-count overhead of LoRAGen (for Stage 1) compared to the T2L baseline? How much do the adapter-level losses and MoE decoder add to the computational cost?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.  The paper is grounded in a clear and compelling empirical analysis (Figure 1). The two observations (non-uniqueness and heterogeneity) are well-demonstrated and provide a strong \"why\" for the proposed method.\n2.   The proposed solutions map directly to the identified problems. The adapter-level supervision (direction and spectral loss) is a very clever way to address the non-uniqueness issue. The authors also correctly note the importance of an *efficient* implementation (Appendix A.3), which avoids materializing the $d \\times d$ matrix and makes the approach practical.\n3.   The primary goal of such a model is to generalize to new tasks. The 5% absolute improvement on zero-shot generation (Table 3) over the T2L baseline is a significant and meaningful result, demonstrating the value of the structure-aware approach.\n4.  The method is shown to be effective on both encoder-decoder (FLAN-T5) and decoder-only (Gemma-2) architectures, suggesting the principles are general.\n5.  The paper is well-organized, and the progression from observation to method to results is logical and easy to follow.", "weaknesses": "1. The main weakness is the ablation study in Table 4. The model with just the MoE decoder and a standard reconstruction loss (\"X\", \"X\", \"✓\") achieves 95.2% average accuracy. The full model, with the novel adapter-level losses (\"✓\", \"✓\", \"✓\"), achieves 96.0%. This 0.8% difference on in-distribution tasks seems to *undermine* the importance of the adapter-level supervision ($L_{ang}$ and $L_{spec}$), which is presented as a primary contribution.\n2.  Related to Weakness #1, the paper argues that the adapter-level losses are critical for *generalization* and avoiding \"memorizing\" specific decompositions, which is key for zero-shot performance. However, the ablation study in Table 4 is *only* performed on in-distribution tasks. The most critical ablation—showing the zero-shot performance of the \"MoE + reconstruction\" model (the 95.2% one)—is missing. Without this, the central claim that the novel losses are *necessary* for zero-shot generalization is not fully substantiated.\n3.  The \"Average LoRA\" baseline results in Table 2 are clearly an error. The values appear to be copy-pasted from Table 1, and they show scores (e.g., 96.8 on ArcC) that are vastly higher than the \"Task-specific LoRAs\" (the oracle, 76.7). This is a sloppy error that should have been caught.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PXOkmXyxt1", "reviewer": "Reviewer_nPHN", "summary": "The paper tries to directly predict LoRA adaptation weights from fine-tuning task descriptions with a DiT. With empirical findings that $\\Delta W$ correlates with task but not $A$ or $B$, and weights at different layers have different spectral entropy, the model is supervised by $\\Delta W$ and conditioned on the layer position.", "questions": "The meaning of Fig 1 (Left) is quite unclear, and the explanation in L184-187 is ambiguous. \"Similarity\" is something to be calculated in pairs, but the text doesn't really make it clear. What is the \"representative task\"? I assume that the authors compute the similarity between 112 other tasks and a single \"representative task\", i.e. duorc_gqba. Why was this task selected? Then to what extent they are correlated? Is the rho in the Figure Spearman's coefficient? Please state that more explicitly. From the figure I can't really see the correlation clearly, possibly because the two series are at quite different scales; possibly separating the (A, B) and $\\Delta W$ series into two figures will make it clearer. Particularly, the correlation looks dominated by certain samples with the largest task similarity; then are the observations applicable to other tasks? Also, $\\Delta W$ is not unique either. What will happen if you collect more samples per task?\n\nDiscussions on empirical observations are repeated in page 2 and page 4, and it's difficult to check Fig 1 when reading page 4. I suggest only briefly mentioning the empirical conclusions in the Intro part and moving Fig 1 to page 4.\n\nL357: What is \"element-wise averaged LoRA\"?\n\nL372-373: How is the model applied to a decoder-only base model? I assume that the model is trained on FLAN-T5, including the module/layer embeddings.\n\nAlso, the target model Gemma-2-2B-Instruct itself can already get good performance on many tasks. Hence I have concerns about the cross-model generalization. I think this is critical for the practical use of this series of methods, as training hypernetwork is expensive, LoRA-tuning itself is already cheap, and there is still a considerable performance gap between generated and trained LoRA.\n\nI also have doubts about FLAN-T5's low performance on the benchmarks. Isn't FLAN-T5 already trained on those tasks?\n \n\nL135-136; L365: Should use \\citep here  \nL170: observations  \nL348: Team?  \nL361: Missing label?  \nL710: full?", "limitations": "", "rating": 2, "confidence": 2, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "An interesting exploration of weight space learning is presented.", "weaknesses": "The empirical observations on which the assumption is made are unclear.  \nThe experiment is only carried out on a specific base model with concerns about cross-model generalization.  \nThe presentation is not very clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bjasYy10JJ", "reviewer": "Reviewer_6LiF", "summary": "In this paper, the authors proposed LoRAGen, a structure-aware method that directly synthesizes LoRA parameters from natural language descriptions, to address a critical limitation of traditional LoRA workflows: the need for costly, task-specific training to generate LoRA parameters. LoRAGen can eliminate the need for task-specific data collection and training and is grounded in two key empirical observations about LoRA weight spaces.", "questions": "Please see Weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "1. The motivation of this paper is strong. It provides two observations to inspire the method.\n\n2. LoRAGen generates LoRA parameters directly from task descriptions, bypassing the need for task-specific data collection, annotation, and training.\n\n3. LoRAGen tackles LoRA parameter generation from two key properties of LoRA spaces: 1. the non-uniqueness of Low-rank decomposition and the heterogeneous weight distributions across modules.\n\n4. The empirical results are promising.", "weaknesses": "1. The writing should be improved. For example, in Line 189, “the the cosine” => “the cosine”. In Line 188 “the adapter similarity similarity” => “the adapter similarity”\n\n2. For decoder-only architecture LLMs, the authors did not present weight distribution analysis.\n\n3. The experiments are only conducted on two small models (FLAN-T5-large and Gemma-2-2B). The authors should involve more popular LLMs, like Qwen3-8B and Llama2-7B, to make the empirical findings more convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OSsnJmbVE2", "reviewer": "Reviewer_mT9U", "summary": "This paper proposes LoRAGen, a structure-aware framework for generating LoRA adapters from natural language, addressing the need for scalable and efficient model customization. The methodology is divided into two stages:\nStage 1: Learning a Structured LoRA Latent Space. A LoRA weight autoencoder is trained to map adapter weights to and from a latent space. Its key contribution is a module-aware Mixture-of-Experts (MoE) decoder, where different experts specialize in generating weights for different parts of the network architecture (e.g., attention vs. feed-forward layers). To overcome the issue that multiple low-rank matrices can produce the same adapter, training is supervised directly on the full adapter matrix, ensuring a more robust and meaningful latent space.\nStage 2: Text-Conditioned Latent Generation. A diffusion model is trained as a conditional prior over the learned latent space. At inference time, this model takes a natural language task description, encodes it, and generates a corresponding latent vector, which is then decoded into a full set of LoRA weights by the frozen decoder from Stage 1.\nThe primary contributions are the novel module-aware architecture for weight generation, the robust adapter-level supervision strategy, and the demonstration of strong zero-shot performance, which significantly outperforms existing baselines.", "questions": "See weaknesses\n\n* **Why diffusion (vs other generative priors)?**\n  What concrete gains do you observe from a diffusion prior over simpler/cheaper alternatives (e.g., MLP/linear prior, VAE, normalizing flows, consistency/CM/rectified flow) on the *same latent space*?\n\n* **Sampling cost and scaling to larger adapters.**\n  Diffusion can be slow when sampling many adapter locations for deeper network. What is the end-to-end latency per full adapter set, and how does it scale with (i) number of LoRA locations and (ii) LoRA rank (r)?  If LoRA rank or the number of locations increases, does the autoencoder need redesign (latent size (d_z), decoder depth/width), or does performance remain stable?\n\n* **On missing baselines (Drag-and-Drop).**\n  Could you clarify why **Drag-and-Drop LLMs (prompt→weights hypernetwork)** was not included in benchmarks? Was this due to incompatibility, unavailable code, or scope? Given its direct relevance, a brief comparison or discussion of expected differences would help position your contribution.\n\n* **Decoder architecture/topology details.**\n  Please specify the module-aware MoE decoder precisely: number of experts (E), top-(K), shared vs per-module expert pools, router temperature, load-balancing objective, structural embeddings (module/layer dims), expert MLP widths/depths, normalization/activation/residual scheme, and per-module head parameterization (predicting full (\\Delta W) vs low-rank factors). A parameter-count and FLOPs breakdown per component would also clarify capacity vs. performance.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**Originality**\n\nThe originality of this work extends beyond just using a Mixture-of-Experts decoder. Its primary innovation lies in designing a module-aware MoE decoder specifically tailored to the structural properties of Transformer networks. Rather than using a monolithic decoder, this approach allows different experts to specialize in generating weights for distinct components (e.g., attention vs. feed-forward layers), which is a novel and highly effective concept. Furthermore, the paper introduces a unique adapter-level supervision strategy (using direction and spectral losses) to directly address the ambiguity of low-rank matrix factorization—a subtle but critical problem that previous methods have largely overlooked.\n\n**Quality**\n\nThe paper demonstrates high quality through its rigorous methodology and strong empirical evidence. The entire approach is well-grounded, with its core design principles directly motivated by clear empirical analysis of the LoRA weight space. The experimental validation is comprehensive and convincing, using multiple model architectures and strong baselines. The inclusion of a thorough ablation study, which isolates the contribution of each key component, further attests to the quality and soundness of the research.\n\n**Clarity**\n\nThe paper is written with exceptional clarity. The authors do an excellent job of building a logical narrative, starting with two core empirical observations, proposing solutions directly tailored to them, and then validating these solutions through experimentation. Complex concepts are broken down and explained in a way that is easy to follow, making the paper highly accessible despite its technical depth.\n\n**Strengths**\n\n**A Novel and Robust Generation Framework:** The paper significantly improves upon previous weight generation methods by introducing a more robust and structure-aware framework. The combination of the module-aware MoE decoder and adapter-level supervision provides a more principled way to learn the geometry of the LoRA weight space.\n\n**Principled Design Grounded in Strong Empirical Analysis:** The method isn't just a collection of techniques; it's a carefully designed solution based on a solid analysis of the problem space. This analytical rigor is a key strength and provides a strong foundation for the paper's claims.\n\n**Comprehensive and Convincing Empirical Validation:** The experiments are a major strength. The method achieves performance on par with, or even outperforming, task-specific LoRAs in some cases. Crucially, it demonstrates excellent generalization in both in-distribution and challenging zero-shot (out-of-distribution) settings, proving that it learns a meaningful mapping from language to weights rather than just memorizing trained adapters.", "weaknesses": "**Scalability.**\nEvidence is limited to FLAN-T5-Large (~780M) and Gemma-2-2B. The method’s practical value at the scales where LoRA matters most (7B–70B+) is not demonstrated; compute, memory footprint, and stability characteristics at those sizes remain unclear.\n\n**Baselines and citation coverage.**\nThe evaluation focuses on diffusion-based generators while omitting recent non-diffusion prompt-to-LoRA approaches that also leverage pretrained knowledge, notably **Drag-and-Drop LLMs** and **LoRA-Gen**. This gap weakens novelty positioning and makes robustness claims harder to interpret relative to the broader literature.\n[1] *Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weights*\n[2] *LoRA-Gen: Specializing Large Language Model via Online LoRA Generation*\n\n**Reproducibility and configuration transparency.**\nKey implementation details are not fully specified in the paper: the autoencoder’s exact topology, latent dimensionalities, MoE settings (E, top-K, routing temperature), per-module head parameterization, training schedules for both stages, and per-module LoRA ranks. Although code is provided, the manuscript itself lacks a consolidated description of these hyperparameters, limiting reproducibility from the text alone.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)