Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weights
Authors: Zhiyuan Liang
Dongwen Tang
Yuhao Zhou
Xuanlei Zhao
Mingjia Shi
Wangbo Zhao
Zekai Li
Peihao Wang
Konstantin Schürholt
Damian Borth
Michael M. Bronstein
Yang You
Zhangyang Wang
Kai Wang
Venue: NeurIPS 2025 poster ()
Keywords: Parameter Generation
Parameter-Efficient-Fine-Tuning
Large Language Models
Primary area: general_machine_learning
TL;DR: Generate high-performing LoRA parameters from prompts that is unseen in training.

Abstract: Modern Parameter-Efficient Fine-Tuning (PEFT) methods such as low-rank adaptation (LoRA) reduce the cost of customizing large language models (LLMs), yet still require a separate optimization run for every downstream dataset. We introduce \textbf{Drag-and-Drop LLMs (\textit{DnD})}, a prompt-conditioned parameter generator that eliminates per-task training by mapping a handful of unlabeled task prompts directly to LoRA weight updates. A lightweight text encoder distills each prompt batch into  condition embeddings, which are then transformed by a cascaded hyper-convolutional decoder into the full set of LoRA matrices. Once trained in a diverse collection of
prompt-checkpoint pairs, DnD produces task-specific parameters in seconds, yielding i) up to
\textbf{12,000$\times$} lower overhead than full fine-tuning, ii) average gains up to \textbf{30\%} in performance over the strongest training LoRAs on unseen common-sense reasoning, math, coding, and multimodal benchmarks, and iii) robust cross-domain generalization improving \textbf{40\%} performance without access to the target data or labels. Our results demonstrate that prompt-conditioned parameter generation is a viable alternative to gradient-based adaptation for rapidly specializing LLMs.
We open source \href{https://jerryliang24.github.io/DnD}{our project} in support of future research.

Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weights
Zhiyuan Liang"
Dongwen Tang'
Yuhao Zhou'
Xuanlei Zhao'
Mingjia Shi!
Wangbo Zhao Zekai Li
Peihao Wang"
Konstantin Schürholt
Damian Borth
Michael M. Bronstein
Yang You?
Zhangyang Wang2*
Kai Wang
National University of Singapore, PT Austin,
University of St. Gallen, +Oxford University
Abstract
Modern Parameter-Efficiee Fine- Tinenn (PEFT) methods such as low-rank adap- tation (LoRA) reduce the cost of customizing large language models (LLMs), yet still require separate optimization run for every downstream dataset. We intro- duce Drag-and- -noooppp LLMs (DnD), prompt-conditioned parameter generator that climinates per-task training by mapping a handful of unlabeled task prompts directly to LoRA weight updates. A lightweight text encoder distills each prompt batch into condition embeddings, which are then transformed by a cascaded hyper- convolutional decoder into the full set of LoRA matrices. Once trained in diverse collection of prompt-checkpoin pairs, DnD produces task- sesk-specificc parameters seconds, yielding up 12,000 lower overhead than full fine-tuning, ii) average gains up to 30% in performance over the strongest training LoRAs on unseen common-sense reasoning, math, coding, and multimodal benchmarks, and iii) robust cross-domain generalization despite never seeing the target data or labels. Our results demonstrate that prompt-conditioned parameter generation is viable alternative to gradient-basee adaptation for rapidly specializing LLMs. Our project is available https://eerryliang44 git.bbbbiiii.DDDDD...
Introduction
I Large Language Models (LLMs) such as GPT-4, Llama 2/3, Qwen2.5, and DeepSeek have rapidly become the backbone of contemporary natural-language processing and artificial intelligence more broadly, thanks to their internet-scale pre-training and transformer architectures [1. 19, 63, 36]. This pre-training endows a single model with broad zero-shot competence across mathematics, coding, reasoning, and even multimodal understanding [24, 12, 62, 29]. Yet, real-world deployments rarely stop at zero-shot use; they demand task-specific behavior that reflects internal data, domain jargon, or bespoke response styles. Parameter- Eirrmietr-Efficient Fine-Tuning (PEFT) aims satisfy this demand by inserting a small set of trainable weights, most prominently the low-rank adapters of LoRA [25]. While LoRA allows maintain the number of trainable parameters and storage overhead small by keeping the model frozen, the wall-clock cost remains very high: for example, adapting the lightest 0.5 B--arrmeter Qwen2.5 with LoRA still occupies four A100 GPUs for half day [63]. Moreover, each downstream user or dataset requires its own optimization run, which quickly becomes the computational bottleneck for practitioners when deploying PEFT at massive scales.
We observe that a LoRA adapter is nothing more than a function of its training data: gradient descent "drags" the base weights towards task-specific optimum (Figure 1). If that mapping from prompts to weights can be learned directly, we could bypass gradient descent altogether. Early work parameter generation has shown that dyper-network minutes (51, 43, 59, 49, 53]. Yet they either ignore task conditioning or use simple binary embeddings.
synthesize billions of weights in
"hiyuan, Zhangyang, and Kai are core contributors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

novel dataset 1
tuning with LLMS
novel model 1
novel dataset
nove model
hours
w/o tuning
:
DnD
:
novel dataset
tuning with LLMS
novel
novel dataset
seconds
novel model
hours
model
(a) Frrameter-Efficien--in---
(b) Drag-and-Drop
Figure I: Left: Parameter-efficien methods such as LoRA need hours to optimize LLMs in order to adapt them to novel datasets. Right: Our method adapts LLMs by directly generating LoRA matrices for novel datasets in seconds without any tuning.
Recent progress makes this goal attainable. RPG [58] is one the first approaches to condition on task information and generate an entire classifier in single pass, matching from-scratch training previously unseen image classes zero-shot. Translating that success to language, however, raises new obstacles. First, linguistic prompts carry orders of magnitude binary embeddings used by RPG. A practical generator must therefore ingest rich task descriptors and preserve their nuances. Second, an LLM in production may face hundreds of heterogencous workloads, so the conditioning mechanism must scale gracefully while injecting task-specific cues with high fidelity. These challenges bring the need for compact yet expressive representation that both captures salient features of the input texts and steers the hyper-network towards the corresponding region of LoRA weight space. Designing such a representation the central challenge that our method, introduced next, is built to address.
semantic variation than the
We introduce Trag-and-Droo LLMs (DnD), prompt-conditioned hyper-generatee that converts handful unlabeled task prompts into complete set of LoRA adapters in seconds, eliminating any per-task optimization. DnD employs off-the-shelf, lightweight text encoder to compress given batch of prompts into conditional embeddings, which a cascaded byper-convolutional decoder then expands into LoRA updates for every transformer layer.
common-sense reasoning. mathematics, sede-generation, and multimodal benchmarks, DnD cuts adaptation overhead by up to 12,000x while yielding up to 30% increased performance on unseen datasets compared with the strongest training LoRAs, and transfers seamlessly from 0.5B to 7B parameter backbones. By collapsing the classical " ggaddients weights" loop into single forward step, DnD challenges the notion that gradient descent indispensable for model specialization and opens new path where weights themselves become new data modality and generative target conditioned on concise task descriptors.
Our main contributions are outlined as follows:
New LLM adaptation paradigm. We cast LoRA adaptation direct generation of task task-specific weights from raw prompts in a novel dataset and realize this mapping with a scalable which is way more efficient than traditional tuning.
Practical Architecture. frozen text encoder coupled with hyper-convoluttonall decoder is able to generate large scale parameters while reducing adaptation overhead by four orders of magnitude.
Comprehensive evaluation. Experiments across reasoning, math, coding, and multimodal show up to 30% zero-shot gains on unseen datasets and smooth transfer across model sizes, highlighting DnD's impressive efficiency and versatility.
2 Drag-and-Drop Your LLMs
2.1 Preliminary
Parameter-Efffciiet Fine- Fine-Tuning, Strameter-Efficien Fine-Tuning (PEFT) saves training costs via introducing and tuning only a small number of additional parameters while keeping the original model weights frozen. This approach has been applied to LLMS and other foundation models, particularly in Low-Rank Adaptation (LoRA) manner, such as LLaMA [19] and Stable Diffusion [48]. The optimization process can be formulated as:

---

datasets
prompt-ckpt pairs
Rossetta
Alpaca
ShareGPT
text encoder
embedding
tuning with foundation model
checkpoints
loss
parameter generator
LLMosstttt
LLpopo
SharLGGT
1. training data preparation
2. training of DnD
in-domain drag
cross- domain drag
GlaiveAs
DnD
LLM Gluivals
Science
DnD
3. evaluating DnD novel datasets
Our approach obtains dragging ability via two processes: prepare training data (upper lef)) and training the parameter generator (upper right). When preparing training data, we explicitly pair parameters with dataset-specific conditions. During training, DnD takes condition as input and generate parameters, using original parameters as supervision.
Figure 2:
min L(W, - BA, A,B
(1)
where Wo is the frozen original weight matrix, low-rank matrices B [gdr and A Bg with r < min (d, k) are the only trainable parameters and 'D represents the fine-tuning dataset. Based on Equation 1, we can conclude that LoRA uses data D raw material and optimization the driving force to obtain weight space shift AW BA, thus making AW and D strongly associated.
Parameter generation. This approach treats models or trained weights as data, aiming to synthesize high-performing neural network parameters without conventional training. Recent advances, such as COND P-DIFF [27], RPG [58], SANE [53], and ORAL [28] have achieved controllable generation by incorporating conditioning mechanisms, allowing primitive personalized generation of parameters for simple datasets. The parameter generation process shares fundamental commonalities with PEFT, where conditions serve as raw materials and the parameter generator provides the driving force to produce target weights with specific properties.
One questions remains: Can utilize parameter generation to effectively "drag-andddropp LLMS' weights towards configurations better suited for given novel task? By "drag-and-drop", we draw an analogy our simple, tuning-free process that directly generates task-specific weights, associating to the intuitive action of dragging a file and dropping it into place without further configuration. Key challenges. Before addressing the above question, we analyze the potential challenges.
Challenge I: How to equip the parameter generator with effective "drag-and-drop" ability? The generator should produce parameters that can effectively adapt LLMs towards specific tasks.
Challenge 2: How to enable adaptation without task-specific training? Traditional PEFT methods typically require training on new tasks, but can achieve comparable performance by directly generating parameters without any fine-tuning on the target task?
Challenge 3: How to make the drag-and-drop function user-friendly and accessible? The generation mechanism should be simple and intuitive, enabling broader adoption and practical deployment.
2.2 Overview of DnD
To address the identified challenges, we present DnD illustrated in Figure As preparation, we first train and save LoRA adapters various datasets. To develop the (drag-and-drop capability, our approach should be aware of parameters' correlations with datasets. Consequently, we randomly pair collected checkpoints with prompt batches from their training data. A pre-trained text encoder then extracts embeddings from the prompts and feed them our parameter generator. The generator features a simple architecture of cascaded pure convolutional decoder blocks (details in Section 2.5 and Appendix A.3). We optimize this generator using mean squared error (MSE) loss between the generated and original tokenized model weights. During inference, we evaluate our approach in both
3

---

in-domain and cross-domain scenarios: simply feeding prompts from novel datasets (not seen during DnD to obtain tailored model parameters with one single forward pass.
training)
2.3 Data Preparation of DnD.
Checkpoint collection. We collect the checkpoints across various datasets, i.e. serving as diverse supervisions, to equip the capability of DnD. The collection process follows previous parameter generation works [59, 58]: training for specified epochs, then performing iterative fine-tuning while preserving checkpoints at each iteration (more details in Appendix A.
The role of prompts. Recent studies [38, 66] demonstrate that samples from different datasets exhibit distinct features, i.e, samples could be considered as "fingerprints" of specified datasets (tasks). Based on this observation, we utilize data samples (prompts) as representative proxies for their respective datasets (tasks). To establish data-parameter mapping, we incorporate prompts from the datasets used to train these checkpoints. These prompts contain dataset-specific features, enabling the generator to infer appropriate "dragging" directions for models across various tasks.
Prompt-checkpoint pairing. Based on the above analysis, the next important question is: how to utilize these elements to equip DnD with strag-and-droy ability in training? Given dataset P. we first divide it into non-overlapping prompt batches [pI,.... .Pi. ... pr]. We note the trained LLM checkpoints of this dataset as M [m,,'... Myj.... m.y]. We randomly pick batch of prompts and corresponding checkpoint. The process can be formulated as,
randomly pick
mp) randomly pick
[pr.... Pi, Pr]
[m]," my, m.),
(2)
where (p:; m2} serves as a pair for parameter generator training. Prompt pt and checkpoint m, serve as the input and supervision, respectively.
2.4 Prompt Embedding.
For each batch of prompts, employ a open-sourced text encoder extract embeddings that serve as inputs for the parameter generator. This extraction process can be formally represented as:
Encoder(ps, 8),
where Encoder(:;)) denotes the embedding extraction function parameterized by 0, and e represents the extracted embedding corresponding to prompt p;. By default, we leverage an encoder-based language model architecture [14] for prompt embedding. In the experimental section, we further explore and quantitatively evaluate alternative embedding approaches, including word2vec represen- tations [44], encoder-decoder architecture [46], and decoder-only language models [63].
2.5 Training and Inference of DnD.
Figure 3: Block details of parameter genera- tor. Each block of conellccnvolttion contains three hyper-convolution modules, extracting and fusing features in different dimensions. More details are in Appendix A.3.
Convw
(B. N. L, C)
ConvL
(B. Nw. Lw. Cw)
Convh
convolution
summation
specilled dimension
Structure of parameter generator. Different from diffusion-based parameter generation [59, 27, 58], we use a conpoutiooollti decoder to learn the mapping between input prompts and parameters. That design mainly considers efficiency, as the decoder-only structure has shown its superiority in LLMs [1, 19, 63, 22]. We show the block details of the decoder in the right part (Figure 3). We assume the dimension of input prompt embeddings is [B, N, T., C). where B. N, L,, and C denote batch size, length of prompt batch (i.e., number of prompts), sequence length, and hidden dimension, respectively. The cascaded convolutional blocks transform prompt embeddings match the dimensions of tokenized weights. Here, we refer the output of the last block as [B, N.. Lw, Cw]. Learning objective. The learning objective is simple: we calculate the mean squared error (MSE) loss between the output of the last block of the parameter generator and the corresponding tokenized checkpoints. Similar to RPG [58], we tokenize each layer's parameters into non-overlapping segments and apply padding, ensuring checkpoints have a consistent shape of [B, Na., J.w,, C.r). Formally, we write the MSE loss below,

---

prompt embeddings
parameter generator
LMSE
lokenization (58, corresponding checkpoints.
Inference. We expect the parameter generator to develop effective "drag-and-drop"" ability, particu- larly for novel datasets or tasks not encountered during training. Therefore, our evaluation primarily focuses on performance across novel datasets. The inference process consists of four steps: 1) sampling prompts from novel datasets, 2) extracting embeddings from these prompts, 3) feeding the embeddings into the parameter generator, and 4) evaluating the generated parameters on the novel datasets. To comprchensively demonstrate the "drag-an -nd--nddooo ability of our approach, examine performance both in-domain scenarios (e.g., common cense-to-commen sense) and cross-domain scenarios (e.g., common sense-to-science)
3 Experiments
3.1 Implementation Details
We choose Qwen2.5 [63] series as foundation model and conduct experiments on common sense reasoning, coding, math, and multimodal tasks. The details of involved model sizes and datasets for each task are listed in the table below. The default text encoder is Sentence-BEie [47], and length of prompt batch is set to 128, 64, 64 and 32 for common sense reasoning, math, coding, and multimodal task, respectively. For other
settings, please refer Appendix A.1.
task
#model size (B)
datasets
common sense
0.5
ARC-c ARC-c [11], BoolQ [10], OBQA [41],
HelaSwag [65], PIQA [6]. WinoGrande [50]
coding
1.5,7
Evol-Instrrct-688K- [40], Glaive-Assistant-V V2 [18].
Cython-Codes225K [16), 2aa-774k-ShareGP [2), Rosetta-Code [13],
LLaMA-Pythour Sowerhoonnoooes3030- [15]. CodeAlpaca-2009 [8]
math
1.5
CempetiiioonMMat//22
Math-IlO-68K-Minn [45]
Math-Plus [56]. Mu-Math [64]. ToT-Math- -1 [42]
multimodal
Math [55]
3.2 Common Sense Reasoning
Evaluating setting. We employ LoRA [25] to fine-tune Qwen2.5-0.5B on seven common sense reasoning datasets and save the checkpoints training data. In every column of Table 1, the specified dataset as test set (i.e,, not used in training) and train DnD on other datasets' LoRAs.
method \ test
ARC-e
OBQA
ARC-c
PIQA
HellaSwag
BoolQ
WinoGrande
training LoRAs
37.5
30.2
39.5
40.5
22.4
13.5
38.8
DnD
68.6
40.8
51.6
87.9
25.9
44.9
50.0
average accuracy improvement: 21.0 training LoRAs
Table 1: Generalization on novel (test) datasets. Our approach significantly outperforms LoRAs used in training in terms of accuracy (%) across all unseen datasets. Bold entries are the best results.
Analysis. We report the average accuracy of training LoRAs and our generated ones in Table 1. Several observations can be made from these results: i) Our method consistently outperforms LoRAs used for training on unseen datasets, indicating it manages drag-an and- and-drop LLM parameters to sask-specific distribution specified via condition. ii) This frru--nnddroy ability holds across different datasets, showing strong robustness towards various data inspirations.
Cross-domain Drag-and-Dree To further ex- plore DnD's i zero-shot ability. we not only use in-domain novel datasets in inference, but also test it on cross-domain task. In this part, we use the checkpoint trained on common sense reasoning tasks and feed it with prompts from science-dataset [34]. The generated parameters are compared with training LoRAs on the science dataset. From Table 2, we can observe that DnD surpasses its training LoRAs' average accuracy,
testset science
training LoRAs
DnD
improvement (t)
35.6
45.3
9.7
Table 2: Our approach also succeeds on cross- domain scenario (i.e., both novel data and task).
5

---

indicating our method manages to moug-and-drop LLMs for cross- tasks (i.e., from common sense reasoning to science). Note training LoRAs are trained on ARC-e, OBQA, PIQA, HellaSwag. WinoGrande, BoolQ, and they've ve never seen science dataset.
Coding (HumanEval)
Math
Multimodal
method task
pass@1
1
pass@5
@
10
gsm8K
MATH
Math- Vision
Math-Vistr
training LoRAs
17.6
28.6
33.2
42.9
14.8
23.0
61.5
DnD
32.7
55,3
64.1
66.3
23.9
24.3
62.3
improvement (t)
15.1
26.7
30.9
23.4
9.1
1.3
0.8
Table 3: DnD can generate parameters for more complex tasks like math, code and multimodal question answering. Our method continues to show strong zero-shot ability on these tasks.
3.3 Generalization to coding, math, and multimodal tasks
To further validate our method's applicability in more complex scenarios, we also employ DnD for coding, math, and multimodal tasks. The empirical results and findings are as belows.
Coding. Similar common sense reasoning task, use LoRA to fine-tune Qwen2.5-1.5B on seven coding datasets and save the checkpoints as training data. Evaluation is carried out on HumanEval [9] benchmark using pass @k [33] score
5. 10). Note that neither LoRA fine-tuned models nor DnD has seen any samples of the benchmark in their training. Therefore, we directly test training LoRAs and our synthesized ones on HumanEval. From Table 3, we can draw some findings: i) Our method yields promising results, with improvement over average pass @ 1 = 15.1, pass@5 = 26.7, pass @ 10 30.9. ii) Despite training LoRAs perform poorly on the test set, DnD still obtains good performance. This means instead of memorizing parameters seen in training, it learns to fit novel datasets given condition as inspirations.
Math. We fine-tune Qwen2.5- 1.55B on six math datasets and save the checkpoints. We adopt gsm8K [12] and MATH [24] as our benchmarks and accuracy as evaluation metrics. Results listed in Table 3 show commonalities with those regarding common sense reasoning and coding tasks, underscoring the superiority of our method across a wide range of scenarios.
Multimodal. The above results confirm our method's s effectiveness in text modality. In the following. we explore its greater potential by pacing towards other modalities. We fine-tune Swee2.55VL--3. [5] on V36KK [55], save the checkpoints, and evaluate using Math-Vision [60] and Math-Vista [39]. Results in Table 3 show that DnD perform well in multimodal tasks, revealing that our method can be adapted to modalities other than texts and has promising application potential.
Based on the above results and comparisons, DnD is high-performing zero-shot learner with strong robustness and wide applicability, as reflected by the significant improvements compared to its training data and its promising performance across various scenarios. In the following, we continue to explore more interesting features of our proposed approach.
Takeaway:
3.4 Ablation Studies
This section mainly aims at exploring series of interesting features about our approach. For those exploring various settings in our experiment, we report them in Appendix B.3 for thoroughness. If not stated, use ARC-c the test set and other common sense reasoning datasets for training. What types of data will help Drag-and-Drop LLMs better? As introduced in Section 2.3, prompts as conditions to inspire DnD. Can this drag-and-dee and-drop ability holds when condition types changes (e.g., answers)? We carry out ablation studies by changing condition types as prompt, prompt + answer and their mixture (prompt: answer=4: 1) and report the results in Table 4a.
It can be observed that the prompt + answer group surprisingly lead to poor performance. We conclude that it is because answers common sense reasoning datasets lack diversity (i.e, A/B/C/D) and combining them with prompts may detriment dataset-specific representations. This shall hinder generator to distinguish different datasets and generate specific parameters. Consequently, we advise not to use answer alone inspirations. However, conclusions may be different for some tasks where answers are more complicated and diverse and we show the results on these tasks in Appendix B.3. How does the choice of condition extractor affect DnD's performance? Our default condition extractor is Sentence-BEn [47], yet it is interesting to explore other models' potentials. To ensure

---

condition type
accuracy
condition extractor
accuracy
dataset arrangement improves.
prompt
51.6
Glove
50.8
6 train. E test
12.1
prompt + answer
27.0
Sentence BERT
51.6
4 train, 3 E test 11.4
mix
49.7
T5-base
50.2
3 train, 4 E test
0.8
training LoRAs
39.5
Qwen2.5-7B
fail
2 train, E test
-1.4
(a) Condition types. Pure prompts used as inspirations yield the best results compared other formats.
(b) Extractor structure. Several eneoder-based extractors perform better than decnder-only ones.
(c) Train-test set arrangements.
More diverse training data intro- duces higher improvements.
length
DnD
improves.
method
w/o cvg.
cvg.
method
full
10%
8
33.2
-6.3
training LoRAs
27.3
39.5
training LoRAs
39.5
29.8
32
47.3
7.8
DnD
46.3
51.6
DnD
51.6
49.1
128
51.6
12.1
improvement (t)
19.0
12.1
improvement (t)
12.1
19.3
(d) Length of prompt batch. As prompt batch grows longer, DnD's performance improves accordingly.
(e) Checkpoint quality, 'evg.' de- notes 'converged', DnD maintains strong even with weak LoRAs.
Dataset size. DnI maintains strong even with only 10% data (51.6% 49.1%).
Table 4: Ablation studies about condition types, condition extractor's type, train-test set arrangement, length of prompt batch, checkpoint quality, and dataset size. This series of explorations validate several designs of DnD.
thorough comparisons, we include classical word2vector method Glove [44], the default encoder-only Sentence-BERT [47], encoder-decoder model T5 [46] and decoder-only Qwen2.5-7B [63].
Results in Table 4b reveal several insights: i) Even traditional methods such as Glove can help DnD to obtain promising result, indicating method fit plenty of text encoders. ii) Qwen2.5-7B performs not as good as expect, which has two possible causes: First, its heaviness limits the number of conditions paired with parameters per iteration, leading poor awareness of novel datasets. Similar conclusions can be drawn from our experiments in Appendix B.3. Second, Qwen2.5-7BB decoder-only architecture may constraint conditions' diversity, since it encodes prompts to answers. What property of training data equips our method with drag-and-drop ability? By default, we train on several datasets and test on 1 novel dataset. In this part, we explore DnD's robustness by shrinking the number of train sets and test on more datasets. Train-test set arrangements are: 6-1, 4-3, 3-4 and 2-5. Generated parameters are compared with training LoRAs' average accuracy on the unseen datasets and their average improvements in accuracy are reported in Table 4c.
can be observed that: i) Generally, more training datasets lead to better performance improvement. This is expected since more data ensures better coverage of condition-pann parammmeee correlations and lead to better robustness for novel data. ii) DnD fails to drag-and-drop LLMs to novel datasets when training samples are few. As datasets used for training lessen, the average improvement of DnD drops accordingly. It hardly improves over training LoRAs for the 2-5 case. We can conclude that basic amount of training samples are needed for DnD to learn condition partmmeee correlations.
How does DnD's performance compared with foundation LLMs? Given massive pretraining LLMs often take, fine-tuning on a small down- stream dataset may detriment their zero-shot per- formance on novel test sets. Aware of this phe- nomenon, we compare DnD generated weights' performance with foundation LLMs across tasks involved in our experiment. Specifically, for foundation I.LMs, we adopt Dwen2.5-0.5B for common sense reasoning. 1.5B for math and coding, and Qwen2.5-VIL-3B for multimodal task. Results in Table again show our approach's 's superiority: DnD outperforms foundation L.MMs across all tasks. Its "drag-and-drop" ability can generate task-specific parameters, with performance better than foundation LLMs that go through abundant pretraining.
testset \ method foundation LLM
DnD
improves. (f)
ARC-c
38.3
51.6
13.3
HumanEval
32.3
64.1
31.8
gsm8K
64.4
66.3
1.9
Math-Vision
22.7
24.3
1.6
Table 5: DnD surpasses foundation LLMs across various tasks, showing the "drag-and-drop" effect.
How does the length of prompt batch impact DnD? We change the length of prompt batch, turning it to 8, 32, 128 and explore their influence on DnD. From Table 4d, when prompt batch is too small (e.g., 8), DnD's performance not good. but can surpass training LoRAs with larger prompt batch
(c.g. 2 32). As prompt batch grows longer, DnD's performance improves accordingly.

---

How does the quality of LoRAs impact DnD? In addition to collecting well-traines checkpoints, we collect checkpoints the beginning of training trajectory, i.c., checkpoints not converged on the training sets. As shown in Table 4e, DnD maintains good performance even with worse-performing LoRAs. Its performance drops with training LoRAs that are insufficiently tuned, but still surpasses converged training LoRAs on zero-shot test set. Therefore, Dnl) works well with worse training LoRAs, which reveals that our method has strong data adapting ability.
How does dataset size impact LoRA training? In our paper, we use the entire dataset to train and collect LoRA checkpoints. To ablate the influence of dataset size, we use 10% of each dataset to train LoRAs, then adopt these LoRAs for DnD's training, and evaluate their performance. Based on the results in Table 4f, even trained with only 10% data, DnD still maintains good performance (51.6%- + 49.1%). As expected, training LoRAs' performance drops largely (39.5% + 29.8%) with only 10% training data. DnD' performance drop is much smaller than training LoRAs novel datasets (i.e., ARC-c). Therefore, DnD is less sensitive to dataset size than traditional LoRA tuning, demonstrating its strong dataset-adapting ability.
3.5 Open Explorations and Analysis
Pairing. In this part, we explore other pairing strategies influence on performance than that introduced in Section 2.3. We test condition pairing strategies:
Strategy 1: We fix the total number of prompts to be 128, 256, 512, 1024, 2048, 5000 and use all those prompts to pair with parameters every iteration (æ =).
Strategy 2: We fix the length of prompt batch to be 128 in every iteration and randomly picks these prompts from 128, 256, 512, 1024, 2048, 5000 candidate prompts (128 <- r).
70-
60-
strategy 2:128
stratogy 1:X=x
46
-
49
51
80-
1u1-shet (more terstion on
Ni1-shet
TrD
55
1.5C
2.0K
saving 120- 60000x overhead
1.0k
30
PS 26c
KL acC
KL awathezd
coathead
30
26
500
20-
0.1
16
4- 120
x
256
512
A-
1024 5000
perforannce
overtwad
16
64
256
(a) Random selection and pairing is better than using the same condi- tions for all parameters.
(b) DnD can reach comparable or even better performance than full- shot tuning and ICL before 256 shot while being 2.5K x faster. shots while avoiding using answers.
(c) DnD outperforms popular few-
Figure 4: Explorations about DnD's cunditio--paameet pairing strategy, compare DnD with state- of of-the-art art methods, and its superiority over few-shot tuning and in-context learning.
Based on the results in Figure 4a, we can draw several conclusions: i) With limited number of conditions, DnD fails to generalize over novel data, since it can hardly learn comprehensive knowledge about condition-paranne
mapping's 's landscape. ii) As number of condition increases, Strategy 2's performance skyrockets since DnD is exposed to sufficient condition-parameter pairs. This indicates Strategy 2 may help DnD to converge efficiently. iii) Strategy needs more conditions to reach comparable performance as Strategy 2. With large number of conditions, Strategy 1 suffers from out of memory issues. The same-condition per-parameter strategy may hinder DnD to associate conditions with specific datasets. Conclusively, Strategy 2 is superior than Strategy 1 both model generality, convergence speed, and memory consumption. These conclusions are consistent with findings in Table 4c: Diversity of training data equips our method with drag-and-die -nd--rop ability.
DnD vs full-shot tuning. In this part, we compare DnD with full-shot tuning in accuracy and overhead. Specifically, we test ARC-c tuned LoRA's (==5 iterations for ARC-c, detailed in Appendix A.4) full-shot performance using ARC-c's s test set. Since LoRA's performance might improve with more iterations, we fine-tune LoRA on ARC-c for 300 iterations and test its performance as well. These results are compared with zero-shot DnD in Figure 4b: i) using mild tuned checkpoints for training, DnD already yields better results than full-shot tuning. This showcases DnD's impressive zero-shot ability even surpasses full-shot tuning. ii) DnD is incredibly efficient, with better performance than full-shot tuning while being 2500 x faster. Moreover, as training continues, though full-shot prevails, our method shows minimal performance gap with it while being 12,000 x more efficient. Efficiency analysis of DnD. In additional to full-shot tuning, in-context learning (ICL.) and few-shot tuning (FS) are also popular methods in LLM fine-tuning. In Figure 4c, conduct experiments
8

---

investigating the celtorrmaane--ffciiinee trade-off of them. Several observations can be drawn: i) Both ICL and FS' results are poor when shots are few, but their overhead rises as shots increase. ii) DnD can reach better performance than FS and ICL. before 256 shots with negligible overhead. iii) noteworthy that both few-shot and ICL. use answers for instructing the LLM to obtain better performance. On the contrary, DnD relies on only 128 unlabeled prompts. Based on the above results, we anticipate DmD is a powerful and efficient zero-shot learner.
Scalability of DnD. In this part, we explore DnD's scalability. Since common sense reason- ing is simple and 0.5B model suffices, we fo- cus on math and coding tasks while increasing foundation model size from 1.5B to 7B. We use original math and coding datasets in Section 3.3, evaluate using gsm8K for math and more diffi- cult benchmark LiveCodeBench [26] for coding. We report accuracy math task and pass @ 1 for coding in Table 6. It can be observed that: i) DnD consistently surpasses training LoRAs in both tasks under 7B model setting, underscoring its eminent scalability for larger foundation LLMs. ii) With more difficult coding benchmark, DnD maintains superior performance with training LoRAs, i.., improvement over average pass @1 20.3. It reveals DnD's capacity for generalizing more complex benchmarks, showing promising application potential and robustness. Comparisons with previous methods. We compare our
testset method training LoRAs
DnD
improves (1)
LiveCodeRench gsm8K
13.0
33.3
20.3
65.9
83.1
17.2
Table 6: DnD scales well with larger 7B foundation models, and maintains strong performance in more complex benchmark LiveCodeBench.
spen
792
method with most recent parame-
8
ter generation method RPG [58].
We explore both methods' perfor- mance in two scenarios: Close set generation: generate param- eters seen training. Open set generation: generate parameters
20
B180))
Figure 5: DnD and RPG perform well in most close-set tests. However, RPG can hardly generate parameters for novel dataset while DnD still presents strong zero-shot ability on open-set test. for unseen datasets. Figure 5 shows both methods work well on close set generation, but RPG fails on open set generation, showing that our designs (prompts as conditions, condition-paranee pairing) have certain robustness and generality towards novel datasets.
Visualization for the effect of drag-and-drop In this part, we visualize original and generated parameters in Figure 6. It can be seen that original parameters exhibit diverse patterns in the weight space, forming different clusters. Moreover, even param- eters close to those tuned on the target dataset (i.e., ARC-c) can have large performance gap (i.e., 19.1% compared with 40.7%). After training on these models that have distinct features, DnD original ones in the weight space and promising performance. can generate parameters for the target dataset in a zero-shot manner. The generated parameters are close the original ones in the weight space and with even superior performance than full-shot tuning (i.e., 51.6% compared with 40.7%). This brings the "drag-and-drop" effect to life.
Acc: 49.4% -51.6%
Acc: 19.1% 46.7%
Acc: 24.5% 27.3%
Acc: 40.1% -40.7%
Acc: 47.6% 50.1%
Acc: 49.7% 51.3%
training LoRAs (except ARC-c)
full-shot training on
DnD generated ARC-c
Figure 6:
DnD generates parameters with close distribution to
Exploration of the domain gap influence. Without loss of generality, we test cross-domain with large domain gap, i.e., ooding-to-math. We use datasets and checkpoints in math and coding tasks to train DnD jointly. Based on the results in Table 7, DnD improves largely over its training LoRAs on gsm8K, showing it has the potential of cross complex domains.
method training LoRAs
accuracy (%)
46.3
DnD with joint training
65.3
improvement (f)
19.0
Table Performance on gsm8K: train- ing LoRAs vs. DnD with joint training.

---

4 Related Works
Parameter-Effiiient Fine-Tuning (PEFT). LLMs' size rapidly scales up in the 2020s, making full parameter fine-tuning infeasible. To address this, Low-Rank Adaptation (LoRA) [25] has been proposed, leveraging L.MSs inherent sparsity and substantially reduces fine-tuning costs by optimizing two low rank matrices instead of the original weights. Multiple variants of LoRA emerge afterwards, such as DoRA [37], LoRA+ [23], VeRA [31], and DyLoRA [57]. However, all of them have one potential flaw: they requires tuning model parameters for every novel dataset, therefore lacks generality. This can still induce extra costs as model size enlarges and training data increases.
Parameter generation. Parameter generation trains on model checkpoints and aims at generate high-performing parameters, both seen and unseen in training. Previous works [7, 20, 30, 17] focuses on learning distributions over the parameters, yet struggle to reconstruct original models' performance. With the development of diffusion models, Hyper- Representationss [51, 52, 54] and p-diff [59]. use the latent diffusion architecture to generate frrhh-perfomiig parameters. Armed with Mamba [21] and appropriate tokenization strategy. RPG [58] can generate 200M of parameters in minutes. Regarding conditional generation, COND P-DIFF [27], Tina [35] and ORAL [28] explores text-controlled parameter generation method. RPG even generate parameters for novel datasets binary embedding classification task for CIFAR-I0 [32]. However, these methods can hardly keep promising zero-shot ability on more complex tasks, hindering parameter generation's is greater potential. On the other hand, our method use prompts in novel datasets as conditions, captures parameters' relations with datasets better, and is able to generate competent parameters for novel datasets.
5 Discussion and Conclusion
PEFT offers efficient solution reduce costs when customizing LLMs/VLLMs for downstream tasks, yet it is still expensive when models are large and tasks are diverse. In our study, we train a parameter generator map prompt-weight pairs, which can produce customized weights for novel tasks by processing unlabeled task prompts. Notably, our approach can transform unlabeled task prompts directly LoRA weights update seconds. This prompt-to-wend to-weight paradigm, which can generate task-specific weights without further tuning, sheds lights on a promising new direction for efficient LLM and VLLM customization.
Previous research [51, 59, 58] and our approach demonstrate that neural network weights can be effectively synthesized. It appears that network parameters be viewed as a new form of data modality. To excavate this emerging field's potential, several challenges remain to be tackled. First, scaling parameter generation larger models (7B-70B parameters) requires novel architectural and algorithmic advances. Second, leveraging existing pre-trained checkpoints from the Internet could enhance the practicality of parameter generators. Third, generating structurally diverse models adaptable to various hardware configurations would improve deployment flexibility.
Acknowledgments and Disclosure of Funding
We sincerely appreciate Yuxiang Li, Jiaxin Wu, Zhiheng Chen, Lei Feng, Jingle Fu, Hesen Yang, Bohan Zhuang, Ziheng Qin, Zangwei Zheng, Zihan Qiu, Zexi Li, Gongfan Fang, Xinyin Ma, and Qinglin Lu for valuable discussions and feedbacks during this work.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[2] ajibawa 2023.
Code-744-ShareCCie 2023.
[3] Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Soncel-Kedzionst Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation- based formalisms. In NAACL, 2019.
10

---

[4] Anonymous. Towards personalized parameter generation via data-conditiones mapping. In Submitted to The Fourteenth International Conference on Learning Representations, 2025. under review.
[5] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
[6] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. AAAI, 2020.
[7] Léon Bottou et al. Stochastic gradient learning in neural networks. NeuroNimes, 1991.
[8] Sahil Chaudhary. Code alpaca: An instruction instuwction-folloiing llama model for code generation. https://g/ gi thub. com/ sahi 1280114/ 1/ac///aacccaaaa.../aa 2023.
[9] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
[10] Christopher Clark, Kenton Lee, Ming-Wet Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACL, 2019.
[11] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint urXiv:1803.05453 2018.
[12] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110, 14161,68,, 2021.
[13] Rosetta Code. Rosetta code rosetta code., 2022. [Online; accessed 8- B-December-2022).
[14] Jacob Devlin, Ming-Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
[15] flytech.
llama-yython-codes-300. llama-python-codes--000 2023.
[16] flytech. flytachhppthhon-ccddee-2255 python-codes-25k, 2024.
[17] Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In ICML, 2016.
[18] glaiveai. glavveudglavve.cod--asitttt-- glaive-code-assistate 2024.
[19] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama
3 herd of models. arXiv preprint arxiv:2407.2178. 2024.
[20] Alex Graves. Practical variational inference for neural networks. In NeurlPS, 2011.
[21] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In COLM, 2024.
[22] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12945 2025.
[23] Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large models. In ICML. PMLR, 2024.
[24] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In NeurIPS, 2021.

---

[25] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 2022.
[26] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. ICLR, 2024.
[27] Xiaolong Jin, Kai Wang, Dongwen Tang, Wangbo Zhao, Yukun Zhou, Junshu Tang, and Yang You. Conditional lora parameter generation. arXiv preprint arXiv:2408.01415, 2024.
(28) Rana Muhammad Shahroz Khan, Dongwen Tang, Pingzhi Li, Kai Wang, and Tianlong Chen. Oral: Prompting your large-scale loras via conditional recurrent diffusion. arXiv preprint arXiv:2503.24354 2025.
[29] Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Iision-and--e ann-lang transformer without convolution or region supervision. In ICML, 2021.
[30] Diederik P Kingma and Max Welling, Auto-encoding variational bayes. 1rXiv:1312.61 14, 2013.
arXiv preprint
[31] Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M Asano. Vera: Vector-based random matrix adaptation. In ICLR, 2024.
[32] Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
[33] Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy S Liang. Spoe: Search-based pseudocode to code. In NewrlPS, 2019.
[34] LawalAfeez. Lawalafeez/sciincc-daaase 2023.
[35] Zexi Li, Lingzhi Gao, and Chao Wu. Text-to-model: Text-conditioned neural network diffusion for frain-lnce-for-all personalization. arXiv preprint arXiv:2405. /4132, 2024.
[36] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint 1943 2024.
[37] Shih- Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. In ICMI., 2024.
[38] Zhuang Liu and Kaiming He. A decade's battle on dataset bias: Are we there yet? In ICLR, 2025.
[39] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In ICLR, 2024.
[40] Ziyang Luo, Can Xu, Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma. Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. In ICLR, 2024.
[41] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? dataset for open book question answering. In EMNLP, 2018
[42] moremilk. Tot-math-v1. ToT-Math-V 11, 2023.
[43] William Peebles, Ilija Radosavovic, Tim Brooks, Alexei A Efros, and Jitendra Malik. Learning to learn with generative models of neural network checkpoints. arXiv preprint arXiv:2209. 12892, 2022.
[44] Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In EMNLP. 2014.
[45] PrithivMLmods. Math-ioo-6kkmini. Math-I10668K-Minn 2023.
12

---

[46] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. JMIR, 21(140):1-67, 2020.
[47] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In EMNLP, 2019.
[48] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[49] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wad- hwa, Michael Rubinstein, and Kfir Aberman. Hyperdreambooth: Hypemetworks for fast personalization of text-to-image models. In CVPR, 2024.
[50] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In AAAI, 2020.
[51] Konstantin Schürholt, Boris Knyazev, Xavier Giró-i Nieto, and Damian Borth, Hyper- representations for pre-training and transfer learning. In NeurlPS, 2022.
[52] Konstantin Schürholt, Boris Knyazev, Xavier Giró-i Nieto, and Damian Borth. Hyper- representations for pre-training and transfer learning. arXiv preprint arXiv:220.109551, 2022.
[53] Konstantin Schürholt, Michael W Mahoney, and Damian Borth. Towards scalable and versatile weight space learning. ICML, 2024.
[54] Konstantin Schürholt, Michael W Mahoney, and Damian Borth. Towards scalable and versatile weight space learning. arXiv preprint arXiv:2406.09997, 2024.
[55] Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See Kiong Ng. Lidong Bing, and Roy Lee. Math-llava: Bootstrapping mathematical reasoning for multimodal large language models. In EMNLP, 2024.
[56] TiGER-Lab. Math-plus. MATH-plus, 2023.
[57] Mojtaba Valipour, Mchdi Rezagholizadch, Ivan Kobyzev, and Ali Ghodsi. Dylora: Parameter- efficient tuning of pre-trained models using dynamic rearch-free low-rank adaptation. In ACL, 2023.
[58] Kai Wang. Dongwen Tang, Zhiyuan Liang, Wangbo Zhao, Konstantin Schürholt, Zhangyang Wang, and Yang You. Recurrent diffusion for large-scale parameter generation. NeurIPS, 2025.
[59] Kai Wang, Dongwen Tang, Boya Zeng, Yida Yin, Zhaopan Xu, Yukun Zhou, Zelin Zang, Trevor Darrell, Zhuang Liu, and Yang You. Neural network diffusion. arXiv preprint arXiv:2402.1314. 2024.
[60] Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. In NeurIPS, 2024.
[61] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self- Aligning language models with self self-generated instruc- tions. In ACL, pages 13484-13508, 2023.
[62] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe Le, Denny Zhou, et al. Chain-of-thouugh prompting elicits reasoning in large language models, In NeurlPS, 2022.
[63] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024.
13

---

[64] Shuo Yin, Weihao You, Zhilong Ji, Guoqiang Zhong, and Jinfeng Bai. Mumath-code Combin- ing tool-use large language models with multi perspeetrree data augmentation for mathematical reasoning. arXiv preprint 4rXiv:2405.07551, 2024.
[65] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In ACL, 2019.
[66] Boya Zeng, Yida Yin, and Zhuang Liu. Understanding bias in large-scale visual datasets. In NeurlPS, 2024.
14

---

We organize our appendix as follows.
Hyper-Parameete Settings
Section I. Training Recipe.
Section A.2: Description of Datasets,
Section A.3: Detailed Structure of Hyper-Convolutiona Decoder.
Section A.4: Details of Trained Checkpoints Collection.
Additional Experiment Results
Section B.I: 1: More Results of Common Sense Reasoning, Math, and Coding.
Section B.2: Inference Efficiency Analysis.
Section B.3: More Ablation Studies.
A
Hyper-Parameter Settings
A.1 Training Recipe
In this section, provide details of our training recipe and various hyper-parameter settings. We
incorporate multiple tasks in language models, each involves different foundation model sizes, different generator architecture, and training schedules. We report settings for every task in Table 8.
training setting
common sense
coding
math
multimodal
batch size
128 (0.5B)
128 (L.5B), 48 (7B)
128 (L.5B). 48 (7B)
64 (3B)
optimizer
AdamW
AdamW
AdamW
AdamW
learning rate
3e-5
30-5
3e-5
3e-5
length of prompt batch
128
32
16
training step
5,000
5,000
5,000
5,000
weight decay
0.1
0.1
max grad norm noise aug. amplitude
1.0
1.0
1.0
1.0
le-4
le-4
le-4
le-4
Table 8: Training recipe for different tasks in Section 3.2 and Section 3.3.
Length of prompt batch. has been introduced in Section 2.3 that every parameter is grouped with certain amount of texts each iteration. Due the length of prompt in different datasets varies and induces variable training costs, the length of prompt batch also varies.
Revision statement. During preparation of camera ready version, we fix some minors of the code and list them here: 1) the tokenization process, 2) the warm up scheduler. We updated the code to our GitHub repository.
A.2 Description of Datasets
In this section, we introduce the datasets used in the paper, including those for common sense reasoning, math, coding, and multimodal tasks.
Common sense reasoning. ARC dataset [11] contains grade-schodd level, multiple-choice science questions and is splited into easy and challenge sets. OBQA [41] aims to promote research in advanced question-anse answering with salient facts summarized as an open book. PIQA [6] focuses on everyday situations with preference for atypical solutions. HellaSwag [65] instructs models to select from choices that best finish the sentence among ground truth and an adversarial set of
machine-generated wrong answers. WinoGrande [50] features fill-in-a-blank task with binary options for commonsense reasoning questions. BoolQ [10] is question answering dataset for yes/no questions containing various factual problems.
Coding. Evol-Instruce [40] contains evolutionary prompts tailored for code-related tasks and incorporates code debugging and time-space complexity constraints. Glaive-A [18] is dataset of code problems and solutions generated using Glaive's synthetic data generation platform. Python-Codes [16] is cleaned Python dataset covering instructional tasks. Code-ShareGPT [2]
15

---

consists of conversations along with detailed Python code explanations. It generated using GPT-3.5, GPT-4 etc. Rosetta-Code [13] presents solutions the same task in as many different languages as possible, to aid a person with grounding in one approach to problem in learning another. I. Python-Codes [15] primarily focuses on instructional tasks in Python, tokenized specifically for the Llama architecture. It is a blend of GPT-4 generated content, custom codes, behavioral approaches and tasks. Code-Alpaca [8] dataset is generated by the techniques [61], with some modifications. Math. Competition-Matt [24] consists of problems from math competitions, including the AMC 10, AMC 12, AIME, and more. Math-110-68KK-10
[45] mathematical questions and with corre- sponding step-by-step solutions. MathQA [3] is large-scale dataset math word problems that are densely annotated with operation programs. Math-Plus [56] is augmented dataset with GPT-4. Mu-Math [64] is a meta-cvaluation dataset derived from the U-MATH [64] benchmark, intended to assess the ability of LLMs to judge free-form mathematical solutions. ToT-Math- V1 [42] prioritizes reasoning and explanatory problem-solving over provided answers.
Multimodal. Math [55] consists 40K images from datasets and 360K question-answer pairs. It is used to enhance the multimodal mathematical reasoning capabilities of MLLMs.
A.3 Detailed Structure of Sepeo-Convllutional Decoder
Details of condition extractor. As introduced in Section 3, we use Sentence-BERT [47] as our condition extractor in default Gall-MiniLM--L--e specifically). Since BERT's supported sequence length is only 512, for longer sequences, we need to preprocess the input sequences. Specifically, we first pad the sequence the length that can be divided by 512, slice it into multiple sub-strings, and encode the sub-strings respectively. The input length for different tasks is in the table below.
common sense
coding
math
multimodal
length
384
26624
4608
1536
Details of hoper-convoluttinall decoder.
In this part, we delve into hyper-convolutional decoder's inner architecture introduced in Section 2.5.
The decoder consists of multiple cascading hyper-convolutional blocks, each containing 5 2D convolution modules, Specifically, we divide convolutions into three categories: i) width convolution that operates on (C, L) dimension, ii) height convolution that operates on (L, N) dimension) iii) layer-wise convolution that on (N, L) dimension), with notations Convw, Conv и. and Conv z. In the above convolution modules, the input tensor transposed to the shape that specified dimensions act as feature maps, the remaining dimension act as channel dimension like conventional convolution. Each byper-convolutional block consists of two Convw, two Conv y and one Conv Given this, the forward operation of hyper-convolutional block can be formulated as:
c Conv (Conv (c' 1));
CH = Conv (Conv (c' (-1-)) 1));
d' Conv &w + cH +b, 3
where ct is hidden state output by the th layer, c² is prompt embedding encoded by the condition extractor, and b is learnable bias. Through this process, transforms the input with shape of [B, N. L, CJ to [B, N", I', C"). Model architecture used in different tasks. In this part, we show the architecture of hyper- convolutional decoders. We use the three element tuple (N, L, C') to represent decoder structure since it reflects input conditions" changes in the network. Also, B dimension is omitted since it doesn't 't. affect the model architecture, Note that for math and coding tasks, we first project the dimension to 1000 to reduce overwhelming memory costs induced by giant convolution kernels.
A.4 Details of Trained Checkpoints Collection
In this section, we discuss how we collect checkpoints in detail. It is noteworthy that all checkpoint collection process go through two phases: i) Pretraining on the target dataset for specified steps. ii) Fine-tuning on the target dataset for certain additional steps, while saving checkpoint at each step.
Therefore, the essence of checkpoint collection the pretrain and fine-tune phase's 's learning rate, training steps, number of samples, and batch size. Note that except for learning rate and training
16

---

task
foundation model
channel
common sense
0.5B
(128, 384, (1443344)2(128 200, 300) 100, 256)
(256, 50, 200)>+(112, 50, 200)=+(1024, 23, 200)
(1024, 10, 200)
202)-, 128)
coding
1.5B
(32, 1000, 384) 500, 300)->(256, 500, 300)->(11, 125, 300)
(1024, 64, 256)-2 32, 256) (x4508, 16, 256)
(16, 1000, 384))>(64, 500, 384)-x(25,, 125, 400)-> 400)
(1024, 64, 400)) 32, 400), (4928, 16, 512)
math
1.5B
(16, 1000, 364) 500, 300))x 125, 300)->(512, 64, 300)
(1024, 64, 2520) 32, -45)-= 16, 256),
7B
(16, 1000, 384)-(64, 500, x82), 125, 400)->(512, 64, 400)
(1024, 64, 400) (2000), 32, 400), (4928, 16, 512)
multimodal
3B
(16. 1536, 384)->(64, 500, 300)- 125, 300)
(1024, 64, 300)), 16, 253)-+ 16, 256)
Table 9: Model architectures for different tasks.
steps, other haper-parameters are kept the same. Detailed settings are Table For those
datasets contain less samples than the number specified below, use the entire dataset for training.
pretrain
finetune
task setting
training step
batch size
#num. samples
training step
common sense
le-4
75
32
5000
lc-5
50
coding
le-4
4000
64
10000
le-6
100
math
le-4
4000
10000
1c-6
multimodal
le-4
8000
100000
le-6
Table 10: Details for checkpoint collection.
The number of checkpoints used for each dataset during model training impacts the learning dynamics of mapping data to weights. Empirical experiments revealed that (1) insufficient checkpoints may hinder the model's 's ability to capture nuanced parameter interactions, while (2) excessive checkpoints increase computational overhead without proportional gains. To balance efficiency and performance, we configured checkpoint counts as follows.
task
number chockpoints
common sense reasoning
50
coding/math 1.5B, math 7B
80
coding 7B
100
multimodal 3B
200
Table 11: Number of checkpoints per dataset, selected to balance effectiveness and training cost.
B Additional Experiment Results
B.1 More Results of Common Sense Reasoning, Math, and Coding
In this section, delve into details about each group of training LoRAs' performance test sets, report their performance and discuss further findings.
Common sense reasoning. In this part, we show training LoRAs' performance on each test set in Table 12. Each row specifies the dataset LoRA is trained on, and each column denotes the dataset used for testing. Consequently, the diagonal elements are full-shot results of training LoRAs, which are marked in red. We also incorporate their average accuracy (exclude diagonal elements, marked in gray). ), 2w.0525--..BB' performance, and DnD's zero-shot results (marked in green
17

---

train set test set
ARC-e
OBQA
ARC-e
PIQA
HellaSwag
BoolQ
WinoGrande
training LoRA ARC-e
59.4
46.2
46.7
80.4
30.6
44.6
52.2
of OBQA
64.3
38.4
53.4
56.1
29.0
1.3
of ARC-c
57.2
38.2
40.7
65.9
46.7
23.6
of PIQA
27.9
27.0
24.7
66.2
24.4
9.2
50.3
of HellaSwag of BoolQ
57.2
43.2
41.0
40.4
23.4
0.5
52.8
fail
fail
52.0
fail
fail
22.11
fail
of WinoGrande
18.6
26.8
19.1
0.1
3.8
1.6
50.5
Qwen2.5-0.55
54,8
16.6
38.3
16.6
26,5
37.0
50.2
average of training Ls
37.5
30.2
39.5
40.5
22.4
13.5
38.8
DnD
68.6
40.8
51.6
87.9
25.9
44.9
50.0
Table 12: More results for common sense reasoning task. Red marks full-shot results, gray shows the average of training LoRAs, and green marks the results of DnD. DnD not only consistently surpasees average of training LoRAs, but also outperform their full-shot performance in most cases.
It can be observed that: i) original LoRAs generally perform poorly novel datasets, e.g., LoRAs tuned on BoolQ fail (accuracy=O.0) on half of zero-shot datascts, this may because training on one specific dataset will fit the parameters for this dataset (binary True/False in BoolQ's s case) and limit their generality (multiple choices question of other datasets). ii) DnD even outperforms training LoRAs' full-shot performances in most cases, showing it has incredible zero-shot ability with great efficiency. These findings further illustrates that: fitting parameters for certain data may lack generality, learning the correlations between data and parameters may be better.
Coding. In this part, we elaborate on training LoRAs' performance coding datasets, along with their average (marked gray ), Qwen2.5-1.5BB77 and DnD's 's performance (marked in green
Share
Evol Instruct
Glaive Assistant
LLaMA
Qwen2.5
train set avg.
DnD
test train set
Python
Rosetta
Alpaca
GPT
Python
1.5B
pass@1
28.8
40.0
14.0
17.6
13.7
23.2
14.7
17.6
32.7
5
46.2
53.4
27.8
19.1
28.6
25.5
31.0
26.5
28.6
pass 10
52.9
56.4
35.0
23.8
33.2
29.9
34.1
32.3
33.2
64.1
testset:
HumanEval, average improvement: pass @
15.1,
pass@ 26.7, pass@ = 30.9
(a) @a (k 5, 10) scores of foundation LLM, original and generated LoRA for Qwen-1.58 HumanEval. DnD shows improves largely over its training data and base Qwen, validating its effectiveness.
Share GPT
Evol
Glaive
LLaMA
Qwen2.5
train set avg.
DnD
test set train set
Python
Rosetta
Alpaca
Instruct
Assistant
Python
7B
pass @1
22.4
23.2
24.6
fail
fail
fail
fail
34.1
13.0
33.4
pass@5
28.4
28.2
32.7
fail
fail
fail
fail
41.6
16.4
42.1
pass @ 10
30.8
30.3
35.8
fail
fail
fail
fail
43.8
17.6
46.0
testset: LiveCodeBench, average improvement: pass@ 20.3, pass@5 25.7, pass @ 10 28.4
(b) DnD constantly present promising results at more complex benchmarks (i.e., LiveCodeBenchh, even when over half of its training LoRAs fail (pass @k = 0.0) on this dataset.
Table 13: DnD surpasses the average pass @k score of training data on zero-shot coding benchmarks.
Empirical results in Table 13 indicates that: i) our approach is able to generalize to complex real- world problems, generating parameters for zero-shot benchmarks with promising results (reflected in improvement over training data). ii) DnD's good performance on the 7B group, despite poor performance of training LoRAs (pass@k=0.0) k=0.0), stressing that our method is not simply memorizing seen parameters, but learn to establish data- parameter mappings and adapt LLMs for novel datasets. Math. In this part, we report more results for math experiments in Table 14. Similar to common sense reasoning and coding tasks, DnD continues to work well on math datasets, improving largely over the average accuracy of training data on zero-shot benchmarks. This showcases its drag-and-drop ability has wide application scenarios. Also, results on Qwen2.5-7B prove its promising scalability.
18

---

test set train set IIO-Mini
ToT-Mini
Math-Plus Mu-Math Competition Math- Q.A
Qwen
train set avg.
DnD
gsm8K
68.6
35.4
68.3
22.8
31.0
31.4
64.4
42.9
66.3
MATH
30.0
30.2
7.2
16.5
3.5
29.3
14.8
23.9
average accuracy improvement: 23.4 on gsm8K, 9.1 on MATH.
(a) Even with some pww-erformming training LoRAs (i.e., accuracy less than 50% of base Qwen), DnD still maintains good zero-shot performance, showing the drag-and-drop ability to fit L.L.s novel datasets.
test set \ train set Mini
ToT-Mini Math -luth Mu-Math Competition Math QA Qwen train avg. DnD
gsm8K
88.3
50.7
68.3
61.1
88.3
60.7
81.2
65.9
83.1
average accuracy improvement: 17.2 on gsm8K.
(b) Using larger Qwen2.5-7B5 7B, DnD continues to drag-ande anng-ood-drop it, indicating method has good scalability. Table 14: DnD continues work well on math tasks, showing our method has broad applicability.
B.2 Inference Efficiency Analysis
In this part,we we further analyze DnD's efficiency by presenting memory usage and inference time for generating various LLMs respective tasks. We show the cost of generating single model on a NVIDIA A100 80G GPU in Table 15.
metrics
common
math
coding
multimodal
0.53 (I.5B)
0.70 (L.5B)
time (second)
0.11
0.61
0.55 (7B)
0.73 (7B)
15.43 (1.5B)
19.17 (1.5B)
memory cost (GB)
9.59
20.31
16.22 (7B)
20.48 (7B)
Table 15: Inference time and memory cost for different LLMSs generation. All metrics are measured on a single NVIDIA A 80G GPU. The time and memory is the cost to generate single model.
B.3 More Ablation Studies
Can answer serve as condition in more complex scenarios? Section 3.4, we've explored different condition types and come up with the conclusion: simple, identical answers will limit the diversity of conditions. this part, we explore using more complex answers from math datasets in pairs and training DnD.
Section 3.3 as conditions for building condition-parame
From results in Table 16, we can observe that more complex, diverse answers can lead to better perfor- mance. This may because their diversity able to provide DnD with comprehensive view of condition- parameter mapping. However, as answers are typi- cally much longer than prompts in math and coding datasets due to problem complexity, we still recom- mend to use prompts conditions for DnD.
condition type
accuracy gsm8K(S))
64.0
prompt
66.3
Table 16: Answer in math task can serve as conditions, but prompts still work better.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
All the claims are supported by experiment results in Section 3.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Section 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
20

---

Justification:
This paper does not contain theoretical proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All results in main paper and appendix are fully reproducible and we provide detailed recipe in Appendix A.I.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [Yes]
Justification: All datasets and models in this paper accessible via the Internet, and we will update the relevant code subsequently, it is highly recommended to access the latest version of our codes.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
SaaiiessCooeeubmisse
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results, See the NeurIPS code and data submission guidelines (https:
wtwc//ggess/eeeeeeee hwt////oooeesssaaeeeeeeee ssionPolccy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide detailed recipes in Appendix A.I.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
All experiment results are average results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
22

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We discuss compute resources in appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that NeurIPS Code of Ethics https: http:p:////rrpps..pppbbi..ccc httpe///c..//e//..ccc...eeeeeeee
didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the
cwwucieluuliiiiiee ines?
Answer: [Yes]
Justification: This paper strictly follows the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss it in Section 5 and appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All assets used in this paper are properly cited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
25

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: This paper's development does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https://netll http://///.ooeeeeeee/////////// for what should or should not be described.
/
26


Reviews:
[{"review_id": "LPkTHwz8ak", "reviewer": "Reviewer_2zUf", "summary": "This paper introduces Drag-and-Drop LLMs, a hypernetwork approach where prompt-conditioned generator maps a small batch of unlabeled task prompts directly to LoRA weight updates. Experiments on several text-based tasks show that DnD outperforms existing PEFT methods.", "questions": "Please see weaknesses", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "- The motivation is clear: the proposed framework eliminates the need for per-task optimization while also sharing information across tasks via the parameter generator. The idea is, to my knowledge, novel. The efficiency gains on novel text tasks is very compelling.\n- The paper provides several insightful ablations: in-domain vs cross-domain, conditioning, extractor, train/test split.\n- Experimental results show that DnD consistently outperforms LoRA training\n  -  For this experiment, I'd imagine the conclusion has a strong dependence on dataset size. For a representative set of datasets, could you report how LoRA performance scales with dataset size (while comparing to DnD numbers)?\n- The learning objective was somewhat counterintuitive to me. You minimize an MSE loss to the fine-tuned LoRA updates, which would make sense when the set of good LoRA weights for a given task is unimodal, i.e. if you trained multiple times with different random seeds, the resulting weights would be close together. Is this something you see empirically? If not, why does the MSE loss work?", "was_revised": true, "final_justification": "After reading through the discussion, I am deciding to maintain my score.", "rebuttal": "Thanks reviewer 2zUf for recognizing our work's novelty and efficiency, and for raising insightful comments. We address the comments one by one as follows.  \n\n> **Q1:** How LoRA performance scales with dataset size (while comparing to DnD numbers)?  \n  \n**A1:** While we are not completely certain about the specific dataset size that Reviewer 2zUf refers to and would appreciate a clarification, we did our best to provide as comprehensive an answer as possible based on our understanding. \n\n- **Dataset size used for training LoRAs:** We think reviewer 2zUf most likely refers to the dataset size used to train LLM LoRAs, that is, the **text dataset size**. Based on this, we perform an experiment to explore **dataset size**'s influence on **training LoRAs** and **DnD**.\n\n    - **Setting:** In our paper, we use the **entire dataset** to train and collect LoRA checkpoints. To ablate the influence of **dataset size**, we use **10\\% of each dataset** to train LoRAs, then adopt these LoRAs for DnD's training, and evaluate their performance. Due to limited time, we only report test results on **ARC-c** (training LoRAs are obtained from ARC-e, OBQA, PIQA, WinoGrande, HellaSwag, and BoolQ).\n    - **Results:** \n\n        |dataset size|avg of training LoRAs|DnD|\n        |-|-|-|\n        |full (our paper)|37.5|51.6|\n        |10\\%|29.8|49.1|\n \n    \n    - **Findings:** \n        - Even trained with only 10\\% data, DnD still maintains **good performance** (51.6\\% $\\rightarrow$ 49.1\\%).\n        - As expected, training LoRAs' performance drops largely (37.5\\% $\\rightarrow$ 29.8\\%) with only 10\\% training data.\n        - DnD' performance drop is much smaller than training LoRAs on **novel** dataset (i.e., ARC-C)\n\n    - **Conclusions:**\n        - DnD is **less sensitive** to dataset size than **traditional LoRA tuning**.\n        - Despite the performance **drop** in training LoRAs, DnD **maintains** good performance. This indicates our method has **strong dataset adapting ability.**\n          \n    - **Action:** We've added these results to **Table 4f** in **section 3.4** of the paper.\n\n- **Data sample number used to train LoRAs and inspire DnD respectively:** **Dataset size** may also refer to **the data** used to train LoRAs and inspire DnD. Take common sense reasoning as example, in **Table 7** and **Table 9** of the paper, LoRAs is trained on **5000 sample** for each task, while DnD accepts **128 prompts** to generate a LoRA adapter.\n    - We put respective data in the **table below** for the convenience:\n\n        |samples used for training LoRAs (DnD training data) | length of prompt batch (DnD testing)|\n        |--|--|\n        |**entire** dataset | **batch** of dataset, default: 128/1000|\n\n    - **Conclusion:** DnD uses very **few samples** as conditions to generate parameters, yet still obtains superior performance compared to training LoRAs (51.6\\% vs 37.5\\%).\n\n\n- **Dataset number used to train DnD:** reviewer 2zUf may refer to the **number of datasets (prompt-checkpoint pairs)** used to train DnD. In **Table 4c**, we conduct an ablation studies named **datasets arrangement** to explore, we put the table here for the convenience, and hope this can address your concern.\n    - **Table:**\n\n        |datasets arrangement|$\\Delta$ with training LoRAs|\n        |--|--|\n        |6 $\\in$ train, 1 $\\in$ test|  $\\uparrow$ 12.1 |\n        |4 $\\in$ train, 3 $\\in$ test|  $\\uparrow$ 11.4 |\n        |3 $\\in$ train, 4 $\\in$ test|  $\\uparrow$ 0.8 |\n        |2 $\\in$ train, 5 $\\in$ test|  $\\downarrow$ 1.4 |\n    \n    - **Conclusions:** \n        - Generally, **more** training datasets lead to **better** performance improvement. \n        - As datasets used for training **lessen**, the average improvement of DnD **drops accordingly**. \n        - Therefore, **basic amount** of training samples are needed for DnD to learn condition-parameter correlations.\n        - more datasets $\\rightarrow$ more prompt-checkpoint pairs $\\rightarrow$ deeper understanding of **data-to-weight transition** $\\rightarrow$ **better generality**\n    \n\n\n- **Length of prompt batch:** **Dataset size** may also refer to the **number of prompts used as inspiration** (length of prompt batch, default as 128) introduced in **line 139** of the paper, and we conduct more ablation studies accordingly.\n\n    - **Setting:** We both shrink and enlarge the length of prompt batch, turning it to 8, 16, 32, 64, 512 and explore their influence on DnD. Due to limited time, we only test on **ARC-c**.\n\n\n    - **Results:**\n        \n        |length of prompt batch|avg of training LoRAs|DnD|$\\Delta$ with training LoRAs|\n        |-|-|-|-|\n        |8|37.5|33.2|$\\downarrow$ 4.3|\n        |16|37.5|45.4| $\\uparrow$ 7.9|\n        |32|37.5|47.3| $\\uparrow$ 9.8|\n        |64|37.5|48.2| $\\uparrow$ 10.7|\n        |128|37.5|51.6| $\\uparrow$ 14.1 |\n        |512|37.5|52.0| $\\uparrow$ 14.5 |\n\n    - **Findings:**\n        - When **prompt batch** is **too small (8)**, DnD's performance is not good.\n        - DnD can **surpass** training LoRAs with small prompt batch, i.e., **16**.\n        - As prompt batch grows **larger**, DnD's performance **improves** accordingly.\n\n    - **Conclusion:** \n        - DnD can achieve good performance with small prompt batch such as 16, showcasing our method's potential in **data shortage** scenarios.\n        - As prompt batch grows **larger**, DnD obtains better understanding of the given tasks (i.e., better likelihood estimation), reflected in DnD's performance **improvement**. We will further enlarge the prompt batch to obtain better performance in the future.\n        \n    - **Action:** We add these results to **Table 4d** in **Section 3.4** to consolidate our work.\n\n- **Number of collected checkpoints for each dataset:** The number of checkpoints used in each dataset may also be the answer. \n    - It can be conclude that **too few** checkpoints fails to learn the data-to-weight transitions for each dataset. \n    - While **too much** checkpoint can induce **extra training cost**. We use the settings below that have passed our **empirical examination.**\n\n    - **Setting:** Note that we use one dataset for **multimodal**, so the number of checkpoints is large.\n    \n        | task | number of checkpoints |\n        |----|--------|\n        |common sense reasoning| 50 |\n        |coding/math 1.5B, math 7B| 80 |\n        |coding 7B| 100 |\n        |multimodal 3B|200| \n    - **Action:** We add these settings to **Table 9** in revision for better clarity.\n\n- **Conclusion:** \n    - We summarize our understanding of Reviewer 2zUf's question, and hope to have addressed their concern. We would appreciate a clarification and will be glad to provide further responses. \n\n>**Q2:** The design choice of MSE loss.\n\n**A2:** \n\n\n- **Checkpoint collection:** We collect checkpoints **for a given task** by:\n    - Performing **iterative LoRA fine-tuning** on one foundation model for specified epochs.\n    - Collecting checkpoints **at the end** of training trajectory.\n\n- **Distrubtion of LoRA adapters:** We agree with Reviewer 2zUf that **MSE makes sense when LoRA weights for a given task is unimodal**. So we conduct the following experiment to clarify:\n    - **Setting:** We compute the **average L2 distance** between parameters trained on the **same dataset** and those trained on **different datasets**. Additionally, we perform **Hartigan’s Dip Test** to assess unimodality, and report the **proportion of parameter elements** with **p-value > 0.05**. Due to time limits, we only conduct experiment on **ARC-c**.\n\n    |avg L2 distance (within ARC-c) |avg L2 distance (with other 6 common sense reasoning datasets)| ratio of p-value > 0.05 (dip test)|\n    |--|--|--|\n    | 4.0 e-09 | 2.2 e-06 | 0.865 |\n\n\n    - **Findings:** \n        - **L2 distance** within the same dataset is much **smaller (with magnitude of 500)** than with other datasets.\n        - **Most parameters (more than 85 percent)** of collected checkpoints conform to distribution with **p-value>0.05**, meaning that we can **accept the unimodal hypothesis**.\n\n\n\n- **Why we collect checkpoints only from one training trajectory for each dataset?**\n    - **Target:** Obtaining model parameters that perform well.\n    - **Single or multiple parameter patterns:** Based on our target, we foucs more on  developing mapping from data to single parameter pattern rather than decoding multiple patterns from given prompts.\n    - **Optimization difficulty consideration:** Mapping to single pattern is much easier to optimize. Since checkpoints collected from **multiple trajectory** is **not simple unimodal distribution**, making optimization much harder.\n\n- **Generating various parameter patterns is challenging but interesting.**\n    - We are aware that generating multiple high-performing and diverse parameters is difficult just as reviewer 2zUf mentioned.  \n    - **Potential Solution:** It's possible to devise a **conditioning mechanism** that encodes **different random seeds'** or **network symmetries'** information in parameter generator's training.\n\nWe appreciate the comments from reviewer 2zUf and hope the answers have addressed the concerns."}, {"review_id": "wLBBRhC14H", "reviewer": "Reviewer_BsLp", "summary": "This work proposes a method to generate LoRA adapter weights conditioning on task-relevant prompts. The weight generator is trained from pairs of prompt and finetuned LoRA weights. Experiments demonstrate the proposed method can achieve better zero-shot accuracy on unseen task and cross-domain task compared to other trained LoRA adapters", "questions": "1.\tIn line 169, how is “the average accuracy of training LoRAs” calculated? Suppose the test set is ARC-e, does “training LoRAs” correspond to each LoRA adapter trained from the other tasks (OBQA, ARC-c …) respectively? If so, why not compare DnD with one LoRA adapter finetuned on all other test sets? Since the training sample of DnD generator includes adapters parameter produced from all other test sets.\n2.\tIn Table 3, why is the improvement of DnD in Multimodal task minor compared to other tasks?\n3.\tIn practical deployment, does DnD require sufficient number of prompts (which is determined by the length of prompt batch hyperparameter during training) to start generating the weights for a task?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths\n1.\tThe idea of generating LoRA weights using task prompt is interesting and novel.\n2.\tConsiderable performance boost in the zero-shot accuracy in various tasks compared to other trained LoRA adapters.\n3.\tExtensive evaluations on key design factors of the proposed method.\n\n## Weakness\n1.\tThe author did not explain why DnD has good generalization even in cross-domain task. In Table 2, the DnD generated weights even outperforms the LoRA adapter trained on the target domain, which appears counter-intuitive given DnD is never trained on adapters related to science-dataset.\n2.\tThe comparison of DnD’s performance with foundation LLMs should be included in Table 1-3 instead of conducting a separate ablation study. It seems that most of the training LoRAs have a degraded task accuracy than the base LLM. Therefore, the actual performance gain brought by DnD may be exaggerated.\n3.\tThe emphasis on 12000x speedup of DnD compared to full-shot tuning does not look reasonable to me. The building of DnD dataset already involves training a number of LoRA adapters for extended iterations, which takes much longer than training one specialized LoRA adapter.", "was_revised": true, "final_justification": "The authors have well addressed my concerns. I thus increase the score to 5.", "rebuttal": "Thanks reviewer BsLp for acknowledging the novelty, performance gains and extensive evaluations of our work. We answer reviewer BsLp's questions below.\n\n> **Q1:** Why does DnD outperform those LoRAs tuned on the science dataset? \n\n**A1:** Thanks for this question.\n- **Clarification:**\n    - In **Table 2**, the training LoRAs are **trained on ARC-e, OBQA, PIQA, HellaSwag, WinoGrande, BoolQ (common sense reasoning datasets, NOT science datasets)**. \n    - These training LoRAs are **never trained on science dataset**.\n- **Action:**\n    - We add **\"Note training LoRAs are trained on ARC-e, OBQA, PIQA, HellaSwag, WinoGrande, BoolQ, and they've never seen a science dataset.\"** in **line 174**, paragraph **Cross-domain Drag-and-Drop.** for better clarity.\n\n> **Q2:** Why does DnD succeed in the cross-domain (source $\\rightarrow$ target) dragging?   \n\n**A2:** We answer as follows.\n- DnD learns the **data-to-weight mapping**, instead of **fitting its parameters for a given dataset** like traditional LoRA tuning.\n- Moreover, DnD learns a parameter generator that is **data-adapting**.\n- Therefore, in cross-domain scenarios, it utilizes target domain data as **inspiration** and generate respective parameters, which surpasses source domain tuned LoRA in terms of **generality**.\n- For convenience, we summarize the differences between DnD and traditional LoRA tuning below: \n\n    |property|LoRA tuning|DnD| \n    |-|-|-|\n    | objective | fitting data in the source domain | learning data-to-weight mapping |\n    |origin of generality | domain gap | domain gap+target data inspiration |\n   \n- The generality of our method also depends on the domain gap, and since the science dataset is **similar** to common sense reasoning, DnD is easier to generalize.\n\n- To explore the domain gap's influence, we test cross-domain with a larger domain gap, i.e., **coding-to-math**. (More in **A4 of reviewer AWwr**.)\n    - **Setting:** We use datasets and checkpoints in **math and coding tasks** to train DnD jointly.\n    - **Results:**\n        |test set|avg of training LoRAs| DnD with joint trainig |\n        |---|----|-----|\n        |gsm8k (accuracy)|46.3|65.3|\n    - **Conclusion:**\n        - DnD improves largely over its training LoRAs on gsm8k, showing it has the potential of **cross complex domains: from coding to math**.\n    - **Action:** We add these results to **Section 3.5** and **Table 7** in the revision.\n\n\n\n> **Q3:** The comparison of DnD with foundation LLMs should be included in Table 1-3.\n\n**A3:** Thanks for this advice, we change **Table 1-3** as follows. \n\n- **Modification:** \n    - We add the results in **Table 5** to **Table 1-3**.\n    - The relevant analyses are also moved to the respective sections in the revision.\n- **Performance Comparison among different methods:**\n    - Overall, DnD achieves the best performance among training LoRAs and base LLMs.\n    - Despite being trained on **worse-performing** LoRAs compared to base LLMs, DnD still obtains **better performance** than base LLMs on target tasks, showcasing our method has **strong perception and adaptation** of target datasets.\n\n> **Q4:** The emphasis on 12000x speedup of DnD compared to full-shot tuning does not look reasonable since the building of the DnD dataset already involves training LoRA adapters.\n\n**A4:** Thanks for the comment. We answer it as follows:\n- Training LoRAs on **zero-shot** test sets performance may be sub-optimal, and a practical solution is to use **full shot tuning** to get **dataset-specific** LoRA.\n- Our method, on the contrary, is a **once-for-all** parameter generator, that can adapt to other **novel** tasks unseen in training with proper training.\n- Therefore, the cost spent on DnD's training set building is **not targeted at the test set**. For test set adaptation, DnD takes **only one single forward pass**.\n- We appreciate the suggestion and make the following **modifications**:\n     - In **line 10** of the paper, \"up to 12,000× lower overhead than full fine-tuning\" $\\rightarrow$ \"up to 12,000× lower novel task adaptation cost once finished training\".\n     - In **Figure 4b** of the paper, \"comparable to full-shot tuning while being 12000× faster.\" $\\rightarrow$ \"performance comparable to full-shot tuning while saving 12000× adaptation cost after training\".\n     - In **line 278** of the paper, \"while being 12000× faster\" $\\rightarrow$ \"being 12000× faster in adaptation once trained\".\n\n> **Q5:** How is “the average accuracy of training LoRAs” calculated? Why not compare DnD with one LoRA adapter finetuned on all other test sets?\n\n**A5:** Thanks for the comment. We answer as follows:\n\n- **Calculation of avg acc**: The assumption is correct, “training LoRAs” are LoRAs trained on other tasks, and average accuracy is calculated via:  $\\text{avg acc}=\\frac{ \\sum_i^n \\text{acc}_i }{ \\sum_i^n i }$, where $i$ starts from 1.\n- DnD takes prompt-checkpoint pairs to transform data to weights, i.e., **pairing** checkpoints with prompts they are **trained on**. \n- Training one type of LoRA adapter on all datasets will lead to prompts-checkpoints pairs being prompts from **various datasets** paired with **only one type of checkpoints**.\n- Trained on this data will instruct the generator to transform **all kinds of prompts** to **only one kind of weights**, limiting its generality for novel tasks.\n- To ensure **the number of datasets** used to train DnD and LoRAs is consistent, and considering the tight time of rebuttal, we design the experiment as follows: \n    - **Setting:** We train DnD on **ONLY one dataset PIQA** and use prompts from **one unseen test set ARC-c** to inspire the generator. **This guarantees the number of datasets used to train DnD and LoRAs is exactly the same**.\n    - **Results:** We report **accuracy on ARC-c**.  \n\n        |training LoRAs (PIQA)|DnD generated LoRA|improvement |LoRA tuned on ARC-c (upper bound)|\n        |-|-|-|-|\n        |19.1|28.8| $\\uparrow$ 9.7 |55.1|\n\n        Experiment results on **multimodal tasks** can also address the concern: (Note that Math-Vision and Math-Vista are zero-shot benchmarks that **don't contain a training set**, so there are no full-shot tuning results.)\n\n        |benchmark|training LoRAs|DnD generated|improvement|\n        |-|-|-|-|\n        |Math-Vision|23.0|24.3|$\\uparrow$ 1.3|\n        |Math-Vista|61.5|62.3|$\\uparrow$  0.8|\n            \n    - **Findings:** \n        - With **same** number of training datasets, DnD obtains better **(9.7 improvement in accuracy)** performance than training LoRAs on novel test set.\n        - DnD trained on **only one dataset** is hard to generate competing parameters as full-shot tuning for novel datasets in inference.\n        - On multimodal tasks, DnD obtains better performance with training LoRAs with **minor improvement**.\n    - **Conclusion:** \n        - DnD's surpassing training LoRA indicates it works even with **limited knowledge** of data-to-weight transition.\n        - Using **only one kind** of checkpoints can't give DnD enough information about data-to-weight mapping, which limits DnD's generality.\n        - Results on multimodal tasks also prove that only one kind of dataset limits DnD generality to zero-shot benchmarks.\n    - **Action:** We add results and more experiment results (ongoing) in **Appendix B.3** in the revision.\n\n> **Q6:** Why is the improvement of DnD in the Multimodal task minor compared to other tasks?\n\n**A6:** Thanks for the comment. We answer it as below:\n- As discussed in **A4**, the number of training datasets is crucial in developing DnD's zero-shot ability.\n- In **line 164** of the paper, we use only **MathV360K** for multimodal task, due to **computation resources limit**. For example, CoMM takes up **2 TBs** of space and has more than**1B tokens** for training.\n- This limits the **diversity** of training LoRAs and makes DnD relatively **hard** to generalize well, so the improvement is minor. \n- **Plans:** We anticipate that the performance of the multimodal task can be improved via:\n    - incorporating **more datasets** in DnD's training, thus increasing the type of prompt-checkpoint pairs. \n    - **manually split MathV360K** into subsets (according to chartQA, mapQA, geometry, and other categories), collect LoRAs **tuned on each subset**, and train DnD on them. \n\nThanks again for the comment. We would like to discuss more with reviewer BsLp.\n\n\n> **Q7:** In practical deployment, does DnD require a sufficient number of prompts?\n\n\n**A7:** Thanks for the comment, we answer this question as follows.\n\n\n- **Experiment:** We conduct more ablation studies on the **length of prompt batch**.\n    - **Setting:** We shrink the length of prompt batch to 8, 16, 32, 64 and evaluate their performance on **ARC-c**.\n    - **Results:**\n    \n        |length of prompt batch|avg of training LoRAs|DnD|$\\Delta$ with training LoRAs|\n        |-|-|-|-|\n        |8|37.5|33.2|$\\downarrow$ 4.3|\n        |16|37.5|45.4| $\\uparrow$ 7.9|\n        |32|37.5|47.3| $\\uparrow$ 9.8|\n        |64|37.5|48.2| $\\uparrow$ 10.7|\n        |128|37.5|51.6| $\\uparrow$ 14.1 |\n        |512|37.5|52.0| $\\uparrow$ 14.5 |\n        \n    - **Findings:**\n        - When **prompt batch** is **too small (8)**, DnD's performance is not good.\n        - DnD can **surpass** training LoRAs with small prompt batch, i.e., **16**.\n        - As prompt batch grows **larger**, DnD's performance **improves** accordingly.\n\n    - **Conclusion:** \n        - DnD can achieve good performance with a small prompt batch, such as 16, showcasing our method's potential in **data shortage** scenarios.\n        - As the prompt batch **grows larger**, DnD obtains a better understanding of the given tasks (i.e., better likelihood estimation), reflected in DnD's **performance improvement**. We will further enlarge the prompt batch to obtain better performance in the future.\n        \n    - **Action:** We add these results to **Table 4d** in **Section 3.4** to consolidate our work.\nWe thank reviewer BsLp again for the comments."}, {"review_id": "aeQR5ZAAGb", "reviewer": "Reviewer_MX6i", "summary": "The paper introduces Drag-and-Drop LLMs, a new way to quickly adapt large language models to new tasks without any extra training. Instead of fine-tuning a model for each task, DnD takes a few example prompts and directly generates the necessary weight updates using a lightweight encoder and a special decoder. It’s super efficient, up to 12,000 times faster than traditional methods, and still manages to beat or match performance on tasks like reasoning, math, coding, and even multimodal problems. Best of all, it works out-of-the-box on new tasks it hasn’t seen before.", "questions": "(1) How sensitive is DnD to the diversity and quality of the training prompt-checkpoint pairs?\n(2) Why was a hyper-convolutional decoder chosen over more common alternatives, and how does it compare? I'm still a llitle bit confused why this architecture.\n(3) What are the main failure cases or limitations of DnD in practice?", "limitations": "Please see weakness and Questions. Several questions need to be further addressed.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "This paper has some strengths:\n(1) The method achieves dramatic reductions in adaptation overhead—up to 12,000× faster than traditional fine-tuning—making it highly practical.\n(2) Well-written and easy to read.\n(3) Extensive evaluations, ablation studies, scalability tests, and comparisons with both traditional and recent methods (e.g., RPG) build a solid case for the method’s robustness and utility.\n\nThis paper has some weaknesses:\n(1) Several important implementation details are deferred to the appendix, which can interrupt understanding when reading the main paper.\n(2) Since DnD is trained on LoRA outputs, it may not generalize well to tasks or models where LoRA performs poorly. How should I address this?", "was_revised": false, "final_justification": "", "rebuttal": "Thanks reviewer MX6i for appreciating the efficiency, extensive evaluations, and novelty of our work. We also thanks the helpful suggestions and insightful questions offered by reviewer MX6i, we address these as follows.\n\n\n> **Q1:** Several details are deferred to the appendix, which can interrupt understanding of the main paper. \n\n\n\n**A1:** Thank for the comment, we make several **modifications** in the revision:\n- In **Section 2.5**, **line 135**, we put **Tabel 8** in **Appendix** here to delve into **hyper convolutional decoder's structure**.\n- In **Section 2.5**, **line 145** of the paper, we formulate the detailed **tokenization process** for better clarity.\n- In **Section 3.1**, **line 162** of the paper, we put **Table 7**, **Table 9** in **Appendix** here, to give a thorough introduction of **our training setting** and **checkpoint collection**.\n\n- **Conclusion:** \n    - Following reviewer MX6i's suggestion, we will add more details in the revision.\n    - We apologize for interrupting the understanding of readers, and we will continue to polish our work.\n\n> **Q2:** How should we apply DnD to scenarios where LoRA performs poorly?\n\n**A2:** Thanks for the comment. We admit that LoRA may perform poorly and not be as good as foundation LLMs in some scenarios. We clarify our method's characteristics:\n- Our work mainly explores **LoRA scenarios in language tasks**. However, our method is **independent** of model structures, which means we can also generate **entire** foundation LLMs.\n- By arithmetic estimation, we can generate full parameters of Qwen2.5-**0.5B** on **8 H200 GPUs**. But we are currently **limited by resources** to generate foundation LLMs of larger sizes, e.g., 7B or larger.\n- We will continue to explore **scaling up** of parameter generation, broadening the application scenario of this technology. \n- reviewer MX6i's question can also be interpreted as DnD's **robustness** after training on **low-quality** LoAR adapters (Please feel free to correct us if there is any misunderstanding).\n- In some scenarios, we may only have **worse-performing** LoRAs for training, and our method still shows strong **data-adapting** ability. Please kindly refer to **A3** for more details.\n\n\n> **Q3:** How sensitive is DnD to the diversity and quality of the training prompt-checkpoint pairs? \n\n\n**A3:** Thanks for the question. We answer as follows:\n\n- **Diversity:** We conduct ablation study of **datasets arrangement** in **line 233** and **Table 4c** of the paper. For convenience, we put the results here.\n    - **Results:**\n        |dataset arrangement | $\\Delta$ with training LoRAs|\n        |-|-|\n        | 6 $\\in$ train,1 $\\in$ test |$\\uparrow$ 12.1 |\n        | 4 $\\in$ train, 3 $\\in$ test |$\\uparrow$ 11.4 |\n        | 3 $\\in$ train, 4 $\\in$ test |$\\uparrow$ 0.8 |\n        | 2 $\\in$ train, 5 $\\in$ test |$\\downarrow$ 1.4 |\n    - **Conclusions:** \n        - Generally, **more** training datasets lead to **better** performance improvement. \n        - As datasets used for training **lessen**, the average improvement of DnD **drops accordingly**. \n        - Therefore, **basic amount** of training samples is needed for DnD to learn condition-parameter correlations.\n        \n- **Quality:** Thanks reviewer MX6i for the question, and we conduct more ablation studies as follows.\n    - **Setting:** In addition to collecting **well-trained** checkpoints, we collect checkpoints **at the beginning** of training trajectory, i.e., checkpoints **not converged** on the training sets.\n    - **Results:** (training LoRAs are trained **ARC-e, OBQA, PIQA, HellaSwag, WinoGrande, BoolQ** and test set is **ARC-c**)\n            \n        |checkpoint quality|avg of training LoRAs|DnD generated LoRA|improvement|\n        |-|-|-|-|\n        |w/o converged|27.3|46.3| $\\uparrow$ 19.0 |\n        |converged|37.5|51.6| $\\uparrow$ 14.1 |\n        \n    - **Findings:**\n        - DnD **maintains** good performance even with **worse-performing** LoRAs.\n        - DnD's performance **drops** with training LoRAs that are **insufficiently tuned**.\n        - Even DnD tuned on worse LoRAs surpasses **converged** training LoRAs on **zero-shot** test set.\n        \n        \n    - **Conclusions:** \n        - DnD works well with worse training LoRAs, which reveals that our method has strong **data adapting** ability.  \n        - Training on sub-optimal LoRA can affect DnD's performance, since it isn't aware of **good weights' distribution**.\n        \n        \n    - **Action:** We add these results and analysis to **Section 3.4**, **Table 4e** in the revision.\n\n> **Q4:** Why was a hyper-convolutional decoder chosen over more common alternatives? \n\n\n\n**A4:** Thank for the comment, we answer as follows. \n- According to previous works [4,6,7,8,9], we can conclude that common **backbones** in parameter generation can be:\n    - **Pure latent diffusion[3]:** [4,6,7]\n    - **Conditional latent diffusion:** [6]\n    - **Mamba [5] + latent diffusion:** [9]\n    - **LSTM [1] + latent diffusion:** [9]'s ablation\n    - **Casual transformer[2] + latent diffusion:** [9]'s ablation\n- We would like to summarize different backbones' properties as follows:\n    | backbone | large scale generation | single step generation | conditional generation | \n    |-|-|-|-|\n    |latent diffusion| $\\times$ | $\\times$ | $\\times$ |\n    |conditional latent diffusion| $\\times$ | $\\times$ | $\\checkmark$ **(text discriptions)**|\n    |RPG with Mamba|$\\checkmark$ **(up to 200M)** | $\\times$ | $\\checkmark$ **(binary embeddings)**|\n    |RPG with LSTM|$\\checkmark$ **(up to 200M)** | $\\times$ **(slower than Mamba)**| $\\checkmark$ **(binary embeddings)**|\n    |RPG with causal Transformer| $\\checkmark$ **(up to 200M)**| $\\times$ **(slower than Mamba)**| $\\checkmark$ **(binary embeddings)**|\n    |**Hyper convolution (ours)**| $\\checkmark$ **(up to 0.5B)**| $\\checkmark$ **(0.1s)**| $\\checkmark$ **(prompts in data samples)**|\n    (Generation parameters are tested on **8 H200 GPU**.)\n    - **Findings:**\n        - **Pure** latent diffusion models can not generate large models.\n        - **Mamba, LSTM, and transformer** can generate parameters of larger scale **(up to 200M)**, but this manner of generation **lacks efficiency**.\n        - Hyper convolution can achieve both large scale **(up to 0.5B)** and conditional **(data as conditions)**. Moreover, it can achieve **real-time** generation with only **0.1s**.\n- **Broader impact of real-time parameter generation:** \n    - **Fast Domain Adaptation**\n    - **Customizing Models for Users**\n    - **Others**\n\n**References:** \n\n[1] Hochreiter, Sepp. et al. Long short-term memory, MIT press 1997 \n\n[2] Vaswani, Ashish. et al. Attention is all you need, NeurIPS 2017\n\n[3] Rombach, Robin. et al. High-resolution image synthesis with latent diffusion models, CVPR 2022.\n\n[4] Wang Kai. et al. Neural network diffusion, arXiv 2024.\n\n[5] Gu, Albert. et al. Mamba: Linear-Time Sequence Modeling with Selective State Spaces, COLM 2024\n\n[6] Jin, Xiaolong. et al. Conditional lora parameter generation. arXiv 2024\n\n[7] Soro, Bedionita. et al. Diffusion-based neural network weights generation, arXiv 2024.\n\n[8] Li, Zexi. et al. Text-to-model: Text-conditioned neural network diffusion for train-once-for-all personalization, arXiv 2024\n\n[9] Wang Kai. et al. Recurrent diffusion for large-scale parameter generation, arXiv 2025.\n\n\n> **Q5:** What are the main failure cases or limitations of DnD in practice?\n\n\n**A5:** Thank for the comment. DnD's current limitations are:\n\n- **Fixed structure generation:** Currently, DnD focuses on generating models with the same structure for different tasks. It currently doesn't support generating models of **different structures**. We realize that **variable structure generation** is a crucial and promising direction, and we will explore it in the future.\n- **Cross Domain Generalization:** Though we successfully drag common sense reasoning to science dataset in **Table 2** of the paper, crossing domains with larger gap remains challenging, as suggested in our **A1 to reviewer BsLp**.\n    - **Potential Solution:** We are going to incorporate a large amount of diverse training LoRAs (as many as possible), aiming at empowering DnD with stronger cross-domain ability.\n\nWe add these limitations and relevant analysis to the revision. We appreciate reviewer MX6i's opinion and hope to discuss more."}, {"review_id": "a9sgS8mXDY", "reviewer": "Reviewer_AWwr", "summary": "The paper proposes DnD to replace PEFT. DnD takes unlabeled task prompts as input to generate LoRA parameters. The paper shows that the idea is working on many datasets and provides several ablations.", "questions": "I'll decrease score if non of the weaknesses is solved and raise to 6 if most of the weaknesses are solved.", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strength:\n\n1. It's crazy for me to see (if) the idea is working.\n\nWeakness:\n\n0. The tokenization in equation 4 is not clear. The author should formulate the tokenization out. (I had a glance on RPG, there are lots of implementation details. The author should detail the implementation used in this paper with formulation, possibly in appendix, rather than simply use tokenization[58] and force readers to go through RPG.)\n1. It's not clear that what if different datasets use different settings of lora? Such as the low rank is different among different datasets.\n2. Table 2 is not comprehensive, making the results unconvincing: the table only shows science as results of cross domain. What about others? The author should show many. It's fine if the results are bad. Bringing the whole picture is important.\n3. For all experiments (let me know if I'm wrong), DnD is trained on only one domain. The paper should consider multiple domains for training DnD.\n4. The paper misses an important result. What is the DnD's performance on dataset A, if DnD is trained on datasets A+B+C+D+...? So A is in the training data. This let us know the DnD's effect on training tasks.\n5. It's not clear why in Figure 5, DnD performs better than RPG.\n6. The setting of Figure 6 id not clear. What is DnD trained on? Multiple Fig. 6 are supposed be shown in different datasets.\n\n[let me know if the any of my question is solved by appendix, I did not go through all the appendix]", "was_revised": true, "final_justification": "As comment to the authors' rebuttal.", "rebuttal": "Thanks reviewer AWwr for constructive comments. We are delighted to discuss the raised questions below and sincerely hope these answers can address the concerns.\n\n> **Q1:** The tokenization in equation 4 is not clear. The author should formulate the tokenization out, possibly in appendix.\n\n**A1:** Thanks for the comment. We formulate the tokenization process as follows:  \n- **1. Normalize each layer's weight of collected checkpoints:** Each layer's weight is normalized by its element-wise mean and std: $\\mu = mean(W), \\sigma= std(W), \\hat{W} = \\frac{W-\\mu}{\\sigma}$, where $W^{i}$ are the $i$ th layer's parameters.  \n- **2. Slice model parameters of each layer into tokens of specified size:** We first slice model weights into equal-sized tokens: $\\hat{W}^{i} = [ \\hat{W}_1^i , \\hat{W}_2^i , ... \\hat{W}_n^i  ]$, where $\\hat{W}_j^i$ is the $j$ th token of $i$ th layer.\n- **3. Stack all tokens together:** All tokens are stacked together to form the shape of **[N′,L′,C′]** mentioned in **line 146**.\n- **Shorter version:** For the convenience, we also present tokenization process in the form of **mind chains**: **weights normalization** -> **weights tokenization** -> **stack tokenized weights**.    \n- **Action:** We follow reviewer AWwr's advice and take the following actions:\n    - We add the above formulations and details to **Section 2.5**, **line 143** in the revision.\n    - We sincerely apologize for causing obstacles in reading for the readers and we will carefully polish and constantly improve our work.\n\n\n> **Q2:**  It's not clear what if different datasets use different settings of LORA? Such as the low rank is different among different datasets.\n\n**A2:** Thanks for the comment.\n\n- DnD works well in parameter generation with **consistent structures**, e.g., LoRA adapters with the **same rank**.\n- Currently, DnD does not support generating LoRA adapters with **varying ranks**.\n- **Variable structure parameter generation** is a very promising and valuable insight, especially for tasks related to **model deployment**. Specifically, we can generate different parameters satisfying various **hardware requirements**.\n- To achieve this, we plan to conduct the following explorations:\n    - Devising **tokenization method** that supports variable structure parameter generation.\n    - Embedding this structural information as **condition** in generation process.\n    - Generating LoRA adapters that is **robust to pruning on ranks** may also be a solution.\n\nOnce more, we thank reviewer AWwr for the constructive advice, and we are going to explore variable structure parameter generation in the future.\n\n\n> **Q3:** Table 2 only shows science as the cross-domain results. What about others? \n\n**A3:** Thanks for the comment. \n\n- Before answering the question, we show the **max prompt length** of different tasks as follows:  \n\n    |task|max prompt length|\n    |-|-|\n    |common sense reasoning| 384 |\n    |science|416|\n    |coding| 26624 |\n    |math|4608|\n    |multimodal|1536 |\n\n- **Findings:**\n    - common sense reasoning and science dataset's max prompt length is **close**.\n    - common sense reasoning dataset's max prompt length **differs significantly** from math and coding datasets.\n\n- **Why can't we drag-and-drop common sense reasoning to the math/coding dataset?**\n    - **Overall Summarization:** The gigantic difference in max prompt length makes DnD trained on common sense reasoning hard to encode math/coding datasets' prompts, **losing information** for the given task.\n    - In **line 563**, **Appendix A.3** of the paper, we introduce that the model processes sequences of fixed length (512).\n    - Therefore, for **common sense reasoning**, we padding **384** to **512** as the sequence length dimension.\n    - **Math** and **coding** datasets' prompt length is **much larger** than 512. If we want to use math or code prompts to inspire DnD trained on common sense reasoning without introducing extra modules, we need to **average pool** or **randomly drop** **26624** or **4608** to 512.\n    - This operation is an adding-on to our method **(NOT in an end-to-end manner)** and **may lose much information** for the given task. Therefore, we fail to drag common sense reasoning to math or code domains.\n\n- **Why we can drag-and-drop common sense reasoning to science dataset**\n    - Max prompt length in **science dataset** is smaller than 512, so its information for the given task can be **well captured** in DnD's processing.\n\n- **Whole picture demonstration:**\n    - **Direct math-to-coding cross domain:** Similar to the above discussion, math/coding datasets' max prompt length also varies **(4608 vs 26624)**, compressing coding prompts to math's input length will also cause **information loss**.\n    - **Mixed domain training:** Train math and coding datasets **together** may be a solution for cross-domain dragging, as reviewer AWwr's **Question 4**. Please refer to our **Answer 4** for more details.\n\n\n\n> **Q4:** Train DnD on multiple domains.\n\n**A4:** Thanks for reviewer AWwr's comment. We conduct the experiment as follows:\n\n- **Setting:** We involve datasets and checkpoints **in math and coding tasks** to train DnD.\n- **Implementation Details:** Due to time limits, we directly apply BERT's padding to **math** prompt embeddings to ensure all embeddings are of **equal sequence length**. Simply speaking, **4608 + padding $\\rightarrow$ 26624**.\n(Note: Due to dramatic differences **(4608 vs 26624)** in prompt lengths, we are not certain this is the best solution. We will continue exploring better solutions.)\n- **Results:**\n\n    |test set|avg of training LoRAs| DnD with joint trainig |$\\Delta$|\n    |-|-|-|-|\n    |HumanEval (pass@10)|39.4|29.6|$\\downarrow$ 9.8|\n    |gsm8k (accuracy)|46.3|65.3|$\\uparrow$ 19.0|\n\n- **Findings and Conclusions:**\n    - Mixed domain training doesn't help DnD drag to tasks with longer prompt length.\n    - Mixed domain training enables DnD to cross domains with shorter prompt length.\n\n- **Action and Future Plans:**\n    - We add these results to **Section 3.5** and **Table 7** of the paper, with a paragraph named **Mixed domain training.**\n    - We will continue exploring mixed domain training with plans as below:\n        - Using **feature compression** methods to obtain sequences of **similar length**.\n        - Applying **data augmentation** methods instead of **padding**, since it may cause information loss.\n\nWe are still improving DnD for the better, and thanks reviewer AWwr for offering the advice.\n\n> **Q5:** What is DnD's performance on dataset A, if DnD is trained on datasets A+B+C+D+...? \n\n**A5:** Thanks for the comment. We answer as follows:\n- We conduct this experiment in **Section 3.5**, **line 302**, paragraph **Comparison with previous methods**, and show the results in **Figure 5**. For the convenience, we summarize and put the results in the table below.\n- **Settings:** We train DnD on **all 6 common sense reasoning datasets except ARC-c** and evaluate on training datasets.\n- **Results:**\n\n    |dataset|training LoRAs|DnD|\n    |--|---|--|\n    |ARC-e|67|69|\n    |OBQA|44|51|\n    |PIQA|30|36|\n    |HellaSwag|51|51|\n    |WinoGrande|81|38|\n    |BoolQ|46|64|\n\n- **Findings and Conclusions:**\n    - DnD surpasses training LoRAs in most cases, revealing good ability also in **close-set generation**. \n\n\n> **Q6:** It's not clear why in Figure 5, DnD performs better than RPG.\n\n**A6:** Thanks for the comment. We conclude that DnD outperforms RPG for the following reasons:\n\n\n\n- **Conditioning Mechanism: data feature vs binary embedding** \n    - DnD directly utilizes **input prompts' embeddings** as conditions, which equips it with an understanding of the transition from data space to weight space. \n    - RPG uses only **binary embeddings** as conditions, which, though it succeeds on **simple tasks such as CIFAR-10**, fails in complex scenarios such as language tasks.\n    - In close-set training, datasets in common sense reasoning have **subtle differences**, but binary embedding is hard to serve as conditions for this fine-grained scenario. Therefore, it is more appropriate to use data prompts as conditions.\n    - Binary embedding is **discretely distributed** while prompt embeddings are **continuous in semantic space**. In open-set testing that requires generality, prompt embeddings as conditions can have better zero-shot ability.\n\n\n> **Q7:** The setting of Figure 6 is not clear. What is DnD trained on? Multiple Fig. 6 are supposed be shown in different datasets.\n\n\n**A7:** Thanks for this suggestion, and we will make the following adjustments to the paper.\n\n- **What is DnD trained on?** Training datasets are **ARC-e, OBQA, PIQA, HellaSwag, WinoGrande, BoolQ**. **Gray circles** contains parameters whose **distribution is close** after PCA.\n\n- **Add more visualization Figures:**\n    - We add coding, math, and multimodal weights' visualization Figure in revision's **Figure 7**, **Figure 8**, and **Figure 9**.\n    - Though NeurIPS 2025 **can't upload Figures** in rebuttal phase, we provide the **figure captions** here:\n    - **Coding caption:** In complex scenarios, DnD manages to drag-and-drop LLM weights towards the desired location with good performance.\n    - **Math caption:** DnD also works well on math datasets, showing robust generality and wide application scenarios.\n    - **Multimodal caption:** Our method continues to show its magic on multimodal tasks, indicating it has broad applicable potential without the limitation of modality.\n    - Moreover, we add visualization of **other common sense reasoning datasets** to **Appendix C**.\n\nWe sincerely hope our response can address the concerns, and hope to hear more from reviewer AWwr."}]

Author rebuttal:

Meta-review: This paper proposes a framework named darg-and-drop LLMs to efficiently generate task-specific LoRA adapter weights based on downstream task prompts, avoiding the overhead of gradient-based optimization for LoRA adapters. Experiments on multiple adaptation tasks demonstrate the effectiveness and generalization ability of the method. The proposed method is novel and the zero-shot drag-and-drop manner for efficient adaptation is interesting and may inspire future research. All reviewers acknowledge that their questions and concerns have been addressed and favor the acceptance of this paper. The AC agrees with the reviewer comments and recommends accept for this paper.
Decision: Accept (poster)