Title: Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weights
Venue: NeurIPS 2025
Accepted: yes
Keywords: Parameter Generation

Parameter-Efficient-Fine-Tuning

Large Language Models
Abstract: Modern Parameter-Efficient Fine-Tuning (PEFT) methods such as low-rank adaptation (LoRA) reduce the cost of customizing large language models (LLMs), yet still require a separate optimization run for every downstream dataset. We introduce \textbf{Drag-and-Drop LLMs (\textit{DnD})}, a prompt-conditioned parameter generator that eliminates per-task training by mapping a handful of unlabeled task prompts directly to LoRA weight updates. A lightweight text encoder distills each prompt batch into  condition embeddings, which are then transformed by a cascaded hyper-convolutional decoder into the full set of LoRA matrices. Once trained in a diverse collection of
prompt-checkpoint pairs, DnD produces task-specific parameters in seconds, yielding i) up to
\textbf{12,000$\times$} lower overhead than full fine-tuning, ii) average gains up to \textbf{30\%} in performance over the strongest training LoRAs on unseen common-sense reasoning, math, coding, and multimodal benchmarks, and iii) robust cross-domain generalization improving \textbf{40\%} performance without access to the target data or labels. Our results demonstrate that prompt-conditioned parameter generation is a viable alternative to gradient-based adaptation for rapidly specializing LLMs.
We open source \href{https://jerryliang24.github.io/DnD}{our project} in support of future research.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes DnD to replace PEFT. DnD takes unlabeled task prompts as input to generate LoRA parameters. The paper shows that the idea is working on many datasets and provides several ablations.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n1. It's crazy for me to see (if) the idea is working.\n\nWeakness:\n\n0. The tokenization in equation 4 is not clear. The author should formulate the tokenization out. (I had a glance on RPG, there are lots of implementation details. The author should detail the implementation used in this paper with formulation, possibly in appendix, rather than simply use tokenization[58] and force readers to go through RPG.)\n1. It's not clear that what if different datasets use different settings of lora? Such as the low rank is different among different datasets.\n2. Table 2 is not comprehensive, making the results unconvincing: the table only shows science as results of cross domain. What about others? The author should show many. It's fine if the results are bad. Bringing the whole picture is important.\n3. For all experiments (let me know if I'm wrong), DnD is trained on only one domain. The paper should consider multiple domains for training DnD.\n4. The paper misses an important result. What is the DnD's performance on dataset A, if DnD is trained on datasets A+B+C+D+...? So A is in the training data. This let us know the DnD's effect on training tasks.\n5. It's not clear why in Figure 5, DnD performs better than RPG.\n6. The setting of Figure 6 id not clear. What is DnD trained on? Multiple Fig. 6 are supposed be shown in different datasets.\n\n[let me know if the any of my question is solved by appendix, I did not go through all the appendix]\n\nQUESTIONS:\nI'll decrease score if non of the weaknesses is solved and raise to 6 if most of the weaknesses are solved.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-01"}

comments: SUMMARY:
The paper introduces Drag-and-Drop LLMs, a new way to quickly adapt large language models to new tasks without any extra training. Instead of fine-tuning a model for each task, DnD takes a few example prompts and directly generates the necessary weight updates using a lightweight encoder and a special decoder. It’s super efficient, up to 12,000 times faster than traditional methods, and still manages to beat or match performance on tasks like reasoning, math, coding, and even multimodal problems. Best of all, it works out-of-the-box on new tasks it hasn’t seen before.

STRENGTHS AND WEAKNESSES:
This paper has some strengths:
(1) The method achieves dramatic reductions in adaptation overhead—up to 12,000× faster than traditional fine-tuning—making it highly practical.
(2) Well-written and easy to read.
(3) Extensive evaluations, ablation studies, scalability tests, and comparisons with both traditional and recent methods (e.g., RPG) build a solid case for the method’s robustness and utility.

This paper has some weaknesses:
(1) Several important implementation details are deferred to the appendix, which can interrupt understanding when reading the main paper.
(2) Since DnD is trained on LoRA outputs, it may not generalize well to tasks or models where LoRA performs poorly. How should I address this?

QUESTIONS:
(1) How sensitive is DnD to the diversity and quality of the training prompt-checkpoint pairs?
(2) Why was a hyper-convolutional decoder chosen over more common alternatives, and how does it compare? I'm still a llitle bit confused why this architecture.
(3) What are the main failure cases or limitations of DnD in practice?

LIMITATIONS:
Please see weakness and Questions. Several questions need to be further addressed.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a method to generate LoRA adapter weights conditioning on task-relevant prompts. The weight generator is trained from pairs of prompt and finetuned LoRA weights. Experiments demonstrate the proposed method can achieve better zero-shot accuracy on unseen task and cross-domain task compared to other trained LoRA adapters\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1.\tThe idea of generating LoRA weights using task prompt is interesting and novel.\n2.\tConsiderable performance boost in the zero-shot accuracy in various tasks compared to other trained LoRA adapters.\n3.\tExtensive evaluations on key design factors of the proposed method.\n\n## Weakness\n1.\tThe author did not explain why DnD has good generalization even in cross-domain task. In Table 2, the DnD generated weights even outperforms the LoRA adapter trained on the target domain, which appears counter-intuitive given DnD is never trained on adapters related to science-dataset.\n2.\tThe comparison of DnD’s performance with foundation LLMs should be included in Table 1-3 instead of conducting a separate ablation study. It seems that most of the training LoRAs have a degraded task accuracy than the base LLM. Therefore, the actual performance gain brought by DnD may be exaggerated.\n3.\tThe emphasis on 12000x speedup of DnD compared to full-shot tuning does not look reasonable to me. The building of DnD dataset already involves training a number of LoRA adapters for extended iterations, which takes much longer than training one specialized LoRA adapter.\n\nQUESTIONS:\n1.\tIn line 169, how is “the average accuracy of training LoRAs” calculated? Suppose the test set is ARC-e, does “training LoRAs” correspond to each LoRA adapter trained from the other tasks (OBQA, ARC-c …) respectively? If so, why not compare DnD with one LoRA adapter finetuned on all other test sets? Since the training sample of DnD generator includes adapters parameter produced from all other test sets.\n2.\tIn Table 3, why is the improvement of DnD in Multimodal task minor compared to other tasks?\n3.\tIn practical deployment, does DnD require sufficient number of prompts (which is determined by the length of prompt batch hyperparameter during training) to start generating the weights for a task?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper introduces Drag-and-Drop LLMs, a hypernetwork approach where prompt-conditioned generator maps a small batch of unlabeled task prompts directly to LoRA weight updates. Experiments on several text-based tasks show that DnD outperforms existing PEFT methods.

STRENGTHS AND WEAKNESSES:
- The motivation is clear: the proposed framework eliminates the need for per-task optimization while also sharing information across tasks via the parameter generator. The idea is, to my knowledge, novel. The efficiency gains on novel text tasks is very compelling.
- The paper provides several insightful ablations: in-domain vs cross-domain, conditioning, extractor, train/test split.
- Experimental results show that DnD consistently outperforms LoRA training
  -  For this experiment, I'd imagine the conclusion has a strong dependence on dataset size. For a representative set of datasets, could you report how LoRA performance scales with dataset size (while comparing to DnD numbers)?
- The learning objective was somewhat counterintuitive to me. You minimize an MSE loss to the fine-tuned LoRA updates, which would make sense when the set of good LoRA weights for a given task is unimodal, i.e. if you trained multiple times with different random seeds, the resulting weights would be close together. Is this something you see empirically? If not, why does the MSE loss work?

QUESTIONS:
Please see weaknesses

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-03