WeGeFT: Weight‑Generative Fine‑Tuning for Multi‑Faceted Efficient Adaptation of Large Models
Authors: Chinmay Savadikar
Xi Song
Tianfu Wu
Venue: ICML 2025 poster (2025)
Keywords: Parameter-Efficient Fine-Tuning
Primary area: general_machine_learning->transfer_multitask_and_metalearning
TL;DR: An approach to generate fine-tuning weights from pretrained weights while preserving compute and memory efficiency

Abstract: Fine-tuning large pretrained Transformer models can focus on either introducing a small number of new learnable parameters (parameter efficiency) or editing representations of a small number of tokens using lightweight modules (representation efficiency). While the pioneering method LoRA (Low-Rank Adaptation) inherently balances parameter, compute, and memory efficiency, many subsequent variants trade off compute and memory efficiency and/or performance to further reduce fine-tuning parameters. To address this limitation and unify parameter-efficient and representation-efficient fine-tuning, we propose Weight-Generative Fine-Tuning (WeGeFT, pronounced  *wee-gift*), a novel approach that **learns to generate fine-tuning weights directly from the pretrained weights**. WeGeFT employs a simple low-rank formulation consisting of two linear layers, either shared across multiple layers of the pretrained model or individually learned for different layers. This design achieves multi-faceted efficiency in parameters, representations, compute, and memory, while maintaining or exceeding the performance of LoRA and its variants. Extensive experiments on commonsense reasoning, arithmetic reasoning, instruction following, code generation, and visual recognition verify the effectiveness of our proposed WeGeFT.

WeGeFT: Weight-Generative Fine- Tunnng for Multi-Faceted Efficient Adaptation of Large Models
Chinmay Savadikar' Xi Song? Tianfu Wu
Code: https::/savaad: sasaaaaaaaaaraa.
github. kaw.
wwwefff
Abstract
and challenging research topic. In recent years, large feature backbones with open weights, such as LLaMA (Touvron et al., 2023a;b; Meta, 2024), have become ubiquitous. Training such models from scratch is infeasible with lim- ited resources, and fine-tuning them entirely can also be prohibitively costly. This raises two critical questions: (i) which parts of pretrained model should be fine-tuned (of- ten treated as a hyperparameter), and (ii) how those parts should be fine-tuned. In this paper, we focus on the lat- ter question by leveraging module/layer selection strategies widely adopted prior art.
Fine-tuning large pretrained Transformer mod- els can focus on either introducing a small num- ber of new learnable parameters (parameter effi- ciency) or editing representations a small num- ber of tokens using lightweight modules (represen- tation efficiency). While the pioneering method LoRA (Low-Rank Adaptation) inherently bal- ances parameter, compute, and memory efficiency, many subsequent variants trade off compute and memory efficiency and/or performance to fur- ther reduce fine-tuning parameters. To address this limitation and unify parameter-efficien and representatioon-effiieen fine-tuning, propose Weight-Generann Fine-Tuning (WeGeFT, pro- nounced wee-gift), a novel approach that learns to generate fine-tuning weights directly from the pretrained weights. WeGeFT employs a simple low-rank formulation consisting of two linear layers, either shared across multiple layers of the pretrained model or individually learned for different layers. This design achieves multi- faceted efficiency in parameters, representations, compute, and memory, while maintaining or ex- ceeding the performance of LoRA and its variants. Extensive experiments on commonsense reason- ing, arithmetic reasoning, instruction following, code generation, and visual recognition verify the effectiveness of our proposed WeGeFT.
Low-Rank Adaptation (LoRA) (Hu al., 2022) is a pio- neering and widely adopted approach that achieves built-in efficiency in parameters, compute, and memory. LoRA learns fine-tuning weight residuals in low-rank forms for pretrained weights on a layer-specific basis (see the left of Fig. 1). Thanks to its strong applicability and promising performance, many follow-up works have emerged, such DoRA (Liu al., 2024) and VeRA (Kopiczko al., 2023). However, these variants often sacrifice compute efficiency (training wall time) and/or GPU memory effi- ciency to achieve reductions in learnable parameters or per- formance gains on certain downstream tasks. As we demon- strate experiments, DoRA, while matching or slightly surpassing LoRA's performance, increases training wall time by more than 5x and consumes around 3GB more GPU memory. On the other hand, VeRA, though significantly re- ducing the number of learnable parameters, performs much worse than LoRA while drastically increasing training wall time (by more than 20x) and consuming similar additional GPU memory. These trade-off's motivate us to seek a formulation that can significantly reduce the number of learnable parameters, achieve superior on-par perfor- mance compared with LoRA, and retain its efficiency in compute and memory.
1. Introduction
Fine-tuning pretrained deep neural networks (DNNs) as fea- ture backbones for downstream tasks has been an important
'Deparmeent of Electrical and Computer Engineering, North Carolina State University, Raleigh, USA FAn Indepen- dent Researcher. Correspondence to: Chinmay Savadikar <csavadi @ncdi.cusuuu...... Tianfu Wu <twuii@ncuu.cdu>.
Towards these objectives, the recently proposed ReFT (Wu et al., 2024) introduces promising framework that focuses on lightweight learning weight residuals, as LoRA does. ReFT is inspired by causal intervention mechanisms (Geiger al., 2024) and operates in layer-specific manner. While ReFT methods
cirrseentttionediiiin modules instead of
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

WeGeFT (wee-gift): Geightt Fine- Tine-Tuning
&nu E R Nidus
x I Nicdae
AW B¹.
B E BRASCKK
W E Rdaiida
A' R xm
Q E R Lixihae xednn
WeGeFT
f2 R Lixdus xdi
R Niddan
z/ R Vxdie
(a) LoRA: Low-Rank Adaptation
(b) WeGeFT: Weight-Generative Fine-Tuning
Figure 1: Comparisons between (a) LoRA (Hu et al., 2022) and (b) our proposed WeGeFT.
reduce learnable parameters and retain compute and mem- ory efficiency comparable to LoRA, they often show inferior performance, confirmed in our experiments. Additionally, selecting where to intervene within the model to achieve strong downstream task performance is non-trivial. For in- stance, DiReFT, one of the two ReFT formulations, can be interpreted as applying LoRA directly to hidden representa- tions at specific intervention points. This motivates us to seek a unified perspective between carameter-efficiene and representation-effiicee fine-tuning that enables sim- pler formulations while achieving on-par better per- formance.
that performance can be further enhanced, especially for stronger pretrained models. Therefore, we aim to param- eterize fine-tuning weight residuals in weight-aware manner.
Layer-Specifie vs. Shared Adaptation: In LoRA, B and A' are layer-specific, which ensures layer-local adap- tation. Recent methods like Tied LoRA (Renduchintala et al., 2024) propose sharing B and across selected lay- ers, while VB-LoRA (Li et al., 2024) introduces shared vector bank compose all low-rank matrices via differ- entiable top-k admixture. Although these approaches re- duce the number of learnable parameters, our experiments show that they often underperform or significantly de- grade performance. For instance, Tied LoRA reduces pa- rameter count but sacrifices expressivity, while VB-LoRA adds unnecessary complexity via top-k modeling. We ad- vocate for parameter sharing across layers but emphasize that the expressivity of the fine-tuned model should rely pretrained weights to compensate for the reduction in learnable parameters. Our results highlight the criti- cal role of pretrained weight awareness in enabling this trade-off.
In summary, our experiments reveal the limitations of LoRA variants such as DoRA and VeRA, as well as alternative methods like ReFT, highlighting their drawbacks in com- pute, memory, and performance trade-offs. Outperform- ing LoRA while maintaining multi-faccted efficiency in parameters, representations, compute, and memory re- mains significant challenge. In this paper, we propose novel approach to address this challenge.
To clarify the foundation of our proposal, we first review the formulation of LoRA (Hu et al., 2022). Denote the pretrained weights of a layer E L of a Transformer model by W E addouexdn, and the fine-tuned weights by W E Rodt xdn, LoRA is defined as:
Additive vs. Multiplicative Updates: LoRA uses ad- ditive weight residuals (B' AA'), which can be limiting in terms of expressivity. Alternatively, multiplicative up- dates may enable richer, structured transformations, For example, DiReFT (Wu et al., 2024) applies multiplicative updates in the representation space. Let y E Rdo denote the activation output (representation) for the i-th token at layer 1. DiReFT updates as:
(L) where B' Xr and A' E Br are two low- rank matrices representing new learnable model parame- ters introduced during fine-tuning, and r is the rank (r & min(d;,, (din, dout)). Typically, B' is initialized to zeros, while A' is randomly initialized.
LoRA: W = W ww'+B'
DiReFT: o = 2/ + B' (A' of +b))
(2)
(I+ A') 1/ BB
We rethink LoRA from three key aspects:
where B¹ E R.ooxx A' E Rxdouut and D' R
parameters of the representationn-ediing module. is the identity matrix. While DiReFT introduces multiplicative residuals into the representation space, it is coupled with token intervention search selkei-elecctive..
Pretrained Weight Awareness: LoRA imposes no con- straints on B' and A' beyond their low-rank structure, enabling downstream task data to dictate the fine-tuning process. However, the pretrained weights W' encode "carry-over" knowledge that is expected to be useful for downstream tasks. By making fine-tuning weight resid- uals aware of the pretrained weights, we hypothesize
To achieve multi-faceted efficiency across parameters, rep- resentations, compute, and memory, we propose Weight- Generative Fine- Tuning (WeGeFT, pronounced wee-gift),
2

---

WeGeFT (wee-gift): Weight-Generative Fine-Tuning
Commonsense
Code (HumanEval)
Math (GSMBK)
88.2
LoReFT
WeGeFT
DoRA
19.9
WeGiFTA
LORA-Gr GA
DoRA
55.0
WeGeFT+
LoRA-GA
85.8
DiReFT
LoRA
DoRA
18.6
LoRA+
51.7
LoRA+
Accuracy
Accuracy
AdaLoRA
83.3
VeRA
Pass
17.3
AdA
48.5
rsLoRA PISSA
IsLoRA
80.9
16.0
45.2
PISSA
VoRLoRA
VB-LoRA
VB-LoRA
14.7
LoRA
42.0
LoRA
78.5
0.01
0.02
0.04
0.07
0.14
0.27
0.52
1.0
0.01
0.05
0.22
0.80
0.01
0.05
0.22
0.80
Parameters (%)
Parameters (%)
Parameters (%)
Model
LLaMA-I (7B)
Llama 2 (7B)
Llama (8B)
Figure 2: Comparisons of performance vs. trainable parameters between our WeGeFT and baseline methods on three tasks using the Llama model family. Figure shows that WeGeFT maintains the compute and memory efficiency of LoRA, thus achieving very strong multi-faceted efficiency across parameters, representations, compute and memory. See Section 4 for experimental details.
a simple yet effective formulation (see the right of Fig. 1):
21.5
Our WeGeFT: W 1 + W
(3)
15.5
W (I+ v)
9.5
Ratio
4.5
where e E Rddox and sh Bexdi. low-rank matrices shared across layers, and r is the rank. See Appendix C for gradient analyses between LoRA and our WeGeFT.
1.5
1.0
0.5
WeGeFT be viewed as weight- variant of LoRA, where the layer-specific B' in LoRA becomes weight-aware (B'
Weight-Aware Parameter Sharing:
0.0
DoRA
WeGeFT
VB-LoRA
VeRA
GPU Memory
Training Wall Time
W .) and the layer-specific A' becomes layer-agnostic (A x.). Compared to Tied LoRA (Renduchintala et al.,
Figure 3: Comparison of the ratio of the GPU Memory (Training Wall Time) for various PEFT methods with the GPU Memory (Training Wall Time) of LoRA. WeGeFT maintains the efficiency of LoRA, as opposed to DoRA and VeRA. Note that while VB-LoRA maintains the memory and compute efficiency, it performs worse than LoRA as seen Figure 2.
2024), WeGeFT retains layer-specific information via B!, preserving performance. Unlike VB-LoRA (Li et al., 2024), WeGeFT avoids the need for complex shared vector bank and top-k admixture, instead relying pair of shared low-rank matrices (Ø, v)) for simplicity and stability. WeGeFT achieves significant improvements in parameter efficiency without sacrificing performance and even enables performance gains when stronger pretrained models (e.g., LLaMA vs. Llama 3) are used.
ciency of our proposed WeGeFT. Fig. shows the efficiency comparisons of GPU memory footprint and training wall time.
Residual Learning in Weight Space: WeGeFT extends the residual learning principle of ResNets (He et al., 2016), [x), into the weight space. In contrast, DiReFT (Wu et al., 2024) applies this principle in the representation space with causal intervention treatments. WeGeFT, however, eliminates the need to search for spe- cific intervention positions and token-agnostic, ensuring it retains LoRA's compute and memory efficiency while offering greater flexibility (see Sec. 3.2).
Visual Inspection of WeGeFT in Computer Vision Tasks: Let Cd = W o be the transformation using the first linear layer WeGeFT for an output projection layer in MHSA (see Sec. 4.5). We show that Cd ded xr can be used tuned model, the activation of the output projection layer is, ÿ' E I. x where N the number of visual tokens in raster order. We compute heatmaps for visual token clustering by,
token-clustering head. Using the fine-
Fig. 2 shows result comparisons on three benchmark datasets, which demonstrate the overall multi-faceted effi-
Hl'xr -W C.. Cdouxrr
(4)
3

---

WeGeFT (wee-gift); Weight-Generative Fine-Tume Fine-nuning
Layer 10 Cluster
Layer 10
Layer 10 Cluster 4
Layer 10 Cluster 10
Input
Cluster
Input
Figure 4: WeGeFT can play the role of a r-way token-clustering head that can localize meaningful objects/parts on images. More examples can be found in Figure 5 in the Appendix.
which can highlight semantically meaningful parts of the image. We normalize the heatmaps to (0, individually and use 0.5 as the threshold in visualizations (Fig. 4).
works, i.e, neural networks that generate the parameters for other neural networks, in language modeling tasks by generating the weights of an LSTM (Hochreiter & Schmid- huber, 1997). Hypernetworks have previously been applied for few-shot classification (Zhao et al., 2020; Zhmoginov et al., 2022), transfer learning (Requeima al., 2019) and continual learning (von Oswald et al., 2020; Yin al., 2022). Similar our proposed approach, (Requeima al., 2019) learns adapt global feature extractor through adapta- tion network. In few shot continual learning setup, (Vla- dymyrov et al., 2023) uses a hyper- Transformer to generate the parameters for a separate Convolutional Neural Network (ConvNet), which use as inputs both a support set images of the current task and the ConvNet parameters generated for the previous tasks. HyperFormer++ (Mahabadi al., 2021b) uses Multi-Layer Perceptron (MLP) to generate the parameters from layer embedding and a latent vector for Adapters (Houlsby et al., 2019) introduced across layers of a pretrained model a multitask setting. Unlike (Mahabadi et al., 2021b), we directly use the weights the frozen pretrained model, thus eliminating the need for embeddings.
2. Related Work and Our Contributions
Parameter Efficient Fine-tuning (PEFT). The goal of PEFT methods is to reduce the computational resources (memory footprint, wall time, etc.) required for fine-tuning large models such as Transformers (Vaswani et al., 2017) and Vision Transformers (ViTs) (Dosovitskiy et al., 2021). Prompt-hased methods either append prompts to the input tokens (Lester et al., 2021; Jia et al., 2022), or the inter- mediate layers (Li Liang, 2021: Liu al., 2021; Zhang al., 2023b). Early work PEFT used sequential/paraalel learnable adapters added after the Multi-Head Self Atten- tion and/or FFN blocks (Houlsby et al., 2019; Bapna & Firat, 2019; Pfeiffer et al., 2021; 2020; Rücklé et al., 2021; Mahabadi et al., 2021a; Chen et al., 2022). LoRA (Hu et al., 2022) and its variants (Zhang et al., 2023a; Dettmers et al., 2023; Lialin et al., 2023; Kopiczko et al., 2023; Gao al., 2024; Liu al., 2024) learn residuals to the pretrained weight matrices in the form of low-rank factorization, moving the added inference cost in adapter based methods. BitFit (Zaken et al., 2022) fine-tunes all the bias terms in pretrained backbone. MEND (Mitchell et al., 2022) edits a pretrained model by learning fine-tur tooitunnng weights from the gradient inputs with low-rank MIP parameterization. FacT (Jic & Deng, 2023) shares the trainable parameters Self-Attention and MLP blocks. While this reduces trainable parameters, it has drawback: Transformer mod- ules serve distinct functions attention layers largely handle syntactical and in-context learning abilities (Bietti et al., 2023; Voita et al., 2019), while MLP layers encode factual knowledge (Meng et al., 2022). Sharing parameters across these modules may therefore be suboptimal. Tied-LoRA (Renduchintala et al., 2024) shares the residual weights across layers, and also across Query, Key and Value compo- nents. In Section 5.2, we show that the in WeGeFT is essential for parameter sharing across layers, enabling it to outperform Tied-LoRA.
Neural Functionals: Our approach is related to neural functionals that aim to learn deep neural networks acting on the weights of other neural networks. For toy problems, equivariant architectures have been explored for tasks like classifying implicit neural representations (Navon et al., 2023; Zhou et al., 2023a;b; Kofinas et al., 2024), adapting model architectures to new domains (Navon al., 2023), predicting model generalization performance (Zhou et al., 2023a:b; Kofinas et al., 2024; Lim et al., 2023), and learned optimizers (Zhou et al., 2024). However, our work is the first to explore fine-tuning of a model using it's own weights. We do not use equivariant architectures, but note that this direction of work is orthogonal to ours, and can be further explored in the future.
Our Contributions summarized as follows:
Weight-Generatie Fine-Tuning Framework: WeGeFT introduces novel formulation where fine-tuning weight residuals are parameterized directly using pretrained weights, leveraging their inherent knowledge for better expressivity, and leading multiplicative updates in the
Hypernetworks. (Ha et al., 2016) introduced Hypernet-

---

WeGeFT (wee-gift): Geneatt Fine-Tuning
weight space for richer and more structured transforma- tions compared to additive methods.
in weight space, including the from-scratch-training of the pretrained weights W themselves, are mostly sim- ple updates based on SGD. Nonlinear transformations may be destructive to the "carry over" knowledge in the pretrained weights, thus negatively impact weight- awareness. More importantly, with a nonlinear 9o. our WeGeFT will sacrifice the multiplicative updates in Eqn. W W (I + Ø v). to additive updates, W W + go o) 4/, which will also significantly impact compute and memory efficiency.
Multi-Faccted Efficiency: WeGeFT achieves multi- faceted efficiency across parameters, representation, com- pute and memory by using shared low-rank matrices (ø, 2)), significantly reducing learnable parameters while retaining LoRA's simplicity without sacrificing perfor- mance (See Figure 2 and Figure 3).
Unified Parameter and Representation Efficiency:
WeGeFT bridges the gap between parameter-effiiien and representatioon-effcciee fine-tuning by unifying weight- generative paramcterization and shared low-rank matrices. Strong and Scalable Performance: WeGeFT consis- tently matches or outperforms LoRA and its variants, achieving superior scalability with stronger pretrained models like Llama 3.
WeGeFT can be applied along the dous dimension too. In Eqn. 3, (Ø, 4)) are applied along the d,n dimension of pre- trained weights W It is straightforward to apply WeGeFT along the dout dimension by,
W = W + (Ø TTTT)).T.. WW,
where & E Rod and vb E Brxdow.
3. Approach
WeGeFT Without Parameter Sharing, It is straightfor- ward to apply our WeGeFT (Eqn. 3 and Eqn. 7) without sharing (Ø, v)) across layers (denoted by WeGeFT-Sep) which will increase the learnable parameters of the counter- part (WeGeFT with parameter sharing), and to the same as LoRA. We show that WeGeFT-Sep can obtain on mn-par or bet- ter performance than LoRA, demonstrating the advantage of weight-awareness. This flexibility allows WeGeFT to scale more elegantly to larger and diverse datasets, which cannot be achieved by prior methods like VeRA and VB-LoRA.
In this section, we elaborate on our simple formulation of WeGeFT (Eqn. 3) from the more general parameter- generation perspective, which can provide deeper insights.
3.1. Weight Generation for Explicit Weight-Awareness
For a layer/ / E L of pretrained Transformer model, to learn its fine-tuning weights, denoted by AW', that are aware of pretrained weights W' "carry over" their knowledge, general formulation is to train generator network,
AW G(W θ), VI L,
(5)
3.2. WeGeFT as Token-Agnostic ReFT
where θ is the learnable parameters of the weight generator.
Consider a linear layer with pretrained weights W and the pretrained bias term b', and WeGeFT weights W (T + Φ +)- W' (Eqn. 3). For input se, the output repre- sentation/activation at this layer is,
G(:; O) needs to be parameterized in way to meet the desired multi-faceted efficiency. Inspired by the low-rank parameterization scheme in LORA, we have,
G(W*; = S o go o So(""), VI L., where both Jo RRdout diin + Rdoue and [x(x) Bdout xr + Rdoui xd, are linear projection layers (without bias terms) with representing the "rank". go : X x Rr realizes latent transformations in the low r-dim space. θ (Ø, v 8) collects all learnable parameters.
(6)
8' = x' WT = 2l W +
(8)
where a' is the "fine-tuned" input representation/aaciia tion using the same WeGeFT parameters. Hence, our WeGeFT can be equivalently applied to the input ac- tivation, rather than the pretrained weights, to achieve the same fine-tuning effect, maintaining the memory and compute efficiency of LoRA in implementation. Unlike the ReFT (Wu et al., 2024) that entails a dedicated search for where the representation interventions should apply at the token level, our WeGeFT eliminates the need of search, en- abling token-agnosticity. Thanks to the parameter sharing, our WeGeFT can retain the representation efficiency.
We note that this design offers a very flexibile way to cap- ture underlying contingency between the fine-tuning weight residuals and the pretrained weights in all T. layers. Sur- prisingly, we observe that we do not need ge based on our ablation studies (see Section 5.1). In other words, go is an identity transformation, leading to the simple formulation in Eqn. 3. Our (post-hoe) intuitive understanding is:
If go includes only linear transformations, it can be natu- rally absorbed into fo and/or fu.
4. Experiments
If go is an overall non-linear transformation, G(WL; 0) applies the non-linear transformation in the weight space, which may not be necessary. After all, iterative updates
We conduct extensive experiments across Natural Language Generation and Visual Recognition, and compare our two- linear-layer parameterized WeGeFT with various PEFT
5

---

WeGeFT (wee-gift); Geneatt Fine-Tuning
Table I: Results of fine-tuning Llama 2 (7B) on the Meta- MathQA dataset and evaluating it on the GSM8K test set. All baseline results are obtained from (Wang et al., 2024), except VB-LoRA is trained by together with our proposed WeGeFT. We train our WeGeFT and VB-LoRA by following the same settings in (Wang et al., 2024).
Table 2: Results of fine-tuning Llama and 2 (7B) on the Math10k benchmark. The Mem. refers to GPU memory, and Wall Time is the time required to complete epoch of training. All results are obtained by us using our code base for fair comparisons, except those by DiReFT and LoReFT using LLaMA 1 are from (Wu et al., 2024).
Method
% Trainable
GSM8K (Acc.)
Params (%)
(CB)
Full
100
54.20.042
AQUA
GSMMK
MAWPS
SVAMP
Avg. Ape.
LoRA (Hu et al., 2022)
0.297
42.08:0004
Wall
Method
PISSA (Meng et al., 2024)
0.297
44.54
IsLoRA (Kalajdzievski, 2023)
0.297
45.62=a010
LoRA
0.416
18.01
0:43
23.5
38.5
85.3
56.4
50.9
LoRA+ (Hayou et al., 2024)
0.297
52. 12.1
LLAMA-1
DoRAmis
WeG.PP-SSpp....
0416
0.427
20.37
18.01
2.36
0.46
23.8
21.5
37.9
37.9
84.5
86.0
55.3
54.2
50.1
50.2
DoRA (Liu et al., 2024)
0.317
53.07
A (Zhang et al., 2023a)
0.445
50.72+1.99
LoRA-=
0.052
17.74
0.43
23.1
34.6
83.9
54.1
48.9
DoRA
0.065
20.09
2.36
21.1
34.6
84.0
53.8
48.4
LaRA-GA (Wang et al., 2024)
0.297
53.60020:0
VeRA
0.042
34.0
50.7
47.2
VB-LoRA (Li et al., 2024)
1.194
44.93+152
FucT-TT
0.051
17.74
0.52
21.5
30.7
80.3
50.3
45.7
Our WeGeFT
0.068
54.89
FacT-TK
0.062
17.75
21.3
34.8
82.2
51.9
47.5
VB-LeRA
0.840
18.33
0.42
21.26
29.3
78.9
49.5
44.7
WeGelTT,.
0.052
17.74
36.5
50.0
DiRcFT
0.031
31.42
0.26
21.3
24.1
74.5
42.7
40.6
methods and ReFT. We also conduct ablation studies on the different parameterization schemes of WeGeFT. More de- tails can be found in Appendix D. In all the experiments, we follow the baselines in selecting the layers to fine-tune for downstream tasks for fair comparisons. For clarity, we use WeGeFT a.. (Eqn. 3), WeGeFTa.w. (Eqn. 7), and WeGeFT- Sepa.. (Eqn. 3 but without parameter sharing).
LoReFT
0.831
55.42
0.29
21.4
26.0
76.2
46.8
42.6
WeGelT
0,016
17.71
0.36
20.74
33.0
80.8
53.5
47.0
LoRA
0.466
18:01
0.43
24.5
43.4
86.1
57.2
52.8
DoRA--1e
0.429
20.37
2.35
24.1
41.4
87.1
57.1
52.4
Llona 2.78))
WeGeFTT-Sep....
0.416
18.01
0.46
26.1
42.4
85.9
58.6
53.3
LoRA
0,052
17.74
24.7
40.2
85.0
51.5
DoRA
0.065
20.09
2.35
24.0
40.6
84.6
56.0
51.3
VeRA
0.042
20.65
9.00
23.5
38.7
85.5
54.3
50.4
VB-LaRA
0.840
18.33
0.43
22.4
33.4
81.4
52.4
47.4
FacT-TT
0.051
17:74
24.9
38.3
81.9
56.2
50.3
FacT-TK
0.062
17.75
0.59
24.5
41.0
85.7
54.4
51.4
4.1. Arithmetic Reasoning
WeGeFT,
0.052
17.74
0.50
23.6
42.4
84.2
57.4
51.9
DiReFT
0,031
31.42
0.26
20.5
27.9
77.5
45.9
42.9
We demonstrate the multi-faceted efficiency of WeGeFT with experiments on the Math10k benchmark (Hu et al., 2023) for Arithmetic Reasoning, which is a small scale dataset enabling comprehensive evaluations. We conduct further experiments with MetaMathQA (Yu et al., 2024), a larger and higher quality fine-tuning dataset, to understand the impact of higher quality data on WeGeFT. In both exper- iments, evaluate the model on the final answer following the same protocol used in prior works. Experimental details and hyperparameters are provided in Appendix D.2.
LoRelT
0.031
55.42
0.29
24.8
31.7
79.6
50.9
46.7
WeGieT
0,016
17.71
0.39
26.1
38.0
83.1
57.3
51.1
parable parameter count ( (0.55%) WeGelTT outperforms all baselines (LoRA, DoRA, VeRA, VB-LoRA) and nearly matches LoRA at r 16. This demonstrates that gen- erating fine-tuning residuals from pretrained weights improves parameter efficiency by enabling adaptation with minimal parameters, whereas LoRA and DoRA with r = 2 struggle to perform well.
Results: Table shows the results of finetuning using the MetaMathQA datasets, and evaluating on the GSMSk test set. WeGeFT outperforms all the prior methods with 4 times fewer parameters, and slightly outperform the full fine- tuning. We use the relatively smaller Math benchmark (Table 2) to run comprehensive evaluations for WeGeFT with respect to multi-faceted efficiency.
Computational and Memory Efficiency: WeGeFT main- tains compute and memory efficiency while achieving equal or higher accuracy than prior PEFT methods, unlike DoRA and VeRA, which compromise efficiency. Although VeRA reduces trainable parameters, it requires large interme- diate dimension for fixed random weights (12288 here). On Math10k, VeRA takes ~9 hours/epoch and 20.65GB GPU memory, whereas LoRA and WeGeFT need only ~0.5 hours/epoch and 17.74GB under the same setup. Table 10 in Appendix A confirms that reducing the intermediate di- mension to 1024 (as in (Kopiczko et al., 2023)) lowers time and memory costs but causes severe accuracy drop. Table 11 in Appendix B shows that WeGeFT is efficient even with mixed preed-precisoon float16. Thus, Similar to LoRA, WeGeFT can be used even with consumer GPUs.
Parameter Efficiency: WeGeFT efficiently adapts both strong and weak models, as shown by finetuning LLaMA-1 and LLaMA-2 (7B) on the Math10k benchmark in Table 2. LLaMA-1 (7B), a weak model for arithmetic reason- ing (11% zero-shot accuracy GSM8k (Touvron et al., 2023a)); requires substantial adaptation. Table 2 shows WeGeFT adapts LLaMA-1 (7B) far more effectively than prior PEFT methods. Reducing parameters in LoRA and DoRA significantly degrades performance LoRA's accu- racy drops from 50.9 (r = 16) to 48.9 (r = 2). At a com-
Impact of Data Quality: WeGeFT "reacts" positively to the quality of training datasets, as can be seen by the relative
6

---

WeGeFT (wee-gift): Weightt Fine-Tuning
Table Results of fine-tuning LLaMA-1 7B, Llama 7B and Llama 8B on eight Commonsense Reasoning benchmarks
(Commonsensel100s ReFT results are obtained from (Wu al.. 2024).
Method
Params (%)
BoolQ
PIQA
SIQA
HellaS.
WinoG.
ARC-e
ARC-c
OBQA
Avg
LoRA (Hu et al., 2022)
0,416
73.3
84.5
80.4
94.2
85.5
87.6
72.6
85.6
83.0
LLaMA-1 (7B)
DoRA (Liu et al., 2024)
0.427
73.4
84.8
80.7
94.4
85.7
87.4
72.4
85.9
83.1
VeRA (Kopiczko et al., 2023)
0,023
70.4
82.4
79.9
91.4
81.8
83.3
67.0
80.6
79.6
VB-LoRA (Li et al., 2024)
0.840
70.5
82.6
79.3
92.5
83.1
84.5
68.1
81.7
80.3
WeGeFTa..
0.052
72.8
84.7
80.8
93.9
84.3
86.7
72.3
85.1
82.6
DiReFT (Wu et al., 2024)
0.031
69.5
83.0
79.0
92.5
80.5
82.2
68.0
77.5
79.0
LoReFT (Wu et al., 2024)
0,031
69.3
84.4
80.3
93.1
84.2
83.2
68.2
78.9
80.2
WeGeFTT
0.016
71.5
83.4
81.1
93.6
83.7
86.1
72.0
83.9
81.9
LoRA (Hu et 2022)
0.416
74.8
85.9
80.8
94.8
86.3
88.3
75.4
85.9
84.0
DoRA (Liu ct al., 2024)
0.427
74.6
86.2
81.1
94.9
86.8
89.1
75.9
86.7
84.4
Llama 2 (7B)
VeRA (Kopiczko et al., 2023)
0,023
71.9
82.2
80.0
92.2
83.3
84.3
68.8
80.5
80.4
VB-LoRA (Li et al., 2024)
0.840
71.6
83.2
79.7
92.8
83.5
84.8
69.3
81.9
80.8
WeGelFTa..
0.052
73.9
85.7
82.1
94.6
85.6
88.3
74.7
85.4
83.8
DiReFT (Wu et al., 2024)
0,031
70.8
83.6
80.2
93.6
82.1
84.8
70.4
81.5
80.9
LoReFT (Wu et al., 2024)
0.031
71.1
83.8
80.8
94.3
84.5
85.6
72.2
82.3
81.8
WcGeFTa,..
0.016
73.4
85.2
81.8
94.3
85.3
87.7
74.9
83.8
83.3
LoRA (Hu al., 2022)
0.392
74.6
89.4
81.3
95.9
87.7
91.9
81.7
87.7
86.3
DoRA (L.u et al,, 2024)
0.361
76.2
90.8
82.1
96.7
89.0
93.5
83.4
89.1
87.6
Llama 3 (8B)
VeRA (Kopiczko et al., 2023)
0,018
71.6
85.7
80.7
93.8
85.2
87.6
75.6
84.1
83.0
VB-LoRA (Li et al., 2024)
0.712
72.3
83.7
80.0
93.1
84.
85.1
70.5
81.9
81.3
WeGeFTa...
0.049
76.0
89.7
83.1
96.7
89.1
93.0
84.4
89.8
87.7
DiReFT (Wu et al., 2024)
0.026
73.4
88.7
81.0
95.6
85.5
91.8
81.8
85,4
85.4
LoReFT (Wu et al., 2024)
0,026
75.1
90.2
82.0
96.3
87.4
92.4
81.6
87.5
86.6
WeGeFT
0.013
75.7
89.9
82.5
96.4
88.7
92.5
82.3
86.3
86.8
performance improvement from Math10k benchmark (Table 2) to the MetaMathQA benchmark (Table 1).
of our initial experiments, WeGeFT-Sep does not show sig- nificant improvement over WeGeFT. Hence, we focus on evaluating WeGeFT on the Commonsense 70k benchmark. We summarize the observations as follows:
Comparison with ReFT: To compare with DiReFT and LoReFT, we apply WeGeFT following their fine-tuning strategy (Fqn. 2). WeGeFT achieves higher average accu- racy than both ReFT variants while using half the parame- ters on LLaMA-1 and 2. ReFT's higher memory usage (and lower wall time) is due to its implementation lacking gradient checkpointing, whereas our HuggingFace PEFT-based implementation uses checkpointing for better scalability with large models.
The weight-awareness of our WeGeFT is positively cor- related with the expressivity of the pretrained models. WeGeFT slightly outperforms DoRA in fine-tuning Llama
3 (8B) using 8x fewer parameters, while DoRA outper- forms WeGeFT in fine-tuning both LLaMA and Llama 2 the expense of 8x more parameters and more expen- sive training cost. Considering the trend of increasingly powerful pretrained large foundation models, WeGeFT shows a very promising potential due to its efficiency and strong performance.
4.2. Commonsense Reasoning
We use combined training data of eight benchmarks (i.e.,
Although VeRA uses less parameters, its performance is much worse and the training cost is very high, sim- ilar the observations on 10th Furthermore, our WeGeFT outperforms VeRA with even less parame- ters and much more efficient training.
Commonsense 170k, containing total of 170k training samples), and evaluate on their test sets individually, follow- ing the same protocol used in (Hu et al., 2023) and (Wu et al., 2024). The examples in the Commonsensel 70k are formu- lated as multiple choice questions and consist of BoolQ (Clark et al., 2019), PIQA (Bisk et al., 2020), SIQA (Sap al., 2019), HellaSwag (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2021), Are-e and Are-c (Clark et al., 2018), and OBQA (Mihaylov et al., 2018) datasets. We experiment with 1 (7B), Llama 2 (7B) and Llama 3 (8B) mod- els. Experimental details including hyperparameters are provided in Appendix D.3.
Compared with DiReFT and LoReFT, our WeGeFT is still better and reduces the parameters by half.
4.3. Instruction Following
We fine-tune LLaMA-2 (7B) on a 52k subset of the Wiz- ardLM dataset (Xu et al., 2024), filtering out samples con- taining "Sorry" and "As an AI" following (Wang et al., 2024). We evaluate the instruction following ability on the
Results are shown in Table 3. Based on the observations

---

WeGeFT (wee-gift): Weieht- Fine- Fine-Tuning
Table 4: Results of fine-tuning Llama 2 (7B) on the Wiz-
LoRA-GA and DoRA.
ardLM dataset (Xu et al., 2024) and evaluating it on the
MT-Bench (Zheng et al., 2023). All the results except VB- LoRA are obtained from (Wang al., 2024). We train VB-
4.5. Visual Recognition
I LoRA and WeGeFT following the same settings as (Wang
Data. We evaluate WeGeFT on the VTAB-IK benchmark (Zhai al., 2019) and the fine-grained visual classification (FGVC) benchmark containing Caltech-UCSD Birds (Wah et al., 2011), NABirds (Horn et al., 2015), Oxford Flowers (Nilsback & Zisserman, 2008), Stanford Cars (Gebru et al., 2017), and Stanford Dogs (Khosla et al., 201 1).
et al., 2024).
Method
Params (%)
First Turn Score
Full
100
5,56.10.9
LoRA (Hu et al., 2022)
0.297
5.6110.10
PISSA (Meng et al., 2024)
0.297
5.30+am.
IsLoRA (Kalajdzievski,, 2023)
0.297
5.25
LoRA+ (Hayou et ul., 2024)
0.297
5.71
Models. We use the ViT-B/16 architecture (Dosovitskiy et al., 2021) pretrained on ImageNet21k dataset (Deng et al., 2009) using supervised objective, with the checkpoints from the t timm package (Wightman, 2019). We apply LoRA and WeGeFT to the output projection layers in MHSA, which is inspired by observations in (Savadikar et al., 2023), All hyperparameters are provided in Appendix D.6.
DoRA (Liu al., 2024)
0.317
5.9710.0
AdaLoRA (Zhang al., 2023a)
0.445
5.57=0.u5
LoRA-GA (Wang ct al., 2024)
0.297
5.95+0.16
VB-LoRA (Li al., 2024)
1.194
5.57.0.08
WeGeFTar.
0.068
5.75
MT-Bench dataset (Zheng et al., 2023), which spans do- mains such math, roleplay, reasoning, and coding. We report the the single turn score by prompting an LLM judge (GPT4) to rate the responses from the fine-tuned model from 1-10. Experimental settings and hyperparameters can be found in Appendix D.4).
Results: Tables and show that our WeGeFT performs better than other PEFT methods on both FGVC, while using fewer parameters. The GPU memory consumption is similar among the different methods with negligible differences. With 5.9 times less parameters used (0.025M vs 0. 1.147M), on FGVC tasks, our WeGeFT improves LoRA by 0.68% Top-1 accuracy.
Results are shown in Table Our WeGeFT is on-par with LoRA-GA and DoRA with times fewer parameters. It also slightly outperforms the full fine-tuning.
Table 6: Results on the finegrained visual classification (FGVC) tasks with V pretrained on ImageNet2 lk. The number of trainable parameters are reported without the classification head (which has the same number of pa- rameters for all the methods).
4.4. Code Generation
We fine-tune Llama 2 (7B) using the
dataset (Zheng et al., 2024), which is multi-turn dataset contain- ing execution and human feedback. We evaluate the fine- tuned models on HumanEval (Chen et al., 2021), containing Python problems evaluated for functional correctness. Ex- perimental settings and hyperparameters are in Section D.5.
Method
Params (M)
CUBS
Bird
Flower
Dog
Avg
VPT
0.046
87.88
84.79
98.98
84.51
82.89
87.81
Birit
0.083
87.75
84.61
99.32
85,23
84.01
88.18
LORA
0.147
88.00
84,94
99,32
85.36
85.92
88.71
WeGelT
0.025
89.71
86.28
99.22
87.44
84.28
89.39
Table 7: Results the VTAB benchmark (Zhai et al., 2019)
Table 5: Results of fine-tuning Llama 2 (7B) on the Code- Feedback dataset (Zheng et al., 2024) and evaluating it on the HumanEval (Chen et al., 2021). All results except VB-LoRA are obtained from (Wang et al., 2024). We train
with ViT-B/16 pretrained on ImageNet21k. Trainable pa- rameters are reported the same way as Table 6.
Method
Params (M)
Natural
Specialized
Structured
Avg
VPT
0.046
81.0
85.7
58.9
72.7
VB-LoRA and WeGeFT following the same settings as
BitFit
0.083
81.8
85.2
57.8
72.4
(Wang et al. 2024).
0.147
82.0
74.0
FacT-TT
0.040
79.8
86.0
58.0
71.9
Method
Params (%)
Pass 1
FacT-TK
0.069
80.0
86.8
60.9
73.4
Full
100
19.87
WcGeFTa,.
0.025
82.0
86.3
61.1
74.1
LoRA (Hu al., 2022)
0.297
14.76=0.11
PiSSA (Meng et al., 2024)
0.297
16.02
IsLoRA (Kalajdzievski, 2023)
0.297
16.01
5. Ablation Studies
LoRA+ (Hayou et al., 2024)
0.297
18. 00.52
DoRA (Liu al., 2024)
0.317
19.75=0.11
5.1. Diffferntt Parameterization Schema for WeGeFT
AdaLoRA (Zhang et al., 2023a)
0.445
17.80=0044
As mentioned in Section 1, simple linear transformation of the pretrained weights works surprisingly well in gen- erating fine-tuning residual weights. To verify effects of non-linear go() in Eqn. We compare,
LoRA-GA (Wang et al., 2024)
0.297
19.81
VB-LoRA (Li et al., 2024)
1.194
14.92
WeGeFT de..
0.068
19.39
Results are shown in Table Our WeGeFT is on-par with
Transformer: We treat the set of shared pretrained weights
8

---

WeGeFT (wee-gift): Geneatt Fine-Tuning
across L layers batch of L sequences each consist- ing of dout tokens in r-dim space (after the first linear project layer fa), denoted by W. We then apply single Transformer block (Vaswani et al., 2017).
6. Remarks on the Effectiveness of WeGeFT
Based on the experimental results, we may draw intuitive and potentially deeper understanding of PEFT and ReFT methods using pretrained Transformer backbones: Pre- trained Transformer backbones "distill" general and diverse knowledge from large-scale pretraining dataset, encoded in the pretrained weights. When fine-tuning them at down- stream task, to "absorb" new information in the training data of the downstream task, one of the simplest updates that minimally "distorts" and maximally "preserves" the pretrained knowledge is defined by Eqn. or Eqn. 7, thanks to the low-rank factorized linear projection in the parameter space. The newly "absorbed" information from the down- stream task is also linearly expressed in the space spanned by the pretrained weights (knowledge).
MI.P-Mixers: Similar to vanilla Transformers, we apply a single MLP-Mixer (Tolstikhin et al., 2021) block.
Malei-Layer Perceptrons (MLPs): e.g.. 9(W:)) where 01 E Rmr-x+tm- and 0g E * consist of weights and bias terms of the two linear layers with the MLP latent dimension ratio (e.g.,
Linear( Linear( (LLLU
2).
funtctioon functions without learnable parameters (i.e., 9 = 0): e.g., g(W) = Sigmoid(W) or g(W) = GELU
Element-wise non-linearity
Table 8: Comparisons between various non-linear transfor- mations for go on the FGVC benchmark.
Schema
Params (M)
CUBS
Bird
Flower
Dog
Car
Aug
7. Conclusion
Identity
0.025
86.28
99.22
87.44
84.28
Sigmoid
0.025
89.56
84.61
99.20
86.69
84.04
88.82
We present Weight-Grenrratve Fine-Tuning (WeGeFT) for adapting pretrained Transformer backbones downstream tasks. Our WeGeFT learns to generate the fine-tuning weight-residuals for layers selected in fine-tuning directly from their frozen pretrained weights. It is parameterized using two-linear-layers (without bias terms). It achieves muutii-accted efficiency across parameters, representations, compute and memory in comparisons with LoRA and its variants, and ReFT. We conduct experiments across various tasks, including Natural Language Generation (instruction following, commonsense reasoning, code generation, and arithmetic reasoning). and Visual Recognition. WeGeFT shows strong performance while retaining multi- facetee efficiency.
Gel.U
0.025
89.70
85.30
99.19
86.71
83.81
88.94
MLP
0.036
89,06
99.30
87.17
84.24
88.84
Transformer
0.027
89.56
86.23
99.24
86.31
84.26
89.12
MLP Mixer
0.125
88.76
86.21
99.25
86.35
85.66
89.25
Through ablation studies on the FGVC benchmark, we ver- ify that using any non-linear transformation for go results in degradation in performance. We use the same settings Section 4.5. As seen from Table 8, the simple two-linear layer formulation achieves better or equivalent performance than all other schema at a lower parameter cost. While we do not have a theoretical understanding yet, we hypothe- size that the superior performance of the identity operation over more complex and non-linear operations is because of difficulty in optimization. We only study the non-linear formulation small models on simpler tasks due to compu- tational constraints, and note that this presents an interesting avenue for future research.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
5.2. A Alternative Formulation of Tied LoRA
Tied LoRA (Renduchintala et al., 2024) uses sophisticated design of sharing weights across layers. We test a straightfor- ward parameter sharing LoRA, i.e., AW' = B- A VI E L, where (B, A) is shared across layers. Table 9 shows that this strategy leads to much lower performance than our WeGeFT, which justifies the advantage of weight-awarenesss
A Acknowledgments
This research is partly supported by NSF IIS- ARO Grant W91 INIIN1110295, ARO Grant 1911NF2210010, NSF CMMI-2024688 and NSF IUSE-2013451. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or im- plied, of ARO, NSF or the U.S. Government. The U.S. Gov- emnment is authorized reproduce and distribute reprints for Govermmental purposes not withstanding any copyright annotation thereon.
Table 9: Comparisons of Shared LoRA and WeGeFT on eight commonsense reasoning benchmarks.
Method
Params (%)
Avg
Shared LoRA
0.052
78.0
L (7B)
WeGeFT 4...
0.052
82.6
Shared LoRA
0.052
78.3
1 (7B)
WeGeFT
0.052
83.8
Shared LoRA
0.044
76.1
Llama-3 (8B)
WcGeFT
0.044
87.7
9

---

WeGeFT (wee-gift): Geneatt Fine-Tuning
References
trained on code. CoRR, abs/2 2021. URL http:://aaxxv..rrg/eaa// /twppa///21007033..
AI@Meta Llama 3 model card. 2024. URL https: //github. com/ meta- http//1aaaa3//////// //blbb/m SinnMODDE__AAA md.
Chen, S., Ge, C., Tong, Z., Wang, J., Song, Y., Wang, J., and Luo, P. Adaptformer: Adapting vision transformers for scalable visual recognition. In NeurIPS, 2022. URL
Bapna, A. and Firat, O. Simple, scalable adaptation for neural machine translation. In Inui, K., Jiang, J., Ng. V., and Wan, X. (eds.), Proceedings of the 2019 Confer- ence Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNIPP 2019, Hong Kong. China, November 3-7, 2019, pp. 1538-1548. Association for Computational Linguistics, 2019. doi: 10. 10.11853/V1/D19-11 1165. URL httpps://dii.ww /rrg:///////////////////oo 10. 1 (11188653vll1/ 9-1165
http:///apetssiiss.ccppppeee___ er / 20122222haa///622224900077711110 c555990f8 Abt
Clark, C., Lee, K., Chang, M., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surpris- ing difficulty of natural yes/no questions. In Burstein, J., Doran, C.,, and Solorio, T. (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pp. 2924-2936. Association for Computational Linguistics, 2019. doi: 10. 10.18653/V1/N199 1300. URL https://doi org / 00..0.1855553/0 /000/////////111110..
Bietti, A., Cabannes, V., Bouchacourt, D., Jégou, H., and Bottou, L. Birth of a transformer: A memory view- point. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine. S. (eds.), Advances in Neural In- formation Processing Systems 36: Annual Conference Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http: http:://paperse /ttpppaapaspea/aa/eeeeeaaeaeeeea .npperrssniiiss cc/paper _ iles htte:/pppee//222//hh/////1/33333 c8cd2ef0e5Ccfafe aart:eeesscoaffffaaaaaaaaaeeaaee Abst Conference. html.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the A12 reasoning challenge. CoRR, 00011/833005557 2018. URL http: /trtpii//axiv... org/ /t///////// /10.//113.... 05457.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. CoRR, abs/21 141668 2021. URL https: htt//axxxivirgge/ee httgg//bs//00 /210 14168.
Bisk, Y., Zellers, R., Bras, R. L., Gao, J., and Choi, Y. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Arti- ficial Intelligence, AAAI 2020, The Thioty-Second Inno- vative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA February 7-12, 2020, pp. 7432-7439. AAAI Press, 2020. doi: 10. 10.1609/AAAAL.34405..0oo https: https://doii: /ttp:://doi.ooo http:/i..rg...o.o/oo wwt.://g//.............. waw.///////a/a..//eeeeaaaeeaa..a v34105. 6239.
Deng, J., Dong, W., Socher, R., Li, L.,, Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pp. 248-255, IEEE Com- puter Society, 2009. doi: 10..110//VPP..202. 2009 0000940000 URL www.https/// /ttp:///oi..e. http///i..oggg/////////o 11t9////9..... 5206848.
URL
Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J.. Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R.. Krueger, G., Petrov, M.. Khlaaf, H., Sastry, G., Mishkin, P.. Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavar- ian, M., Winter, C., Tillet, P., Such, P., Cummings, D., Plappert, M. Chantzis, F., Barnes. E., Herbert- Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S., Jain, S., Saun- ders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J. Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., Mc- Grew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W. Evaluating large language models
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. Qlora: Efficient finetuning of quantized llms. CoRR, abs/2305. 14442 2023. doi: 10.48550/ARXIV.2305. 4. URL https: /ww./oiooooo oaciiorg/10.0 wtt.oo/gg......./.// 48550/ arxiv. 305.1433 14314.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL
10

---

WeGeFT (wee-gift); Weieht- Fine-Tuning
http:://ppeneeeeee...e//orrrrrrr
Horn, G. V., Branson, S., Farrell, R., Haber, S., Barry, J., Ipeirotis, P., Perona, P., and Belongie, S. J. Building bird recognition app and large scale dataset with citizen scientists: The fine print in fine-graines dataset collection. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pp. 595-604, IEEE Computer Society, 2015. doi: 0tW.99/CCVVR2211.228888888 URL https: / /oii. org/10. 110.0.1199//00 7291
FdNTTy.
Gao, Z., Wang, Q., Chen, A., Liu, Z., Wu, B., Chen, L., and Li, J. Parameter-effiieent fine-tuning with discrete fourier transform. arXiv preprint arXiv:2405.0m 003000...0 2024.
Gebru, T., Krause, J., Wang, Y., Chen, D., Deng, J., and Fei-Fei, L. Fine-grained car detection for visual cen- sus estimation. In Singh, and Markovitch, S. (eds.), Proceedings of the Thirty-Firs AAAI Conference on Arti- ficial Intelligence, February 4-9, 2017, San Francisco, California, USA, pp. 4502-4508. AAAI Press, 2017. doi: 00.1609/AAAA..3311.1111 //doi org 10. 1 609 aaai. v3111. 11174.
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., de Laroussilhe, Q. Gesmundo, A., Attariyan, M., and Gelly, S. Parameter-effiiient transfer learning for NLP, In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceed- ings of the 36th International Conference Machine Leurning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 2790-2799, PMLR, 2019. URL http:://roott// http//poooeddnng////eeeee mwr. pte.s//9.// hwt//////uuuooo /houl sby1 9a ha.m..
URL https:
Geiger, A., Wu, Z., Potts, C., Icard, T., and Goodman, N. Finding alignments between interpretable causal variables and distributed neural representations. In Locatello, F. and Didelez, V. (eds.), Proceedings of the Third Confer- ence on Causal Learning and Reasoning, volume 236 of Proceedings of Machine Learning Research, pp. 160-187. PMLR, 01-03 Apr 2024. URL ngs m1m press / wv23/v2223662 httpg/iggger222 pttpp/////////eeeeeeeeaaaa...
Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, I.., and Chen, W. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://
Ha, D., Dai, A. M., and Le, Q. V. HyperNerworks. In International Conference on Learning Representations, October 2016. URL httpt:t/p///oc. httpppp://ppeeee///e /ppp//peeeevveeeeeeeeeeeeeeeeeee net/f orum?id grumpid=rkpAAell1.
attps://ppenzeview hppp//ppeeeewe.eeeeeeeeee net fere/fooommtt ?id-nZeV KeeFYf9.
Hu, Z., Wang, L., Lan, Y., Xu, W., Lim, E., Bing, L., Xu, X., Poria, S., and Lee, R. K. LIm-adapters: An adapter family for parameter-efficite guage models. In Bouamor, H., Pino, J., and Bali, K. Empiri-
Hayou, S., Ghosh, N., and Yu, B. Lora+: Efficient low rank adaptation of large models. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https: btctp:///ppenevieew.rt wttp://wwww.etoooutummoeooo YaBRGO.
fine-tume tine-tuning of large lan-
(eds.), Proceedings of the 2023 Conference cul Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 5254-5276. Association for Computational Linguistics, 2023. doi: 10. N0.18653/V1/220 12055//W2222..MMLPMMMMMX //doi. wat.i.orgg/ae 18.33 319.
He, K., Zhang, X., Ren, S., and Sun, J. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In 2015 IEEE International Con- ference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pp. 1026-1034. IEEE Computer URL 123. He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016. doi: 10. 1091/.20/1. 6.90. URL. htt.t:://///// org 100.1/10///// CVPR. 016.90.
https: emnlp-main
Jia, M., Tang, L., Chen, B., Cardie, C., Belongie, S. J., Hariharan, B., and Lim, S. Visual prompt tuning. In Avi- dan, S., Brostow, G. J., Cissé, M., Farinella, G. M., and Hassner, T. (eds.), Computer Vision ECCV 2022 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIII, volume 13693 of Lec- ture Notes in Computer Science, pp. 709-727. Springer, 2022. doi: 10. 1011077/978-33001111997777117
Society, 2015. doi: 10. 20.1109//CC..2011000 https: www.doo..oooooooooooooooooooo.o. http://......//...
URL
https: //:::/doi.. htt.://///1...10///.11.. /ttpp:///obl.cee//0.000////11-3
27 41.
Jie, S. and Deng, 7. Fact: Factor-tuning for lightweight adaptation on vision transformer. In Williams, B., Chen, Y., and Neville, J. (eds.), Thirty- Seerey-Seeetth AAAI Conference on Artificial Intelligence, AAAI 2023. Thirty- Confer- ence on Innovative Applications of Artificial Intelligence,
Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural Comput., 98): nov 1997. ISSN 0899- 7667. doi: 10w.1162mecoo.1997.98... // //doi. /1/ //000..o////0.1115///rrr...///
URL https:

---

WeGeFT (wee-gift): Geneatt Fine-Tuning
IAAI 2023, Thirleenth Symposium on Educational Ad- vances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023, pp. 1060-1068. AAAI Press, 2023. doi: 10. (0.1609/AAALV3311.5555775 btttps///do .tgp:/..rg///// 609//aai wt3.0///a................ v3711.2 5187.
15179. URL wwtpt:t///o....oo/ /10 48550/arX 1v.2405.15179
Lialin, V., Shivagunde, N., Muckatira, S., and Rumshisky, A. Stack more layers differently: High-rank training through low-rank updates. CoRR, abs/2307 0.060.5,.6, 2023. doi: 10. 10.44550/ARXXI.22007..0 A00.AARXXIII.2000005555 URL https: / o//.i..org/....... 48550 http:////....05002/211.... 2307 05695.
URL
Kalajdzievski, D. A rank stabilization scaling factor for fine-tuning with lora. CoRR, abs/2312.03732, 2023, doi: 00.48550/ARXIV.2312..7777 URL https::///// //d::///doo.. org/10. 111110..855000 48050/8550///0 2312, 03732.
Lim, D., Maron, H., Law, M. T., Lorraine, J., and Lucas, J. Graph metanetworks for processing diverse neural architectures. CoRR, Abs/2312.04501. 2023. doi: 10.485 50/ARXIV.3312.05500 URL btttps://do.../ 10. 1...855//axxi..221 02.311.04500.
Khosla, A., Jayadevaprakash, N., Yao, B., and Fei-Fei, L. Novel dataset for line-grained image categorization. In First Workshop on Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and Pattern Recog- nition, Colorado Springs, CO, June 2011.
Liu, S., Wang, C., Yin, H., Molchanov, P., Wang, Y. F. Cheng, K., and Chen, M. Dora: Weight-decomposed low-rank adaptation. CoRR, abs/2402.09353,, 2024. doi: 000S550/ARXXV.24402..00
Kofinas, M., Knyazev, B., Zhang, Y., Chen, Y., Burghouts, J., Gavves, E., Snoek, C. G. M., and Zhang, D. W. Graph neural networks for learning equivariant represen- tations of neural networks. CoRR, abs/2403. 121443, 2024. doi: 011448000ARXXIV2200.1122222
URL https: /doi. org org/10. 484544855////0 03955.
Liu, X., Ji, K., Fu, Y., Du, Z., Yang, Z., and Tang, J. P-tuning v2: Prompt tuning can be comparable to fine-tuning uni- versally across scales and tasks. CoRR, abs/21 10.07602, 2021. URL / 7602.
URL https:
Kopiczko, D. J., Blankevoort, T., and Asano, Y. M. Vera: Vector-based random matrix adaptation. CoRR, abs/2310. 1454, 2023. doi: 10,48550/RXXIV2230000 454. URL https::////. http://ddiioei http:///t...og..//////oo///// 48550/arxiv .23110 11454.
Mahabadi, R. K., Henderson, J., and Ruder, S. Compacter: Efficient low-rank hypercomplex adapter layers. In Ran- zato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems 34: Annuul Conference on Neural Information Processing Systems 2021, NeurIPS 2021, De- cember 14, 2021, virtual, pp. 1022-1035, 202 URL https: http:///oceeeddeeeeeeo htw.:occeednnns..uiuisio....o 77001/haah/00011eefFFFF000000000 6ef70c0- cosstracc.html.
Lester, B., Al-Rfou, R., and Constant, N. The power of scale for parameter-efficient prompt tuning. In Moens, M., Huang, X., Specia, L., and Yih, S. W. (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November. 2021, pp. 3045-3059. Association for Computational I. Linguis- tics, 2021. doi: 10. wwwooorg/..../. 185138663//// URL https:ttp/..mmm /tt///dd..// http//di..rrgg/ .emmlppmaee i. 243.
Mahabadi, R. K., Ruder, S., Dehghani, M., and Henderson, J. Parameter-efficient multi-task fine-tuning for trans- formers via shared hypernetwrkss. In Zong, C., Xia, F., Li, W., and Navigli, R. (eds.). Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the I 11th International Joint Conference on Natural Language Processing, ACL/JJCNLP 2021, (Vol- une 1: Long Papers), Virtual Event, August 1-6, 2021, pp. 565-576. Association for Computational Linguistics, 2021b. doi: 10. 10..8866/1//2211AAL--LOOC https: //doi wttp//....13//1122/22 - -long.47 47.
Li, and Liang, P. Prefix-tuning: Optimizing continuous prompts for generation. In Zong, C., Xia, F., Li, W., and Navigli, R. (eds.), Proceedings of the 59th Anmual Meeting of the Association for Computational Linguisties and the 1 1th International Joint Conference Natural Language Processing, ACL/JCCNP 2021, (Volume 1: Long Pupers), Virtual Event, August 1-6, 2021, pp. 4582- 4597. Association for Computational Linguistics, 2021.
URL
doi: 10. 10.16665//2221AAALLLLOG.... //t/:///oorrr//../////////////// act.-onn...c1nnnnn...
URL https:
Meng, F., Wang, 7., and Zhang, M. Pissa: Principal singular values and singular vectors adaptation of large language models. CoRR, abs/2404.02944 2024. doi: 10.48550/A 029440 URL https: /tw////iii// wot./org//... 110.4 8550 2.50/arxiv..m 2404.44400 02948.
353.
Li, Y., Han, S., and Ji, S. Vb-lora: Extreme param- eter efficient fine-tuning with vector banks. CoRR, a5s/2405.15179, 2024. doi: 10.48550/ARXIV..
12

---

WeGeFT (wee-gift): Geneatt Fine-Tuning
Meng, K., Bau, D., Andonian, A., and Belinkov, Y. Locating and editing factual associations in GPT. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans,
transfer learning. In Merlo, P., Tiedemann, J., and Tsar- faty, R. (eds.), Proceedings the 16th Conference of the European Chapter of the Association for Computational Linguisties: Main Volume, EACL 2021, Online, April 19 23, 2021, pp. 487-503. Association for Computational Linguistics, 2021. doi: 10.18633/1/2222.EACLLLM 39. URL https: /oii org 101300...66611v11 #..eacl-main.
LA, USA, November 28- December 9, 2022, 2022. URL http:/ wtwppspppppseeaa//eee nips cc wwtppppppppp.... httpc//ppppp___aeeeeeeee files/pall /pap er 2022 (ww.shhh61143355da22 33bf 2er-02--bsrrttte Abssractt-oofeeeencttt Conference...eeec html.
9b0
Renduchintala, A., Konuk, T., and Kuchaiev, O. Tied-lora: Enhancing parameter efficiency of lora with weight ty- ing. In Duh, K., Gómez-Adorno, Adorno, H., and Bethard, S. (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pp. 8694-8705. Association for Computa- tional Linguistics, 2024. doi: 10. (0.116553V1/20224.. CL-LONG.481. URL https: / wwooiioogg/111../ 53/v1/2024 naacl - long......... 481.
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct clectricity? A new dataset for open book question answering. In Riloff, E., Chiang, D., Hock- enmaier, J., and Tsujüi, (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31- November 4, 2018, pp. 2381-2391. Association for Computational Linguisties, 2018. doi: 10. 18653/11/D1 1260. URL https: /tw.i//iiio/////////oo www.ooggooo..ooooooooooo 186.1166633/333 v1 d1 1260.
Mitchell, E., Lin, C., Bosselut, A., Finn, C., and Manning, C. D. Fast model editing at scale. In International Con- ference Learning Representations, 2022.
Requeima, J., Gordon, J., Bronskill, J., Nowozin, S., and Turner, R. E. Fast and Flexible Multi-Task Classifica- tion using Conditional Neural Adaptive Processes. In Advances in Neural Information Processing Systems, vol- ume 32. Curran Associates, Inc., 2019. URL https:
Navon, A., Shamsian, A., Achituve, I., Fetaya, E., Chechik, G., and Maron, H. Equivariant architectures for learning in deep weight spaces. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), International Conference 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp- 25790-25816 6. PMLR, 2023. URL proceee press / 02/02/222222/ navon23a......am.am... htm.22l....mm..........
/ /uppoceedinggg.nnuurpe....c//c/e / .htrra
Machine Learning. ICML
Rücklé, A., Geigle, G., Glockner, M., Beck, T., Pfeiffer, J., Reimers, N., and Gurevych, I. Adapterdrop: On the effi- ciency of adapters in transformers. In Moens, M., Huang, X., Specia, L., and Yih, S. W. (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, EMNLP 2021, Virtual Event Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 7930-7946. Association for Computational Linguistics, 2021. doi: (0..865//1/222.EENL--NNNM https: 1tt.oo..000....1111111112//212 1p-main. 626.
Nilsback, M. and Zisserman, A. Automated flower clas- sification over large number of classes. In Sixth In- dian Conference on Computer Vision, Graphics & Image Processing, ICVGIP 2008, Bhubaneswar: India, 16-19 December 2008, pp- 722-729. IEEE Computer Society, 2008. doi: 10. 109910I/I 2ttp://CCCII...0...4... 47. URL https: //doi. org 10 11 1wt.11119//vv11 2008 47.
URL emn
Sakaguchi, K., Bras, L., Bhagavatula, C., and Choi, Y. Winogrande: an adversarial winograd schema challenge at scale. Commun. ACM, 64(9):99-106 2021. doi: 10.1 145/3474381. ht /ttp::::////oo...//..... 1145/ 3474381.
Pfeiffer, J.. Vulic, I., Gurevych, I,, and Ruder, S. MAD-X: an adapter-based framework for multi-task cross lingull transfer. In Webber, B., Cohn, T., He, Y., and Liu, Y. (eds.), Proceedings of the 2020 Conference on Empiri- cal Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 7654-7673. Association for Computational Linguistics, 2020. doi: URL 1ttp://1...1113//111/1 emnlp-main
Sap, M., Rashkin, H., Chen, D., Bras, R. L., and Choi, Y. Socialiqa: Commonsense reasoning about social in- teractions. CoRR, abs/1904.097288, 2019. URL http:
(0.18853/V 1200:/01112000EMML-PMMMMM //doi. ort. 180.3/88633333 617.
/ //arxiv. wrg.abs/////oo........o....... 1904. ....90444000 09728.
Savadikar, C., Dai, M., and Wu, T. Learning to grow artifi- cial hippocampi in vision transformers for resilient life- long learning. arXiv preprint arXiv:2303.082511 2023.
Pfeiffer, J.. Kamath, A., Rücklé A., Cho, K., and Gurevych, Adapterfusion: Non-destruetive task composition for
13

---

WeGeFT (wee-gift); Geneatt Fine-Tuning
Shi, B., Gai, S., Darrell, T., and Wang, X. Refocusing is key to transfer learning. arXiv preprint arXiv:2305./5552, 2023.
Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998-6008, 2017. URL https: http://oooeeeeenneeeeee cttp://s..sstiirrcco...ooooooo cttpippp.ccpaaaaaa cappa Fbobo fbd053c1c
Srivastava, N., Hinton, G. E.,, Krizhevsky, A., Sutskever, I., and Salakhutdinov, Dropout: a simple way to prevent neural networks from overfitting- J. Mach. Learn. Res., 15(1):1929-1988 2014. doi: 10.555/2667743522708113 URL hwwps:tt//.t.m / //1l. acm. wwtp/oog/////oooooooo///.//.o.t/ do 10. 5555/2 627435. 2670313.,
4a845aa-Abstrace html.
Vladymyrov, M., Zhmoginov, A., and Sandler, M. Con- tinual few-shot learning using hypertransformers. arXiv 2023.
preprint arXiv:2301.00
Tolstikhin, O., Houlsby, N., Kolesnikov, A., Beyer, L.. Zhai, X., Unterthiner, T., Yung, J., Steiner, A., Keysers, D., Uszkoreit, J., Lucic, M., and Dosovitskiy, A. Mlp- mixer: An all-mlp architecture for vision. Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6- 14, 2021, virtual. pp- 24261-24272, 2021. URL https: /roceeding s. s. cttp:ppsccppppppee
Voita, E., Talbot, D., Moiseev, F., Sennrich, R., and Titov, I. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Korho- nen, A., Traum, D. R., and Märquez, L. (eds.), Pro- ceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume I: Long Papers, pp. 5797-5808. Association for Computational Linguistics, 2019. doi: 10.18653/V 1580. URL https: //doi.org/100 11t.:/.......11....
Abstract. ht
m1.
von Oswald, J., Henning, C., Grewe, B. F., and Sacramento, J. Continual learning with hypemetworks. In Interna- tional Conference on Learning Representations, 2020. URL https: pestps://operreee http///pppereeeeeeeeee net wttt:/ffrmmmiimi SJgwNerKvB.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. Llama: Open and efficient foundation lan- guage models. CoRR, abs/2302. 13971, 2023a. doi: 10.48550/ARXIV www.. 22.22.33271111 URL https: //doi. org/10. , 4 48550 / arxiv. 2302.
Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. The Caltech-UCSD Birds-200-2011 Dataset. Jul 2011.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, I.., Canton-Ferrer, C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao. Y., Martinet, X., Mihaylov. T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E., Tang, B., Taylor, R. Williams, A., Kuan, X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. Llama 2: Open foundation and fine-tuned chat models. CORR, abs/2307. abs/2300,09228,, 2023b, doi: 10.48550ARXIV.2300 288. URL https: / http//dd...o/// oro.ioorg/100. http://...448850 48550/550/// arxiv .2307.00 09288.
Wang. S., Yu, L., and Li, J. Lora-ga: Low-rank adaptation with gradient approximation. CoRR, abs/2407.00 .05000. 2024. doi: (044850/ARXI..240000000 //doi. 48550 /00555///xiii0 . .5000.
URL https:
Wightman, R. Pytorch image models. https: gtth ub ctt.coo///ai// hww.gggttaan/pptooo--mieeee--eee 2019.
Wu, Z., Arora, A., Wang, Z., Geiger, A., Jurafsky, D., Man- ning, C. D., and Potts, C. Reft: Representation finetun- ing for language models. CoRR, abs/2404.03592. 2024. doi: 10. 10..48550/RRXI...o www..XXX.I.2244005222.. URL https: // ///o/. wwo.iorrg/10.00 htt.////.....5///2////././// 2404 03592.
Xu, C., Sun, Q. Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., Lin, Q. and Jiang, D. Wizardlm: Empowering large pre-trained language models to follow complex in- structions. The Twelfih International Conference on Learning Representations, ICLR 2024, Vienna, Austria. May 1- 2024. OpenReview.net, 2024. URL https: / / net http://ffruummmcccceceeenee
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In Guyon, I., von Luxburg, U., Bengio, S., Wallach, H. M., Fergus, R., Vishwanathan, S. V. N., and Garnett, R. (eds.), Advances Neural Information
Yin, L.,, Perez-Rua, J. M., and Liang, K. J. Sylph: A hy- pernetwork framework for incremental sew- object detection. In JEEE/CVF Conference on Computer Vision
14

---

WeGeFT (wee-gift): Geneatt Fine-Tuning
and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 9025-9035. IEEE, 2022. doi: ALL109/CVVRS2268222222.000 i. o.rg//0. wtw.:00.1110///P222222
Zhao, D., Kobayashi, S., Sacramento, J., and von Oswald, J. Meta-learning via hypernetworks. In 4th Workshop on Meta-Learning at NeurlPS 2020 (MetaLearn 2020), NeurIPS, 2020.
URL httph::////// //do 2022. 00883.
Yu, L., Jiang, W., Shi, H., Yu, J., Liu, Z., Zhang, Y., Kwok, TT, Li, Z.., Weller, A., and Liu, W. Metamath: Boot- strap your own mathematical questions for large language models. In The Twelfth International Conference Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: //openreviews net http://oummmmmmo i www.rrrrmmniiNNNNhggnnNNNA
Zheng, L.,, Chiang, W., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y. Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging auddge with mt-bench and chatbot arena. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http: //papers. nips cc wtw.ppp.ccpppppppeppppppp http:/papeeeeeeeeeaea
Zaken, E. B., Goldberg, Y., and Ravfogel, S. Bitfit: Sim- ple parameter-efficient fine-tuning for transformer-based masked aanuuage-mdells In Muresan, S., Nakov, P., and Villavicencio, A. (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguis- tics ( (Volume 2: Short Papers), 2022, Dublin, Ireland, May 22-27, 2022, pp. 1-9. Association for Computational Linguistics, 2022. doi: 10. 10.11665311/20222AAL- .1. URL https: /tt////oi.oo/ org/ 1ttp://....111///// 22. acl 22lacshhorr....
1L52--Abetract-Dttteetts...o ks.html.
Zheng, T., Zhang, G., Shen, T., Liu, X., Lin, B. Y., Fu, J., Chen, W., and Yue, X. Opencodeinterpreter: Integrating code generation with execution and refinement. In Ku, L., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pp. 12834-12859. Linguistics, 2024. doi: Association for Computational CL.762. URL httpp:///ad..rrrr10..185////// 2024. f idi moddingg-acl... 762.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? In Korhonen, A., Traum, D. R., and Marquez. L. (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2. 2019, Volume 1: Long Papers, pp- 4791-4800 Association for Computational Linguistics, 2019. doi: 10. 10.186533V1//1199 1472. URL https: //doi.org//11
Zhmoginov, A., Sandler, M., and Vladymyrov, M. Hy- perTransformer: Model generation for supervised and semi-supervised few-shot learning. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Confer- ence Machine Leurning, volume 162 of Proceedings of Machine Learning Research, pp. 27075-27098, PMLR, 17-23 Jul 2022. URL https: / http:////occeeeeeeeeee/ee .m1 r r.press/ / v1 62/ wwwogghoo222aa...mmmmmmo html.
Zhai, X., Puigcerver, J., Kolesnikov, A., Ruyssen, P.,
Riquelme, C., Lucic, M., Djolonga, J., Pinto, A. S., Neumann, M., Dosovitskiy, A., Beyer, L.,, Bachem, O., Tschannen, M., Michalski, M., Bousquet, O., Gelly, S., and Houlsby, N. The visual task adaptation bench- mark. CoRR, abs/1910.00867, 2019. URL http: //arxiv. org/ aws/ 04867.
Zhou, A., Yang, K., Burns, K., Cardace, A., Jiang, Y., Sokota, S., Kolter, J. Z., and Finn, C. Permutation equivariant neural functionals. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Conference tion Processing Systems 2023, NeurlPS 2023, New Or- leans, LA, USA, December 10 - 16, 2023, 2023a. URL http: httt:::/papeers.nsss.c.//. nip...i.pppp.................... ex / 2 html.
Zhang, Q. Chen, M., Bukharin, A., He, P., Cheng, Y., Chen, W., and Zhao, T. Adaptive budget allocation for parfmeter-efffcient fine-tuning. In The Eleventh Interna- tional Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, / http://nnweieeeeee/eteetee d=1g62uWRJjiy.
Neural Informa-
2023a. URL https:://///
Zhang, R., Han, J., Zhou, A., Hu, X., Yan, S., Lu, P., Li, H., Gao, P., and Qiao, Y. alapter: Efficient fine- tuning of language models with zero-init attention. CoRR, abs/2303. 161999, 2023b. doi: 10.48550/ARXIV 2303. 16 199. URL https:/ / // //w...iii///////////../o wrg/10 41w5:10..8555//// 2.303.1699)
Zhou, A., Yang, K., Jiang, Y., Burns, K., Xu, W., Sokota, S., Kolter, J. Z., and Finn, C. Neural functional transform- ers. In Oh, A., Naumann, T., Globerson, A., Saenko, K.,
15

---

WeGeFT (wee-gift): WeiehttGenerative Fine- Tuninn
Hardt, M., and Levine, S. (eds.), Advances in Neural In- formation Processing Systems 36: Annual Conference Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023b. URL htt.tttp://papere: nips. pape /LLLe//aaaaaa//0022haaaaa1111112
ea01 2.015577ecca60556cccctttaee Abstract-tttttttooaoooooo Confere nce. html.
Zhou, A., Finn, C., and Harrison, J. Universal neu- ral functionals. CoRR, abs/2402.05232,, 2024. doi: AA8S50/ARXI..2202055222222 URL https: sttpps///ot //ooi. org/10. 48550 arXiv 2402. 05232.
16

---

WeGeFT (wee-gift); Weight-Generatiie Fine- TinenTuning
Appendix
A. Detailed analysis of Training Time
To show the advantage WeGeFT over VeRA, we conduct further experiments by setting intermediate rank VeRA to be 1024 (as used in (Kopiczko et al., 2023)). Table 10 shows that while reducing the dimension lowers the training time and memory costs, it causes severe drop in accuracy.
Table Results of fine-tuning Llama and (7B) on the Math10k benchmark. The Mem. refers to GPU memory, and Wall Time is the time required to complete epoch of training. All results obtained by using our code base for fair comparisons, except those by DiReFT and LoReFT using LlaMA are from (Wu al., 2024).
Params (%)
Mem. (GB)
Wall Time
GSMSK
MAWPS
SVAMP
AQuA
Method
LoRA 16
0.416
18.01
0.43
23.5
38.5
85.3
56.4
50.9
DoRA"
0,427
20.37
2.36
21.5
37.9
86.0
55.3
50.2
LLaMA-1 (7B)
WeGeFF-SSep.,
0.416
18.01
0.46
23.8
37.9
84.5
54.2
50.1
LoRA
0.052
17.74
0.43
23.1
34.6
83.9
54.1
48.9
DoRA
0,065
20.09
2.36
21.1
34.6
84.0
53.8
48.4
TeR25=12288
0.042
20.65
9.01
21.3
34,0
82.8
50.7
47.2
VeRA-=102 1014
0.015
1.15
23.0
30.5
48.4
WcGeFTA..
0.052
17.74
0.51
24.3
36.5
82.4
56.9
50.0
WcGeFT2...
0.016
17.71
0.36
20.7
33.0
80.8
53.5
47.0
LoRAr=16
0.416
18.01
0.43
24.5
43.4
86.1
57.2
52.8
DoRA 16
0.429
20.37
2.35
24.1
41.4
57.1
52.4
Liama 2 (7B)
WeGGeFT-Sepa..
0.416
18.01
0.46
26.1
42.4
85.9
58.6
53.3
LoRA -2
0.052
17.74
0.42
24.7
40.2
85.0
56.0
51.5
DoRA m-2
0.065
20.09
2.35
24.0
40.6
84.6
56.0
VeRA T- 12268
0.042
20.65
9.00
23.5
38.7
85.3
54.3
50.4
VeRA TM 1024
0,015
17.80
1.15
23.6
35.5
82.1
53.3
48.6
WeGeFT....
0.052
17.74
0.50
23.6
42.4
84.2
57.4
51.9
WeGeFT 0.016
17.71
0.39
38.0
51.1
B. Performance of WeGeFT with mixed precision float16
We conduct additional experiments with LLaMA-1 (7B) using mixed-precising float16 instead of mixed-preeiisin bfloat The table below shows that WeGeFT and LoRA experience similar relative performance drop with float16 while maintaining comparable memory and wall-time, consistent with float16's known limitations compared bfloat16. The performance drop due floatl6 can be offset by inereasing the number of trainable parameters in WeGeFT. As shown in the table, WeGeFT with a rank of 128 outperforms LoRA even with float1 6, while using four times fewer parameters. These results further confirm WeGeFT's compatibility with any device that supports LoRA.
Table 11: Results of fine-tuning Llama (7B) the Math 10k benchmark with pretrained weights and activations converted to 16 and b precisions.
Params (%)
Mem. (GB)
Wall Time
GSM8K
MAWPS
SVAMP
Avg. Acc.
AQuA
Method
LoRA 16
0.416
18.01
0.43
23.5
38.5
85.3
56.4
50,9
bfloat16
WeGeFTu=61
0.052
17.74
0.51
24.3
36.5
82.4
56.9
50.0
di..
LoRA--Ih
0.416
18.07
0.42
21.8
37.9
84.7
57.1
50.4
float16
W6
0.052
17.75
0.43
22.7
36.0
83.5
54.9
49.3
di.,
WeGeFFT din
0.104
17.80
0.43
22.7
38.5
84.6
56.1
50.5
17

---

WeGeFT (wee-gift); Weight-Generative Fine- Tine-Tuning
C. Analysis of WeGeFT vs. LoRA
To rigorously distinguish WeGeFT from LoRA, we perform an analytical gradient comparison in the context fine-tuning Transformer-basee architectures.
C.I. Gradient Analysis
Consider a simplified Transformer layer with pretrained weights W for layer L. LoRA fine-tunes by introducing additive low-rank residuals:
WLRA = W + B'A'
where B¹ and A are learnable low-rank matrices. In contrast, consider Eqn. 3, WeGeFT fine-tunes through multiplicative residual explicitly dependent pretrained weights:
Wi WeGerT x
(10)
with shared low-rank matrices & and vh.
Let L denote scalar loss function (e.g., cross-entropy). For LoRA, the gradient computations with respect to the matrices
A' and B¹ are:
Oc 0A = Tx"
Oc OB = (A))
(11)
where X" (.2) x aggregates local gradient information.
For WeGeFT, gradients for 6 and d include information aggregated across layers due parameter sharing:
Oc TJ "(w')x"
Oc
Σ (W"))x" x
(12)
0Φ
Here, WcGeFT gradients inherently integrate knowledge from pretrained weights across multiple layers, encapsulating broader contextual and structural dependencies than LoRA.
C.2. Implications for Optimization Dynamics
Layer-wise vs. Global Updates: LoRA updates parameters in isolation per layer, restricting interaction. In contrast, WeGeFT updates global, considering inter-layer correlations and leading more cohesive and stable optimization trajectories.
Pretrained Knowledge Utilization: By explicitly multiplying residuals with pretrained weights, WeGeFT exploits existing model structure, preserving crucial pretrained information, potentially yielding superior convergence and generalization.
Expressivity and Efficiency Trade-off: WeGeFT maintains expressivity through multiplicative updates despite substantial parameter sharing, balancing parameter efficiency without compromising learning capacity, unlike traditional additive methods such as LoRA.
C.3. Summary of Advantages
Improved Parameter Efficiency: Explicitly leverages pretrained weights to achieve stronger fine-tumin fine-tuning results with fewer Icarnable parameters.
Optimized Gradient Flow: Gradients leverage global information, enabling coordinated fine-tuning across layers.
This analysis underpins the empirical advantages of WeGeFT observed in extensive experimentation, highlighting funda- mental theoretical distinctions from LoRA.
18

---

WeGeFT (wee-gift): Weight-Generative Fine-Tume Tine-Tuning
D. Implementation Details and Hyperparameter Tuning
In practice, we use scaling factor of _ for residuals as done in LoRA (Hu et al., 2022). We also use dropout (Srivastava al., 2014) on the pretrained weights before transforming using WeGeFT parameters means of regularization: = Dropout(W")) e v
(13)
We omit this in the main section for ease of notation and simplicity, as it does not affect the analysis. In experiments, we initialize t to all zeros and & to Kaiming Uniform initialization (He et al., 2015).
D.I. Computing Resources and Code
All our experiments are run on single Nvidia A100 GPU. Our code is provided in the supplementary materials.
D.2. Arithmetic Reasoning
On the Math10k, we follow (Wu et al., 2024), and tune the hyperparameters by fine-tuning the LLaMA-1 (7B) model on the GSM8K dataset (Cobbe et al., 2021) using a separate validation set constructed from the training set, and use the same hyperparamters for Llama-2 (7B). Table 12 shows the hyperparameters used in our experiments. We perform hyperparameter search using the seed 42, and report the final results by averaging over three runs with seeds 42, 43, and 44. We use greedy decoding scheme during inference, with a maximum new token number of 512. For experiments on fine-tuning Llama 2 (7B) MetaMathQA and evaluating on GSM8k, use the hyperparameter's from (Wang et al., 2024), and tune the learning rate on validation split from Meta-MathQA. We report average scores across 3 runs with seeds 42, 43, 44.
Table 12: Hyperparameters used for the MathlOk experiments. We greedy sampling following (Wu al., 2024)
Hyperparameter
Value
Max Sequence Length
512
Optimizer
AdamW
Weight Decay
0.0
LR Scheduler
Linear
Batch Size
16
Epochs
3
Learning Rate
4 x 10-4
Rank
64
WeGeFT din
Scaling Factor
128
Warmup Ratio
0.1
Dropout
0.1
Fine-Tuned Layers
Query, Key, Value, Up Projection, Down Projection
Learning Rate
7 x 10-4
Rank
64
WeGeFT dost
Scaling Factor
64
Warmup Ratio
0.06
Tune-Tuned Layers
Out Projection, Down Projection
D.3. Commonsense Reasoning
We tune the hyperparameters for commonsense reasoning by fine-tuning the LLaMA-I model the BoolQ dataset (Clark et al., 2019) using separate validation set constructed from the training set. Table 14 shows the hyperparameters used in our experiments. We search for the hyperparameters using LLaMa-1 (7B) and use the same hyperparameters for LLaMA-I (13B), Llama (7B) and Llama 3 (8B) models. We perform hyperparameter search using the seed 42, and report the final results by averaging over three runs with seeds 42, 43, and 44. We use greedy decoding scheme during inference, with a maximum new token number of 32.
19

---

WeGeFT (wee-gift): Weight-Generative Fine-Tuning
Table 13: Hyperparameters used for fine-tuning on MetaMathQA and evaluating on GSM8K.
Hyperparameter
Value
Max Sequence Length
1024
Optimizer
AdamW
Weight Decay
0.0
LR Scheduler
Cosine
Batch Size
Epochs
1
Learning Rate
5 10
Rank
64
WeGeFT
Scaling Factor
128
Warmup Ratio
0.03
Dropout
0.0
Fine-Tuned Layers
All linear layers (excluding vocabulary projection and head)
Generation: Temperature
0.8
Generation: top-p
0.95
Table 14: Hyperparameters used for the commonsense reasoning experiments. We use greedy sampling following (Wu et al.,
2024)
Hyperparameter
Value
Max Sequence Length
512
Optimizer
AdamW
Weight Decay
0.0
IRR Scheduler
Linear
Batch Size
16
Epochs
3
Learning Rate
9 10-5
Rank
WeGeFT
Scaling Factor
128
Warmup Ratio
0.1
Fine-Tunce Layers
Query, Key, Value, Up Projection, Down Projection
Learning Rate
6
10-4
Rank
WeGeFT (Output)
Scaling Factor
64
Warmup Ratio
0.06
Dropout
0.0
Fine-Tuned Layers
Out Projection, Down Projection
D.4. Instruction Following
For fine-tuning Llama 2 (7B) on WizardLM and evaluating on MT-Bench, we use the hyperparameters from (Wang et al., 2024) and use the same learning rate as MetaMathQA experiments. We report average scores across runs with seeds 42, 43, 44.
D.5. Code Generation
For fine-tuning Llama 2 (7B) on Code-Feedbacc dataset (Zheng et al., 2024) and evaluating on HumanEval, we use the hyperparameters from (Wang et al., 2024) and tune the learning rate on a separate validation split from Code-Feedback. We report average scores across 3 runs with seeds 42, 43, 44.
D.6. FGVC Experiments
For all the experiments, we use ViT-B/16 model (Dosovitskiy et al., 2021), which contains 12 transformer blocks, each Self-Attention (MHSA) blocks, and dimension of 768. We use checkpoints from the
with 12 heads in the
20

---

WeGeFT (wee-gift): Weight-Generative Fine-Tuning
Table Hyperparameters used for fine-tuning on Code-Feedback and evaluating on JamanEEall
Hyperparameter
Value
Max Sequence Length
1024
Optimizer
AdamW
Weight Decay
0.0
LR Scheduler
Cosine
Batch Size
Epochs
1
Learning Rate
4 10
Rank
64
WeGeFT
Scaling Factor
128
Warmup Ratio
0.03
Dropout
0.0
Fine-Tuned Layers
All linear layers (excluding vocabulary projection and head)
Generation: Temperature
0.8
Generation: top-p
0.95
Table 16: Hyperparameters used for fine-tuning on WizardLM and evaluating on MT- MT-Bench.
Hyperparameter
Value
Max Sequence Length
1024
Optimizer
Adam W
Weight Decay
0.0
LR Scheduler
Cosine
Batch Size
Epochs
Learning Rate
5 x 10
Rank
64
Scaling Factor
128
Warmup Ratio
0.03
Dropout
0.0
Fine-Tuned Layers
All linear layers (excluding vocabulary projection and head)
Generation: Temperature
0.8
Generation: top-p
0.95
model pretrained on the ImageNet21k (Deng et al., 2009) under the supervised training protocol provided by the immm package. For both VTAB and FGVC experiments, we use a hyperparameter search using the validation sets and use the vraldiinttalliattion data during the final run and report the results on the in our experiments in provided in Table 17. We use the same train, validation and test splits as (Shi et al., 2023), except for Stanford Cars dataset (Gebru et al., 2017). Due to the unavailability of the dataset from the original source, and the difference in the format of the data provided by the updated source, we create our own training and validation split (with the same number of images as (Shi et al., 2023)) and use the official testing split. We initialize á with zeros and if with Kaiming uniform initialization.
The hyperparameter search space used
21

---

WeGeFT (wee-gift): Weieht-Generative Fine-Tuning
Table 17: Hyperparameter search space used for FGVC experiments. During the search, we use 25 epochs due to computational constraints, and use 100 epochs in the final run with the selected hyperparameters
Hyperparameter
Values
Learning Rate
1e 3 1.5e 33 2.55 .e 53,3, 1e-2
BitFit
Weight Decay
0.0
Learning Rate Weight Decay
le 1.5e 3, 2e 2e,3, 2.5c
3, 5e 1
VPT
0.0
Num. Prompts
5
Learning Rate Weight Decay
le 3 1.5e 3
2e 3
2.5e
3 5e 2
LoRA
0.01, 0.001, 0.0001, 0.0
Rank r
8
Learning Rate Weight Decay
1e 2.5e
5e 1e 2.5e
3, 3
WeGeFT
0.01, 0.001, 0.0001, 0.0
Rank r
16
Optimizer
A Cosine
LR Scheduler
Warmup Epochs
Epochs
100
Batch Size
32
E. Visual Inspection of Our Two-Linear-Layer Parameterized WeGeFT
22

---

WeGeFT (wee-gift): Weight-Generative Fine-Tuning Tuning
Layer
Layer
Layer
Layer Cluster
Layer Cluster
Layer
Cluster
Cluster
Layer Cluster
Layer Cluster
Layer 10
Layer Cluster
Layer Cluster
Layer
NABirds
Cluster
VGG-Flowers
Cluster
Layer Cluster
Layer Cluster 16
Layer Cluster 2
Layer Cluster
Layer Cluster 15
Layer Cluster
Stanford Dogs
Layer Cluster
Layer Cluster
Layer Cluster
Stanford Cars
Layer Cluster
Layer Cluster
Layer 10 Cluster
Figure 5: More examples of the visual interpretability of our two-linear-layer parameterized WeGeFT tested on the FGVC benchmark. We show examples of head, wings and legs of birds in the top-left, examples of flower petals in the top-right, examples of head, ears and legs of dogs in the bottom-left and examples of tires, windshield and bumper of cars in the bottom-right.
23


Reviews:
[{"review_id": "DruKIM0nln", "reviewer": "Reviewer_UHSy", "summary": "This paper introduces Weight-Aware Fine-Tuning (WAFT), a novel parameter-efficient fine-tuning method for large pre-trained Transformer models. WAFT proposes to generate fine-tuning weights directly from the pre-trained weights using a low-rank formulation with shared linear layers across multiple transformer layers. By sharing parameters across layers, the authors claim that WAFT achieves multi-faceted efficiency in parameters, representations, compute, and memory, while maintaining or exceeding the performance of LoRA and its variants.\nExperimental results are presented for various NLP tasks (commonsense reasoning, arithmetic reasoning, instruction following, code generation) as well as visual recognition to demonstrate the effectiveness of WAFT.\n\n## update after rebuttal\n\nI thank the authors for their rebuttal and willingness to improve clarity on the weight-aware term. I will maintain my score.\n\nMy 2) concern is indeed about having a $\\mathbb{W}^l$ in the weight update that is not the pretrained weight which I suppose bears similarities to VeRA as suggested. There is not clear reason why this approach would not converge since VeRA does. \n\nThis study would help enhance the insights of WAFT to understand whether  the algorithm actually uses knowledge from the pretrained weight or if it simply uses properties of the learned matrix that arise during the pre-training stage (independence of features for example).\n\nI suggest the authors include such study in the paper (or supplementary) if they obtain interesting results for the camera ready.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer UHSy,\n\nThank you for your feedback and efforts in reviewing our submission. We address your concerns (C1-C5) as follows, which will be carefully incorporated in the revision.\n\n> **C1: Weight-awareness in LoRA and WAFT**\n\nIn general, any PEFT methods have to be (pretrained) weight-aware to be effective for downstream tasks. However, prior art like LoRA has `implicit` weight-awareness between the fine-tuning residual ($\\Delta W^l = B^l\\cdot A^l$) and the pretrained weights ($W^l$) via gradient backpropagation in learning (see Eqn.10 for a toy example in Appendix A). Our proposed WAFT harnesses `explicit` weight-awareness (in the forward computation) via learning $\\Delta W^l = W^l\\cdot \\phi \\cdot \\psi$. In the abstract (lines 025-027), we define **WAFT, a novel approach that learns to generate fine-tuning weights directly from the pretrained wegiths**. (the generic formulation is in Sec.3.1). To address your concerns, we propose to use **generative weight-aware fine-tuning (GenWAFT)** to better highlight the characteristics of our proposed method in revision. The comprehensive experiments have verified its effectiveness.\n\n> **C2: Training time for VeRA**\n\nVeRA's long training time results from the large intermediate dimension used in our experiments. Since VeRA relies on fixed random matrices, a large dimension is necessary for reasonable accuracy. As shown in the table below, **reducing the dimension to 1024 (as in the original paper) significantly degrades performance**, despite lowering training time and parameter count. In contrast, our proposed WAFT maintains parameter and training efficiency while achieving comparable or superior performance to state-of-the-art methods.\n\nMethod|Params (%)|Mem. (GB)|Wall Time|AQuA|GSM8k|MAWPS|SVAMP|Avg\n-|-|-|-|-|-|-|-|-\n-|-|-|-|LLaMA 1|-|-|-|-\nVeRA $^{r=12288}$ |0.042|20.65|9.01|21.3|34.0|82.8|50.7|47.2\nVeRA $^{r=1024}$ |0.015|17.80|1.15|23.0|30.5|79.1|48.4|45.2\n-|-|-|-|Llama 2|-|-|-|-\nVeRA $^{r=12288}$ |0.042|20.65|9.00|23.5|38.7|85.3|54.3|50.4\nVeRA $^{r=1024}$ |0.015|17.80|1.15|23.6|35.5|82.1|53.3|48.6\n\n> **C3: Effect of weight-aware finetuning in cases of weak zero-shot performance of models**\n\nThis is certainly a valid concern, especially when evaluating methods that reduce parameter count. Our experiments on the Math10k benchmark with LLaMA-1 (7B) address this issue (Table 2). Given LLaMA-1 7B’s low zero-shot accuracy of 11.0 on GSM8k [1], it can be considered weak for arithmetic reasoning which would require significant adaptation to perform well on Math10k. The results show:\n\n- Simply reducing parameters by lowering the rank in LoRA and DoRA leads to a significant performance drop. LoRA's accuracy with rank=16 is 50.9 vs. 48.9 with rank=2. With an equivalent parameter count (~0.05%), WAFT outperforms all baselines (LoRA, DoRA, VeRA, and VB-LoRA), and achieves accuracy close to LoRA $^{r=16}$. This demonstrates that generating the fine-tuning residuals from the pretrained weights improves adaptation with minimal parameters, whereas LoRA $^{r=2}$ struggles to achieve good performance.\n\nWe have also included further experiments with the VTAB benchmark as suggested. We follow the same settings as the FGVC experiments (Section 4.5). The table below shows that WAFT can perform on par or better than baseline methods even on Specialized and Structured tasks on the VTAB benchmark.\n\nMethod|Params (M)|Natural|Specialized|Structured|Avg\n-|-|-|-|-|-\nVPT|0.046|81.0|85.7|58.9|72.7\nBitFit|0.083|81.8|85.2|57.8|72.4 \nLoRA |0.147|**82.0**|85.9|61.0|74.0\nWAFT|0.025 |**82.0**|**86.3**|**61.1**|**74.1**\n\n\n> **C4: More exploration could be done on the relevance of using the pre-trained weight for the matrix multiplication, for example is a properly scaled random matrix performing close to the pretrained weights?**\n\nWe are not certain that we fully understand your concern. Were you suggesting to test a variant like this: converting WAFT update from $\\Delta W^l= W^l\\cdot \\phi\\cdot \\psi$ to $\\Delta W^l= \\mathbb{W}^l\\cdot \\phi\\cdot \\psi$ in learning fine-tuning residual weights, where $\\mathbb{W}^l$ is ``a properly scaled random matrix\"?  It seems very unlikely that a random matrix can do better than the pretrained weights. With the random matrix, the overall fine-tuning residual weights $\\Delta W^l$ would behave similar in spirit to VeRA. Could you kindly elaborate your concern?\n\n> **C5: Ablations on using other types of matrices for the multiplicative updates**\n\nThe purpose of our simple formulation is that it enables memory and computationally efficient updates, since the low-rank structure and linear updates result in smaller matrix multiplications. Our ablations studies (Table 7) show that more complex updates are not necessary. Hence, we leave the further exploration to future work, and we would welcome any suggestions from the reviewers and the broader community.\n\n---\n[1] Touvron et al., \"LLaMA: Open and Efficient Foundation Language Models\", ArXiv 2023", "claims_and_evidence": "This paper proposes to modify LoRA's weight additive paradigm to a matrix-multiplication one. This is a valid idea to pursue although I do not feel that the weight-aware narrative pushed by the authors is very convincing.\n\nI have two main concerns about the author's weight-aware claims:\n\n(1) LoRA is weight-aware to an extent as the update is formulated as (W + BA) so the BA updates takes W in consideration when learning a solution. This is a bit hand-wavy from my part. Maybe the weight-aware term is not characteristic enough in this scenario which weakens the narative pushed by the paper to justify the multiplicative update.\n\n(2) It is not always clear that the pre-trained weight are a good starting basis for finetuning. What about tasks where the zero-shot model is highly un-adapted ? In this case, starting without the pre-trained weights constraints may be an advantage. The cars dataset in Table 6 could be an example or any of the VTAB1k [1] specialized or structured subsets.\n\nMore exploration could be done on the relevance of using the pre-trained weight for the matrix multiplication, for example is a properly scaled random matrix performing close to the pretrained weights ?\n\n[1] Zhai, Xiaohua, et al. \"A large-scale study of representation learning with the visual task adaptation benchmark.\" ICLR 2020", "methods_and_evaluation_criteria": "Good range of test datasets which adequately justify the performance of the proposed algorithm", "theoretical_claims": "Did not check the correctness of proofs", "experimental_designs_or_analyses": "Good design but more ablation study should have been conducted on using the pre-trained weight matrix for the multiplicative update.", "relation_to_broader_scientific_literature": "PEFT is a relevant field, it is good to see contributions that veer away from LoRA's formulation.", "essential_references_not_discussed": "SOTA is well represented", "other_strengths_and_weaknesses": "Discussed above", "other_comments_or_suggestions": "In general the idea of multiplicative update is valid and appear to be effective experimentally. \n\nThe current justification as to why this is a good idea theoretically is not good enough as I do not find the weight-aware narrative compelling and ablations are missing on using other types of matrices for the multiplicative updates.", "questions_for_authors": "I was surprised of the 9 hours per epoch of VeRA line 307 which is almost 20 times longer than LoRA. Is this because of the very large intermediate dimension of 12k ? What about using a representation of 1024 as suggested in the original VeRA paper ?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dTk8cqWZBL", "reviewer": "Reviewer_NsTU", "summary": "1. Fine - tuning large pretrained Transformer models has two main focuses: parameter - efficient and representation - efficient fine - tuning. LoRA is a pioneering method, but its variants often sacrifice compute and memory efficiency or performance. ReFT is another approach, yet it has performance limitations.\n\n2. The paper proposes Weight - Aware Fine - Tuning (WAFT), which generates fine - tuning weights from pretrained weights. It uses a simple low - rank formulation with two shared linear layers, aiming for multi - faceted efficiency in parameters, representations, compute, and memory.\n\n3. WAFT is related to hypernetworks and neural functionals. It innovatively tunes models using their own weights. Its contributions include a novel framework, multi - faceted efficiency, unifying parameter and representation efficiency, and strong and scalable performance.\nExperiments on arithmetic reasoning, commonsense reasoning, instruction following, code generation, and visual recognition show that WAFT outperforms many baseline methods. It can achieve better performance with fewer parameters while maintaining LoRA's compute and memory efficiency.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer NsTU,\n\nThank you for your feedback and efforts in reviewing our submission. We address your concerns as follows, which will be carefully incorporated in the revision.\n\n> **Performance in constrained environments**\n\nWe use an A100 GPU to accelerate experiments, but the implementation is not tied to any specific GPU architecture. WAFT’s memory and wall-time requirements are comparable to or lower than LoRA, while VeRA and DoRA significantly increase both. Table 2 in the manuscript provides a detailed analysis, confirming WAFT’s suitability for any device that supports LoRA.\n\nFor further verification, we conduct additional experiments with LLaMA-1 (7B) using mixed-precision float16 instead of mixed-precision bfloat16. The table below shows that WAFT and LoRA experience a similar relative performance drop with float16 while maintaining comparable memory and wall-time, consistent with float16’s known limitations as compared to bfloat16.\n\nThe performance drop due to float16 can be offset by increasing the number of trainable parameters in WAFT. As shown in the table, WAFT with a rank of 128 outperforms LoRA even with float16, while using four times fewer parameters. **These results further confirm WAFT’s compatibility with any device that supports LoRA**.\n\n\nMethod|Params (%)|Mem. (GB)|Wall Time|AQuA|GSM8k|MAWPS|SVAMP|Avg. Acc.\n-|-|-|-|-|-|-|-|-\nLoRA $^{r=16}$ (bfloat16)|0.416|18.01|0.43|23.5|38.5|85.3|56.4|50.9\nWAFT $^{r=64}$ (bfloat16)|0.052|17.74|0.51|24.3|36.5|82.4|56.9|50.0\nLoRA $^{r=16}$ (float16)|0.416|18.07|0.42|21.8|37.9|84.7|57.1|50.4\nWAFT $^{r=64}$ (float16)|0.052|17.75|0.43|22.7|36.0|83.5|54.9|49.3\nWAFT $^{r=128}$ (float16)|0.104|17.80|0.43|22.7|38.5|84.6|56.1|50.5\n\n> **Generalization to other architectures**\n\nWe recognize the importance of exploring generalization. However, Transformers have proven to be highly scalable and flexible, and have emerged as the standard architecture due to their scalability and adaptability. Consequently, both our study and most prior PEFT research focus on Transformers. We believe extending WAFT to other architectures is a valuable future direction, and its strong performance in the context of prior research on fine-tuning Transformer models makes it a meaningful contribution.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes. Compete and correct proofs.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "1. The paper's WAFT approach builds on the prior work of LoRA and its variants. While LoRA achieved efficiency in parameters, compute, and memory, its successors often traded off these aspects. WAFT addresses this by directly generating fine - tuning weights from pretrained weights, offering multi - faceted efficiency and better performance.\n\n2. ReFT, another related method, focused on lightweight representation - editing modules. However, it had performance issues. WAFT unifies parameter - efficient and representation - efficient fine - tuning, improving upon ReFT by leveraging pretrained weight knowledge and achieving on - par or better performance.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Novelty: \n\n1. WAFT innovatively generates fine - tuning weights from pretrained weights. This is a new approach in fine - tuning Transformer models. It sets itself apart from LoRA and its variants by using weight - aware parameterization, which enables more effective utilization of pretrained knowledge. It unifies parameter - efficient and representation - efficient fine - tuning. By combining these two aspects, WAFT offers a more comprehensive solution. This unified approach simplifies the fine - tuning process and can potentially be applied to a wider range of tasks.\nThe design of WAFT, with two shared linear layers, is simple yet effective. It reduces the number of learnable parameters while maintaining or improving performance. This simplicity also contributes to its scalability and ease of implementation.\n\n2. Experiments\n\nThe paper conducts extensive experiments across multiple tasks. It tests WAFT on arithmetic reasoning, commonsense reasoning, instruction following, code generation, and visual recognition. This wide range of tasks validates the method's generality and effectiveness in different scenarios. The experiments include comparisons with various baseline methods. By comparing WAFT with LoRA, DoRA, VeRA, and ReFT, among others, the paper clearly demonstrates WAFT's advantages. The results show that WAFT can achieve better performance with fewer parameters and maintain compute and memory efficiency.\nAblation studies are carried out to verify the effectiveness of WAFT's components. These studies test different parameterization schemes and alternative formulations. They help to understand the impact of each part of WAFT, providing insights into its design and performance.\n\n3. Reference Integrity\n\nThe paper comprehensively reviews the related literature. It covers parameter - efficient fine - tuning methods, hypernetworks, and neural functionals. This shows a deep understanding of the field and places WAFT in the context of existing research.\nAll the references are properly cited. The authors use a wide range of sources, from conference papers to arXiv preprints. This gives credit to the original work and allows readers to further explore related research.\nThe reference list is up - to - date. It includes recent papers published in 2024, ensuring that the research builds on the latest developments in the field. This indicates the paper's relevance and timeliness.\n\nWeakness\n\n1. Limited Generalization Exploration\nAlthough WAFT shows good performance on the tested tasks, its generalization to other architectures and tasks is not fully explored. The paper mainly focuses on Transformer - based models. It's unclear how well WAFT would work in models with different structures or in emerging applications.\nThere is a lack of analysis on the long - term stability of WAFT. As the field of deep learning evolves, new data and tasks may emerge. It's not clear if WAFT can maintain its performance and efficiency over time without further adjustments.\nWAFT's performance in resource - constrained environments is not well - studied. The experiments are mostly conducted on a single Nvidia A100 GPU. It's unknown how the method would perform in devices with limited computational power and memory, which are common in real - world applications.", "other_comments_or_suggestions": "N.A.", "questions_for_authors": "N.A.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WZyscaRqVp", "reviewer": "Reviewer_M1uG", "summary": "The authors:\n1. Identify current limitation of existing PEFT methods -- the replicated structures compromise time and memory efficiency. \n2. Propose a novel approach that shares PEFT paratemers across layers. \n3. Evaluate the proposed method on different testbed, including image classification, reasoning and generation tasks. \n4. Perform ablation study on the effect of different latent operators g, ultimately finding that the identity operator is sufficient.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer M1uG,\n\nThank you for your feedback and efforts in reviewing our submission. We address your concerns as follow, which will be carefully incorporated in the revision.\n\n> **Comparison with FacT**\n\nWe acknowledge that Factor Tuning (FacT) shares a similar parameter-sharing concept with WAFT. However, unlike WAFT, FacT shares parameters across both layers and modules (Self-Attention and MLP blocks). While this reduces trainable parameters, it has a drawback: Transformer modules serve distinct functions - attention layers largely handle syntactical and in-context learning abilities [1, 2], while MLP layers encode factual knowledge [3]. Sharing parameters across these modules may therefore be suboptimal.\n\nFor a comprehensive evaluation, we compare FacT on the VTAB and Math10k benchmarks. We use the open-source implementation from the original paper for VTAB, employing the same model as in our FGVC experiments. For Math10k, we adapt FacT to the HuggingFace PEFT package and evaluate it on LLaMA-1 and LLaMA-2 (7B). The table below shows that WAFT outperforms both FacT variants on VTAB while using fewer parameters.\n\nMethod|Params (M)|Natural|Specialized|Structured|Avg\n-|-|-|-|-|-\nVPT|0.046|81.0|85.7|58.9|72.7\nBitFit|0.083|81.8|85.2|57.8|72.4\nLoRA|0.147|**82.0**|85.9|61.0|74.0\nFacT-TT|0.040|79.8|86.0|58.0|71.9\nFacT-TK|0.069|80.0|**86.8**|60.9|73.4\nWAFT |0.025|**82.0**|86.3|**61.1**|**74.1**\n\nWAFT outperforms FacT on the Math10k benchmark, with a larger performance gap on LLaMA-1 than Llama 2. Since LLaMA-1 is weaker and requires more adaptation for arithmetic reasoning, this highlights WAFT’s ability to adapt both weak and strong models effectively, whereas FacT struggles with weaker models.\n\nMethod|Params (%)|Mem. (GB)|Wall Time|AQuA|GSM8k|MAWPS|SVAMP|Avg\n-|-|-|-|-|-|-|-|-\n-|-|-|-|LLaMA 1|-|-|-|-\nFacT (TT) |0.051|17.74|0.52|21.5|30.7|80.3|50.3|45.7\nFacT (TK) |0.062|17.75|0.59|21.3|34.8|82.2|51.9|47.5\nWAFT|0.052|17.74|0.51|**24.3**|**36.5**|**82.4**|**56.9**|**50.0**\n-|-|-|-|Llama 2|-|-|-|-\nFacT (TT) |0.051|17.74|0.52|**24.9**|38.3|81.9|56.2|50.3\nFacT (TK) |0.062|17.75|0.59|24.5|41.0|**85.7**|54.4|51.4\nWAFT|0.052|17.74|0.50|23.6|**42.4**|84.2|**57.4**|**51.9**\n\n> **Effectiveness of the identity transformation**\n\nWhile we do not have a theoretical understanding yet, we hypothesize that the superior performance of the identity operation over more complex and non-linear operations is because of difficulty in optimization.\n\n---\n\n**References**\n\n[1] Alberto Bietti, Vivien Cabannes, Diane Bouchacourt, Hervé Jégou, Léon Bottou: Birth of a Transformer: A Memory Viewpoint. NeurIPS 2023\n\n[2] Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, Ivan Titov: Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned\n\n[3] Kevin Meng, David Bau, Alex Andonian, Yonatan Belinkov: Locating and Editing Factual Associations in GPT. NeurIPS 2022", "claims_and_evidence": "All of the authors’ claims are generally convincing.", "methods_and_evaluation_criteria": "The evaluation benchmark is comprehensive. Providing convincing evidence for the performance of proposed method.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experiments cover a wide range of tasks for foundation models, offering a comprehensive and sufficiently thorough evaluation.", "relation_to_broader_scientific_literature": "The idea of describing a shared weights between PEFT layers is enlighting. I believe this methodology offers valuable insights for future PEFT research, specifically, enlights people on design of more effective and efficient architectures.", "essential_references_not_discussed": "See section \"Questions For Authors\".", "other_strengths_and_weaknesses": "Strenths: \n\nA nice work overall. \n1. The paper is well-written and well-organized. \n2. The motivation is clear and proposed method is effective, easy to understand. \n3. Experiment design are comprehensive and convincing. \n\nWeaknesses:\n\nSee section \"Questions For Authors\".", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. The concept of sharing information across PEFT layers appears very similar to approach proposed in [1]. I'm kind of superised the authors cite that work without comparison with them. I would be very interested in seeing a detailed comparison between these two mehods, particularly, how to explain if any difference in performance. \n2. I'm interested in the performance of identify transformation conducted in section 5.1. Seems the authors present them with explanation. Do the authors have any hint on its superior performance versus other operators?\n\nI don't mind raising my score if my questions (especially 1) are well-addressed. \n\n[1] Jie, S. and Deng, Z. Fact: Factor-tuning for lightweight adaptation on vision transformer. In Williams, B., Chen, Y., and Neville, J.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Jxo4ZAN5Te", "reviewer": "Reviewer_Zn72", "summary": "The paper proposes Weight-Aware Fine-Tuning (WAFT) to generate fine-tuning weights directly from the pretrained weights.\nResults show promising results on three datasets based on the LLaMA series of models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer Zn72,\n\nThank you for your feedback and efforts in reviewing our submission. We address your concerns as follow, which will be carefully incorporated in the revision.\n\n> **Experimental designs, especially for using what foundation models and on what datasets, are not logically clear**\n\nWe follow prior works [1, 2, 3] in selecting foundation models and datasets, all of which are standard in the PEFT literature. We appreciate any suggestions on how to further clarify our choices.\n\n> **The claimed scalable performance is not validated by experiments**\n\nWe have evaluated WAFT at scales of 7–8B parameters on large fine-tuning datasets for arithmetic reasoning, code completion, commonsense reasoning, and instruction following, all of which are tasks of practical interest. Based on these results, we believe our experiments demonstrate scalability. However, we welcome further insights on how to strengthen this validation.\n\n**References**\n\n[1] Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, Roy Ka-Wei Lee: LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models. EMNLP 2023: 5254-5276\n\n[2] Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D. Manning, Christopher Potts: ReFT: Representation Finetuning for Language Models. NeurIPS 2024\n\n[3] Shaowen Wang, Linxi Yu, Jian Li: LoRA-GA: Low-Rank Adaptation with Gradient Approximation. NeurIPS 2024", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "Yes.\nTable 1 shows the results of fine-tuning Llama 2 (7B) on the MetaMathQA dataset and evaluating it on the GSM8k test set, while Table 2 presents the results of fine-tuning Llama 1 and 2 (7B) on the Math10k benchmark.\nIn addition, Table 3 gives the results of fine-tuning LLaMA-1 7B, Llama 2 7B and Llama 3 8B on Commonsense170k.\nThe rationality of foundation model selection and benchmarking method is not clear.", "relation_to_broader_scientific_literature": "PEFT methods are meaningful for various applications based on foundation models.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n1. Weight-Aware Fine-Tuning Framework is novel.\n2. Experimental results are presented clearly.\n\nWeaknesses:\n1. Experimental designs, especially for using what foundation models and on what datasets, are not logically clear.\n2. The claimed scalable performance is not validated by experiments.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper is focused on fine-tuning, seeking to improve on LoRA-style parameter efficient techniques. The authors introduce a simple multiplicative low-rank adapter (as opposed to the additive one in standard LoRA); the idea is related to other techniques sometimes used in representation editing. 

The authors’ approach is very simple and works well in a range of settings. Reviewers were largely enthusiastic, with questions on how to extend the applicability of the work and on complexity. The authors cleared up all concerns. Solid paper overall.
Decision: Accept (poster)