Transformer-Squared: Self-adaptive LLMs
Authors: Qi Sun
Edoardo Cetin
Yujin Tang
Venue: ICLR 2025 Poster (2025)
Keywords: Transformers
Self-Adaptation
Fine-tuning
Dynamic System
LLMs
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose Transformer-Squared, a new framework that dynamically adapts the weights of LLMs based on test time conditions

Abstract: Self-adaptive large language models (LLMs) aim to solve the challenges posed by traditional fine-tuning methods, which are often computationally intensive and static in their ability to handle diverse tasks. We introduce Transformer-Squared, a novel self-adaptation framework that adapts LLMs for unseen tasks in real-time by selectively adjusting only the singular components of their weight matrices. During inference, Transformer-Squared employs a two-pass mechanism: first, a dispatch system identifies the task properties, and then task-specific 'expert' vectors, trained using reinforcement learning, are dynamically mixed to obtain targeted behavior for the incoming prompt. Our method consistently outperforms ubiquitous approaches such as LoRA, with fewer parameters and greater efficiency. Furthermore, Transformer-Squared demonstrates versatility across different LLM architectures and modalities, including vision-language tasks. Transformer-Squared represents a significant leap forward, offering a scalable, efficient solution for enhancing the adaptability and task-specific performance of LLMs, paving the way for truly dynamic, self-organizing AI systems.

Published as a conference paper at ICLR 2025
TRANSFORMER-SQUAREE SELF-ADAPTIVE LLMs
Qi Sun 1.2%, Edoardo Cetin' Yujin Tang"
Sakana AI, Japan PInstitute of Science Tokyo. Japan gissu,, edo, yujintang sakana. ai
Equal contribution
ABSTRACT
Self-adaptive large language models (LLMs) aim to solve the challenges posed by traditional fine-tuning methods, which are often computationally intensive and static in their ability to handle diverse tasks. We introduce Transformer (Tranfformer-qquared) novel self-adaptation framework that adapts LLMs for unseen tasks in real-time by selectively adjusting only the singular components of their weight matrices. During inference, Transformer' employs a two-pass mech- anism: first, dispatch system identifies the task properties. and then task-spec "expert" vectors, trained using reinforcement learning. are dynamically mixed to obtain targeted behavior for the incoming prompt. Our method consistently out- performs ubiquitous approaches such as LoRA, with fewer parameters and greater efficiency. Furthermore, Transformer" demonstrates versatility across different LLM architectures and modalities, including vision- laiiual tasks. Transformer represents significant leap forward, offering scalable, efficient solution for hancing the adaptability and task-specific performance of LLMs, paving the way for uuly dynamic, organoiggn AI systems. We provide our full source code httpps: htt/ggttuue github..h.... com Satp//aaaaaaaaaaaaaa selfe adapt adaptive--o ivertiieeee 11ms
INTRODUCTION
Self-adaptive large language models (LLMs) would represent significant advancement in artificial intelligence, providing a framework where mod- els can adjust to varied tasks and dy- namic contexts in real time. This con- cept draws inspiration from the long- standing idea of neural networks modifying their own weights to adapt to tasks dynamically (Schmidhuber 1993 Irie et al. 2022) and neural net- works generating weights for other networks, as popularized by Hyper- Networks and related methods (Hal et al.] 2017 (Stanley et al.] 2209). While compositionality and scalabil- ity are crucial for effective adapta- tion. current LLM training method- ologies fall short of achieving both these properties simultaneously. Our research aims to present pioneering solution to realize this vision and address these gaps.
Thhl a math quedion"
Anower to User Query
Matik mult pliation
SVD of Weights
Hosten Statee
VT
Flemenhaaiee mudtppiration
Self-Adapterion Wectors
poso
User Query
Second pass
Figure 1:
Overview of Transformer? In the training
phase, we tune the scales of the singular values of the weight matrices to generate a set of "expert" vectors, each which specializes in one type of tasks. In the inference phase, two-pass process is adopted where the first applics the task- specific expert and the second generates the answer.
Traditionally, LLM post-training has sought to optimize model for wide range of capabilities in a single, extensive training session. While this "one-shot" fine-tuning framework is ideal from simplicity perspective, it is also difficult achieve in practice. For instance, post- training is still resource-intensive. Additionally, expanding data breadth often creates performance trade-offs, making it difficult to simultaneously address overfitting and task interference.

---

Published as a conference paper at ICLR 2025
In contrast, self-adaptive models offer more flexible and efficient approach. Rather than attempting to train an LLM for all tasks in one step, expert modules can be developed offline and augmented to the base LLM on-demand (Kang et al. 2024). This allows the model to dynamically modify its behavior based on the task at hand, without the need for constant re-tuning. In addition to the bene- fit of having independent components, this modularity also supports continual learning, enabling the model to add new skills over time without catastrophic forgetting. Moreover, self-adaptive LLMs mirror a well-established principle in neuroscience and computational biology, where the brain activates specific regions depending on the task at hand (Loose et al. 2017) and dynamically
reconfigures its functional networks in response to changing task demands (Davison et al.) (005)).
In principle, the first step toward achieving self-adaptive LLMs can be realized through the devel- opment of specialized expert modules, each fine-tuned (Kaplan et al. 2020) via techniques such as low-rank adaptation (LoRA) (Hu et al. 2021). These expert modules can then be dynamically composed at runtime based on the task demands, a process that can be efficiently managed through Mixture of Experts (MoE)-like systems (Tianlong et al. 2024). However, several challenges need to be addressed to make this approach both scalable and compositional. First, fine-tuning LLMs to cre- ate multiple expert modules significantly increases the number of parameters that need to be trained. In practice, even with parameter-efficiee ules can quickly escalate, leading increased storage and computational demands. Second, these expert modules are often prone to overfitting, a phenomenon especially prevalent when training on smaller datasets or narrow task domains. Third, the flexible composition of these expert modules also presents largely unresolved challenges currently posing as open research problems.
methods like LoRA, the cumulative size of these mod-
To overcome these limitations, we first propose Singular Value Fine-tuning (SVF), a novel parameter-efficient fine-tuning (PEFT) method to obtain effective building blocks for self- adaptation. SVF works by extracting and tuning only the singular values within the model's weight matrices. By focusing on this principled parameterization, our approach mitigates the risk of over- fitting, drastically reduces computational demands, and allows for inherent compositionality. We show these properties enable us to cheaply obtain a set of effective domain-specifie "expert" vectors by training on narrow datasets with RL, directly optimizing task performance on individual topics.
We then introduce our full Transformer" through the underlying principles of self-adaptation. Given a prompt from an unknown task, Transformer" entails a two-pass inference mechanism which we illustrate in Figure 1 During the first pass, Transformer executes the model and observes its test-time behavior, gathering the rele- vant information to understand the necessary skills tackle the current problem. During the second pass, our framework uses this information to combine the available expert vectors and provide a new modification to the base weights of the LLM specifically tailored to its test-time conditions. We design three different adaptation strategies that can be used within Transformer', which we show provide monotonic performance benefits with increasing access to the test-time conditions.
Sounsformer-Squaree framework to empower LLMs
We evaluate SVF and the full Transformer? framework through extensive experiments across a di- verse range of LLMs and tasks. First, when trained on domain-specific datasets, we show that SVF consistently outperforms traditional strategies for efficient fine-tuning such as LoRA, and at the same time, with orders of magnitudes fewer parameters. Then we show that Transformer? is able to push performance far further, effectively adapting the weights of the base model even in entirely out-of-distribution applications such as visual question answering. Finally, we analyze the proper- ties of our new framework, validating that it provides increasing benefits with additional access to its current test-time conditions and even allow for recycling pre-trained SVF experts across model architectures. In summary, our key technical contributions are the following:
The development of Transformer as a pivotal self-adaptation framework for LLMs, pro- viding universal blueprint to dynamically adapt the behavior of LLMs from a growing set of pre-trained skills.
The introduction of SVF, novel PEFT method trainable with RL on small datasets, pro- ducing compact expert vectors with inherent compositionality, all key properties necessary for our scalable self-adaptation framework.
The implementation of three adaptation strategies within Transformer", effectively dis- patching SVF-trained experts with properties designed to cope with different requirements and deployment scenarios.
2

---

Published as a conference paper at ICLR 2025
2
RELATED WORKS
Self-adaptive LLMs We define self-adaptive LLMs as a group of LLMs or a standalone LLM that can evaluate and modify its behavior in response to changes in its operating environment internal state, without external intervention. This dynamic adjustment has parallels to concepts like fast- weight memories, which enable networks update weights in response to task demands (Schmid-) huber 1992] Gomez & Schmidhuber 2005), and neural network weights being treated as dynamic programs (Schmidhuber] 20155. Recently, Panigrahi et al. (2023) introduces an approach where a smaller auxiliary transformer is updated dynamically within a larger model, aligning with the prin- ciples of self-adaptive behavior.
This adaptation can be explored from two perspectives: macroview, where multiple LLMs collab- orate and/or compete, and a microview, where internal adaptations allow a single LLM to specialize in different tasks.
Macroview: From this perspective, the system directs queries to L.LMs with domain specific exper- tise, prioritizing outputs from expert models, thereby achieving higher accuracy and task-specific optimization. Such task-specific ensembles can be realized through various mechanisms: multiple LLMs playing distinct roles and coordinate toward a shared goal (Zhuge et al., 2023), engaging in mutual listening and debate (Du et al. 2023), or using meticulously crafted prompt construc- tions (Zhang et al. (202) to integrate knowledge library and skill planning. Naturally, the improve- ment in the specialization and adaptive capabilities of individual LLMs in the ensemble enhances the collective performance. Thus, in this paper, we focus on the microview of self-adaptive LLMs.
Microview: MoF. in LLMS plays a critical role in this perspective (Tianlong et al. 2024). In MoF
systems, inputs are dynamically routed to a subset of specialized modules or layers (e.g., MLPs) containing domain-specific knowledge (Rajbhandari et al., 2022, Fedus et al. 2022). To reduce inference time, researchers introduce sparsely activated MoE where only a subset of the experts are selected per token Jiang et al. (2024): Qwen Team (2024). While it is possible to view Transformer" loosely as a type of MoE, there are two major differences. In the aforementioned systems, self- adaptation is achieved through token-level routing, whereas Transformer employs a sample-level module selection strategy. The second difference lies in the construction of expert modules. In traditional MoE systems, expert modules are either trained from scratch (Fedus et al. 2022 Jiang al. 2024) or dense models (e.g., upcycling) (Qwen Team 2024 Zhu al. 2024). without an auxiliary loss to ensure module specialization. In contrast, Transformer' specifically trains expert making them true experts.
vectors with RI. to acquire domain
Low-rank adaptation PEFT methods such as LoRA (Hu et al., 2021 works by freezing the original model's parameters and introducing small trainable low-rank matrices for task-specific updates. It significantly lowers the computational and memory costs while providing performance comparable to full fine-tuning. Inspired by LoRA's design, various modifications have been proposed (Zhang
al. 2023 Kopiczko al., 2023 Liu et al. 2024 (Bllzy al., 2024: Cetoli 2024 Kaushik al., 2025
Transformer" does not rely on low-rank matrices, and instead scales the singular vectors of the original parameter matrix that span the full rank space.
SVD for LLM Fine-tuning SVD is increasingly being used an inductive bias for PEFT in LLMs. For example, Wang et al. (2024) decompose a weight matrix and use the minor singular components, associated with noisy or long-tail information, to initialize low-rank matrices for LoRA fine-tuning. Earlier work proposed using compressed forms like DCT coefficients for generating weight matrices in neural networks (Koutnik et al. 2010), offering efficiency in memory-constrainee environments, which resonates with our approach. In a similar vein, SVD is employed to approximate an original weight matrix with the top r singular vectors, corresponding to the highest singular values. A small trainable matrix is then introduced on top of the truncated singular value matrix to adjust the magnitude and orientations within this top-r subspace (Batazy et al. 2024 Cetoli 2024). However, the drawback of this approach is that retaining only the top singular components can result in the loss of important information, particularly when the singular values distribution is less skewed. The work most similar to ours is concurrent effort by Lingam et al. (2024), where they introduce various sparsification methods that utilize the SVD of the weights. However, it is not for self adaptive LLMs and does not use RL to enhance learning efficiency.
3

---

Published as a conference paper at ICLR 2025
3 METHODS
3.1 PRELIMINARIES
Singular value decomposition (SVD) offers fundamental view of matrix multiplications. In the context of neural networks, each weight matrix W E Rax can be decomposed into three compo- nents W U2VT, yielding semi-orthogonal matrices U RRxx and V _gax together with an ordered vector of r singular values (in descending order) arranged in the diagonal matrix E E RR"x" The linear operation defined by applying W onto æ, can be then decomposed into a sum of indepen- dent terms, derived from mapping each column from V into the corresponding column from as y Σi= auu;o z. Hence, each singular component represented by the rank-1 matrix u independently processes the input, providing orthogonal contribution to the layer's outputs, with the singular values o, modulating the degree of the contributions.
Cross-entropy method (CEM) is a Monte Carlo method for importance sampling and optimiza- tion [Rubinstein & Kroese 2004). The method is based on the concept of minimizing the KL divergence between two probability distributions DKL (P) where P is the target distribution and Q is a maintained distribution. At its core, CEM repeatedly generates a set of samples from Q, evaluates these samples with performance function, and then updates the distribution with the characteristics of the elite samples that have performed best. In the standard setup employed in most applications, Q is set to diagonal multivariate Gaussian, reducing the problem simply estimating the empirical mean and standard deviation of the latest elites until stopping criterion is met. We illustrate a complete CEM step in the Python pseudocode below.
def cem_step (mu, sigma, num_elites, num_samples):
samples np. normal (loc=mu, scale-sigma, size=num size-num_samples)
scores eva luate (samples)
elites = samples [np aapgortsrt (scores) [ [-num_elites:]]
new_mu mean (elites, axis=0)
new sew_sigma = np std (elites, axis=0)
return (new_mu, new_sigma)
3.2 TRANSFORMER
The construction of Transformer comprises two main steps, for which we provide illustrative overview in Figure 2] First, we introduce Singular Value Fine-tuning (SVF), method to learn with RL compact and compositional expert vectors based on the SVD of the base model's weights. Then, describe three different adaptation strategies within Transformer?, inspired by three thogonal principles, which adaptively combine the SVF-trained expert vectors during inference. We motivate how the properties of SVF are highly complementary Transformer" an effective and scalable framework for the design of new self-adaptive LLMs.
adaptation strategies, making
parameters
Replaced with one leameed vactor
U
VT
Replaced with mixture of the learned vectors:
Norm
leyers
matrices
Learnable parametens trsined with RIL
A) Prompt based adoptation, or B) Job classifier based adaptation
C) Mixture-based adaptation
Layer Norm
Training Time
Inference Time
Figure 2: Method overview. Left) At training time, we employ SVF and RI to learn the "expert" vectors z's that scale the singular values of the weight matrices. Right) At inference time, we propose three distinct methods to adaptively select/combine the learned expert vectors.
Singular value fine-tuning is key building block in Transformer?. offers an extremely efficient parameterization for fine-tuning and provides inherent compositionality for adaptation. Conven-

---

Published as a conference paper at ICLR 2025
tional fine-tuning techniques often aim to augment pre-trained models with new capabilities by mod- ifying their weight matrices. However, in large-scale transformers, these weights are already rich repositories of abstracted knowledge, thanks to the breadth of the pre-training data and expansive architectural design. In fact, as evidenced in much of the prior literature, the requisite capabilities for solving many downstream tasks appear to already exist within these pre-trained models (Sharma iet al.) 2023). Therefore, instead of seeking to add new features, an efficient fine-tuning approach should focus on making these latent capabilities more expressible. Motivated by these considera- tions, for any weight matrix W, SVF learns a simple vector 2 E IK" that provides targeted modifica- tions to each singular component of W independently, yielding a new weight matrix W UVVV, where Σ' = Σ @ diag(z). This essential parameterization enjoys several benefits:
Negligible parameters: Learning only a vector 2 for each weight matrix allows for very efficient fine-tuning with orders of magnitudes fewer optimized parameters even when compared to prior approaches specifically designed for efficiency. For example, the widely popular LoRA approach requires (m++n) x' learnable parameters per weight matrix, where r' ís a paparr-rrratter that gen- erally needs to be set large enough for expressivity. While recent extensions, such LoRA-XS (Balazy et al. 2024), try to push efficiency even further, they often introduce limiting assumptions that curb applicability in several practical scenarios (see examples in Appendix [ In contrast, while SVF only needs r min(m,.)) n) parameters, show it empirically does not display the same shortcom- ings thanks to working on a highly-meaning space provided by the latent expressiveness compressed in the weights of modern LLMs. SVF's scaling only the singular values may seem to lead to limited expressiveness, we wish to point out that the ability to affect the weight matrix in a full-rank manner technically provides more information than low-rank approaches.
High compositionality: Decomposing the weights in independent singular components makes the learned x vectors highly composable and interpretable, opening numerous possibílities for adapta- tion via algebraic manipulations. Instead, LoRA based methods inherently lack these properties. For instance, even if two LoRAs learned on the same task were to learn exactly the same adjustments for each W. directly interpolating between their compressed A and B matrices is unlikely to preserve any of their original behavior, given the countless number of equivalent parameter permutations they might have converged to.
Principled regularization: Exclusively modifying the magnitude of pre-existing singular compo- nents provides a principled and effective form of regularization. In practice, this property enables us to tine-tune for arbitrary downstream tasks with only hundreds of data points without the risk of severe collapse or overfitting.
End-to-end optimization with RL. We train a set of SVF vectors 0Σ = (=1.... _NxM) } to fine- tune arbitrary language model Now parameterized by Ow with RL, optimizing directly for task performance. Here, Ow = (W,,.. ... x } is the set of weight matrices, where N is the number of layers and M is the number of weight matrices fine-tune per layer. We use the seminal RE- INFORCE algorithm (Williams) 1992) and label each generated answer yi (for the prompt 2; E D) with a unitary reward based on its correctness for optimizing LLMs (Ouyang et al., 2022), we regularize the REINFORCE objective by adding a KI. penalty for deviating from the original model's behavior, weighted by a small coefficient A E R Thus, our final objective function can be written as:
[1, Inspired by related applications of RL
J(O.) E [log (Te (ūs 2:)) r(h,, v(]] ADKL (TT ||
where we use Te to denote the resulting language model after substituting the original weight matrices W with W'. While RL is generally considered less stable than next-token prediction ob- jectives, we find the regularization properties of SVF avoid many of the failure modes of prior less- constrained parameterizations (see Section 4.3). Thus, combining these complementary components effectively enables us to avoid relying on expensive fine-tuning procedures with large hand-designed datasets as proxies, and directly maximize task performance end-to-end.
In general, SVF with RL puts lower requirement on the dataset it trains on For example, LoRA fine-tuning requires "explaining texts" to perform next token predictions, which puts a higher re- quirement on the dataset (e.g., imagine LoRA fine-tuning on a GSM8K dataset where no reasoning text but only the final number is provided). This benefit allows SVF to be more general and effective. One possible caveat SVF can face is the sparse rewards caused by a weak base model, which we discuss this further in Section 5
5

---

Published as a conference paper at ICLR 2025
Self-adaptation is critical mechanism in nature that has established itself as core guiding princi- ple in modern system design (Klös et al. 2013). Our initial efforts toward self-adaptive foundation models focus on the inference stage of LLMs, where devise simple two-pass adaptation strat- egy that combines K' sets of base "expert" vectors 21:K trained with SVF to provide different kinds of capabilities (e.g., coding, math, etc). The mapping between capability and the dataset we train on can be acquired in the dataset's ' meta data. In the first inference pass, given a task or an individ- ual input prompt, Transformer executes the model and observes its test-time behavior to derive a new z' vector tailored to its test-time conditions. This adapted 2' is then used in the second infer- ence pass to provide an actual response with the newly adapted weights. The interaction between SVF-trained expert vectors and the adaptation strategies ensures seamless integration, where ex- pert vectors provide modular capabilities, and the adaptation strategies dynamically determine and compose the most suitable combination to address the input task. In this first work, we propose three simple approaches to produce the vector z' during the first inference pass, implementing self- adaption with distinct methods and requirements. Below, we provide an outline of each method and refer to Appendix A for additional implementation details.
A) Prompt engineering: Our most basic approach involves constructing a new "adaptation" prompt which we use to directly ask the LLM to categorize the input prompt. Based on its response, we then extract one category out of the set of domain topics used to pre-train each SVF expert and, thus, we select the corresponding z' directly from 21: In our adaptation prompt, we also explicitly provide the option for a generic "others" " category, allowing the model to use its base weights in case no expert provides appropriate capabilities. We show the format used to construct the adaptation prompt in Figure
B) Classification expert: A direct extension of the prompt engineering approach comes from using a specialized system handle task identification. Fol- lowing the principles of self-adaptation, we ap- ply SVF fine-tune the base LLM itself to han- dle this task. In particular, we collect dataset D (= 1), 1).... (Ii,k, ......)) from the K SVF training tasks, where Ti,k is the i-th example from the k-th expert task. Each tuple (II.R, k) then forms an example to pre-train an additional job classifica- tion expert 2 learned in the same fashion as the oth- ers. During the first inference pass, simply load 2", intending to improve the inherent task classifica- tion capabilities of the base model to select more appropriate z' to handle the input prompt.
Anclyze the oven cuestion end clasffy it into one of four categer 'code, math 'rassning or 'others'. Follow these gulsetires:
Code Questions aseng programming solutions..
Moth: Questions imvtving mothematioal colculatioos..
Others: Quections dlarryy iit tiiking...
categories..
Inttrctions:
Consier the primary locus, sells, and knowledge required to answer
question.
spant
Prouid cisssiication withi (sboeed notation Exemple bunsefresssening
Format your responge es follows: classtirmtio (asssd//catggrr))
Figure 3: Prompt based adaptation. Self- adaptation prompt used by Transformer" to classify the task prompt into pre-defined cat- egories.
C) Few-shot adaptation: Our third approach leverages additional task information by assuming extended access to its test-time conditions beyond individual prompts. Our approach is inspired by popular few-shot prompting techniques, which have been shown to provide consistent performance improvements and even allow LLMs to "in-context" learn tasks that were entirely unseen prior to inference (Brown 2020). For each optimized W, our approach entails producing an entirely new
z'' = -E R=1 Or by linearly interpolating between the K learned SVF vectors, each weighted by the coefficients ag. We employ CEM to search over the possible values of each og based on the performance on a set of "few-shot prompts", which are specifically held out from the rest of the test prompts and used evaluate CEM's population samples. In the case of multiple population samples obtaining the same score on these held-out prompts, we break ties by favoring the one with the highest average log-likelihood across its own generated correct answers. Crucially, we only need to perform this process once for each target task, avoiding the need to increase the length of each question prompt, a relevant downside of traditional few-shot prompting. We refer to Section A.4 for additional details and an extended discussion of this final approach.
KK
4 EXPERIMENTS
We extensively evaluate Transformer on multiple tasks and models with the purpose of: (1) as- sessing the efficiency and effectiveness of SVF; (2) demonstrating self-adaptiveness through the

---

Published as a conference paper at ICLR 2025
Codlng
Seasoriing
Vi Language
0.43
6.00
0.00
Sett
0.4.
0.7
0.50
0.5
0.2
6.2
0.0
2:10
289
Figure 4: SVF learning curves. The dashed lines indicate the performance of LLAMA3-8B. INSTRUCT on the test split of each task. SVF effectively fine-tunes to surpass the base performance. While we use the best validation score to select our checkpoint for evaluation (marked by red dots), we present the entire training curve without early stopping to demonstrate SVF's learning capabili- ties. Tasks with only hundreds of training samples like Coding and Reasoning were stopped early. In our experiments, we update the parameters at the end of each epoch.
three proposed adaptation strategies; (3) conducting in-depth analysis and ablation studies aimed at understanding and interpreting the properties of our new framework.
4.1 EXPERIMENTAL SETUPS
To validate the generality of Transformer' we consider three pre-trained LLMs ranging across dif- ferent model families and architecture sizes: LLAMA3-BB-INSTRUCT For each model, we obtain three sets of SVF-trained vec-
MISTTAL-7B-INSTRUUR
v0.3, and LLAMA3-70B-INSTRUCE tors to maximize performance for GSM8K (Cobbe et al. 2021), MBPP-pro (Austin et al.) 2021), and ARC-Easy (Clark et al., 2018), respectively. Additionally. we also train set of z vectors for LLAMA3-SB-INSTRUR 2019). in order to assess SVF's applicability to the vision-language modeling (VLM) domain. We provide SVF's main learning curves each of these tasks in Figure 4 Finally. evaluate the full Transformer" adaptation framework on four unseen tasks: MATH (Hendrycks et al. 2021 Humaneval (Chen et al.) (02T), ARC-Challenge (Clark et al., 2018), and OKVQA (Marino et al, 2019). In all our adaptation experiments, we only consider experts obtained in the pure-language set- tings, assessing its test-time applicability even for the distinctive vision domain. Please refer to the Appendix A for additional details and summary of the hyper-paune parameerrssterss used in the experiments.
when applied the language backbone for TextVQA (Singh et al.)
4.2 EXPERIMENTAL RESULTS
SVF performance We provide results after training on each considered task with the LLAMA3- and CLLAMA--70B-INSTEN [ Remarkably, we find that SVF provides considerable and consistent performance gains across nearly all tasks and base models. Instead, LoRA experts yield smaller gains and even sporadic per- formance degradation. (These LoRA experts are trained with next token prediction. While we also have LoRA experts trained with RL in Table 4 SVF.) This observed trend extends also to the vision-language domain, as fine-tuning LLAMA3- LLAVA-NEXT-SB with SVF bolsters the base model's performance by over 39% (see Figure To ensure a fair comparison, we provide extensive ablations to both our model and the LoRA baseline considering different architecture and optimization objectives in Appendix 4.3 Due its essential
8B- IN--INST MISTRAL-TT -
base models in Ta-
RL seems work less well with LoRA than with
Table 1: Fine-tuning results. LLM performance on the test splits of math, coding and reasoning. Normalized scores are in the parentheses.
Lama
Method
GSM8K
MBPP-Pro
ARC-Easy
LLAMA3-BB-INSTIE
75.89 (1.00)
64,65 (1.00)
88.59 (1.00)
+ LoRA
77.18 (1.02)
67.68 (1.05)
88.97 (1.00)
+ +SVF (Ours)
79.15 (1.04)
66.67 (1.03)
89.56 (1.01)
MISTRA--BB-ASTTRCCYVV...
42.83 (1.00)
49.50 (1.00)
81.65 (1.00)
+ LoRA
44.66 (1.04)
51.52 (1.04)
81.19 (0.98)
+ SVF (Ours)
49.74 (1.16)
51.52 (1.04)
85.14 (1.04)
LLAMA3-70B-
85.29 (1.00)
80.81 (1.00)
89.10 (1.00)
Tearivga Figure 5: Results for
OCYOA
+ LoRA
77.26 (0.91)
68.69 (0.85)
88.55 (0.99)
+ +SV (Ours)
88.32 (1.04)
80.81 (1.00)
88.47 (0.99)
the VLM domain.
7

---

Published as a conference paper at ICLR 2025
Table Self-adaptation on unseen tasks. Normalized scores are in the parentheses.
Method
MATH
Humaneval
ARC-Challenge
LLAMA3-8B-INSTER
24.54 (1.00)
60.98 (1.00)
80.63 (1.00)
LoRA
24.12 (0.98)
52.44 (0.86)
81.06 (1.01)
Transformer (Prompt)
25.22 (1.03)
61.59 (1.01)
81.74 (1.01)
+ "ransformer" (CIs-expert)
25.18 (1.03)
62.80 (1.03)
81.37 (1.01)
Transformer" (Few-shot)
25.47 (1.04)
62.99 (1.03)
82.61 (1.02)
MISTRAL-7B I-NNT-B---TT
13.02 (1.00)
43.29 (1.00)
71.76 (1.00)
LoRA
13.16
37.80 (0.87)
75.77 (1.06)
+ Transformer" (Prompt)
11.86 (0.91)
43.90 (1.01)
72.35 (1.01)
Transformer' (Cls-expert)
11.60 (0.89)
43.90 (1.01)
74.83 (1.04)
+ Transformmr" (Few-shot)
13.39 (1.03)
47.40 (1.09)
75.47 (1.05)
MAMA3-70B-INSTRUCE
40.64 (1.00)
78.66 (1.00)
87.63 (1.00)
LoRA
25.40 (0.62)
73.78 (0.94)
83.70 (0.96)
Transformer" (Prompt)
40.44 (1.00)
79.88 (1.02)
88.48 (1.01)
parameterization, we would like to note that training SVF requires considerably fewer resources, with less than 10% of the training parameters of our LoRA implementation.
Adaptation performance With the SVF trained 2 vectors, assess the self-adaptation capability of Transformer" on unseen tasks. For fair comparison with LoRA, we record the performance of this baseline using all checkpoints from the considered training tasks and report only its highest per- formance for each of the test tasks. As shown Table 2 all of our Transformer' adaptation strategies demonstrate improvements across all tasks for -LL-AM3--B---A SB-INSTRUCT base models. and in at least two out of three tasks for both BETTTRL--B-INSRRUUEE contrast, even the best training LoRAs only provide marginal improvements the ARC- ARC-Challenge task and still significantly deteriorate performance on both MATH and Humaneval. This discrepancy suggests that LoRA's parameterization and optimization might be particularly sensitive to overfit- ting. In F we find a similar dichotomy in the OKVQA task, with the performance of the base LLAMAS-LLAVA-NEXTT
and LLAMA3- ----70I-ISSSS INSTRCTT. In
VLM only improving after applying Transformer".
Comparing the three proposed adaptation strategies, we highlight clear monotonic trend with more involved strategies and information about the test-time condition, self-adaptation appears to be increasingly effective. In particular, Transformer with few-shot self-adaptation is almost always the highest-scoring method, providing notable improvements across all settings except for LLAMA3- 70B- INS-UCTT @ @MATH, where we have only SVF-tuned half the layers due to limited GPU resources. This trend shows that providing additional or different kinds of information seems highly benefit our framework, suggesting that Transformer could provide foundation models with new means to continually improve performance when deployed in lifelong settings.
repotss the inference time required by the prompt adap- tation strategy of Transformer", with the time spent on solving the entire problem set presented separately for the 1st and 2nd passes. Notice that the 2nd pass inference time is the time spent on solving the problems, and the 1st pass inference time is the time for self ada-aadat,,,on, 1st to 2nd pass inference time ratios are in parentheses. While the additional inference pass appears to double the overall runtime, it is important to note that inference time primarily depends on the number of tokens generated. In our settings, it is (n)) where n is the length of the input. ARC-challenge have larger cost ratio because they single choice problems and the cost of the 2nd pass is also On)). In general settings, we think it is reasonable to assume this ratio to be closer to those of MATH and Humaneval. For a detailed discussion on improving the efficiency of CEM adaptation methods, please see Appendix D
Table 3: Time cost of 2-pass inference in prompt adaptation strategy of Transformer" for the entire problem set. 1st 2nd pass inference time ratios are shown in parentheses.
Task
1st (s)
2nd (s)
MATH
42.64 (13%)
321.19
Humaneval ARC-Challenge
2.76 (19%)
14.28
13.40 (47%)
28.51
4.3 ANALYSIS
Lastly, we analyze and discuss the properties of our adaptation strategies for which we provide extensions and further discussion Appendix B
8

---

Published as a conference paper at ICLR 2025
Lana Pramge Enghnering
20
Capsiiaete Dosrt
hrmppErg
Gls Isution
LX
962
1:00
6/8 5. V4S
989
2.0
041
3W 1.09
2.9
2.7
0.5
021
3:00
Figure 6: Confusion matrices. These matrices display the classification percentages, where rows represent the task classes (ground truth) and columns indicate the predicted categories. Some sam- ples are misclassified as "Others," which reflected in rows where the totals do not sum to one.
Analysis 1: Job dispatching accuracy In Figure 6 we provide the confusion matrices of classification-base adaptation strategies. Results validate the effectiveness of both our strategies to match each prompt with experts trained in similar domains, as evidenced by the high values along the diagonals. Furthermore, the results from INSTRUCT-VÓ.. also show that using the classification expert consistently provides higher classi- fication accuracy than vanilla prompt engineering. While this difference could explain the higher performance of the relative self-adaptation strategy, we also note that domain similarity might not be the only metric relevant to identifying the best expert for each prompt or task. To this end, we be- lieve many further extensions could be explored in future work, using heuristics such as past expert performance or token-level analysis to further push our framework's 's scalability.
and MISTRAL--7B -B-
Analysis 2: Training tasks adaptation contribution In Figure 7 we show the normalized adaptive coefficients a₁ interpolating between SVF vectors learned via CEM for LLAMA3-8B. CEAAA-----INSTRRCC and MISTRAL- - v0.3 across the unseen tasks. Intuitively, we find that the expert vec- tors from the training tasks sharing similar topics to the unseen ones are often the highest contribu- tors. However, we observe that the MATH task appears as an exception, as the ag for the GSM8K expert is actually the lowest in both models. We hypothesize this reflects the different nature of the mathematics competition problems from MATH as compared to the grade-school problems in GSM8K. In fact, not only is the difficulty of the MATH questions far beyond GSM8K, but large portion of its problems also hinges mainly on logical reasoning, for which a task like ARC might be more aligned. Furthermore, to adaptation in the Llama model. This difference indicates that, due to its higher base performance, the Llama model does not need to rely on any particular set of skills much as Mistral, and can har- ness more holistic benefits from self-adaptation. Note that applying ag uniformly is not a universal solution for leveraging expert vectors. This becomes evident when we look at different model and task combinations (e.g. applying as uniformly on LLAMA3-8B- ---------ISSUCCC for MATH tasks only achieves 24.47, while Transformer (Few-shot) achieves 25.47).
note that the different vectors appear contribute more uniformly
Analysis 3: Ablation studies
Module sensitivity: We compare the performance of SVF on different modules (see trials 1-3). Both individual MLP and attention updates improve performance, with MLP resulting in more pro- nounced gains. Updates to both module types yield even more sigmificant enhancements.
Objective function: We are interested in the performance impact from different objective functions, and we compare the RL objective with next-token prediction (see trials 2, 4). For the latter, we use instruction ine-tuning with official GSM8K solutions as target tokens. Results show clear perfor- mance gains with RL. Conversely, next-token prediction even hinders performance. This highlights RL's ability to handle cases lacking detailed solutions, suggesting its superiority this context.
SVF LoRA: We also evaluate LoRA using the RI. objective (see trials 2, 5). A significant per- formance disparity is observed, primarily attributed to the severe instability of the LoRA training process. Despite exploring wide range of learning rates, LoRA's performance consistently lagged behind. For further illustrations, see Figure in the appendix.
Analysis 4: Cross-model compatibility Finally, we explore the potential for our framework to be applied across different LLMs. In particular, we evaluate whether the SVF vectors trained on LLAMA3-8B. INSIRUCT can benefit MISTRAL-7B-INSTRUCTVV adaptation across the vectors of these two models. We present our main findings in Table [ and refer to Appendix B for additional results. Surprisingly, we find that positive transfer occurs with
and whether we can perform

---

Published as a conference paper at ICLR 2025
Table 4: Ablation studies. We fine-tune LLAMA3-8B-INSTRUU on the GSM8K training split with
different settings and the results on the test split along with zero-shot transfer results on MATH.
Method Objective Function
Module
#Params (4)
GSM8K (t)
MATH (t)
1LAMA-3-8--INSRREEE
75.89 (1.00)
24.54 (1.00)
SVF
Policy gradient
MLP
0.39M
78.62 (1.04)
24.20 (0:99)
SVE
Policy gradient
attention
0.16M
76.19 (1.00)
24.20 (0:99)
3
SVE
Policy gradient
MLP + attention
0.58M
79.23 (1.04)
25.04 (1.04)
4
SVF
Next token pred
attention
0.16M
60.50 (0.80)
18.52 (0.75)
5
LoRA
Policy gradient
attention
6.82M
57.92 (0.76)
15.72 (0.64)
LoRA
Next token pred
attention
6.82M
77.18 (0.98)
24.12 (0.96)
LoRA
Next token pred
MLP attention
35.13M
75.66 (0.96)
22.12 (0.91)
Table 5: Cross-model z vector transfer. Results from transferring the expert vectors trained on with cross model few-shot adaptation.
LLAMA3-8B-INSTRUCE to
Method
MATH
Humaneval
ARC-Challenge
SVF training lask
GSM8K
MBPP-pro
ARC-Easy
MISTRAL-7B- --NLT--UU-STT
13.02 (L.00)
43.29 (1.00)
71.76 (1.00)
Llama SVF (ordered os)
11.96 (0.92)
45. (1.04)
72.01 (1.00)
+ Llama SVF (shuffled o.)
10.52 (0.81)
40.24 (0.93)
70.82 (0.99)
+ Few-shot adaptation (cross-model)
12.65 (0.97)
46.75 (1.08)
75.64 (1.05)
visible benefits in 2 out of 3 tasks. We note these improvements are due to the inherent ordering of the SVF parameterization, as randomly shuffling each SVF vector before applying it to the Mistral model consistently degrades performance.
This operation leads to notable performance degradation across tasks. By performing few-shot adaptation using the SVF vectors from both models, the performance of MISTRAL-755 7B-L---B--TTRCTT v0.3 further improves across the board. We observe that these gains even surpass the best score from adapting MISTRAL-7B INSTRUCT-VO.3 with all the SVF vectors in the ARC-Challenge task reported in Table 2 While these results appear promising, note that the surprising compatibility discovered through our naive transfer approach is potentially tied to the similarity be- tween the architectures of the two considered LLMs. To this end, whether similar transfer can be replicated with models of differ- ent scales remains an open question that could open the doors to recycling task-specific skills for larger models, with important implications for democratization and sustainability.
PAFT
11TL
Figure 7: Or learned weights.
5 CONCLUSION
In this paper, we introduced Transformer", providing a novel blueprint toward realizing self-adaptive LLMs. Within this framework, we first proposed SVF, offering superior performance than prior fine- tuning recipes, together with reduced costs, high compositionality. and overfitting regularization all crucial properties to achieve scalable adaptation. Leveraging a set of SVF experts as building blocks, we developed three effective strategies for self-adaptation, each offering unique advantages, with performance and monotonic performance benefits with increasing access to the test-time conditions.
While Transformer" demonstrates promising results, there remain exciting opportunities for future work. One limitation is that the capabilities of SVF experts are tied to the latent components of the base model. To address this, model merging offers a promising direction ( (Yu et al. 2024: Goddard et al., 2024: AAibba et al., 2024). Additionally, while CEM- adaptation balances performance and efficiency, scaling to large number of specialized domains may increase the initial one-time computational costs. Advances in model merging and efficient adaptation techniques have pro- duced models dominating open leaderboards, making them strong candidates as base models for Transformer' and opening new possibilities for adaptive L.LMs.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. Evolutionary optimization of model merging recipes. arXiv preprint arXiv:2403. 2024.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry Quoc Le, al. Program synthesis with large language models. arXiv preprint arXiv:2108.00 07770222, 2021.
Klaudia Balazy, Mohammadreza Banaci, Karl Aberer, and Jacek Tabor. Lora-xs: Low-rank adapta- tion with extremely small number parameters. arXiv preprint arXiv:2405. 17604, 2024.
Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv: 14/65, 2020.
Alberto Cetoli. Fine-tuning Ilms with singular value decomposition. Hugging Face Blog, June 2024. URL https:://uugiigffcce http:///ww....oooooo////oottteoa
Ac-
cessed: 2024-07-01.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 2021.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 1803 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint 0tti:2110.141660 2021.
Elizabeth N Davison, Kimberly J Schlesinger, Danielle S Bassett, Mary-Ellen Lynall, Michael B Miller, Scott Grafton, and Jean Carlson. Brain network adaptability across task states. PL.S computational biology, 11(1)=e100002, 2015.
Yilun Du, Shuang Li, Antonio Torralba, Joshua Tenenbaum, and Igor Mordatch. Improv- ing factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305. 14325, 2023.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-39, 2022.
Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vlad Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. Arcee's mergekit: A toolkit for merging large language models. arXiv preprint arXiv:2403. 13257, 2024.
Faustino Gomez and Jürgen Schmidhuber. Evolving modular fast-weight networks for control. In International Conference Artificial Neural Networks, pp. 383-389. Springer, 2005.
David Ha, Andrew M. Dai, and Quoe V. Le. Hypernetworks. International Conference on Learn- ing Representations, 2017. URL httlthttt///sseeereeeee httprrewwwwwwwweeeeeeeeeeeeeeeee
?id=rkpACeereee 1x
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Bric Tang, Dawn Song. and Jacob Steinhardt. Mcasuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.038/7 2021.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 1rXiv:2106.09685 2021.
Kazuki Irie, Imanol Schlag, Róbert Csordás, and Jürgen Schmidhuber. A modern self seff-referential weight matrix that learns to modify itself. International Conference on Machine Learning, pp. 9660-9677. PMLR, 2022.
11

---

Published as a conference paper at ICLR 2025
Albert Q Jiaan, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXhv:2401.04088 2024.
Junmo Kang, Leonid Karlinsky, Hongyin Luo, Zhen Wang, Jacob Hansen, James Glass, David Cox, Rameswar Panda, Rogerio Feris, and Alan Ritter. Self-moe: Towards compositional large language models with self-specialized experts. arXiv preprint arXiv:2406. 12034, 2024.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836/ 2020.
Prakhar Kaushik, Ankit Vaidya, Shravan Chaudhari, and Alan Yuille. Eigenlorax: Recycling adapters to find principal subspaces for resource-efficient adaptation and inference, 2025. URL https://arxiv 2ttp.//00000///////20007070
Verena Klös, Thomas Göthel, and Sabine Glesner. Adaptive knowledge bases in self-adaptive sys- tem design. 2015 st Euromicro Conference on Software Engineering and Advanced Appli- cations, pp. 472-478, 2015. doi: 10. N0.1109/SEAA.2015.00
Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki Markus Asano. Vera: Vector-based random matrix adaptation. arXiv preprint arXiv:2310. 1454, 2023.
Jan Koutnik, Faustino Gomez, and Jürgen Schmidhuber. Evolving neural networks in compressed weight space. In Proceedings of the 12th annual conference on Genetic and evolutionary compu- tation, pp. 619-626, 2010.
Vijay Lingam, Atula Tejaswi, Aditya Vavre, Aneesh Shetty, Gautham Krishna Gudur, Joy- deep Ghosh, Alex Dimakis, Eunsol Choi, Aleksandar Bojchevski, and Sujay Sanghavi. Svft: Parameter-efficient fine-tuning with singular vectors. arXiv preprint 0at:22455.19999 2024.
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950-1965, 2022.
Shih- Yang Liu, Chien-Ye Yi Wang. Hongxu Yin, Pavlo Molchanov, Yu- Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprint ar arXiv:2402.09: 00330.99.33,... 2024.
Shih-Yang
Lasse S Loose, David Wisniewski, Marco Rusconi, Thomas Goschke, and John-Dylan Haynes. Switch-independen task representations in frontal and parietal cortex. Journal of Neuroscience, 37(33):8033-88)) 8042, 2017.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. Proceedings of the IEEE/cyf conference on computer vision and pattern recognition, pp. 3195-3204, 2019.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Abhishek Panigrahi, Sadhika Malladi, Mengzhou Xia, and Sanjeey Arora. Trainable transformer in transformer. arXiv preprint arXiv:2307.011 189, 2023.
Qwen Team. Qwen1.5-moe: Matching 7b model performance with 1/3 activated parameters, March 2024. UtLp:ttt::///wennn qit.ub.ittuuu...... htt.://ww..///loo////eeooooooooo
Blog post.
Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Am- mar Ahmad Awan, Jeff Rasley, and Yuxiong He. Deepspeed- Advancing mixture-of-expern inference and training to power next-generation ai scale. In International conference on machine learning, pp. 18332-1834 18346. PMLR, 2022.
12

---

Published as a conference paper at ICLR 2025
Reuven Y Rubinstein and Dirk P Kroese. The cross-entropy method: unified approach com- binatorial optimization, Monte- Monte-Carlo simulation, and machine learning, volume 133. Springer, 2004.
Jürgen Schmidhuber. Learning control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation, 4(1):131-139 1992.
Jürgen Schmidhuber. A 'self-referential weight matrix. In ICANN'93: Proceedings of the Interna- tional Conference on Artificial Neural Networks Amsterdam, The Netherlands 13-16 September 1993 3, pp. 446-450. Springer, 1993.
Jürgen Schmidhuber. On learning think: Algorithmic information theory for novel combina- tions of reinforcement learning controllers and recurrent neural world models. arXiv preprint arXiv:1511.09244 2015.
Pratyusha Sharma, Jordan T Ash, and Dipendra Misra. The truth in there: Improving reasoning in language models with layer-selective rank reduction. arXiv preprint arXiv:2312.13558 2023.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Mareus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Kenneth O Stanley, David B D' A'mrrssi,, and Jason Gauci. A hypercube- based encoding for evolv- ing large-scale neural networks. Artificial life, 15(2):185-122, 2009.
Chen Tianlong, Cheng Yu, Chen Beidi, Zhang Minjia, and Bansal Mohit. Mixture-of-experts in the era of llms: A new odyssey. ICML 2024 presentation slides, 2024. International Conference on Machine Learning (ICML).
Hanqing Wang. Zeguan Xiao, Yixia Li, Shuo Wang, Guanhua Chen, and Yun Chen. Milora: Harnessing minor singular components for parameter-efficient I1m finetuning. arXiv preprint arXiv:2406.09044, 2024.
Ronald J Williams.
Simple statistical pradient-following algorithms for connectionist reinforcement learning. Machine learning, 8:229-256, 1992.
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Ab- sorbing abilities from homologous models as a free lunch. In Forty-first International Conference on Machine Learning. 2024.
Ceyao Zhang, Kaijic Yang. Siyi Hu, Zihao Wang, Guanghe Li, Yihang Sun, Cheng Zhang, Zhaowei Zhang, Anji Liu, Song- Zhu, et al. Proagent: building proactive cooperative agents with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 17591-17599, 2024.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng Hc, Yu Cheng, Weizhu Chen, and Tuo Zhao Adalora: Adaptive budget allocation for parameter- efficient fine-tuning. arXiv preprint arXiv:2303. 105 2023.
Tong Zhu, Xiaoye Qu, Daize Dong, Jiacheng Ruan, Jingqi Tong, Conghui He, and Yu Cheng- Llama-moe: Building ofxturr-of-experts from laaaa with continual pre-training. arXiv preprint arXiv:2406.16554, 2024.
Mingchen Zhuge, Haozhe Liu, Francesco Faccio, Dylan R Ashley Róbert Csordás, Anand Gopalakrishnan, Abdullah Hamdi, Hasan Abed Al Kader Hammoud, Vincent Herrmann, Kazuki Irie, et al. Mindstorms in natural language-baser societies of mind. arXiv preprint arXiv:2305. 17066, 2023.
13

---

Published as a conference paper at ICLR 2025
A IMPLEMENTATION DETAILS AND HYPER-PARAMETER
A.1 SVF TRAINING
We obtain the expert vectors as the base components in Transformer by training the SVF fine- tunes with consistent recipe across the considered training tasks and language models. We divide each dataset to produce equal-sized training and validation splits. We then apply our RL-based approach. optimizing 0₂ with A using a learning rate of 2 x 10 with cosine decay, a batch size of 256, and gradient clipping. We employ early stopping and select the best À (the coefficient of the KL divergence term) based on validation performance. For the apply the SVF half of the layers to reduce memory usage
and Vision tasks experiments, while maintaining considerable performance improvement. During the training of LLAMA3-8B: INSTRUCT the vision language tasks, we apply a small negative reward (-0.1) for training stability.
A.2 LORA TRAINING
We follow community best practices for LoRA fine-tuning, applying to query and value projection layers with learning rates around 5 10- We set 200 total itera- tions with 256 global batch size for suffi- cient training. For feasible LoRA instruc- tion training, we collect solutions for all training tasks (GSM8K, MBPP, Arc- Arc-Easy, TextVQA) from official sources and ap- pend them to question prompts. Table g shows sample math problem used for LoRA fine-tuning. Despite extensive hy- perparameter tuning, we often observe test performance decay as discussed, which can be attributed to the small number of training samples and potential model requirements for instruction fine-tuning data (specifically, the highly detailed thinking process).
instruction
response
appropriately completes the request.
Natalia sold clips as many clips in May. How mony clips did Natalia sell attogether in April and May?
April, then
Notalia sold 48/2 = <c68/2-26os24 clips in May. Natalic sold 48+24 <c488 24--770772 clips altagether April and May. #*## 72
Figure 8: Sample problem and answer. Math data sample used for LoRA instruction fine-tuning, text in blue is the unmasked solution.
A.3 HYPER PARAMETERS
We present a summary of the hyperparameters used in our experiments Table 6 To optimize performance, we conducted sweeps across several hyperparameters and selected the most effective combination based on validation results. For SVF, our primary focus was on adjusting the KL coefficient to enhance training stability. In the case of LoRA, we concentrated sweeping the learning rate and maximum gradient clip norm to identify optimal settings.
A.4 FEW-SHOT ADAPTATION
As described in the main text, few-shot adaptation approach entails producing entirely new z' E 042z for each W by linearly interpolating between the learned SVF vectors, each weighted by the coefficients a E R We employ CEM search for or 's based on the performance on the few-shot prompts, which are specifically held out from the rest of the test prompts and used obtain the elite set each iteration. In the case of multiple sample solutions obtaining the same score on these held-out samples, we break ties by choosing the sample solution with the highest average log-likelihood across the tokens of its generated correct answers.
In all of our main experiments, we reserve only 10 samples of data for self-adaptation and perform up to 100 CEM iterations. For each setting, we consider both per-layer and per- per-vector vector adaptation, where the latter strategy has the advantage of greatly simplifying search (as we only have a coefficients). Moreover, we experiment with both normalizing across the a of different tasks (such that their sum would be fixed to 1) or keeping them unconstrained. Due to the lack of validation set, simply report the performance attained by our best sample from these test configurations at the end of optimization, on the remaining unseen samples for each task.
14

---

Published as a conference paper at ICLR 2025
Table 6: Hyper-parameters used for SVF and LoRA training. We perform sweep on certain sensitive hyper-parameters across methods for fair comparison.
SVF Hyperparameters
Initial mean value of z
0.1
Initial variance value of
1 x 10-3
Global batch size
256
Learning rate
2 x 10-3
Clip max norm
1 10-3
KL coefficient À
0.0, 0.1, 0.2,
LoRA Hyperparameters
Rank
16
LoRA alpha
32
LoRA dropout
0.05
Global batch size
256
Learning rate
2
10-1, 5 x 10-1,2x 10-5, 50-5,5x 10-5, x x, 10-6. 5 10-6,
Clip max norm
1 10 1.0
Table Additional Comparison Experiment. Normalized scores are in the parentheses.
Method
GSMSK
MBPP-Pro
ARC-Fasy
LLAMAA-8B-INSTRICE
75.89 (1.00)
64.65 (1.00)
88.59 (1.00)
78.01 (1.03)
67.68 (1.05)
89.10 (1.01)
DORA
78.09 (1.03)
64.65 (1.00)
89.14 (1.01)
SVF(OOus)
79.15 (1.04)
66.67 (1.03)
89.56 (1.01)
Method
MATH
Humaneval
ARC-Challenge
LLAMA3-8BB LLLMA3-8B-INSTERR SB-INSTRUCT
24.54 (1.00)
60.98 (1.00)
80.63 (1.00)
IA3
23.64 (0.96)
59.76 (0.98)
81.57 (1.01)
DORA
24.44 (0.99)
52.44 (0.86)
81.14 (1.01)
Transformer" (Prompt)
25.22 (1.03)
61.59 (1.01)
81.74 (1.01)
Transformer" (Cls-expert)
25.18 (1.03)
62.80 (1.03)
81.37 (1.01)
Transformer" (Few-shot)
25.47 (1.04)
62.99 (1.03)
82.61 (1.02)
B ADDITIONAL RESULTS
B. BASELINE COMPARISON TO MORE PEFT METHODS
We conduct additional comparison studies against more parameter-efficient fine-tuning methods, including IA IA3Liu et al. (2022), DORA. Liu al. (2024).
As Table 7 shows, SVF still outperforms other methods and shows promising generalized perfor- mance.
B.2 IMPACT FROM NUMBER OF FEW-SHOTS
We investigate the relationship between the number of samples available for few- shot adaptation and downstream perfor- mance. Our analysis focused on the test task where LLAMA3-8B-INSTRRU demonstrates the highest baseline perfor- mance, to prevent the potential for null signal in our CEM-based search.
Table Few-shot adaptation scaling on the Are- Challenge task. Performance varies with number of examples.
Method
Transformer"
100 steps
1900 steps
(1.63
80.63 (1.00)
80.63 (1.00)
aduptation
(1.02)
(1.01)
(0.9%)
5-shot adaplation
82.38 (1.02)
80.89 (1.00)
79.41 (0.98)
10 shot aduptation
82.61 (1.92)
82.00
79.78
20-shot adaptation
(1.02)
81.40 81.40((1.0) (1.01)
79.61 (0.99)
As Table 8 shows, substantial benefits of our few-shot strategy are evident with as few as 3 to test samples. Moreover, performance appears plateau beyond 10 samples, underscoring how essential and inherently regularized SVF pa-
15

---

Published as a conference paper at ICLR 2025
rameterization effectively complements self coff-adaptation. This efficiency enables optimal use of data to enhance understanding the test task.
For completeness, have also conducted experiments with identical settings IA3 (Liu et al. 2022], another method that leverages few-shot examples. All experiments were conducted with full batch size, learning rate of x 10-5, with 100 and 1000 training steps.
Our results indicate that the performance of IA on the unseen test tasks is inferior to CEM-based adaptation for all numbers of few shots considered. We note that in our experiment, we have to considerably limit the number of optimization steps to avoid overfitting the 500,000 parameters of IA on the few-shot samples. However, degree even after only 100 steps, as also validated by the model's perfect training accuracy on this extremely small dataset. This limitation of fine-tuning-bast based adaptation highlights the superior generalization capability of our CEM-base adaptation approach in Transformer".
believe overfitting might still be occurring to some
B.3 CROSS-MODEL SVF TRANSFER ON THE TRAINING TASKS
We provide complementary results to Table 3 in the main text, where we analyze the SVF cross- model transfer performance from training GSM8K, MBPP-pro, and ARC-Easy to consid- ered test tasks. In Table 9 we show the results the same transfer setting this time evaluating DESTRAL-7B-INSTRUCT-VOOO
on the same training tasks where the -LAMA3-8B-INSTRUCE SVF vectors were obtained from. Overall, we recognize a similar trend, albeit with less consistent im- provement from the original model (only in out of 3 tasks), but still much higher performance than the randomly shuffled baseline. These results further confirm that the canonical ordering of the SVF parameterization is key for cross-model transfer, highlighting once more its inherent suitability to empower self-adaptation.
Table 9: Cross-model z Vector Transfer. Results from transfering the SVF expert vectors trained on LLAMA3- -NSSRRCT to MISTRAL--B-INNTTRUCT- in the respective training tasks.
Method
GSM8K
MBPP-pro
ARC-Fasy
MIsTRALL-7B-INsTTRUTT--00 42.83 (1.00)
49.50 (1.00)
81.65 (1.00)
+ Llama SVF (ordered ox:)
42.61 (0.99)
48.48 (0.98)
81.78 (1.00)
Llama SVF (shuffled o;)
41.93 (0.98)
46.34 (0.94)
80.81 (0.99)
B.4 TRAINING CURVE OF LORA AND POLICY GRADIENT
Figure gives the learning curves for LoRA training on the GSM8K task.
Pelley
4.6
Figure 9: Training LoRA with policy gradient. The dashed line shows the performance of ---SSTRUU on the test split. LoRA collapses at the beginning of the training stage and fails to recover, leading to negative effects on test performance. We swept wide range of learn- ing rates (2 x 10 10-4.5 4.5 x 14-4 ...., ...,,,x 10-2, 2 x 10-2), and all learning curves were similar to the one presented.
16

---

Published as a conference paper at ICLR 2025
C PCA ON LLAMA3 AND MISTRAL
To investigate if the singular components that have the highest singular values are able to capture most of the information of weight matrix, we conducted Principle Component Analysis (PCA) on the weight matrices in LLAMA3-8B --INSRUCTT and JISTRAL-7B-INSTRUCT-VO ures [T]. In each figure, we plot the variance that captured by the top components across all the layers in each type of modules for weight matrix W mmxx
(see Fig-
EL Oi Σ
ratio
(m)))
Jj
Here, a's are the ordered (from largest smallest) singular values on the weight matrix W. It is easy to see from the figures that when = 256, less than 50% of the variance is captured by these top components on average. For the MLP layers, this fraction is even lower than 20%. On the other hand, the ranks adopted by LoRA-XS similar methods are much less than 256, resulting even more information loss and restrictions in their modeling power that relies mostly on these components.
pro)
pttj
16-
24
32
CATA
down
3.6
22-
Figure 10: PCA of LLAMA3-8B-INSTRE We show the ratio of the variance captured by the top singular components the y-axis, and the layer indices the x-axis. Except for the Query, Key and Value projection matrices, small values only capture a tiny fraction of variance in singular values the parameter matrices.
D EFFICIENCY CONSIDERATIONS AND IMPROVEMENTS
Our CEM-based adaptation method involves running inference on a small number of sam-
Table 10: 3-shot and light variants Performance with different inference-time adaptation budgets.
ples for each target task (up to 10 in our ex- periments). In typical configuration, this pro- cess is relatively efficient: for example, our CEM-light approach (3-shot with 10 genera- tions) completes the ARC-Challenge task in ap- proximately 11 minutes. As shown in Table I0 this lighter setup reduces the total number of samples to just 3% the original setting while still delivering substantial performance improvements over the base model.
Method
ARC-Challenge
LLAMA3-SB-LNSTARE
80.63 (1.00)
CEM 10 shot adaptation
82.61 (1.02)
CEM 3-shot (30% of prompts)
82.18 (1.02)
CEM light (3% of prompts)
82.08 (1.02)
17

---

Published as a conference paper at ICLR 2025
k
16-
42-
22-
Jown. 5m0
36
Figure 11: PCA of MISTRAL- DA-NALL--T-NST We show the ratio of the variance captured by the top singular components the y-axis, and the layer indices the x-axis. Except for the Query, Key and Value projection matrices, small values only capture tiny fraction of variance in singular values in the parameter matrices.
We acknowledge that CEM-based adaptation entails a trade-off between one-time overhead it spends on searching the optimal combination weights for the SVF-tune vectors and performance. Increasing the number of few-shot samples the number of generations can yield higher performance, but this comes at the cost of additional computational overhead. However, it is important to note that this adaptation cost one-time overhead per task. The cost-per-prompt diminishes significantly when applied to tasks with large number of prompts.
Moreover, practical scenarios, CEM-based adaptation offers better scalability than few-shot prompting methods, which require increasing the length of every prompt, leading to much worse scaling as task sizes grow. In contrast, our method focuses on determining optimal expert vector combinations efficiently and avoids repetitive inference-time costs. However, note that the over- head might be significant for tasks with very few prompts. Thus, the other adaptations methods might be more appropriate for these particular settings.
We also highlight two immediate directions for improving efficiency:
1. Reducing the number of few-shot samples: As shown in our ablation study in Ap- pendix [.22] substantial benefits can be seen even in the 3-shot setting, which requires only evaluation of only 30% of the number of prompts per generation.
2. Reducing the number of maximum generations: In the explored settings. the CEM param- eters tend to converge early on, being very close to the final values after a much lower number of generations than 100.
Finally, in this work only considered CEM due to its simplicity, there exist several different evolution algorithms empirically showing better efficiency and convergence properties that we hope will be explored in future research.
18


Reviews:
[{"review_id": "CiIzm7HtKN", "reviewer": "Reviewer_FCnJ", "summary": "To address the challenge that traditional fine-tuning methods are often computationally intensive and static in their ability to handle diverse tasks, this paper proposes a self-adaptation framework that adapts LLMs by selectively adjusting the singular components of their weight matrices for unseen tasks at inference time. In the training stage for each task, it trains a vector to replace the singular values for each weight matrix. In the inference stage, it first extracts the task-related “singular vectors”, then makes inference with selected task-related “singular vectors”. Compared with Lora, experiments show the effectiveness of the proposed method.", "questions": "1.\tFor evaluation, is one model with three sets of “singular vectors” utilized for all three tasks(i.e., MATH, Humaneval, ARC-Challenge) for proposed methods? For Lora, three models are selected for three tasks, respectively?\n2.\tIt looks that the proposed method has the ability to mitigate catastrophic forgetting problem in continual learning of LLM. The author may also consider exploring this direction to underline the contribution of this work further. (This is just a suggestion.)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.\tThe proposed method of finetuning LLMs within eigenspaces is highly parameter efficient, as it only needs to train r = min(m, n) parameters for each weight matrix rather than $(m+n)\\times r^\\prime$ learnable parameters in Lora. \n2.\tThe proposed method has the ability to mitigate catastrophic forgetting problem in continual learning of LLM, since it can adapt LLMs for unseen tasks at inference time by adjusting the singular components of their weight matrices.", "weaknesses": "1.\tTo demonstrate the effectiveness in parameter efficient finetuning domain, the experimental results are not sufficient as the paper only compares with Lora, which suffers from overfitting in their experiments. How about other parameter-efficient finetuning baselines, like Prefix Tuning, Prompt Tuning, DoRA, BOFT?\n2.\tThe proposed method costs more time at inference time since it needs to make inference twice to get the final generated results. Particularly, for Few-shot adaptation, it needs to perform up to 100 CEM iterations, which costs significantly more time than normal inference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "olQXqXeR5k", "reviewer": "Reviewer_MTB2", "summary": "The paper presents Transformer2, a self-adaptive framework that enhances the adaptability of LLMs for unseen tasks in real-time. It introduces a parameter-efficient approach by adjusting only the singular components of their weight matrices. The inference is a two-pass mechanism which first identifies task properties, and then dynamically combines task-specific expert vectors. The expert vector is trained by reinforcement learning to generate task-specific response. Proposed method is shown to outperform existing parameter-efficient tuning methods like LoRA, demonstrating efficiency and versatility across different LLM architectures and multimodal tasks.", "questions": "Refer to the weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Transformer2 introduces a parameter-efficient adaptation method that selectively tunes only singular components, reducing computational demands compared to full fine-tuning approaches.\n- The use of a two-pass mechanism to dynamically adapt model responses enhances the real-time adaptability of LLMs for unseen tasks.\n- Training expert vectors with reinforcement learning enables the model to tailor its performance, leading to improvements in task-specific outputs.\n- The framework is adaptable to various LLM architectures and applicable to vision-language tasks, showcasing broad utility.", "weaknesses": "- While the paper proposes novel methods, the description of how these components interact and integrate in practice could be clearer.\n- While the use of reinforcement learning to train expert vectors is an innovative aspect of the framework, it introduces significant computational overhead and complexity. This can potentially undermine the paper's claims of efficiency and raises the question of whether these additional costs were considered when comparing the method to others like LoRA. The experiments should include a detailed breakdown of this computational expense to provide a clearer picture of the method's practical applicability and efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EDauDQdJnl", "reviewer": "Reviewer_cbgq", "summary": "This paper proposes a self-adaptive framework called Transformer$^2$ that learns and adapts external expert modules to downstream tasks in real-time. These modules can be used to augment LLMs for specific abilities and provide more flexibility than traditional post-training approaches that directly update the LLM weights. At training time, unlike existing methods that learn the expert modules via LoRA, this paper proposes a method called SVF that tunes only the singular values of a weight matrix to reduce storage and computational costs. At test time, the Transformer$^2$ framework requires two forward passes, one to gather test-task-related signals to select and combine the expert vectors, and one to actually generate the answer with the adapted LLM. Experiments on multiple open-source LLMs and datasets show that the proposed framework outperforms existing self-adaptive methods with reduced adaptation costs.", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of tuning singular values is novel and beneficial for reducing storage and computational costs.\n2. The idea of learning a dispatch module using the LLM itself that determines which expert modules should be used at adaptation time is interesting .\n3. Extensive experiment results show the efficacy of the proposed methods.\n4. Good presentation.", "weaknesses": "1. I think ablation studies in B.1 should be moved to main text because it provides a better understanding of how each component of Transformer$^2$ contributes to the performance gain. Also, is it possible to provide results for LoRA + next-token prediction in Table 5? Ideally with both attention and attention+MLP. Currently it's hard to conclude that SVF with policy gradient does not sacrifice fine-tuning performance.\n2. Scaling to a large number of abilities or specialized domains might be an issue for the few-shot adaptation scheme that uses CEM to search?\n3. How should we reconcile the test time performance on coding in Figure 4? The performance seems to be decreasing over time and worse than non-adapted model\n\nOther minor issues:\n1. Line 74: It's better to add the full name here because the abbreviation SVF appears for the first time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1i6JwomLzc", "reviewer": "Reviewer_Kvfw", "summary": "This work proposes a framework called $\\text{Transformer}^2$ to enable self-adaptation in LLMs.\nIn the initial stage of $\\text{Transformer}^2$, task-specific components of the pre-trained weights are determined by leveraging SVF,  a noverl parameter-efficient fine-tuning approach.\nSVF is based on learning weights for singular values after SVD decomposition on pre-trained weights.\nIn the second stage, $\\text{Transformer}^2$ performs test-time adaptation by leveraging a few samples of test data to infer how to select or mix the task-specific components.", "questions": "- Why is $\\text{Transformer}^2$ bound to SVF as PEFT method, can it not be used with other ones, e.g. LoRA, IA3, etc.?\n- Figure 7 shows approximately uniform weighting of components works well for Llama-3, how does uniform weighting work in general? This would be an insightful ablation study.\n- line 252: \"we find the regularization properties of SVF avoid many of the failure modes of prior less-constrained parameterizations\" - What failure modes of RL in particular does SVF remedy?\n- line 253: \"Thus, combining these complementary components effectively enables us to avoid relying on expensive fine-tuning procedures with large hand-designed datasets as proxies, and directly maximize task performance end-to-end.\" - Why is this only a benefit of SVF, as it can be applied to LoRA as well? \n- How expensive is the reward labelling prior to RL finetuning?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- For the most part the paper is easy to follow and well written.\n\n- SVF reduces the parameter-count compared to standard LoRA.\n\n- $\\text{Transformer}^2$ can adapt quickly to the holdout test tasks.\n\n- Compelling results for cross-model compatibility of task-specific components.", "weaknesses": "- **Weak and missing baseline results**\n\nMost results for LoRA in Table 1 show worse results than the base model, how is this possible? In [1], for example, LoRA even outperforms full fine-tuning on MBPP. A performance drop after fine-tuning indicates that something went wrong with LoRA fine-tuning, maybe suboptimal hyperparameters?  [2] demonstrates importance of learning rate (higher is often better for LoRA), [3] demonstrates sensitivity w.r.t. scaling parameter, maybe there is an issue there? Also, there is no additional info given on the rank used for LoRA and to what weight matrices LoRA is applied to, smaller rank may perform better on smaller datasets.\nThere have also been advances for LoRA in terms of initialization [1,4,5] that have shown to improve upon LoRA and should be considered.\n\nAnother missing baseline is IA3 [6] which is conceptually very similar to SVF as it inserts newly trainable scaling vectors in the model.\nEspecially in scarce data regime, IA3 has shown improvements over LoRA, therefore it should be included in the paper.\nFinally, the MoE approaches are similar to SVF and there are extensions for LoRA [7], but those are not compared to either.\n\n- **Fair Comparisons**\n\nThere is no fair baseline comparison for $\\text{Transformer}^2$, as it leverages test samples.\nFor example, $\\text{Transformer}^2$ (Cls-expert) and $\\text{Transformer}^2$ (Prompt) leverage two forward passes, one to determine the class and another one for generation. Can the authors clarify if this additional forward pass is required for each test sample?\n\n $\\text{Transformer}^2$ (Few-shot) even leverages few-shot exemplars of the test set for determining the weighting factors $\\boldsymbol{\\alpha}_k$.\nTherefore a fair comparison would require the baseline to use an equal amount of compute as the $\\text{Transformer}^2$ variants during inference.\nTo this end, I would suggest fine-tuning the pre-trained IA3 vectors on the few-shot exemplars.\n\nAnother missing comparison is to exchange SVF for the second stage with another PEFT method, e.g. LoRA, or IA3, that were trained via SFT.\nThis would give important insights no the necessity of SVF in combination with RL as PEFT method for the first stage.\n\n- **Limitations**\n\nThe authors claim that $\\text{Transformer}^2$ enables self-adaptive LLMs, even though a lot of components are handcrafted in their framework, i.e. reward function, number of pre-defined classes, training on a number of datasets associated with pre-defined classes, determining of weighting factors by repeated sampling. This limits applicability of the proposed framework and also questions to what extent $\\text{Transformer}^2$ is truly self-adaptive. Therefore, I would suggest to caveat the claims on self-adaptiveness and discuss the mentioned limitations.\n\n$\\text{Transformer}^2$ is limited to information the LLM has obtained during pre-training, i.e. it will not work on tasks that have not been observed during pre-training. While the authors have mentioned in the conclusion that model-merging techniques may be used to overcome this issue, this is impracticable as it requires searching and applying SVF on other pre-trained models until a suitable task-specific component is found. This is a fundamental limitation of $\\text{Transformer}^2$ which makes it less practical.\nAnother limitation that should be made more explicit is the dependence on architecture for cross-model experiments - if architecture differs this is not possible anymore.\n\n* **Overclaiming:**\n  * Claims on overfitting: \n    * line 77: \"our approach mitigates the risk of overfitting\" - figure 4 learning curves show otherwise\n    * line 339: \"highlighting the particular effectiveness of our proposed fine-tuning recipe in narrow spe-\ncialized applications.\" - the smallest dataset is MBPP where SVF only performs better on Mistral\n  * line 332: \"SVF effectively fine-tunes to surpass the base performance.\" - figure 4 curves on coding and reasoning show otherwise\n  * line 350: \"all of our $\\text{Transformer}^2$ adaptation strategies demonstrate improvements across all tasks for both LLAMA 3-8B-INSTRUCT and LLAMA 3-70B-INSTRUCT base models\" - not true for LLAMA 3-70B-INSTRUCT on Math\n  * line 355: \"LoRA’s parameterization and optimization might be particularly sensitive to overfitting, especially when trained with the smaller GSM8K and MBPP-Pro datasets\" - GSM8K is an order of magnitude larger than MBPP\n  * line 364: \"$\\text{Transformer}^2$ with few-shot self-adaptation is always the highest-scoring method, providing notable improvements across all tested settings\" - not true for LLAMA 3-70B-INSTRUCT on Math\n  * line 408: \"$\\text{Transformer}^2$ could provide foundation models with new means to continually improve performance when deployed in lifelong settings.\" - not true if new task is not represented in the base models' weights\n  * line 418: \"Furthermore, they also show that using the classification expert consistently provides higher classification accuracy than vanilla prompt engineering.\" - not shown for LLAMA 3-70B-INSTRUCT\n  * line 476: \"we first proposed SVF, offering superior performance than prior fine-tuning recipes, together with reduced costs, high compositionality, and overfitting regularization\" - there is evidence in figure 4 for overfitting\n\nI recommend rephrasing the above examples which, as far as I can see, are cases of overclaiming.\n\n**References:**\n\n[1] PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models, Meng et al., NeurIPS 2024\n\n[2] LoRA+: Efficient Low Rank Adaptation of Large Models, Hayou et al., ICML 2024\n\n[3] A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA, Kalajdzievski, arXiv 2023\n\n[4] LoRA-GA: Low-Rank Adaptation with Gradient Approximation, Wang et al., arXiv 2024\n\n[5] One Initialization to Rule them All: Fine-tuning via Explained Variance Adaptation, Paischer et al., arXiv 2024\n\n[6] Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning, Liu et al., NeurIPS 2022\n\n[7] Mixture-of-LoRAs: An Efficient Multitask Tuning for Large Language Models, Feng et al., COLING 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)