From Weight-Based to State-Based Fine-Tuning: Further Memory Reduction on LoRA with Parallel Control
Authors: Chi Zhang
REN Lianhai
Jingpu Cheng
Qianxiao Li
Venue: ICML 2025 oral (2025)
Keywords: Low-Rank Adaptation
Parallel Control
Parameter-Efficient Fine-Tuning
Primary area: deep_learning->algorithms
TL;DR: We further reduce the memory cost of LoRA methods by introducing parallel control, which allows training 7B/8B models on GPUs like 3090 or 4090.

Abstract: The LoRA method has achieved notable success in reducing GPU memory usage by applying low-rank updates to weight matrices. Yet, one simple question remains: can we push this reduction even further? Furthermore, is it possible to achieve this while improving performance and reducing computation time? Answering these questions requires moving beyond the conventional weight-centric approach. In this paper, we present a state-based fine-tuning framework that shifts the focus from weight adaptation to optimizing forward states, with LoRA acting as a special example. Specifically, state-based tuning introduces parameterized perturbations to the states within the computational graph, allowing us to control states across an entire residual block. A key advantage of this approach is the potential to avoid storing large intermediate states in models like transformers. Empirical results across multiple architectures—including ViT, RoBERTa, LLaMA2-7B, and LLaMA3-8B—show that our method further reduces memory consumption and computation time while simultaneously improving performance. Moreover, as a result of memory reduction, we explore the feasibility to train 7B/8B models on consumer-level GPUs like Nvidia 3090, without model quantization. The code is available at an anonymous GitHub repository

From Weight-Based to State-Based Fine- Tuning:: Further Memory Reduction on LoRA with Parallel Control
Chi Zhang'
Lianhai Ren
Jingpu Cheng' Qianxiao Li
{czhang24, qianxiao) @nus.edu.sg;
(lianhairen, chengiigguj uuuuus.edu
Abstract
as natural language processing and computer vision. At its core, LoRA leverages low-rank decomposition AW = AB to approximate weight updates with some low-rank matrices A and B, allowing only a small subset of parameters to be trained. Such a design not only maximally preserves the performance of full-tuning but also makes training feasible hardware with limited resources.
The LoRA method has achieved notable success in reducing GPU memory usage by applying low- rank updates to weight matrices. Yet, one simple question remains: can we push this reduction even further? Furthermore, is it possible to achieve this while reducing computation time and preserving performance? Answering these questions requires moving beyond the conventional weight-centrie approach. In this paper, we present a state-based fine-tuning framework that shifts the focus from weight adaptation to optimizing forward states, with LoRA acting as a special example. Specif- ically, state-based tuning introduces parameter- ized perturbations to the states within the com- putational graph, allowing us to control states across entire residual block. A key advantage of this approach is the potential to avoid storing large intermediate states models like transform- ers. Empirical results across multiple architec- tures inclu-inc ViT, RoBERTa, LLaMA2-7B, and LLaMA3-8B show that our method further reduces memory consumption and computation time while preserving performance. As result of memory reduction, we explore the feasibility to train 7B/SB models on consumer-level GPUs like Nvidia 3090, without model quantization. The code is available here.
Yet, such core explanation is challenged by recent fine- tuning approaches (Chen et al., 2022; Zhang et al., 2024b), in which the update rule often takes the form x' f (W,z) + g(AW,z): Here, f represents the pretrained dynamics while g introduces a nonlinear perturbation based the fine-tuned parameters. The presence of the nonlin- ear function g precludes straightforward interpretation of AW as simple low-rank modification to the weight matrix W. As result, this departure from linearity challenges the foundational view of LoRA as merely performing low-rank weight updates. This raises a natural question: if low-rank decomposition no longer fully captures the nature of these adaptations, what alternative frameworks might offer more comprehensive understanding?
Another significant challenge lies in further reducing the GPU memory consumption of LoRA. While approaches like VeRA (Kopiczko et al., 2023), BinaryLoRA (Zhang et al., 2024a), and DoRA (Liu et al., 2024) have made strides by using bbaed, binary LoRAs or halving rank, memory reduction often remains limited. For example, halving the rank in DoRA reduces parameters by nearly 50%, but re- sults in only a limited decrease of 0.256 GB GPU memory on an 8B model. A closer analysis reveals that the primary bottleneck is no longer the storage of parameters, but rather the memory of maintaining forward states. This limitation becomes particularly critical as 7B/8B models become in- creasingly common, yet training them remains infeasible consumer-level GPUs like Nvidia 3090/4090. This shift underscores the need for strategies to address the memory demands associated with activations during the forward pass, enabling more memory-efficied fine-tuning techniques.
1. Introduction
Low-Rank Adaptation (LoRA) (Hu et al., 2021) has emerged as an effective algorithm for fine-tuning pre-trained models, demonstrating notable success across domains such
'Department of Mathematics, National University of Singa- pore 'Institute for Functional Intelligent Materials, National Uni- versity of Singapore. Correspondence Qianxiao <qianx- i edu.edu.ss>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Addressing these challenges requires moving beyond the traditional weight-centric paradigm. While LoRA and its

---

From Weight-Based to State-Based Fine-Tuning: Further Memory Reduction on LoRA with Parallel Control
variants focus on adjusting specific layer weights within this framework, achieving further memory reductions calls for new theoretical insights and the development of alternative fine-tuning frameworks. One promising direction emerges from recent work (Zhang et al., 2024b), which draws con- nections between fine-tuning and control theory. In control theory, directly modifying the core parameters of a system is often impractical. For example, directly adjusting the motor settings of a robot can be difficult; instead, it is typically more effective to regulate system states-such as position or velocity-through feedback control mechanisms (Franklin et al., 2002; Kirk, 2004).
work and greater flexibility in designing parameter-effiieet algorithms. (2) Building on this framework, we demon- strate that memory usage can be further reduced by applying low-rank perturbations to entire components, such as MLP and attention blocks, thus avoiding the need to store large intermediate states. (3) Empirical results show that our ap- proach achieves superior performance while simultaneously reducing GPU memory usage and computation time. (4) We explore the possibility of training large models consumer- level GPUs, allowing more researcherssppracttionne to ben- efit from PEFT studies.
The goal of this paper is to introduce a generalized state- based fine-tuning framework that facilitates direct adjust- ment of network states. By representing the neural network as directed acyelic graph (DAG), this framework enables manipulation of states connected by arbitrary edges, rather than being limited to weight parameters alone. Further- more, the state-based approach supports the adjustment of states that are not necessarily immediate neighbors and can account for hierarchical dependencies within the network. This flexibility allows for more sophisticated control strate- gies over the network's behavior, extending beyond the conventional reliance on low-rank matrix updates.
2. Related Works
Parameter-Efficient Fine-Tuning Owing to the large pa- rameter size of modern transformer-based models (Vaswani et al., 2017; Dosovitskiy et al., 2020), fully tuning these models becomes challenging. Early works (Oquab et al., 2014; Alain, 2016) in transfer learning focused on linear probing, where only the final layer is fine-tuned. These approaches have been extended to selectively tuning the middle or bottom layers in (Lee et al., 2022; Lodha et al., 2023; Kaplun et al., 2023; Nagae et al., 2022). Recent parameter-efficient fine-tuning (PEFT) methods propose to inject new trainable parameters, while keeping the pre- trained model fixed. In particular, Adapter methods and their variants (Houlsby et al., 2019; Rebuffi et al., 2017; Karimi Mahabadi et al., 2021) employ bottleneck structures inserted after self-attention and feed-forward network lay- ers to achieve parameter efficiency. Prompt-base based methods such as Prompt Tuning (Lester al., 2021), P-Tuning (Liu et al., 2021), and mize trainable prompts rather than tuning the weights of the model. LoRA (Hu et al., 2021) is among the most popular PEFT methods. It leverages the idea of low-rank decom- position for the updated weights, significantly reducing the number of trainable parameters. Variants of LoRA (Zhang et al., 2023a; Kopiczko et al., 2023; Zhang et al., 2023b) further optimize efficiency by freezing or sharing the LoRA matrices, reducing memory requirements through quantiza- tion techniques (Dettmers et al., 2023), or using an adaptive strategy. A recent extension, Dora (Liu et al., 2024), decom- poses the magnitude and direction of the low-rank weight matrices, offering improved performance. However; LoRA and these variants remain weight-centric, in comparison to the state based tuning proposed in this paper.
Thanks to this new framework, we can design more efficient algorithms by treating entire components, such as MLP or attention blocks, as single units. A key advantage of this design is the reduction memory usage, as it circumvents the need of storing large intermediate states associated with individual layers or operations. Moreover, by leveraging low-rank matrices, we can reduce both the parameters and the states that need to be stored, resulting in highly memory- efficient algorithms.
(Li & Liang, 2021) opti-
The proposed algorithm outperforms traditional methods across a range of tasks, while using less GPU mem- ory and reducing computation time. These advantages demonstrated across several model architectures, in- cluding ViT (Dosovitskiy et al., 2020), RoBERTa (Liu, 2019), LLaMA2-7B (Touvron et al., 2023), and LLaMA3- 8B (Dubey et al., 2024). Additionally, by further reducing memory usage, we demonstrate the feasibility of training 7B/8B models on a Nvidia 3090 GPU with 24GB mem- ory, without resorting model quantization. Our results reveal that, even with mowory GPUs, the proposed approach achieves performance comparable to setups with higher memory capacity. This makes the approach acces- sible to a wide range of researchers and practitioners with only consumer-level GPUs.
Control in Machine Learning Control theory (Kirk, 2004) focuses on managing dynamical systems by devel- oping models or algorithms that govern system inputs to drive the system toward desired states. From a dynamical perspective, residual neural networks can be viewed as dy- namical systems, with the model representing the system and the layer inputs/outputs serving as the states (F, 2017:
In summary, our contributions are four-fold: (1) We in- troduce a btate- fine-tuning approach that shifts the focus from weight-centric adaptation to optimizing the model's forward states, providing more general frame-
2

---

From Weight-Based to State-Based Fine-Tuning: Further Memory Reduction on LoRA with Parallel Control
Li et al., 2017). This connection between deep learning and control theory has inspired numerous advancements in various domains, including approximation theory (Tabuada & Gharesifard, 2020; Li et al., 2022; Cheng et al., 2023), network structure design (Haber & Ruthotto, 2017; Lu et al., 2017; Nguyen et al., 2024), and optimizer development (Li & Hao, 2018; Benning et al., 2019). The parameter-efficient fine-tuning task can naturally be formulated as a controlled dynamical system (Zhang al., 2024b), where the original model represents the system, and controllers are designed to adapt the system to downstream tasks. Our work provides a more general state-based tuning framework, where this work can be considered as a special case of choosing the control function g-
F(+1 = Ie + J4 (Warr) AW
where fi represents nonlinear functions, such as ReLU or GeLU. Notably, the latter formulation cannot be interpreted as weight adjustment like LoRA, as the presence of nonlin- ear function makes it difficult to explain through the lens of weight decomposition. This limitation highlights the need for a more general fine-tuning framework that can accommo- date broader range of strategies, including those involving nonlinear transformations.
3.2. Inspirations from Control Theory
To proceed, we need to move beyond these classical weight-centric methodologies. In particular, the pioneer- ing work (Zhang et al., 2024b) has demonstrated that Eq (3) align closely with the classical control problem. Consider the continuous- -tmm ordinary differential equation (ODE) with affine control (Franklin et al., 2002):
3. A State-Based Fine- Tunnig Framework
3.1. Weight-Based Fine- The-Tuning (Weight-FT)
We begin by introducing the Low-Rank Adaptation (LoRA) method (Hu et al., 2021), which facilitates parameter- efficient fine-tuning through low-rank decomposition. For a pretrained weight-mante matrix Wo E IR LoRA models the weight update AW E Rdxk with a low-rank decomposition AW- AB, where A E ıddr and B e Ig represent two low-rank matrices with rank min(d, k). Consequently, the fine-tuned weight W' becomes:
L Sa ((()))))) + G())(().
(4)
In state-feedback control systems, the control signal u(t) is typically defined as a function of the state s(t) through a feedback gain matrix K(t), such that u(t) ((t)=(t). Substituting m() into the original system gives:
i fe (W (t)x(t)) + G(t)K(t)x(().
(5)
The central connection lies in the role of the additive terms: the control term G(t) (t)xxt) in the continuous system (5) modifies the state trajectory by applying a control matrix G(() and feedback gain (().. Analogously, Eq (3) can be interpreted as discretized extension of Eq (5), where AWi functions as a control-inspired mechanism that in- fluences state updates.
W' = Wo + AW = Wo + AB,
(1)
where wo remains fixed during the fine-tuning process, and only the low rank matrices A, B are updated.
Following LoRA, many PEFT algorithms further explore and extend such a weight decomposition mechanism, in- cluding VeRA (Kopiczko et al., 2023) and DoRA (Liu et al., 2024) to utilize alternative low-rank structures, EVA (Pais- cher et al., 2024) to introduce data-driven initialization, and rsLoRA (Kalajdzievski, 2023) to stabilize LoRA.
3.3. State-Based Fine- Tine-Tuning (State-FT)
The key insight from the above connection is that these up- dates primarily target the system's 's states {1z],. rather than the parameters governing them. Therefore, in this part, we propose state-based fine-tuning (State-FT) framework that shifts the focus from parameter updates to directly influenc- ing the system's 's states.
In broader terms, these approaches can be categorized as weight-based fine-tuning (Weight-FT) algorithms, as they work on modifying the model's 'ss weight matrices to approxi- mate the low-rank changes during the fine-tuning process. The fundamental hypothesis behind these weight-FT algo- rithms is that fine-tuning updates tend to have a low "intrin- sic rank" (Aghajanyan et al., 2020; Hu et al., 2021). Conse- quently, low-rank matrices can be employed to efficiently adjust the pretrained weights.
Formally, consider a neural network as a directed acyclic graph G = (V, where each node e e V represents a computational state, and each directed edge (u,v) E E represents a transformation applied to the state at node u, producing the state at node
But modern fine-tuning strategies (Chen et al., 2022; Zhang et al., 2024b) have introduced alternative methods, many of which go beyond this weight-FT paradigm. For instance, a pretrained layer can be fine-tuned using either of the follow- ing formulations:
In particular, the computation the edge (u, v) E is defined as:
=== (zw; Ww..),
where WT.. represents the weight matrix for the transfor- mation along edge (u, u), and fo is the function applied, such as a linear transformation or nonlinear activation.
T++1 = zt + fc ((W, + AW.)..)
3

---

From Weight-Based to State-Based Fine-Tuning: Further Memory Reduction LoRA with Parallel Control
State-based tuning involves directly modifying the inter- mediate states Zh in the graph. For example, if an edge (u, v) E is selected for fine-tuning, then the updated state is defined as:
More General Choice of J and Øe More generally, the function f can involve more complex transformations, such as those defined by multiple layers non-linear map- pings that go beyond simple weight matrices. Similarly, the control function De can also be more intricate, potentially incorporating more layers, non-linearities, or even dynamic control mechanisms. For example, assume I consists of two linear layers followed by a nono-llneee non-linearity:
y = Σ TE (E)(
Axy
where A(t) = {ū ū is an ancestor of v) denotes the an- cestors of node v and Art represents the adjustment from the a state Tu. This adjustment can be parameterized as:
S (W Tu) = 02 (W2 (01 (W1x, + bi)) b2),
and keep the control function D as low-rank matrices in Eq (7). As such, we are able control multiple layers with parameter-efficeent control strategy. This general- ization allows for more flexibility in the state-based tuning process, enabling it to model more complex, hierarchical transformations of states during fine-tuning.
Ar 9e (Mu-ou
where -h is the control function, and Mu-r is learnable control matrix to modify the states.
The full update for Ty then becomes:
1/1
Σ (zs: Wa-r) + Te (Mu-,, (E)(())
4. Further Memory Reduction on LoRA with State-FT
For simplicity, we will use J in place of f whenever confusion arises in the following parts.
The flexibility of choosing arbitrary (u, v) E E' within the state-FT framework allows for more general and versatile approaches to model adaptation. For example, we may allow the state F, receive multiple updates Ary, to use weight differences T. Tu. akin classical PID control (Aström Hägglund, 2006). In this section, we present an example to design new garameter-effiieen tuning algorithms, primarily aligned with the PEFT framework, to further reduce LoRA's memory requirements.
The above state-based differs from weight-based based approaches by focusing on the di- rect manipulation intermediate states rather than adjusting the model's weight matrices. While LoRA uses low-rank matrices to update weights, state-based fine-tuning involves modifying state pair (r). This shift allows for more flexible and granular control over the model's behavior, as the state dynamics are influenced by control functions on the system's 's states. Additionally, state-based fine-tuning can incorporate more complex, non-linear transformations, unlike LoRA, which is limited to linear perturbations of the weight matrices. As result, the state-based framework provides a broader range of adaptation strategies.
strategy fundamentally
4.1. State-FT on Complete Residual Block
Modern neural networks are commonly composed of multi- ple blocks, each structured with a residual connection (He et al., 2016). To align with this paradigm, we consider a specific case of the State-FT where layers connected by a shortcut (or residual connection) are grouped into one com- putational block f" By doing so, we are able leverage the inherent structure of these networks and preserve the integrity of the residual connections.
Revisiting LoRA Special Case We demonstrate that LoRA can be viewed as special case of the proposed frame- work by choosing specific forms for f and g. Specifically, LoRA chooses specific edges (u, 1) E E with a linear layer,
Formally, for block of layers connected by a shortcut, let the input to the block be Va. and the output be Z,. The computation for the block can be expressed as:
f W u-- W Wo-
such as the query (Q), key (K), and value (V) blocks. The control function 9d is defined as low-rank update on the weight matrix:
T S (Zui (W...3) + Iu..
where (Wu-r) represents the collection of weight matri- ces for all layers within the block and S aggregates the transformations applied within the block.
9h (u-or; D.)) == Au-ru
where Ag- and Bu-. learnable low-rank matrices, used to update the pre-existing weight matrix W.-.. As such, from the perspective of State-FT, the LoRA algorithm can be seen as a special case in which the selected edges (u, v) are restricted to direct neighbors associated with lin- ear layers.
In the State-FT framework, the updated state for the block incorporates both the transformations and state adjustment term, defined as:
1.y Fu + Folsai (Wu-wr; Ma-ref),

---

From Weight-Based to State-Based Fine- Tinennnnig:: Further Memory Reduction LoRA with Parallel Control
LaRA
Deshae
Mermary (8R))
PNN
Contro)
FEN
Figure 1: Figure (a) illustrates the computational graph in LoRA, where specific edges (Q, K, V) with linear trans- formations are selected for tuning. Figure (b) introduces the control approach, where a new edge (in green color) across the FFN block is introduced. Figure (c) extends this control to both the attention and FFN blocks, resulting in the double-control method.
FEN
Figure 2: Memory consumption analysis for LoRA (top) and Parallel Control (bottom) on two MLP layers. In LoRA, the dominant memory usage comes from the intermediate state 2/. By treating the MLP layers as single unit and bypassing the intermediate states, the Parallel Control sig- nificantly reduces memory requirements.
where F.).) is composite functions defined as:
Fol (Wu-er My (Fui Wa-v) 9 (Ma))
tuning. Consider a typical fine-tuning scenario with a batch size of 16, a token length of 1024, and a feature dimension of 4096. Thanks to the foundational LoRA framework, the trainable parameters A, B, and their gradients require only 5 MB and 15 MB of GPU memory, respectively. The dom- inant memory consumption now comes from the forward states 2ú and the model weights W.. Specifically, the inter- mediate state x contributes the largest portion of memory usage (65.24%), due to the large feature size (16,384 per token).
Here De (M-ss; R.) is the control function that modifies the state using the control matrix Mn--y. The choice of g can significantly influence the behavior and adaptability of the model. For instance, -. can incorporate more complex architectures, such multi-layer perceptrons (MLPs) or non-linear layers, to act as universal function approxima- tors. Alternatively, 9d can be designed as a simple low-rank matrix transformation to reduce trainable parameters and memory footprint.
Consider the transformer architecture. We are now free to pick the starting and ending state of the residual block as an edge to tune. One option is to choose the starting and ending nodes of the feed-forward network (FFN) block, which typically includes a LayerNorm (LN) (Ba, 2016) and multiple linear layers. We refer to this method as the "Parallel Control" approach in Figure 1(b). Alternatively, we can extend this strategy to fine-tune both the attention block and the FFN block within the residual structure, leading to what term the "Double Control" approach. Note that this differs from classical LoRA, which selects specific edges with linear transformations to adjust (e.g., Q K, V in Figure (a)).
Storing this intermediate state is essential in the LoRA algo- rithm, as it is needed for the later gradient computation of Ai w.r.t. the final loss E:
de
Ox7
d[ai]MM oA.
(8)
dA
dxy c..
where Majla 2.
ae would require x!. As such, computing oΛद the system to store I! in the forward pass. Similarly, the state x! needs to be stored, although its size is much smaller.
In contrast, when multiple MLP layers are treated as sin- gle unit, this large intermediate state æ can be released in memory. This is achieved by using the parallel scheme in Figure 2 (bottom left), which bypasses this intermediate state. Consequently, only the comparatively smaller state x needs to be stored, resulting in substantial reduction in GPU memory consumption.
4.2. Reducing Memory by Relcasing Intermediate States
A fundamental question is why we aim to compress the residual block into a single computational unit. To answer this, Figure presents a simple example involving two MLP layers to illustrate GPU memory consumption during fine-
The above example demonstrates why compressing multiple
5

---

From Weight-Based to State-Based Fine-Tuning: Further Memory Reduction on LoRA with Parallel Control
layers into one unit f could potentially save GPU memory. Note the reduction becomes more significant as the batch size increases, the feature length expands, or multiple lin- ear layers are stacked together. In such cases, the memory savings provided by the proposed approach are more pro- nounced, enabling the system to handle larger workloads while keeping memory consumption in check.
such that for all xi E Rdo,
f(xo) = f(x) + Mzo.
(12)
This result implies that in the case of deep linear networks, the parallel control method can always achieve the same expressive power as LoRA, provided they share the same total rank. Consequently, for linear models, it is always safe to use parallel control without loss of expressive capability.
4.3. Reducing Additional Parameter Numbers
In addition to reducing the memory, we show the proposed control mechanism can also reduce the overall parameters or adopt a larger rank with the same parameter budget.
For non-linear blocks, the following result offers insights into the potential advantages of parallel control over a single state.
Theorem 4.2. Let FA,, and Gz, be the mappings AW, Lt+1 E Rd, defined in equations (2) and (3), respectively, If f(()) is singular, then the pushforward of the tangent space at O under Fz, forms a proper subspace of sg In contrast, the pushforward of the tangent space at 0 under G., always spans the entire space R.d, as long as De is non-zero.
Injecting new low-rank matrices would nevertheless intro- duce new parameters, as well as additional intermediate states. For a given state 22 E Rmxdd the standard FFN block of a transformer architecture first expands the feature to 4d and then reduces back to d. Applying the LoRA algorithm to these layers involves introducing low-rank matrices A and B with dimensions r and r 4d. Consequently, tuning the FFN block requires total of 10rd parameters.
This theorem suggests that when V ((+e) is singular, local perturbations in 2, under LoRA are restricted to subspace, limiting control over the state. In contrast, parallel control remains unaffected. This insight highlights that parallel control can potentially provide greater adaptability in the presence of degeneracies in the original model.
In contrast, within the sT framework, multiple MLP layers and the corresponding LayerNorm can be viewed as a single unit f., enabling the application of a unified update across the entire block. By applying the low-rank design to the control function g MM,-o; T.), the update requires only two matrices of dimensions d x and r x d, resulting in total of only 2rd parameters. This approach significantly reduces parameter requirements by bypassing the feature expansion stage in the FFN block. Alternatively, it allows for the adoption of a higher rank within the same parameter budget, thereby enhancing the effectivencss of fine-tuning.
4.5. Rethinking Parallel Control: Affine vs Non-Affine Control Structures
In fact, the difference between equations (2) and (3) extends beyond superficial rearrangement of terms or a simple modification of the computational graph. Instead, these formulations correspond to fundamentally distinct system structures when viewed through the lens of control theory.
4.4. Performance Analysis
Yet, we have to ensure that the reduction in memory and parameters does not compromise overall performance. To validate this, we present an analysis of the expressive power of the proposed parallel control method in this part.
Equation (2), which represents the classical LoRA formula- tion, can be seen as a non-affine control system,
=(t) ((((()).(tt),
For the simplest case of deep linear networks, we have the following result:
where the control input uff), or the trainable matrix AW; in LORA, embedded inside a nonlinear transformation. This makes the system nonlinear in both the state 2, and the control AW, ,. Analyzing such a non-affine control system often tends to be complicated.
Theorem 4.1. Consider a deep linear network defined as:
f :o0 .T, Ft-1 WeTe, t t= 0,...,T, T, (9) and its low-rank adaptation:
In contrast, the "Parallel Control" proposed in this work follows a control-affine structure of the form:
j: x0 + TT. T++11 (W,+ Re)ar,
1 0, 0....,77 T 1,
(10)
where Ie Rd represents the hidden state, W, idd+ is the weight matrix at layer t, and R. Rdeixdd is low-rank matrix with rank rt-
a(t) f(x(t)+++0)))
+ +(())((t).
where the control component is decoupled from the function f. This decomposition separates the dynamics into a nomi- nal term and an additive control perturbation. From a learn ing perspective, this distinction has important implications.
Then, there exists a weight matrix M satisfying
rank(M) & ro + .... TT-1,
(11)
6

---

From Weight-Based to State-Based Fine-Tuning: Further Memory Reduction on LoRA with Parallel Control
Affine and non-affine controls represent fundamentally dif- ferent classes of control design, each with unique stability and efficiency properties. Affine-control systems, in particu- lar, are well-studied in classical and modern control theory, particularly in feedback and optimal control (Franklin et al., 2002; Skaf & Boyd, 2010; Goswami & Palcy, 2021; Li et al., 2023). This structure also aligns naturally with modu- lar fine-tuning strategies, allowing adaptation mechanisms to operate independently of the nonlinear core dynamics f(-). In contrast, general nonlinear (non-affine) control systems like LoRA and its variants embed the control in- put within nonlinear transformations, making analysis and design significantly more complex and often intractable.
proach, requires the storage of multiple intermediate states during the forward pass, resulting in significantly higher GPU memory usage. Specifically, LoRA's memory con- sumption is approximately 46.66% greater than that of the parallel control method, which also results in a 38.24% increase in training time. Despite these gains in memory efficiency and reduced training time, the parallel control approach obtains similar accuracy performance as LoRA. Therefore, the parallel control approach proves to be an effective fine-tuning method, offering lower memory con- sumption and reduced training time while maintaining com- petitive performance.
5.2. GLUE Benchmark with RoBERTa Models
In summary, the proposed formulation not only introduces new edge in the computational graph, but also fundamen- tally redefines the system structure in way that enhances control interpretability and potential theoretical analysis. By adopting control-affine perspective, it opens the door to simpler and interpretable analysis, paving the way for future developments in fine-tuning theory and practice.
We further apply the parallel control approach to the GLUE benchmark (Wang. 2018), where the experiment settings follow the original LoRA paper (Hu et al., 2021). The experimental parameters, including the choice of pretrained model, learning rate, and ranks for the LoRA weights, are consistent with those used in the LoRA paper. Details of these settings available in Appendix B.1. But also make few changes. (1) Unlike the original LoRA paper, which first trains models on MNLI and selects the best checkpoint, we omit this step to reduce complexity and improve reproducibility, especially for datasets with limited samples like MRPC, RTE, and STS-B, as suggested by (Wu et al., 2024). (2) We fine-tune the entire RoBERTa head, which includes multiple MLP layers, instead of just the last layer. While this increases the number of parameters (Table 3), it generally results in more stable performance.
5. Experiments
In this section, evaluate the effectiveness of the proposed State-FT parallel control methods, compared to traditional weight-FT approaches such as LoRA (Hu et al., 2021) and DoRA (Liu et al., 2024). Our experiments cover a diverse range of model architectures, including ViT, RoBERTa, LLaMA2-7B and LLaMA3-8B.
5.1. A Toy Example on Vision Transformer (VIT)
The results, summarized in Table 2, highlight the perfor- mance of the proposed Control method compared to LoRA and DoRA across eight GLUE benchmark tasks. Across these tasks, the Control approach demonstrates superior per- formance on seven out of eight. For tasks with sufficient samples (e.g., MNLI), the improvement is marginal. How- ever, on other tasks, such as CoLA, the Control method outperforms LoRA by 2.05% and DoRA by 1.68%. This improvement demonstrates the method's ability to handle more complex linguistic patterns effectively. The only ex- ception is the STS-B dataset, where the Control method is marginally outperformed by DoRA on RoBERTa-base by 0. 0.1%%
We begin our evaluation with a toy example using the CIFAR-100 dataset (Krizhevsky et al., 2009) on the ViT model (Dosovitskiy al., 2020). The objective of this ex- periment is to demonstrate the fcasibility of treating multiple components as single unit within the State-FT framework. Specifically, we extend the FFN block by adding two addi- tional MI. layers after the activation function. The State-FT framework enables us to treat the LayerNorm and all MLP layers as a unified function St' in contrast to classical LoRA algorithms, which apply low-rank decomposition separately to each layer's weights. To ensure fair comparison, both methods are configured to use an equal number of trainable parameters, and fine-tuned for 50 epochs.
The more important parts are the reduced memory and com- putation time. Table 3 reports the number of parameters, GPU memory usage, and training time for RoBERTa- base on the CoLA dataset (Warstadt, 2019). The control approach demonstrates the lowest GPU memory usage, requiring only 12.634 GB, which is 1.396 GB less than LoRA and 3.652 GB less than DoRA. Furthermore, the control approach achieves the shortest training time, completing in 4 minutes and 59 seconds, making it approximately 10.2% faster than
Table 1: Comparison of algorithms on the extended ViT-B model. [+] Training time is tested on a single Nvidia-3090.
Algorithm
# Params
GPU Memory!
Training Time]"
Accuracy
LoRA
1.27 M 1.27 M
18.010 GB
4h
91.84.0m1
Control
12.280 GB
3h 2:1m
91. 11.96g0.0
Tuning each layer individually, as in the classical LoRA ap-
7

---

From Weight-Based to State-Based Fine-Tuning: Further Memory Reduction on LoRA with Parallel Control
Table 2: Comparison of algorithm performance on the GLUE benchmark. We report the overall (matched and mismatched)
accuracy for MNLI, Matthew's 's correlation for CoLA, Pearson correlation for STS-B, and accuracy for other tasks.
Model Method
MNLI
SST-2
MRPC
CoLA
QNLI
QQP
RTE
STS-B
Avg
Rob (LoRA)
87 87.5310.0s
10.23
88.6410.23
63.27 10.81
98.0610.15
90.79
75.7911.87
90. 90.74
85.62
Robsa (DoRA)
86.28+0.13
94.84
88.40+0.1
63.6440.35
92.96+0.07
90. 1 +0.05
76.05+1.07
90.78
85.38
Robose (Control)
87.57 10.16
96. 95.26gn00 20.00
89.30 10.31
65.32_0..4
93.12_a.0
91.04_0.02
76.89 20.78
90.59_0.11
86.
Robang (LoRA)
90.89 10.12
96.56
90.52.00300
08..13o.12
95.13 1..08
91.79 +0.06
84.48 29
92.00 :0.000..0 18
88.69
Robuge (DoRA)
90.03
46.63
90.85
24
68,82+0.00
95. 15.0.77
90. 00.25
84.72200.5
92. 02.4520..13
88 61.61
Robange (Control)
90.91 10.11
96.67 10.25
90.94 11.24
69.85 _0.31
95.21 0..10
91.94 10.04
86. 10.36
92.45 10.10
89.28
Table 3: Number of parameters, GPU memory and the training time (on single Nvidia-3090) for RoBERTa base on the RTE dataset. Batch size is 256, and epoch is 80.
other hand, the double control approach achieves the lowest GPU memory usage and training time, consuming only 59% of the memory and 47% of the time required by DoRA, while delivering nearly identical performance (79.6 for dou- ble control versus 79.7 for DoRA). For the 8B model, the trends are consistent. The double control method achieves competitive performance (average score 85.3) while requir- ing only 67% of DoRA's memory and 51% of its training time.
Algorithm
# of Params
GPU Memory
Training Time!
LoRA
0.88 M
14.030 GB
5m 33s
DoRA
0.91 M
16.286 GB
7m 10m
Control
0.88 M
12.634 GB
4m 59s
It is important to highlight that the reduction in memory is not primarily due to the decrease in the number of parame- ters. For instance, halving the rank results in approximately 50% reduction in parameters; however, the memory us- age of DoRA only sees marginal decrease of 0.256 GB. This indicates that the primary factor influencing memory consumption is not the parameter count but rather the in- termediate states that need to be stored during the training process. By skipping these intermediate states, the con- trol approaches maximally reduce the memory cost, while preserving or improving the overall performance.
LoRA and 30.4% faster than DoRA.
Finally, the control approach primarily focuses on tuning the MLP layers using a parallel scheme, whereas LoRA and DoRA are designed to tune the Q and V blocks, con- sistent with their original configurations. Furthermore, in Appendix B.2, we provide results demonstrating that the par- allel control approach also outperforms modified version of LoRA adapted to tune the MLP layers.
5.3. Scaling Up to Llama-2 and Llama-3
We extend our experiments to larger backbones, specifi- cally LLaMA2-7B (Touvron et al., 2023) and LLaMA3- 8B (Dubey et al., 2024), on the "Commonsense Benchmark". These experiments were first considered in DoRA (Liu et al., 2024), and we follow this pioneering work with an identical setup. Note that DoRA fine-tunes five components- Q, K, V, U, and D-for optimal performance, and we adopt this design. Specifically, we extend the control approach with two proposed solutions: the first combines control-based tuning for U and D with LoRA-based tuning for Q, K, and V, while the second adopts a unified double-contre approach, tuning all five components using the control mechanism.
5.4. Training 7B/SB Models on Nvidia-3090
Tuning 7B/8B models typically demands high-end GPUs, such as the Nvidia A A100 with 80GB of memory. With the memory reduction provided by our approach, we investigate training these models on consumer-level hardware, specif- ically the Nvidia 3090 with 24GB memory. In contrast to the prior method (Zhao et al., 2024) on the 7B model, we refrain from using quantization in our approach: the pre- trained 7B/SB models are loaded in FP16, while the control weights remain in FP32.
By reducing the batch size to 4 and applying gradient ac- cumulation with a single step, the proposed control- -bsse approach exhibits only a minor performance degradation on the 7B model. For example, the double-control variant experiences a drop of just 0.7 points in accuracy. Despite this slight decrease. the model remains trainable on single NVIDIA 3090 GPU, with memory of only 20.656 GB. Notably, the performance degradation is even smaller the 8B model, highlighting the scalability and practicality of the method on resource-comstrained hardware.
The results, as reported in Table 4, illustrate the perfor- mance of four fine-tuning methods on the LLaMA2-7B and LLaMA3-SB models. For the 7B model, the hybrid ap- proach of (U)+OO((D))
outperforms DoRA in both accuracy and efficiency. With an average score of 80.0, it surpasses DoRA while significantly reducing 21G less memory and requiring almost half training time, demon- strating the advantages of strategically combining control mechanisms and LoRA for selective edge updates. On the
8

---

From Weight-Based to State-Based Fine-Tuning: Further Memory Reduction LoRA with Parallel Control
Table 4: Comparison of algorithm performance on the Commonsense benchmark. Results show that the Control+LoRA and DoubleControl approaches obtain similar performance to DoRA with significant less GPU memories and computation time. Rank is set to 32 for all algorithms. t indicates numbers published in the original DoRA work.
Model
Method
it Params
GPU Memory
Training Time
BoolQ
PIQA
SIQA
HellinSwng
WinoGrande
ARC-e
ARC-c
OBQA
Avg
ChaiGT
73.1
65.4
68.5
78.5
66.1
89.8
79.9
71.8
77.0
LoRA (QKVLID) 1
56.10
44,204 GB
8h37m
68.8
79.9
79.5
83.6
82.6
79.8
64.7
81.0
77.6
LLaMA2-7B
DoRA (QKVUD)
56,98
58.368 GB
14150m
71.8
83.7
76.0
89.1
82.6
83.7
68.2
82.4
79.7
11.91
38,530 GB
7h36mn
73.0
83.5
79.5
89.7
62.6
82.9
68.6
50.4
80.0
DoubleControl (QKVUD)
33.56
35,214 GB
6158m
71.3
82.5
79.2
89.1
83.1
83.0
08.5
79.0
79.6
LoRA (QKVUD) 1
56.62
55.040 GB
9h33m
70.8
65.2
79.9
91.7
84.3
81.2
71.2
79.0
80.8
MLMM3-88
DaRA (QKVUD)
ControlQU.D)4 D)ALLAAAAALAAAOAO V)
57.41
35.65 M
67,284 GB
48.330 GB
15h155
8h1 1m
75.7
74.6
89.3
87.9
80.4
79.9
95.6
95.5
86.3
85.6
90.6
90.5
80.4
79.88
85.8
86.2
85.3
85.2
DoubleContrel (QKVUD)
45.316 GB
7lx44n
86.0
87.8
85.3
Table 5: Training of 7B/8B models on Nvidia-3090. The batch size is set to 4 for the 7B model and 3 for the 8B model, with gradient accumulation set to 2. We omit comparison to LoRA and DoRA, as they are out-of-memory in this setting.
of Params
GPU Memory
Truining Time
Model
Method
BoolQ
PIQA
SIQA
HellaSwag
WinoGrande
ARC-e
ARC-c
OBQA
Avg
LLaMA2-7B
11.91 M
21.871 GB
21h21m
71.4
81.1
75.7
86.7
82.9
82.3
67.2
80.4
78.1
DouhleControl (OKVUD)
33.55 M
20.656 GB
20h09m
70.8
63.0
79.2
84.6
81.5
82.8
68.3
81.2
78.9
LLaMA3-88
35.65
22.9/21 GB
21
87.8
79.9
96-3
85.0
79.0
86.0
DoubleControl (QKYUD)
33.55 M
22,176 GB
20h33m
71.4
86.9
80.4
96.3
85.4
90.1
79.1
85.6
84.7
5.5. Ablation Studies on the Choice of Controlled Blocks
blocks, significantly improving memory efficiency by reduc- ing the need store large intermediate states. Empirical results across multiple model architectures demonstrate that our approach further reduces memory usage and computa- tion time, and also enhances performance.
In this paper, we propose tuning an entire residual block by introducing an additional parallel edge in the computa- tional graph. To motivate this design choice, we conduct an ablation study comparing different tuning granularities, including individual MLP layers, attention modules, and full residual blocks.
While this paper focuses exclusively on edges associated with residual connections, this design choice represents just one possible solution of the broader state-based fine- tuning framework. Whether these edges are indeed the most effective or efficient targets for adaptation remains an open question. Future work may explore tuning edges beyond the original computational graph, including connections (u, v) & E, potentially unlocking new forms of interaction and control within the network. Additionally, we show that repositioning trainable matrices gives rise to a control-affine structure, though a comprehensive theoretical analysis of the resulting control system is left for future work.
Table 6:
Comparison of performance and GPU memory usage across different fine-tuning targets in VIT.
MLP
Attn
Full Block
Performance
91.96go.us
91.93ga.us
91.53_a.11 12.196 GB
GPU Memory
12.280 GB
12.480 GB
These results indicate that controlling either the MLP or at- tention layer individually leads similar performance, with attention control slightly reducing accuracy and increasing GPU memory usage. In contrast, controlling the entire resid- ual block results in more notable 0.43% performance drop. This is likely because each ViT layer consists of two distinct residual blocks, and treating the entire block as a single unit reduces its effectiveness.
A Acknowledgements
This research is supported by the National Research Foun- dation, Singapore, under the NRF fellowship (project No. 1FF-NRFF13-201--000
6. Conclusion and Future Works
Impact Statement
In conclusion, this paper presents a shift from the traditional weight-centric fine-tuning approach to state-based frame- work. Instead of focusing on weight adaptation, our method prioritizes adjusting forward states, with LoRA serving as a special case. By introducing parameterized perturbations to the computational graph, we can control entire residual
This paper presents work whose goal is to advance the field of machine learning, in particular the parameter-efficient fine-tuning. There are limited potential societal conse- quences of our work, none which we feel must be specifi- cally highlighted here.
9

---

From Weight-Based to State-Based Fine-Tuning: Further Memory Reduction LoRA with Parallel Control
References
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Aghajanyan, A., Zettlemoyer, L., and Gupta, S. Intrin- sic dimensionality explains the effectiveness of language model fine-tuning, arXiv preprint arXiv:2012.13255, 2020.
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q. Gesmundo, A., Attariyan, M., and Gelly, S. Parameter-efficient transfer learning for nip. International Conference on Machine Learning, pp. 2790-2799, PMLR, 2019.
Alain, G. Understanding intermediate layers using linear classifier probes. arXiv preprint 1rXiv:1610.01644 2016.
Åström, J. and Hägglund, T. Pid control. IEEE Control Systems Magazine, 1066, 2006.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L.,, and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint vrXiv:2106.09655 2021.
Ba, J. L. Layer normalization. arXiv preprint arXiv: 1607.0660 2016.
Benning, M., Celledoni, E., Ehrhardt, M. J., Owren, B., and Schönlieb, C.-B. Deep learning optimal control problems: Models and numerical methods. arXiv preprint 1rXiv:1904.05657, 2019.
Kalajdzievski, D. A rank stabilization scaling factor for fine-tuning with lora. arXiv preprint arXiv:2312.03732. 2023.
Chen, S., Ge, C., Tong, Z., Wang, J., Song, Y., Wang, J., and Luo, P. Adaptformer: Adapting vision transform- ers for scalable visual recognition. Advances in Neural Information Processing Systems, 35:16664 6664-16678, 2022.
Kaplun, G., Gurevich, A., Swisa, T., David, M., Shalev- Shwartz, S., and Malach, Less is more: Selec- tive layer finetuning with subtuning, arXiv preprint 230.:2230 2023.
Cheng, J., Li, Q. Lin, T., and Shen, Z. Interpolation, ap- proximation and controllability of deep neural networks. arXiv preprint arXiv:2309.0m 00005,,,,, 2023.
Karimi Mahabadi, R., Henderson, J., and Ruder, S. Com- pacter: Efficient low-rank hypercomplex adapter layers. Advances in Neural Information Processing Systems, 34: 1022-1035, 2021.
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.145: 14314, 2023.
Kirk, D. E. Optimal control theory: an introduction. Courier Corporation, 2004.
Dosovitskiy, A., Beyer, I.,, Kolesnikov, A., Weissenborn,
Kopiezko, D. J.. Blankevoort, T., and Asano, Y. M. Vera:
D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. /929, 2020.
Vector-based random matrix adaptation. International Conference on Learning Representations, 2023, doi: 10. 18550
Krizhevsky, A., Hinton, G., al. Learning multiple layers of features from tiny images. 2009.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21788, 2024.
Lee, Y., Chen, A. S., Tajwar, F., Kumar, A., Yao, H., Liang, P., and Finn, C. Surgical fine-tuning improves adaptation to distribution shifts. arXiv preprint arXiv:2210.11 1466, 2022.
E. W. A proposal on machine learning via dynamical sys- tems. Communications in Mathematics and Statistics, (5):1-11,200 2017.
Lester, B., Al-Rfou, R., and Constant, N. The power of scale for parameter-effiiient prompt tuning, arXiv preprint arXi::2104.0869), 2021.
Franklin, G. F. Powell, J., Emaai-Naein A., and Powell, J. D. Feedback control of dynamic systems, volume 4.
Li, B., Wen, S., Yan, Z., Wen, G., and Huang, T. A survey on the control Yyapunov function and control barrier function for nonlinear- control systems. JEEE/CAA Journal of Automatica Sinica, 10(3):584 602, 2023.
Prentice hall Upper Saddle River, 2002.
Goswami, D. and Paley, D. A. Bilincarization, reachability, and optimal control of control-affin nonlinear systems: A koopman spectral approach. IEEE Transactions on Automatic Control, 67(6):2755-272 2021.
Li, Q. and Hao, S. An optimal control approach to deep learning and applications to discrete- discrete-weight neural net- works. In International Conference on Machine Learning, pp. 2985-2994, PMLR, 2018.
Haber, E. and Ruthotto, L. Stable architectures for deep neural networks. Inverse problems, 34(1):01 4004, 2017.
10

---

From Weight-Based to State-Based Fine-Tuning: Further Memory Reduction on LoRA with Parallel Control
Li, Q. Chen, L.., Tai, C., et al. Maximum principle based algorithms for deep learning. arXiv preprint 2017.
Skaf, J. and Boyd, S. P. Design of affine controllers via convex optimization. IEEE Transactions on Automatic Control, 55(11):2476-22
2010.
Li, Q. Lin. T., and Shen, Z. Deep learning via dynamical systems: An approximation perspective. Journal of the European Mathematical Society, 25(5):1671-1709, 2022.
Tabuada, P. and Gharesifard, B. Universal approximation power of deep residual neural networks via nonlinear control theory. arXiv preprint arXiv: 2007.06001 2020.
Li,X, L. and Liang, P. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2/01.001199 2021.
Touvron, H., Martin, L.,, Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Liu, S.-Y., Wang, C.-Y., Yin, H., Molchanov, P., Wang. Y.-C. F., Cheng, K.-T,, and Chen, M.-H. Dora: Weight- decomposed low-rank adaptation. International Confer- ence on Machine Learning. 2024. doi: 10.48550/arXiv. 2402.09353.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems, 30, 2017.
Liu, X., Zheng, Y., Du, Z., Ding, M., Qian, Y., Yang, Z., and Tang. J. Gpt understands, too. Al Open, 2021. doi: ai0016i.aiopenn2223........
Wang, A. Glue: multi-task benchmark and analysis plat- form for natural language understanding. arXiv preprint arXiv: 000.07761, 2018.
Liu, Roberta: robustly optimized bert pretraining approach. arXiv preprint
Warstadt, A. Neural network acceptability judgments. arXiv preprint 1805. 2019.
364, 2019. Lodha, A., Belapurkar, G., Chalkapurkar, S., Tao, Y., Ghosh, R., Basu, S., Petrov, D., and Srinivasan, S. On surgi- cal fine-tuning for language encoders. arXiv preprint arXiv:2310. 10.:24410.77 /7041, 2023.
Wu, M., Liu, W., Wang, X., Li, T., Lv, C.,, Ling, Z., Zhu, J., Zhang, C., Zheng, X., and Huang, X. Advancing param- eter efficiency in fine-tuning via representation editing. arXiv preprint WwXiv:2402.15173 2024.
Lu, Y., Zhong, A., Li, Q., and Dong, B. Beyond finite layer neural networks: Bridging deep architectures and numer- ical differential equations. International Conference on Machine Learning, 2017.
Zhang, C., Cheng, J., and Li, Q. An optimal control view of lora and binary controller design for vision transform- ers. In Computer Vision ECCV 2024: 18th European Conference, Milan, Italy, September 29-0ctober 4, 2024, Proceedings. Part LIII, pp. 144-160, Berlin, Heidelberg, 2024a. Sprrngeer-eerlag ISBN 10. 10.10079978-3-031-7366--- 19. URL https: /oii (tr//10.1007/97----3333
Nagac, S., Kanda, D., Kawai, S., and Nobuhara, H. Auto- matic layer selection for transfer learning and quantitative evaluation of layer effectiveness. Neurocomputing, 469: 151-162, 2022.
doi:
Zhang, C., Jinguu, C., Xu, Y., and Li, Q. Parameter-efficient fine-tuning with controls. In Forty-first International Conference on Machine Learning, 2024b. URL https: //openreview. net/ Forum?id=C4nall
Nguyen, T., Uribe, C. A., Nguyen, T. M., and Baraniuk, R. G. Pidformer: Transformer meets control theory. arXiv preprint arXiv: 2592.9 2024.
Oquab, M., Bottou, L., Laptev, I., and Sivic, J. Learning and transferring mid-level image representations using con- volutional neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1717-1724, 2014.
Zhang, I.., Zhang, I.., Shi, S., Chu, X., and Li, B. Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning. arXiv preprint arXiv: 2308.03303, 2023a.
Zhang, Q., Chen, M., Bukharin, A., Karampatziakis, N., He, P., Cheng, Y., Chen, W., and Zhao, T. Adalora: Adaptive budget allocation for eframeter-efficient fine- tuning. arXiv preprint arXiv:2303. 10512, 2023b.
Paischer, F., Hauzenberger, L., Schmied, T., Alkin, B., Deisenroth, M. P., and Hochreiter, S. One initializa- tion to rule them all: Fine-tuning via explained variance adaptation. arXiv preprint arXiv:2410.07170. 2024.
Zhao, J., Zhang, Z., Chen, B., Wang, Z., Anandkumar, A., and Tian, Y. Galore: Memory-efficien Ilm train- ing by gradient low-rank projection. arXiv preprint WxXiv:443.003500 2024.
Rebuffi, S.-A., Bilen, H., and Vedaldi, A. Learning multiple visual domains with residual adapters. Advances in neural information processing systems, 30, 2017.

---

From Weight-Based to State-Based Fine-Tuning: Further Memory Reduction LoRA with Parallel Control
A. Proof for Theorems
A.I. Proof of Theorem 4.1
Proof. f z0 - 2T is given by
f(xo) = (Wr ...).... ... + Ro)
(13)
Notice that
AW (WT RT)
(Wo -R) WT. .W
(14)
T
T
[-1
T
1 R₂) 1
Σ
( + R₂)
11 W
11
(W,
11
W
(15)
<=t+1
f=1
I
Σ
(W. + Ra)
R
1I W
(16)
(17)
Here, 11i-1 Wi = WWW-1 Wi denotes the left product of a sequence of matrices. Since
( R.) R. (w:))
T
rank
< rank RR) & ne.
(18)
we have rank(AW) <oo ... + TT-1. Therefore, take M = AW gives the result.
A.2. Proof of Theorem 4.2
Proof. Consider AW to be small, Taylor expansion of Fa, zero gives:
(AW) Ia f(((,, + AW;)a;) f((Wæ))) ff O(JAW
(19)
Therefore, V (frr;)WWz; only take values in the image space of f((zz),, which is proper subspace of Rd when ff(;) is singular.
Moreover, recall that
(AW,) = x, f(((Wx))) + AWæ.,
(20)
is obvious that AWax, can take arbitrary value in IR long as 2e non-zero.
B. Experiment Details
We present the details of our experimental setups in this section to provide the configurations and methodologies employed. This includes the specifications of the datasets, model architectures, and hyperparameter configurations, as well as the training strategies and evaluation protocols used our experiments.
B.1. Glue Benchmark with RoBERTa Models
The GLUE (General Language Understanding Evaluation) benchmark (Wang, 2018) widely used suite of natural language understanding tasks designed to evaluate the performance of machine learning models across a diverse range of linguistic challenges. It includes tasks such as sentence similarity (STS-B), natural language inference (MNLI, RTE, QNLI), sentiment analysis (SST-2), textual entailment (WNLI), question answering (QQP), and linguistic acceptability (CoLA). This dataset is widely utilized benchmark in prior studies on LoRA (Hu et al., 2021), serving an evaluation framework for assessing the effectiveness of low-rank adaptation techniques.
For datasets with limited samples, such as MRPC, RTE, and STS-B, -,, the initial LoRA paper employs a strategy of first pre-training models on the MNLI dataset, leveraging its larger size to improve performance, and subsequently selecting the berf-perfo checkpoint as the initialization for fine-tuning. While this approach can enhance downstream task performance, it introduces additional complexity to the experimental workflow and may pose challenges for reproducibility.
12

---

From Weight-Based to State-Based Fine-Tuning: Further Memory Reduction LoRA with Parallel Control
Following the simplified methodology proposed by (Wu al., 2024), omit this pre-training step, prioritizing more straightforward and accessible experimental design.
We utilize the RoBERTa-base and RoBERTa-large models as the backbone architectures, incorporating LoRA, DoRA, or control modules, and directly tine-tuning these enhanced models on each dataset. This straightforward approach eliminates any need for additional pre-training steps. Generally, for datasets with a large number of samples, such as MNLI, the performance remains consistent (e.g., 87.5% the original paper, 87.57% Table 2). However, for smaller-scale datasets, such MRPC STS-B, we observe slight decline performance compared to previously reported results, likely due the absence of specialized pre-training on MNLI, or the shrink for sequence length 128 in our experiment. Despite this, the simplified workflow improves reproducibility while still maintaining similar performance across tasks.
We report the average performance over three random seeds (40, 41, and 42). For each run, the result is determined based on the best-performing epoch, selected according to the validation set performance. The rank of LoRA and DoRA is set as 8, and its alpha is set as 16. Since these methods need to tune both Q and matrices, while Control only needs to tune one block, we set the rank of control to 16. This allows all algorithms to have similar parameters. Details of hyperparameter configurations are listed follows.
Model
Dataset
MNLI SST-2
MRPC
CoLA
QNLI
QQP
RTE
STS-B
Optimizer
AdamW
WarmUp
0.06
Scheduler
Linear
Seq Length
128
Batch Size
128
128
128
64
256
128
128
Robbase
Epochs
30
50
30
80
25
25
80
40
Learning Rate
5e-4
5e-4
4e-4
3e-4
4e-4
5e-4
4e-4
4e-4
Batch Size
32
64
32
32
32
32
64
32
Robjagge
Epochs
10
30
20
10
20
20
10
Learning Rate
3e-4
4e-4
3e-4
5e-4
3e-4
3e-4
4e-4
3e-4
Table 7: Hyperparameters on GLUE benchmark datasets.
B.2. Additional Experiments on GLUE Benchmark
The control method primarily targets tuning the FFN block, whereas LoRA focuses updating the Q and V matrices within the attention block. Our experiments reveal that the control method outperforms LoRA, particularly on datasets such as CoLA. To further investigate, we conduct an additional experiment where we apply LoRA to tune both layers of the FFN block.
Method
Seed 40
Seed 41
Seed 42
Average
LoRA Control
61.92
63.93
60.59
62. 62.15
65.36
64.84
65.77
Table 8: Performance of LoRA and Control on tuning U and D matrices.
Specifically, we repeat this experiment on the CoLA dataset using random seeds ranging from 40 to 42 to ensure the consistency of the results.. The table 3 presents the results of two methods, LoRA and Control, evaluated across these three random seeds with their respective average performances. The results suggest that it is possible to control both MLP layers simultaneously, similar the parallel control approach, rather than tuning each MLP layer individually.
B.3. LLaMA-2 and LLaMA-3
The LLaMA2-7B (Touvron et al., 2023) and LLaMA3-8B (Dubey et al., 2024) models are part of the LLaMA (Large Language Model Meta AI) family, designed for efficient scaling in natural language understanding and generation tasks.
13

---

From Weight-Based to State-Based Fine-Tuning: Further Memory Reduction LoRA with Parallel Control
On the other hand, the commonsense reasoning tasks consist of 8 distinct sub-tasks, each with its own designated training and testing sets. In line with DoRA, combine the training datasets from all 8 sub-tasks to form a unified training set. The pretrained models are first tuned on this combined training set, and then evaluations are carried out separately on the individual testing sets for each sub-task.
This experiment was first explored in DoRA (Liu et al., 2024), and we replicate the experimental setup as outlined in the original paper. In prior work with the RoBERTa model, tuning strategies were restricted to specific components: LoRA and DoRA primarily focus on adjusting the Q and matrices, while the control mechanism targets the and D matrices within the FFN block. However, for more complex architectures like LLaMA, it generally advised (Liu al., 2024) to tune all five components- Q. K, V, U, and D for optimal model performance. To address this, propose two extensions the control approach: the first combines L for tuning the Q, K, and V matrices with control-based tuning for the U and D matrices, while the second utilizes a unified double-control strategy, which tunes all five components through the control mechanism to enhance model flexibility and accuracy.
For the hyperparameter configuration, we use the AdamW optimizer, and the maximum sequence length is set to 256. Following DoRA, the training is conducted over total of 3 epochs, with a batch size of 16 balance computation and memory usage. On Nvidia-3090, the batch size shrinks to 4 for LLaMA2-7B and for LLaMA3-8B model, with another round of gradient accumulation. For LoRA and DoRA, the rank is set to 32, and the alpha value is set 64. In contrast, the rank for the Control approach is set to 64, since it tunes both U and D with one simple control unit.
B.4. Sensitivity the Ranks
We conduct sensitivity analysis the RTE dataset by varying the control rank assess its effect GPU memory consumption, training time, and model accuracy.
Table Effect of rank r on GPU memory, training time, and accuracy.
Rank (r)
GPU Memory
Training Time
Accuracy
12,620 GB
4m 58s
73.05
4
12.622 GB
4m 59s
73.41 +1.8
8
12.626 GB
4m 59s
74.37+0.59
16
12.634 GB
4m 59s
76.89
32
12.674 GB
5m 00s
77.17
12,724 GB
5m 02s
77.01 10.68
As the rank of the control parameters increases, GPU memory usage remains relatively stable, exhibiting only slight increase. This stability stems from the use of low-rank matrices, which efficiently limit the memory overhead associated with storing backward gradients. In terms of accuracy, we observe consistent improvements up rank of 32, suggesting that increased rank enhances the expressiveness of the adaptation. However, at rank 64, the performance slightly declines compared to rank 32, indicating diminishing returns and potential overfitting beyond certain threshold.
Next, we compare the performance of Control, LoRA, and DoRA across different ranks: At lower ranks (r=8), LoRA and
Table 10: Comparison of Control, LoRA, and DoRA across different ranks
Method
Control
LoRA
DoRA
Accuracy (r 8)
74.37 10.59
74.8411.44
74.97
Accuracy (r = 16)
76.89+0.78
75.79+1.57
76.05+1.67
Accuracy (r 32)
77. 77.17
75.97
76.65+1.19
Accuracy (r 64) 77.01+0.66
75.93
76.78+2.01
DoRA achieve slightly higher accuracy than the control method, indicating that traditional weight-centric approaches may have an edge with limited parameter capacity. However, as the rank increases to and beyond, the control method begins to outperform both LoRA and DoRA, demonstrating its ability to better leverage the increased expressiveness. Since raising
14

---

From Weight-Based to State-Based Fine-Tuning: Further Memory Reduction LoRA with Parallel Control
the rank to 16 32 results in only minimal increases in GPU memory usage and training time, we recommend adopting relatively higher ranks to maximize performance gains. Notably, this trend is consistent across other datasets such CoLA and SST-2, highlighting the general effectiveness of the control-based fine-tuning approach.
15


Reviews:
[{"review_id": "Qb5HWqUIN1", "reviewer": "Reviewer_hdej", "summary": "This paper proposes a new state-based fine-tuning framework that allows tuning entire residual blocks or multiple sequential sub-layers instead of adding adapters to each layer. The method significantly reduces the memory footprint by avoiding the storage of large intermediate activations while maintaining fine-tuning performance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank reviewer for the insightful and constructive feedback.\n\n### 1. Sensitivity to the Choice of Controlled Blocks\nControlling either the MLP or attention layer yields comparable performance, as demonstrated by the ViT model results:\n\n||MLP|Attn|Full Block|\n|-|-|-|-|\n|Performance|$91.96\\pm0.05$|$91.93\\pm0.03$|$91.53\\pm0.11$|  \n|GPU Memory|12.280G|12.480G|12.196G|   \n\nThese results indicate that controlling either the MLP or attention layer individually leads to similar performance, with attention control slightly reducing accuracy and increasing GPU memory usage. In contrast, controlling the entire residual block results in a more notable 0.43% performance drop. This is likely because each ViT layer consists of two distinct residual blocks, and treating the entire block as a single unit reduces its effectiveness.\n\nTo further examine the effects of tuning different portions of the MLP, we selectively tune either the first six or the last six layers of the ViT model while doubling the rank to maintain the same parameter number.\n\n||First 6 Layers|Last 6 Layers|All 12 Layers|\n|-|-|-|-|\n|Performance|$91.14\\pm0.03$|$87.23\\pm0.11$|$91.96\\pm0.05$|\n|GPU Memory|11.876G|7.180G|12.280G|\n\nThese results highlight that selectively tuning a subset of the MLP layers leads to a more pronounced drop in accuracy. Specifically, tuning only the first six layers results in a 0.82% accuracy reduction. On the other hand, tuning only the last six layers leads to a more substantial memory reduction, as forward states for the first six layers no longer need to be stored. However, this comes at the cost of a significant 4.73% accuracy drop.\n\nWhile these experiments are conducted on the ViT model, we observe a similar trend across other architectures. These ablation studies will be added to the appendix.\n\n---\n\n### 2. Combining two sets of control edges.\nIf two sets of edges share the same starting and ending nodes on a pretrained model, their combination can be conducted as existing LoRA merging techniques, such as ZipLoRA [1], K-LoRA [2] and SVD methods [3-4]. For example, given two sets of additional edges with weights $\\Delta  W_1$ and $\\Delta  W_2$, we can apply a similar approach as [1] and optimize $\\ell_{\\text{merge}}$ with\n$$\\Delta W = v_1 \\circ \\Delta W_1 + v_2 \\circ \\Delta W_2,$$\nwhere $v_1$ and $v_2$ are vectors, and $\\circ$ denotes element-wise multiplication, such that the j-th column of $\\Delta W$ is scaled by the j-th element of v. The purpose of this lightweight tuning is to mitigate potential conflicts between different sets of low-rank matrices.\n\nIf the two edges have different starting and ending nodes, they can be incorporated into the computation graph independently. However, two key considerations must be addressed. (1) Preserving the Directed Acyclic Graph (DAG): It is essential to ensure that the newly introduced edges do not violate the DAG structure. For example, if the edges include both $A \\to B$ and $B \\to A$, a cycle would be introduced, requiring structural trimming. (2) Resolving Conflicts: Different sets of edges may result in conflicting weight updates. To optimize performance, lightweight tuning remains a recommended step.  \n\n---\n\n### 3. Compatibility with Gradient Checkpointing and Memory-Reduction Methods.\nYes, we use gradient checkpointing when training 7B/8B models to reduce memory. For other methods like quantization, please kindly refer to our response to Reviewer 2 (uZCq).\n\n---\n\n### 4. Overhead During Inference and Deployment on Standard Inference Platforms.\n\nExtra control edges may lead to a slight increase in inference time. In the worst-case scenario, such as the double-control approach with two extra edges, our experiments show that the extra inference time is around $2.1\\% \\pm 0.3\\%$ extra inference time for RoBERTa-base, and $4.6\\% \\pm 0.5\\%$ for the LLaMA2-7B model.\n\nThe introduction of control edges primarily impacts the forward pass by modifying the computation graph. Despite this, our method remains compatible with standard inference platforms like HuggingFace Transformers and Accelerators. While certain platform-specific optimizations, such as quantization, may require some adaptation, we do not anticipate significant changes.\n\n---\n\n### 5. Pseudocode and How to Implement Skipping Intermediate State.\nIn general, steps to skip big intermediate states include:(1) Inherit the original layer and insert low-rank matrices; (2) Define a control function to compute the controlled state $x_c$; (3) Add $x_c$ to the original output.\n\nWe will include the pseudocode in the appendix to provide further clarity and would like to thank you once again for this valuable suggestion.\n\n---\n[1] ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAs. ECCV 2024.\n\n[2] K-LoRA: Unlocking Training-Free Fusion of Any Subject and Style LoRAs. ArXiv 2025.\n\n[3] Task Singular Vectors: Reducing Task Interference in Model Merging. Arxiv 2024.\n\n[4] Model merging with SVD to tie the Knots. ICLR 2025.", "claims_and_evidence": "Overall, authors' claims are backed up by the provided experiments, which appear consistent with their theoretical motivation. More precisely, the experimental results support the paper’s claim that state-centric fine-tuning is more efficient and has same performance as weight-centric fine-tuning.", "methods_and_evaluation_criteria": "Yes, they do. \nIn terms of methods, the paper’s core contribution is to replace weight-based LoRA with a “control-based” injection of low-rank updates in the forward pass. Instead of decomposing weight updates in Q/K/V or feed-forward layers, the method combines entire residual (or multi-layer) blocks into one function and introduces a separate control path that is likewise parameterized by low-rank matrices.\nIn terms of evaluation criteria, the authors rely on standard accuracy metrics for classification and QA tasks (e.g., MNLI accuracy, STS-B Pearson correlation, CoLA Matthew’s correlation, etc.), as well as training resource metrics such as GPU memory consumption and wall-clock training time. These metrics are appropriate for testing whether a method is truly more parameter- or memory-efficient.", "theoretical_claims": "Yes. I checked proofs of theorems 4.1 and 4.2 (guarantee of parallel control method performance). No major issues.", "experimental_designs_or_analyses": "Yes. No major issues. The experiments cover both small-scale (CIFAR-100 on ViT) and medium-scale (GLUE tasks on RoBERTa) fine-tuning, as well as large-scale LLaMA2-7B/ LLaMA3-8B models for common-sense QA tasks. The study ablates memory usage, training speed, and final performance scores. These are well-chosen metrics for a new PEFT algorithm. \nOne potential limitation is that each dataset is evaluated with a specific set of hyperparameters. It would be helpful to see more ablation on rank choices or to confirm that the approach remains stable with smaller ranks on more difficult tasks.", "relation_to_broader_scientific_literature": "The paper is a novel connection between parameter-efficient fine-tuning and optimal control formulation for deep neural networks.  \nThe paper positions itself as a generalization of LoRA in the context of control theory, a line of reasoning that has emerged in prior work linking neural networks to ODEs or closed-loop control systems. This is a valuable perspective The discussion of other memory-efficient techniques (like QLoRA or other quantization-based PEFT methods) is comparatively brief, but the authors’ emphasis is on building upon LoRA-style low-rank methods, so this is understandable.", "essential_references_not_discussed": "The paper covers the main PEFT references (LoRA, DoRA, and adapter/prompt-tuning families) and also cites relevant control-theoretic works. If anything, it might be beneficial to discuss more systematically how these “state-based” methods compare to existing large-activation checkpointing or gradient checkpointing strategies, which also address memory usage. Such a comparison would clarify the practical trade-offs.", "other_strengths_and_weaknesses": "Strengths:\n- Comprehensive and strong experimental results. \n- Significant memory savings allow training 7B/8B models on a single 24GB GPU (Nvidia 3090).\n- The conceptual link between weight updates and states/controls may inspire further research.\n\nWeaknesses:\n- Merging weights or adapters from multiple “control edges” can be more complicated in practice than with purely weight-centric methods. The authors do note an increase in inference time, though they characterize it as modest.\n- More ablation and clarity on how exactly “skipping” big intermediate states is implemented (particularly for widely used frameworks like PyTorch) would strengthen the practical dimension of the paper.", "other_comments_or_suggestions": "Including a clear, step-by-step pseudocode for the “DoubleControl” approach in the Supplementary can help readers adopt the technique.", "questions_for_authors": "1. How sensitive is the method to which blocks get controlled? Have the authors tried controlling only a portion of the MLP layers vs. controlling entire multi-head attention and feed-forward blocks together?\n2. Can the authors elaborate on any overhead or complexity added during inference when “control edges” are introduced? Do you anticipate issues in deployment on standard inference platforms?\n3. Is it feasible to combine two sets of “control edges” learned on different tasks? If so, how would that composition look?\n4. Could gradient checkpointing or other memory-reduction approaches be integrated into this method to push memory usage down even further?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "No ethics concerns."}}, {"review_id": "nbTsY5tOeZ", "reviewer": "Reviewer_uZCq", "summary": "This paper proposes a novel state-based fine-tuning framework named State-FT for parameter-efficient algorithms. The authors shift the focus from traditional weight-based adaptations (e.g., LoRA and its variants) to directly optimizing the model’s intermediate forward states. Based on the inspiration of the control theory, the proposed method referred to as state-based fine tuning (State-FT), directly modifies the intermediate states in the computation graph rather than adjusts the model’s weight matrices. \nThe authors demonstrate that State-FT significantly reduces GPU memory usage by compressing intermediate layers using ‘DoubleControl’ methods. Experiments show that this approach maintains or outperforms the GLUE benchmark and commonsense reasoning tasks. Furthermore, it scales to larger models such as LLaMA2-7B and LLaMA-8B, enabling fine-tuning on GPUs that usual consumers use (under 24GB) without any quantization methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for your positive and insightful feedback.\n\n### 1. Quantization Method like QLoRA/QA-LoRA. Can State-FT use QLoRA method independently?\n\nYes, the State-FT method can independently leverage the QLoRA/QA-LoRA approach. GPU memory usage mainly arises from three sources: model weights, forward states, and backward gradients. QLoRA and QA-LoRA reduce memory by quantizing *model weights* and using low-rank matrices to minimize *backward gradients*. In contrast, State-FT focuses on reducing *forward states* and *backward gradients*. These methods are complementary, addressing different sources of memory cost.\n\nState-FT can be combined with the QLoRA method. This allows control to be operated on a quantization model, hence reducing memory consumption across all three categories. To validate this, we evaluate on the commonsense dataset with 8-bit LLaMa3-8B model. \n\n||GPU Memory|Training Time|Accuracy|\n|-|-|-|-|\n|Double-Control|22.176 GB|20h33mins|84.7|  \n|Q-Double-Control|17.796 GB|39h20mins|84.3|   \n\nAs shown, the use of QLoRA further reduces the memory consumption of the double-control approach by an additional 4.38 GB. However, this reduction in memory comes with trade-offs: training time increases by 91.40% and accuracy drops by 0.4%.\n\nWe further investigated the effects of quantization with 4-bit methods on the RTE dataset. \n\n||GPU Memory|Training Time|Accuracy|\n|-|-|-|-|\n|Control|12.634 GB|4min59s|$76.89 \\pm 0.78$| \n|Q-Control|4.548 GB|9min03s|$73.41 \\pm 1.36$| \n|Q-LoRA|4.688 GB|9min36s|$71.84 \\pm 2.23$| \n|Q-DoRA|4.742 GB|11min03s|$71.00 \\pm 0.94$| \n\n(1) As shown above, 4-bit quantization techniques effectively reduce GPU memory consumption. (2) Like its full-precision counterpart, Q-Control offers slight gains in memory, training time, and accuracy by reducing forward state memory, though overall savings remain modest. (3) Similar to 8-bit, this reduction comes at a cost—Q-Control incurs a 3.48% accuracy drop and an 81.61% increase in training time on the RTE dataset. \n\nIn summary, the proposed state-FT can also benefit from methods like QLoRA. Quantization can effectively further reduce memory consumption for control and other PEFT methods, but comes with trade-offs, such as increased training time and slight performance degradation.\n\n---\n\n### 2. Sensitivity Study of Rank Settings.\n\nWe conduct a sensitivity analysis on the RTE dataset by varying the control rank to evaluate its impact on GPU memory usage, training time, and accuracy.\n\n||GPU Memory|Training Time|Accuracy|\n|-|-|-|-|\n|r=1|12.620 GB|4min58s|$73.05 \\pm 0.34$| \n|r=4|12.622 GB|4min59s|$73.41 \\pm 1.48$| \n|r=8|12.626 GB|4min59s|$74.37 \\pm 0.59$| \n|r=16|12.634 GB|4min59s|$76.89 \\pm 0.78$| \n|r=32|12.674 GB|5min00s|$77.17 \\pm 0.34$| \n|r=64|12.724 GB|5min02s|$77.01 \\pm 0.68$|\n\nAs the rank of the control parameters increases, GPU memory usage remains relatively stable, with only a slight rise. This is due to the use of low-rank matrices, which help reduce the memory consumption of backward gradients. For accuracy, we observe a steady improvement up to a rank of 32. However, at a rank of 64, performance slightly declines compared to rank 32, indicating diminishing gains beyond a certain threshold.\n\nNext, we compare the performance of Control, LoRA, and DoRA across different ranks:\n\n||Control|LoRA|DoRA|\n|-|-|-|-|\n|Accuracy(r=8)|$74.37 \\pm 0.59$|$74.84 \\pm 1.48$|$74.97 \\pm 1.45$| \n|Accuracy(r=16)|$76.89 \\pm 0.78$|$75.79 \\pm 1.57$|$76.05 \\pm 1.67$| \n|Accuracy(r=32)|$77.17 \\pm 0.34$|$75.97 \\pm 2.07$|$76.65 \\pm 1.19$| \n|Accuracy(r=64)|$77.01 \\pm 0.68$|$75.93 \\pm 1.78$|$76.78 \\pm 2.01$| \n\n(1) At lower ranks (r=8), LoRA/DoRA achieve higher accuracy than the control method. (2) As the rank increases (r=16 and above), the control method begins to outperform both LoRA and DoRA. (3)  Given that increasing rank to 16 or 32 has a minimal impact on training time and memory usage, we recommend using a relatively high rank to maximize performance. (4) This trend is also observed on CoLA and SST-2 datasets. \n\n---\n\n### 3. Large Models like 13B, and Different Domains\n\nWe expect State-FT to achieve similar memory reductions on larger models like 13B. Training these models may take several days or even weeks, exceeding the rebuttal period. We are actively working on these experiments. \n\nBelow, we present preliminary results from a multi-task evaluation with VL-BART:\n\n|Method|VQA|GQA|NVLR|COCO|Avg|\n|-|-|-|-|-|-|\n|FT|66.9|56.7|73.7|112.0|77.3| \n|LoRA|65.2|53.6|71.9|115.3|76.5|\n|DoRA|65.8|54.7|73.1|115.9|77.4|\n|Control|65.9|55.1|72.7|115.9|77.4|\n\nIn terms of efficiency, the control method achieves a 31.6% reduction in memory usage and a 65.3% reduction in computation time compared to DoRA.\n\n---\n\n### 4. Essential References Not Discussed\nWe notice [1] is a relevant paper and will include it in our reference list.\n\n---\n[1]AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning. ICLR 2023.", "claims_and_evidence": "The paper’s main claims are convincingly supported by comprehensive theoretical control theory bases and experimental results. The proposed state-based fine-tuning approach shows significant GPU memory reductions (approximately ~3GB on medium-sized models RoBERTa and up to 40$ reduction on large models such as LLaMA2-7B and LLaMA3-8B), as the authors claimed. Additionally, the experimental results consistently show competitive or higher accuracy performance compared to baselines such as LoRA and DoRA, supporting the claim of representational capacity. \n\nHowever, one of the limitations is the lack of explicit comparisons to quantization-based methods (e.g., QLoRA), which also aim at memory reduction. This is one of the concerns that the proposed method is showing effective since it lacks the baselines. Furthermore, experimental validation for large-scale models focuses primarily on NLP and commonsense reasoning tasks, while only a limited CIFAR-100 evaluation is provided for vision tasks. This scope raises concerns about the method’s applicability across diverse domains.", "methods_and_evaluation_criteria": "The method not only introduces a novel framework but also clearly justifies it, showing how the proposed method generalizes and explains the better representational capacity. All of the proposed frameworks (State-FT and double control) are well motivated and logically supported, making effective use of control theory concepts to reduce total GPU memory usage.\n\nThe evaluation involves standard benchmarks such as GLUE and commonsense reasoning datasets previously conducted in the DoRA. It provided GPU memory usage, training time, and parameter counts comparing their approach with the original LoRA and DoRA under the same settings. \nHowever, for the vision domains, the evaluations were limited, showing only the toy example since the proposed method could not be found to be effective.", "theoretical_claims": "Section 3.3 of State-FT provides that the LoRA can be considered a special case of the proposed framework. There seem to be no issues with the theoretical claims.", "experimental_designs_or_analyses": "For the experimental settings, the authors show toy examples (ViT) to large-size datasets (LLaMA2-7B, LLaMA3-8B), GLUE, and commonsense benchmarks. They analyzed a number of parameters, GPU memory, and training time for each experimental setting that could prove their method’s effectiveness.", "relation_to_broader_scientific_literature": "This paper provides new aspects over what the original parameter-efficient training tries to approach (weight-matrices approaches). In addition, they reduce the memory issues without any quantization schemes that were typically conducted for memory reduction. State-FT contributes to PEFT (parameter-efficient training) and memory efficiency areas such as QLoRA or parameter-sharing, while the work more directly addresses activation memory.", "essential_references_not_discussed": "AdaLoRA (Zhou et al., 2023), which is related to dynamic rank allocation, should be additionally referenced for the related works.", "other_strengths_and_weaknesses": "**Strengths**\n\n1.\tProvides novel aspects of the PEFT area that original works did not: As this paper mentions, the authors were inspired by the control theory, which supports new aspects and possibilities in the PEFT area. This novelty distinguishes it from previous PEFT methodologies and shows comparable results.\n\n2.\tProvides strong theoretical justifications: The authors successfully demonstrate how state-based perturbations generalize and enhance the representation capabilities of original LoRA methods. They support their methods both theoretically and experimentally.\n\n3.\tClear justification and motiviation: The necessity of the method is well presented, particularly through detailed analyses breaking down GPU memory usage. The paper highlights critical memory bottlenecks in previous PEFT approaches, clearly motivating the need for their ‘double control’ method. In the end, they successfully show its practical effectiveness on consumer GPU hardware.\n\t\n**Weaknesses**\n\n1.\tLack of comparison with quantization-based methods: Although the paper mentions significantly reducing GPU memory usage, it omits direct comparisons with quantization-based fine-tuning techniques such as QLoRA or QALoRA. Since the paper focuses on memory efficiency, evaluating or discussing results against QLoRA seemed to be needed.\n\n2.\tThe proposed method is not guaranteed to work in other domains, such as vision, since it only shows toy examples. Previous work (DoRA) conducted effectiveness with image-video-text understanding domains, so providing results from other domain tasks might increase the confidence of this paper.\n\n3.\tLimited analysis of parameter sensitivity. This paper lacks sensitivity studies regarding the impact of hyperparameter settings (e.g., rank). Including sensitivity experiments could strengthen the paper’s framework.", "other_comments_or_suggestions": "1.\tThe term ‘parallel control’ in Section 4.1 248 could confuse the readers that it refers to ‘parallel pipelining.’ Using the term ‘block’ in this case seems more appropriate, as the control unit is not working in parallel. In addition, the proposed method is mainly about ‘double control’, so the title seems to make sense ‘parallel control’ into ‘double control.’\n\n2.\tAdding the results (at least QLoRA) might strengthen this paper’s contribution. The quantization method was mainly raised to reduce memory efficient training and complement the performance degradation. If the performance degradation is severe, State-FT’s necessity might be significantly increased. If not, it might be better that State-FT could apply the QLoRA method independently, as DoRA already showed. \n\n3.\tAs the author mentioned their work limitations, it might be comprehensive to provide how much slower they were in the inference phase, at least in the appendix.", "questions_for_authors": "1.\tCould the authors show the results, including the QLoRA and QA-LoRA quantization methods? In addition, could State-FT use the QLoRA method independently?\n\n2.\tCould the authors provide a sensitivity study of changing rank settings?\n\n3.\tThe proposed method seems to be more efficient for large models such as LLaMA-13B or above. Could the authors provide results for larger models that could explain this method’s scalability?\n\n4.\tThe results were only shown for the vision tasks for a small dataset (CIFAR-100). Could the proposed method be adopted for the Image-Video-Text understanding domains that use model backbone VL-BART, such as VQA, GAT, NVLR, and COCO Caption tasks? In the commonsense reasoning tasks, the performance seemed to be competitive with the existing method, but the question remains whether State-FT could remain accurate on multi-modality fine-tuning tasks, as DoRA has already shown. It might strengthen the paper if the state-FT works on even the multi-modality domains.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CJfMNJsjbo", "reviewer": "Reviewer_jhpo", "summary": "This paper presents a state-based fine-tuning framework, which can avoid storing large intermediate states during training. Empirical results show its effectiveness.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable suggestion. \n\n### 1. The idea of state-based fine-tuning seems to apply the LoRA from the QKV matrix to the FFN/ATTN block\n\nWe understand the reviewer’s concern that our contribution may seem limited to proposing an algorithm, and we would like to clarify on this. \n\nFirst of all, our goal of proposing the State-FT method is to provide a **new viewpoint** to link the LoRA method with control approaches. While LoRA is typically framed as a low-rank fine-tuning method that modifies specific model weights, this contrasts sharply with classical control theory, which focuses on adjusting system states, as seen in feedback control. These two areas have largely developed independently. A central argument of our work is that LoRA can, in fact, be viewed as a special case of state-based fine-tuning. As noted by Reviewers hdej and uZCq, our framework aims to establish a direct connection between parameter-efficient fine-tuning and control formulations in deep neural networks. This is the key message we want to deliver when establishing the state-based FT.\n\nSuch a viewpoint also paves the way for new algorithmic design. We propose to treat fine-tuning neural networks as modifications to its computational graph (DAGs), allowing us to rethink tuning as a structural modification of the network itself. This means we can introduce or remove specific edges within the computation graph based on task-specific requirements, rather than being limited to conventional weight adjustments.  For example, we can add edges that may not exist on the original network, to alter the computation flowmodifying the computation flow—something not captured by the original LoRA method. This viewpoint raises several **open questions** within the state-based tuning paradigm. For instance, *which states should be adjusted within the network, and how can we determine the optimal control strategy*? There are many well-established results from control theory, and we would expect them to be applied to finetuning research.\n\nFinally, we introduce an algorithm that tunes entire blocks as a unified unit. Our design philosophy is straightforward: **prioritize simplicity and effectiveness**.  By making the whole architecture simple and making minimal changes, we expect our approach remains broadly applicable across various network architectures. \nIn the meanwhile, we would expect the algorithm to be effectively reduce the GPU memories, as demonstrated in the following experiments. \n\nIn summary, we want to emphasize that our goals would extend beyond one specific algorithm—we aim to bridge weight-based fine-tuning with the well-established principles of state-based control theory. This is the primary contribution of this paper. This perspective not only offers a fresh way to interpret LoRA but also raises several open questions, paving the way for future exploration. We will make these points clearer in the updated version. \n\n---\n\n### 2. The empirical results show some improvements in memory efficiency and training time, but the performance gains are marginal as 0.12%.\n\nAs our title suggests, the core objective of our method is to minimize memory consumption and computational overhead, making it an efficient solution. We demonstrate that our approach reduces memory usage from 18.010GB to 12.280GB on the ViT toy model, achieving a 31.8% reduction. \n\nThe performance evaluations we include serve a specific purpose: to demonstrate that these efficiency gains do not come at the expense of model quality. We hope the reviewer can understand that the 0.12% difference is not intended to suggest a significant improvement over the baseline. Rather, its goal is to convey another message—that our method enhances efficiency without sacrificing effectiveness.\n\n---\n\n### 3. Strong theoretical foundation to justify the state-based tuning framework.\nOverall, we hope the reviewer can understand that this paper is not intended to be a theoretical paper. Instead, the analysis is mainly to support our claims. As alluded to earlier, the primary objective of our algorithm is to demonstrate that improvements in memory and computational efficiency do not come at the cost of model performance. Therefore, our analysis is centered on this key point.\n\nFor linear cases, Theorem 4.1 establishes that the parallel control method retains the same expressive power as LoRA, provided both methods share the same total rank. This guarantees that adopting parallel control does not lead to a loss in expressiveness. For nonlinear cases, Theorem 4.2 further demonstrates that parallel control can offer greater adaptability, particularly in scenarios where the original model exhibits degeneracies. These results reinforce the robustness of our approach across different settings.\n\nIf the reviewer has any additional concerns regarding necessary analyses, we would greatly appreciate the feedback and are willing to address them accordingly.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The proposed methods and/or evaluation criteria (e.g., benchmark datasets) make sense.", "theoretical_claims": "I have checked the correctness of any proofs for theoretical claims.", "experimental_designs_or_analyses": "I have checked the soundness/validity of any experimental designs or analyses. \n\nIn Table 4, the comparison between ChatGPT and other baseline methods seems unfair, as there is no fine-tuning in ChatGPT.", "relation_to_broader_scientific_literature": "The key contributions of the paper are related to the broader scientific literature in PEFT methods, such as LoRA and DoRA.", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "Strengths:\n1. This paper is well-written and easy to read. Figures 1, 2 clearly show the advantage of the proposed method.\n2. The experiments of this paper validate the proposed method's efficient.\n\nWeakness: \n1. The contribution of this work is limited. The idea of state-based fine-tuning seems to apply the LoRA from the QKV matrix to the FFN/ATTN block, which appears to be an incremental extension of existing weight-based methods like LoRA. Moreover, the authors acknowledge that LoRA can be viewed as a special case of their framework (Section 3.3). This suggests that the proposed method is not fundamentally new but rather a generalization of existing techniques.\n2. The empirical results, while showing some improvements in memory efficiency and training time, do not convincingly demonstrate the superiority of the proposed method. The performance gains are marginal (e.g., 0.12% improvement in accuracy on ViT, Table 1)\n3. The paper lacks a strong theoretical foundation to justify the state-based tuning framework.  The connection between control theory and state-based tuning is superficial.", "other_comments_or_suggestions": "n/a", "questions_for_authors": "n/a", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2TFIgPnEFq", "reviewer": "Reviewer_eRj7", "summary": "This paper discussed the PEFT from a new view of control theory.  From control theory, a new State-Based Fine-Tuning (State-FT) is proposed, where the network is modeled as a graph with each edge representing weights and each node representing activations. Thus, any components such as MLP or a couple of layers can be viewed as a unified unit, whose output state (activations) can be modified by the trainable parameters/non-linear function. This method can reduce GPU memory since the middle activations that need to be saved for calculating gradient are reduced. Experiments prove this method's effectiveness.\n\n## update after rebuttal\n\nThe authors have solved my concerns properly and I recommend accepting this paper.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and positive comments. We would like to clarify on a few points to address the concerns raised.\n\n### 1.Discussion about the difference between the current work and the pioneering work (Zhang et al., 2024b).\n\nWe agree with the reviewer that it is important to clearly highlight the differences between our approach and previous work. To address this, we would like to include the following discussion in the revised version.\n\nIn general, the differences between the current work and the pioneering work lie in two aspects.\n\n(1) **Different Aspects of Control Theory**. The previous work primarily focuses on the controllability aspect of control theory, aiming to understand how system parameters can be adjusted to influence system behavior. In contrast, our work introduces the concept of state-based tuning, drawing inspiration from feedback control systems. Rather than focusing on the controllability of the system, our method emphasizes how adjusting the system's state (as opposed to directly manipulating model weights) can lead to improved performance. This distinction highlights that these two papers target different areas of control theory.\n\n\n(2) **Memory and Computation Efficiency**. The most notable difference lies in the memory and computation efficiency. The prior approach enhances the LoRA method by introducing additional cross-patch attention mechanisms to modify the low-rank matrices. While this enhances the model's complexity, it also leads to increased memory consumption and training time due to the added operations and parameters. In contrast, our primary goal is to achieve memory efficiency by reducing the memory footprint, specifically by bypassing large intermediate states during model execution. \n\n---\n\n### 2. Control theory is like theoretical embellishment. \n\nWe notice the reviewer’s concern regarding the role of control theory in our work and would like to clarify its necessity and impact.\n\n(1) **Bridging PEFT and State-Based Control**. While PEFT methods like LoRA primarily focus on tuning model weights, classical control theory is centered around adjusting system states, as seen in feedback control. These two perspectives may initially seem unrelated, but our key argument is that LoRA can be understood as a special case of state-based fine-tuning. This reframing is not just a theoretical exercise—it provides a new lens for understanding and designing PEFT algorithms. This allows our framework to establish a direct connection between PEFT and state-based control in deep NNs. This bridging alone is the key message that authors want to convey in the first part.\n\n\n(2) **Unlocking New Opportunities**. Control theory offers a wealth of results that could inspire and inform the study of PEFT algorithms. By framing LoRA within a control-theoretic perspective, we naturally arrive at several open research questions. For example, an essential question arises: Which states should we tune? This corresponds to selecting the appropriate edges on the computational graph, analogous to choosing control variables in feedback systems.  Many well-established results from control theory could potentially be applied here, but they have not been fully explored in the PEFT community. \n\nThus, the integration of control theory is not mere embellishment; it serves as a foundation that opens new avenues for both theoretical and practical advancements. We will ensure that this motivation is articulated more clearly in the revised version.\n\n---\n### Distinguishing the Proposed Method from LoRA\n\nThe second part of our paper provides a shift where we reposition the low-rank matrices. We acknowledge that our approach is simple and appears structurally similar to LoRA, yet the underlying formulation corresponds to distinct control frameworks.\n\n(1) **Different Forms of Control Systems**. From a control-theoretic standpoint, LoRA and DoRA can be viewed as *non-affine* control mechanisms:  \n\n$\\dot{x}(t) = f(x(t), u(t))$,\n\nwhere the control is deeply entangled with the model’s attention mechanisms. In contrast, our proposed parallel control formulation aligns more closely with *affine control*:  \n\n$\\dot{x}(t) = f(x(t)) + x(t) u(t)$,  \n\nwhere the control component is decoupled from the function f. This distinction is crucial—affine and non-affine controls represent fundamentally different classes of control design, each with unique stability and efficiency properties. Our work leverages this difference to optimize memory consumption.  \n\n\n(2) **Practicality of the Proposed Algorithm**.  Beyond above considerations, a key advantage of our method is its simplicity and ease of integration. By making minimal modifications to existing architectures, our approach remains practical and can be readily applied to various model designs.  \n\n---\n\n### 3. More experimental results if using other units like a total block.\n\nPlease kindly refer to our response to Reviewer 1 (hdej).", "claims_and_evidence": "Yes. There is clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes. I have done my best to check the correctness of Theorem 4.1.", "experimental_designs_or_analyses": "The experimental designs (RoBERTa, LLaMA2-7B, LLaMA2-8B, ) are fair.", "relation_to_broader_scientific_literature": "This paper mainly related to the PEFT area such as LORA and DoRA.", "essential_references_not_discussed": "This paper has discussed related works.", "other_strengths_and_weaknesses": "Strengths:\n\n1. Clear writing and good motivation for this state-based FT method.\n2. Good results.\n\nWeaknesses:\n\n1. Lack of clear discussion about the difference between the current work and the pioneering work (Zhang et al., 2024b).\n2. Although the paper may seem complicated, the method is very simple. I think the previous part about control theory is more like an excessive theoretical embellishment. \n3. To provide more insight, could you provide more experimental results if using other units? For example, using a total block (contain ATTN and FFN) as the unit.", "other_comments_or_suggestions": "No", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes State-FT, a novel state-based parameter-efficient fine-tuning (PEFT) framework based on control-theory.  In contrast to weight-centric PEFT methods, i.e., LoRA and its variants, State-FT directly tunes the intermediate states of the computation graph rather than learning weight adapters.  Importantly, State-FT (which is also shown to generalize LoRA) avoids storing large intermediate activations while maintaining PEFT performance.  For attention-based LLMs, the authors demonstrate the efficacy of State-FT by introducing *parallel control*--wherein feed-forward network (FFN) blocks are fine-tuned--and *double control*--wherein both FFN and attention blocks are fine-tuned.  The authors show the efficacy of their framework across various models/sizes (ViT, RoBERTa, Llama-2-7B, Llama-3-8B), standard NLP tasks (e.g., GLUE and common-sense reasoning), and a toy vision example based on CIFAR-100.  Compared to LoRA/DoRA across the presented evaluations, State-FT methods are shown to significantly reduce GPU memory consumption and wall-clock time without degrading FT performance.

Reviewers agree that this approach is an interesting/novel addition to the long list of existing PEFT methods (while also noting the limitations of the vision results, which could be addressed in follow up works).  Reviewer uZCq noted the lack of comparison to QLoRA as a major concern.  However, the authors did well to show that State-FT and QLoRA are complimentary, effectively addressing these concerns during the rebuttal.  Further discussion included clarifications on the framing of existing PEFT methods in the context of control theory, which the authors showed offers a fresh take on PEFT and generalizes existing approaches.
Decision: Accept (oral)