Study of Training Dynamics for Memory-Constrained Fine-Tuning
Authors: Aël Quélennec
Nour Hezbri
Pavlo Mozharovskyi
Van-Tam Nguyen
Enzo Tartaglione
Venue: ICLR 2026 Poster (2026)
Keywords: Efficient Learning
Energy Saving
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We propose a dynamic channel selection algorithm to perform learning given a memory constraint.

Abstract: Memory-efficient training of deep neural networks has become increasingly important as models grow larger while deployment environments impose strict resource constraints. We propose TraDy, a novel transfer learning scheme leveraging two key insights: layer importance for updates is architecture-dependent and determinable a priori, while dynamic stochastic channel selection provides superior gradient approximation compared to static approaches. We introduce a dynamic channel selection approach that stochastically resamples channels between epochs within preselected layers. Extensive experiments demonstrate TraDy achieves state-of-the-art performance across various downstream tasks and architectures while maintaining strict memory constraints, achieving up to 99\% activation sparsity, 95\% weight derivative sparsity, and 97\% reduction in FLOPs for weight derivative computation.

Published as a conference paper at ICLR 2026
STUDY OF TRAINING DYNAMICS FOR MEMORY- CONSTRAINED FINE-TUNING
Aël Quélennec, Nour Hezbri, Pavlo Mozharovskyi, Van-Tam Nguyen & Enzo Tartaglione
LTCI, Télécom Paris Palaiseau, France
(name) ) (surname] )
@tteeeeeeeeeeee @telecom-paris:
ABSTRACT
Memory-efficiene training of deep neural networks has become increasingly impor- tant as models grow larger while deployment environments impose strict resource constraints. We propose TraDy, a novel transfer learning scheme leveraging two key insights: layer importance for updates is architecture-dependeee minable priori, while dynamic stochastic channel selection provides superior gradient approximation compared to static approaches. We introduce a dynamic channel selection approach that stochastically resamples channels between epochs within preselected layers. Extensive experiments demonstrate TraDy achieves state- of-the-art performance across various downstream tasks and architectures while maintaining strict memory constraints, achieving up to 99% activation sparsity, 95% weight derivative sparsity, and 97% reduction FLOPs for weight derivative computation.
and deter-
Online Network Training
Offline
Layer Selection
B Buen
B
On-device deployment
Random
channel sampling
Training epoch
Training epoch 2
Figure 1: TRady dynamically reselects the subgraph to update within the memory budget Biem-
INTRODUCTION
In the span of a decade, machine and deep learning have become key technologies in the computer science landscape. They have found wide variety of practical applications in fields such Natural Language Processing Vaswani et al. (2017) Tenney et al. 2019). Computer Vision Krizhevsky et al.
(2012): Simonyan & Zisserman 2014); Minaee et al. 2021). Speech Recognition Deng al. (0133: : Nassif et al. (2019). This surge in popularity can be largely explained by the ever-increasing performances of architectures, intimately linked hardware innovations Baji (2018). The design of better-performing parallel computing units (such as GPUs and TPUs) allows the training of large neural networks that feature increasing overparameterizztton compared their predecessors [Sevilla] et al. (2022). If this trend further demonstrates deep learning principles' innate generalization po- tential, raises ecological and technical concerns. Training and exploitation of these architectures require very high energy consumption, and their deployment in real-world environments is impossible without extensive compression, leading to performance worsening.
The research field of efficient neural network compression, consequently, has gained surge of interest in recent years. The main pillars this research area are quantization, low-rank compression, efficient design of compact models, knowledge distillation, and network sparsification (also known as pruning) Cheng et al. (2018); Deng et al. (2020). These methods aim to optimize the trade-off between memory/energy consumption and the inference accuracy of models in resource-constrained

---

Published as a conference paper at ICLR 2026
environments. However, inference is only part of the life cycle of deep neural network, and these works do not provide solutions to perform trained offline and deployed on-device suffer from phenomenon called data drift Sahiner al. (2023) which results in performance degradation over time. Alternatively, enabling on-device learning would improve the viability and efficiency of emhedded Al through several use cases, including user adaptivity or lifelong learning Incel & Bursa (2023).
training. As such, compressed models
One major obstacle making on-device learning practical is the computational and memory burden of backpropagation. For embedded devices, limited memory and computational capacity create hard constraints that cannot be exceeded. Some methods attempt address memory limitations by exploring alternatives to backpropagation, like hyperdimensional computing for tasks like image segmentation Yang et al. (2023a), the Borward-Forwald algorithm Hinnon (2022), and PEPITA Pau & Aymone (023). Although these strategies are promising, they generally don't t match the perfor- mance of backpropagation- based techniques. One direct approach attempting to solve memory and computation issues proposed by Lin et al. (2022) , where static subnetwork is updated for any downstream task. Orthogonally, Yang et (2023b) and more recently Nguyen et al. (2024) reduce memory consumption by compressing the elements store for backpropagation. Each approach, however, comes with its own limitations-eithe compromising accuracy or introducing additional latency.
We propose Training Dynamics (TraDy) for memory-comee constraind transfer learning. Given a pre- trained network, we make three key propositions regarding the training dynamics when performing memory constrriied transfer learning on a downstream task. Building on top of these, we design dynamic channel selection algorithm for efficient transfer learning under memory constraints (Fig. Our main contributions can be summarized as follows.
We show that stochastic gradients exhibit heavy-tailed behavior during transfer learning, creating natural sparsity patterns that facilitate efficient gradient pruning (Sec. 3.2
We show that the relative importance of network layers remains consistent across downstream tasks and primarily depends on network architecture rather than task specifics, enabling a priori layer selection (Sec. 3.3).
We establish that channel importance distributions within layers are task-dependent and cannot be predetermined without task data, while calculating importance metrics for all channels contradicts on-device memory constraints (Prop. 3.2 and Sec. S.c.
We introduce TraDy, a dynamic stochastic channel selection approach that resamples chan- nels between epochs within pre-selected layers, effectively approximating the full gradient while maintaining strict memory constraints (Sec. 3.4).
Our experiments illustrate that TraDy achieves tate-offtthe-rr performance in various downstream tasks and network architectures while respecting memory limitations through high levels of both weight and activation sparsities alongside reduced FLOPs, validating theoretical insights (Sec.
2 RELATED WORKS
Gradient Pruning. Sub-network selection for training, whether static or dynamic, can be referred to gradient pruning. Unlike classical pruning, gradient pruning preserves the complete network during inference, only modifying the backpropagation phase by selectively computing gradients based on specific criteria. While gradient pruning on-device learning primarily addresses memory constraints, other applications focus on accelerating training with minimal accuracy impact Zhang al.) (2024): Bragagnolo et al.] (2022); Li al. (2023); Ye al.) (2020); McDanel et al. (2022). Particularly relevant to our fine-tuning approach is Lee et al. (022)) who explore gradient pruning as a regularization technique. They demonstrate that network blocks can contribute either positively or negatively to downstream task performance, creating task-specific optimal configurations for selective updating. Their work shows that the ratio of gradient norm to parameter norm effectively predicts which blocks should be updated or frozen for optimal transfer learning performance.
On-Device Learning. Our work draws inspiration from three key contributions in the on-device leamning domain, where memory and energy constraints necessitate efficient fine-tuning of pre-trained models rather than training from scratch.
Lin et al. (2022) introduced Sparse Update (SU) schemes, selective parameter updating strategy
2

---

Published as a conference paper at ICLR 2026
that enables fine-tuning extreme edge devices along with operator reordering and quantization- aware scaling. Their approach demonstrated that memory-efficient subnetworks can yield acceptable performance downstream tasks. However, finding adequate SU schemes requires heavy pre- computation through offline accuracy contribution analysis, followed by evolutionary search for each network and memory budget. Additionally, SU applies uniformly across all downstream tasks, implicitly assuming that selected layers and channels are optimal for each individual task and should remain fixed throughout training.
Building on this foundation, Kwon et al. 2024) improved adaptability to new architectures, datasets, and memory budgets. Their approach ranks layers by computing Fisher information on activations from downstream task samples, then applies reweighting by parameter count and MAC operations. Despite increased flexibility, computing Fisher information for all network channels requires more memory than gradient computation itself, contradicting the original memory constraints. Like SU, this approach still employs static selection that does not adapt during training.
Quélennec et al. (2024) propose dynamic subnetwork selection between epochs using "velocity" metric that quantifies neuron output changes when fed with consistent data. Their results demonstrated accuracy improvements over static selection within fixed parameter budgets. While promising and flexible across networks and datasets, this method is limited by its exclusive focus on parameter count without considering activation memory, which represents an equally significant constraint in on-device scenarios Cai et al. (2020).
Our work builds upon these foundations by analyzing transfer learning dynamics in deep neural networks and demonstrating how a theoretically -rrouuddgg dynamic channel selection strategy can overcome limitations of previous approaches while maintaining strict memory constraints.
METHOD
In this section, we unfold our study towards parameter-efficient fine-tuning under extreme memory constraints. After formulating our problem in Sec. [31] we introduce the theoretical foundations of heavy-tailed gradient distributions and our memory-aware gradient norm metric in Sec. 3.2 This theoretical framework guides our analysis of layer behavior in Sec. 3.3 where we demonstrate the sehiitecturedeeeendee nature of layer importance. Building on these insights, we introduce our dynamic channel sampling strategy in Sec. 3.4 which enables efficient transfer learning within strict memory budgets by stochastically resampling channels between epochs from pre- selecee layers.
3.1 PROBLEM FORMULATION AND NOTATIONS
Our goal is to fine-tune pre-trained neural network on a downstream task under specific memory constraints, without prior knowledge of the target task. Although the target device can execute the complete forward pass, the memory limitations prevent training all network parameters simultaneously. Therefore, we aim to strategically select which portions of the architecture to train, optimizing performance while keeping the combined weight and activation memory within the specified budget. Our analysis focuses specifically on standard 2D convolutions within Convolutional Neural Networks (CNNs), excluding bias terms] The CNN then writes sequence of n convolutional layers:
F(X)= (Cw., Cw.-,, o... Cwv, Cw,, (W),
with X' the input of the network and W; E RC'xXXXXD the weight kernels, C' and C' the number of input and outputs channels and D the kernel dimensions.
Given the i-th layer, we note A, E R BBCCxXXW and A4+1 E =X'''" as its input and output activation tensors, where B is the batch size, H and W are the width and height of the feature map. 0c To compute the weight derivatives aw; the loss L calculated at the output of the network and
backpropagated to the 7th layer through the activation derivatives derivatives:
ac We then get the weight
JA4T1
dc
1ΣΣΣ
H' W
[A llllb....o c.
OC
aw.
TA
c.k.
b.o' ,'.."
where h = h¹' x stride + h x dilation, w = w' x stride + I x dilation and A is the padded input.
Similar to unstructured pruning. removing individual parameters does not yield significant compu-
'A similar analysis can be conducted for fully-connected layers.
3

---

Published as a conference paper at ICLR 2026
tational and memory gains, as it creates inefficient unstructured sparse tensors. A more effective approach involves freezing along specific weight dimensions, enabling efficient tensor operations and creating structured gradient sparsity (Bragagnolo et al. (2022). When considering selective freezing, we have four potential dimensions: input channels, output channels, and the two kernel dimensions. However, these options differ significantly in their effectiveness for memory optimization. While freezing along output channels reduces memory needed for storing activation derivative tensors, these derivatives must still be fully computed to ensure accurate gradient propagation through subsequent layers. After analyzing all possibilities, freezing along the input channels dimension emerges as the only approach that simultaneously achieves both weight sparsity and activation sparsity.
From Eq. 2 we observe that updating an input channel requires storing only the corresponding activation values in memory. Specifically, when freezing weight tensors along the input channel dimension, the gradient components form natural groupings that can be treated as independent units. This dual benefit climinates both the storage requirements for the corresponding activations and the computational burden of calculating their associated weight gradients, making input channel freezing optimal for
scenarios.
Based on Eq. 2 we derive analytical expressions for both memory requirements and computational complexity associated with updating a single input channel e within layer i for a single data input. Let C. C' x D D represent the weight memory cost and C. H x W represent the activation memory cost for channel The total space complexity (9 (Qsass)c and time complexity (9 (Qume) are:
Opace) c = C. + CA. (Otie) D'C'H'WW.
(4)
These expressions demonstrate that input channel-level selection provides fine-grained control over both memory and computational consumption while maintaining the structural coherence necessary for effectively exploiting the heavy-tailed sparsity patterns described in the following section, thus achieving the critical combination of weight and activation sparsity essential for memory- efficieen fine-tuning.
3.2 HEAVY-TAILED THEORY AND GRADIENT NORM METRIC
The stochastic nature of gradient descent has significant theoretical implications for our approach. Sim- sekli et al. (019) established that stochastic gradient noise follows a heavy-tailed distribution during training with SGD. Such distributions are characterized by a tail-index parameter E (0, 2) and exhibit power-law decay proportional to 1/|a||++1 When a = 2, this distribution reduces to a Gaus- sian; for all other values of o, the resulting random variable has infinite variance. This heavy-tailed noise can be mathematically formulated as:
U&(W) AW AWW
where AW denotes the true gradient computed using the entire dataset, AWW represents the stochastic gradient estimated from l randomly sampled data points, and Us follows a symmetric o--table distribution Up Sa)((a) In this notation, serves scale parameter controlling the distribution's spread around zero.
Building on this foundation, Wan et al. 2023 demonstrated that injecting heavy-tailed noise during weight updates inherently enhances network compressibility for pruning operations. Their key insight reveals that heavy-tailed noise causes the weight matrix columns follow multivariate heavy-taile distributions independently of each other. Consequently, the norm distribution becomes highly skewed as a small subset of columns exhibits disproportionately large norms while most remain relatively small. This concentration means that the overall weight matrix norm is mostly determined by just few dominant columns, creating an implicit structure that aligns perfectly with sparse update requirements.
In our approach, we extend this theoretical framework to the domain of gradient pruning rather than weight pruning. From (5) we can observe that gradients naturally decomposes the sum of the stochastic gradients and a heavy-tailed noise term Ug. Applying the insights from Wan et al., we expect that gradient norms will concentrate disproportionately in small subset of channels. This creates a natural opportunity for selective gradient computation and parameter updating. To systematically exploit this property, we define the input channel gradient norm as:
((
0L JW,
||
OC
2
=
Ow,
(6)
2l
c'
4

---

Published as a conference paper at ICLR 2026
While the raw gradient norm provides valuable information about update importance, it fails to account for the memory constraints that are central to our scenario. To address this limitation, we introduce memory-aware metric called the Reweighted Gradient Norm (RGN). This metric incorporates both computational significance and memory efficiency by dividing the raw gradient norm by the total memory cost associated with updating that channel. Using the notation established in Sec. 3.1
we define RGN as:
(
OC aw:
RGN C. + C.A.
This reweighting counteracts the bias toward channels with higher parameter counts as they naturally show larger gradient norms. By directly incorporating memory costs, RGN creates different layers and channels' order compared to the raw gradient norm. It is thus well-suited for settings optimizes update efficiency through prioritization of less memory-intensiv channels when raw gradient norms are similar. This allows more parameters to be updated within the same memory budget. potentially improving performance per memory unit. We use this RGN metric throughout our analysis to examine layer and channel importance across different architectures, datasets, and seeds, informing our final solution design.
3.3 LAYERS BEHAVIOR DURING FINE-TUNING
Just heavy-tailed gradient properties create natural sparsity patterns among channels, we hypothe- size that similar dynamics may govern layer-level importance. This section explores how gradient norm distribution across layers influences their relative contribution to the fine-tuning process and how this knowledge can guide our parameter selection strategy. We decide to characterize the layer reweighted gradient norm as follows:
C (8x Σ c=1 c.W. + C.
dc OW
Σ RGN,
Σ
OC aw.
).1:
(8)
(Qspace);
RGN
Proposition 3.1. The relative ranking of layers to their reweighted gradient norm remains largely invariant over time during training and across different downstream tasks. This ranking primarily determined by the network architecture rather than dataset-specijy characteristics.
Based on neural network architecture, certain layers consistently exhibit higher gradient norms than others. This architectural dependency is particularly evident in networks with residual connections. Skip connections mitigate gradient vanishing by effectively reducing the virtual depth of the network for certain computational paths. As a result, we typically observe characteristic pattern in the distribution of gradient norms: the first layer of each residual block generally displays significantly higher gradient norm than subsequent layers within the same block.
We provide detailed analysis of this phenomenon in the appendix, Sec. C and empirically validate such behavior in Sec.4.11
Based on these observations, that naturally receive higher gradients. Recent literature supports this approach, with multiple studies demonstrating that selectively updating certain layers provides significant contributions to model optimization on downstream tasks Kaplun et al. (003); Zhang & Bottou| (0024); Lee et al. (2022). The practical implication is substantial: depending on the similarity between pre-training and downstream tasks, updating only a carefully selected subset of layers can maintain performance comparable to full fine-tuning while significantly reducing memory requirements.
strategically restrict parameter updates to the subset of layers
3.4 DYNAMIC CHANNEL SAMPLING
After analyzing layer-level behavior, we now focus on individual input channels within selected layers.
Proposition 3.2. The distribution of channel gradient norms varies between datasets.
From the weight derivative in (2). two key components emerge: activation maps reflecting network feature extraction and activation derivatives shaping the task-specific loss landscape. Both are fundamentally task-dependent, justifying that channel gradient norms vary between downstream
5

---

Published as a conference paper at ICLR 2026
Algorithm TraDy
Input:
Pre-trained backbone weights W, number of epochs train data D Draiin test data Diest memory budget Bmem. set of relevant layers {LK}.
Function:
for epochs = to n do
Randomly sample channels {C"} within the set of relevant layers {LK} along uniform probability distribution until the memory budget Bmem is met.
Update weights of the selected channels using Dirain
end for
Evaluate the fine-tuned backbone using Diest-
tasks. We provide empirical validation in Sec. While Sec. 3.3 establisss that layers can be predetermined architecturally. static channel selection proves inadequate since RGN distributions downstream data is unavailable offline and memory constraints prevent full gradient computation, directly estimating channel RGN distributions introduces overhead contradicting our efficiency goals. Instead, we propose TraDy, dynamic sampling strategy operating within memory constraints. Our approach randomly selects input channels to update from the predetermined layers, resampling between epochs while maintaining strict memory budget compliance throughout training. This strategy ensures that the combined activation and weight memory consumption remains strictly below the specified memory budget throughout the training process, effectively balancing exploration of the channel space with the practical constraints edge devices.
task-dependent. In real-world scenarios where
Leveraging layer selection from Sec. 3.3 most gradient information concentrates in selected layers,
as predicted by heavy-tailed theory (Sec. 3.2). Random channel selection changing dynamically ensures the expectation of selected gradients approximates the full gradient within efficient layers over time. Let AW, denote the non-null gradient epoch and AW(c)) the sparse gradient from randomly selected set of channels {C"} at epoch i within pre-selected et of layers {Txx) )- Following the principle that stochastic gradient expectation equals full gradient expectation, and due to our layer selection excluding magnttue gradients while the stochastic channel selection follows a uniform distribution, we have:
E
AW
~
E Σ AWioy
The computational complexity of randomly and successively selecting h: elements from n channels (k log(n)), negligible compared gradient computation itself.
We present here TraDy, our dynamic subnetwork update pipeline for transfer learning, under memory constraints, depicted in Alg. [ Given pre-trained backbone and training dataset, channels randomly sampled within the fixed set of layers interest { {Lx} } and updated conditioned on the memory budget (line 4). At the end of the training, evaluate our model's performance on the test dataset (line 7). In the next section, we will present our empirical results.
4 EXPERIMENTS
This section describes the experiment conducted to validate the hypothesis proposed in Sec. Baa well as compare its performance to other sparse update strategies. A complete description of the experimental setup is proposed in Sec. D.I of the appendix.
4.1 GRADIENT STUDY
Heavy-Tailed Stochastic Gradient. We empirically validate the heavy-tailed characteristic of stochastic gradients during fine-tuning, as introduced Sec. Following methodology similar to Şimşekli et al., we use Mohammadi al. (2015) estimator for a-stable distributions. For each fine- tuning epoch t, we collect stochastic gradients of all P trainable parameters across S training steps, constructing a P x S matrix. This matrix is processed by the estimator produce ag, representing the heavy-tailed index of the stochastic gradient distribution during epoch t. Fig. 2 iillustrates the evolution of for three network architectures when fine-tuned three diverse downstream

---

Published as a conference paper at ICLR 2026
meunet-inl
proxyless-wll.
mbv2-w0.33
2.5
1.9
Flowess
1.5
Pets
CUB
1.7
CIFAR-10
CIFAR-130
Food
VWW
1.4
1.44
1.3
50
100 Epocaas
130
2.0
1.2
30
100 Eporaa
150
200
1,3
50
100 Epochs
150
2003
Figure 2: Evolution stochastic gradient heavy-tailed index at.
mbbr2-w0.35
mbb2-w00.35
2.0
CIFAR-10
Flowers
0.8
Flwwrrs
0.8
10
CI
0.5
CUB
CUB
Food
0.1
Food
0.1
www
022
VWw
0.2
Pets
Pets
CIFAR to
Flavess
OIPAR-1 198
CUB
Foot
VIFW
Fids
CIEAR-I J0
Flovers
CWAR- 103
CUB
Food
VWW
Fede
(a) Spearman coneelation layer gradient norm.
gradient norm.
Figure 3: Validation of channel and layer proposition across seeds and datasets for MobileNetV2.
tasks. Consistently across all scenarios, we observe that a remains below two, confirming the heavy- tailed nature of stochastic gradients. Interestingly. MCUNet exhibits significantly more heavy- stived gradient behavior compared to other architectures. We hypothesize this stems from its compressed architecture design, which renders it more parameter-efficient. This increased efficiency likely concentrates gradient information more densely in fewer parameters, intensifying the heavy-tailed characteristic of its gradient distribution.
Layer Gradient Norm Distribution. Next, we examine Proposition 3.1 which posits that the same network produces consistent layer gradient norm topologies across different downstream tasks. We define "layer topology"
vector containing the cumulative gradient norm of each layer across all training epochs. To quantify similarity between topologies, we compute Spearman correlation coefficients between all possible pairs of fine-tuning runs across our seven downstream datasets, using three random seeds per dataset. This yields comprehensive 21 x correlation matrix for each network architecture, as visualized in Fig. 3a with MobileNerV2. The results provide strong empirical support for our proposition- even in the worst-case comparison between the most dissimilar dataset pairs, the correlation coefficient never falls below 0.8. This remarkably high correlation confirms that layer-level gradient importance rankings remain largely invariant across diverse downstream tasks, validating our approach of pre-selecting layers based solely architectural considerations. Channel Gradient Norm Distribution. We now validate Proposition [.2] which addresses gradient behavior the channel level. Using methodology parallel our layer analysis, we construct vectors where each element represents the cumulative gradient norm of a specific channel across all training epochs. This yields dinee-iimnsionnl representation of channel importance for each fine-tuning experiment. To assess whether these channel importance distributions differ significantly between tasks, we employ Student's 's T-test for pairwise comparisons, with results visualized in Fig- (3b The analysis reveals a striking pattern: p-values for all inter-dataset comparisons are effectively zero, strongly rejecting the null hypothesis that channel gradient distributions from different datasets share the same mean values. This confirms proposition that channel-level gradient importance patterns are fundamentally task-dependent and cannot be predetermined offline without access to the target dataset. Notably, the diagonal blocks in our visualization representinggggg comparisons between different random seeds for the same dataset- fataset-mostly feature non-zero p-values. This secondary finding indicates that while channel importance varies dramatically between tasks, it retains some
7

---

Published as a conference paper at ICLR 2026
consistency across different initializations for the same task.
Results for other CNN architectures and transformers architectures show similar patterns (see Sec. D.2 and Sec. D.7of of the appendix).
4.2 MAIN RESULTS
Preamble. To rigorously validate the claims presented in Sec. we systematically compare three distinct
channel selection strategies. For each method, we explore both Static and Dynamic variations. In the Static approach, channel selection occurs once at initialization, with the same channels updated throughout training. The Dynamic approach reapplies the selection rule after each epoch, resulting in different channels being updated over time.
S-Fail Rondom-
S Det RGN
S-Top Random
0.6
Vidotity
0.4
DePal Rondom
RGN-
TraDy
! Full Random, This baseline strategy randomly selects chan- nels from throughout the entire network architecture, without layer-based prioritization. It serves as a control to evaluate the benefit of our layer selection approach.
S-hal Band
S.Dcr AGN
Thipk Band
Vilscity
Dangl Rundon
D-lke DGN
Tadle
② Det RGN: For each training epoch, we first compute the full gradient without updating network weights, then determin- istically select channels with the highest RGN values. While computationally impractical for real-world deployment (as it requires calculating the complete gradient), we expect this oracle-like method to serve performance.
Figure 4: T-test comparisons of average final test accuracies across multiple experimental di- mensions.
upper-bound reference for
3 TopK Random: Randomly samples channels from within the predetermined subset of top K layers. The practical choice of K is defined in the appendix. In its dynamic version, this corresponds to our proposed algorithm TraDy.
We benchmark our method against Lin et al.'s Sparse Update (SU) scheme, which represents the current state-of-the-art static channel selection for memory-constrained fine-tuning, as well as Quélennec et al.' 's Velocity method, which dynamically selects neurons based their output changes between epochs.
Discussion. In Fig. 4 we represent the results of paired t-tests comparing the average final test top-1 accuracies across all experimental conditions. Fach cell represents a statistical comparison testing the hypothesis that the selection strategy the y-axis achieves higher mean test accuracy than the strategy on the x-axis. We provide the complete table of results (Tab. I and Tab along with similar results for transformer architectures D.7) and comparisons with full fine-tuning in the appendix. Regarding our introduced strategies, we observe that each dynamic variant (prefixed with D) outperforms its static counterpart (prefixed with S). Notably, while S-Full Random yields the worst results, our proposed algorithm-whie which restricts selection to top K layers and incorporates dynamic selection- arcection-achieves the best performance, even surpassing D-RGN Deterministic, which was expected to serve an upper bound. Velocity achieves the second-best accuracy performance among all evaluated methods, demonstrating the effectiveness of dynamic selection approaches.
We hypothesize that under extremely constrained memory budgets, D-RGN Deterministie's approach of always selecting channels with maximal RGN effectively leaves many channels with smaller but significant RGN values permanently frozen. This likely causes the training process to follow the direction of maximal gradient slope, potentially leading to local minima. In contrast, TraDy follows, on average, the same direction the non-null gradient while introducing beneficial stochasticity, as layers with negligible gradients are excluded, but dynamic resampling occurs among significant ones. This hypothesis is further supported by S-TopK Random's poor performance (second worst strategy). highlighting that dynamic reselection is crucial for achieving good results.
Efficiency Metries Analysis. Fig. 5 illustrates the temporal evolution of key efficiency metrics during Mob fine-tuning on the Food dataset under the most restrictive memory constraint. These results showease patterns that remain consistent across different network-date baudeett combinations, with complete training metrics available in the following GitHub repository We observe that all our methods achieve similar levels of weight and activation sparsity, respectively in the range of 93% to 99% and 97.5% to 99.5%. SU trades off extremely low activation memory for higher weight memory, possibly linked to the evolutionary search process implicitly maximizing the amount of parameters updated. In comparison, we observe more balanced trade-off
8

---

Published as a conference paper at ICLR 2026
Smail Menrory Budget
Smail Merrory Budget
Small II Meroory Badges
Wagght qpartity (36)
c6
Activatica sparsity 156
07.0
80
0
Full Randam
92.0
Det RGN
90.0
Vilority
be b
Dyramic
85.0
Spadko
Epocho
Epcels
(a) Weight sparsity evolution (br-(b) Activation sparsity evolution (c) Computational savings weight deriva- ing training.
during training.
tive FLOPs.
Figure 5: Efficiency metries comparison across channel selection strategies during MobileNetV2 fine-tuning on Food dataset under memory constraint. Results show evolution of sparsity levels and computational savings throughout training.
with our TraDy algorithm, suggesting that maximizing weight update does not necessarily result in improved task performance.
Notably, while Velocity achieves competitive accuracy (second-best among all methods), its neuron-level selection strategy results in substantially lower activation sparsity (20-40% range) compared to our channel-based approaches (97-99% range). This is because updating even single neuron within a layer requires storing the complete activation map from the previous layer. Consequently, Velocity also achieves lower FLOPs savings (approximately 88%) compared to TraDy and other channel-based methods (97%). Moreover, due to reweighting focusing solely on weight memory, the Velocity selection strategy results in the selection of computationally expensive neurons to update, further increasing FLOPs requirements relative to the other strategies explored in our paper. This is without accounting for the additional computational overhead of computing the velocity metric for all neurons or the memory overhead of storing all activation values necessary to compute this metric.
TraDy consistently requires significantly fewer FLOPs for weight derivative computation compared to both SU and Velocity. This computational advantage stems from the emergence of depthwise convolution layers among the top-ranked layers, which by design have low computational costs during both forward and backward passes.
The combination of both high weight and activation sparsity levels makes our method intrinsically more competitive than strategies that focus on either dimension alone. Methods like Jiang al. (2022) 's Back Razor or Ngguen et al.] (2025)'s 's ASI achieve similar levels of activation sparsity or compression rates but without the weight sparsity, while Velocity optimizes weight memory at the expense of activation memory. TraDy's balanced approach to both dimensions is particularly advantageous for on-device learning scenarios where all memory resources are strictly constrained.
CONCLUSION
In this work, we introduced TraDy, memory-efficien transfer learning approach that dynamically selects channel subsets for update under tight resource constraints. Our method builds on two key insights: stochastic gradients often exhibit heavy-tailed behavior, leading to inherent sparsity, and layer importance remains consistent across tasks while channel relevance varies. By stochastically resampling channels between epochs within architecturally important layers, our approach proves its effectiveness in several challenging transfer learning scenarios, including training on efficient architectures designed for on-device deployment.
Future work will explore connections between stochastic channel selection and optimization theory, and extend approach to broader network architectures for efficient on-device learning.

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
Part of this work was funded by the European Union's HORIZON Research and Innovation Pro- gramme under grant agreement No 101120657, project ENFIELD (European Lighthouse to Manifest Trustworthy and Green AI), by the European Union's Horizon Europe Research and Innovation Programme under grant agreement No. 101 120237 (ELIAS), and by French National Research Agency (ANR-22) 22-PEFT-0000 and and NE-FITNESS project. This work was also supported by the French National Research Agency (ANR) in the framework of the IA Cluster project "Hi! PARIS Cluster 2030" under Grant ANR-23- IACI-0055, and by the Hi! PARIS Center on Data Analyties and Artificial Intelligence. Computation was performed thanks to GENCI-IDRIS ressources (Grant
as part of France 2030, the NF-NAI project
REFERENCES
Toru Baji. Evolution of the gpu device widely used in ai and massive parallel processing. In 2018 IEEE 2nd Electron devices technology and manufacturing conference (EDTM), pp. 7-9. IEEE, 2018.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. nents with random forests. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, 2014.
discriminative compo-
Andrea Bragagnolo, Enzo Tartaglione, and Marco Grangetto. To update or not to updatc? neurons at equilibrium in deep models. Advances in Neural Information Processing Systems, 35, 2022.
Han Cai, Ligeng Zhu, and Song Han. ProxylessNAS: Direct neural architecture search target task and hardware. In International Conference on Learning Representations, 2019. URL pdf
https://arxiv. hwtp://..rgg/dd////111/
Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. Tinytl: Reduce memory, not parameters for
efficient on-device learning. Advances in Neural Information Processing Systems, 33:1 13:11125-11229, 2020.
Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. Model compression and acceleration for deep neural networks: The principles, progress, and challenges. IEEE Signal Processing Magazine, 35 (1):126-136, 2018.
Aakanksha Chowdhery, Pete Warden, Jonathon Shlens, Andrew Howard, and Rocky Rhodes. Visual wake words dataset. arXiv preprint orXiv:/906.0572) 2019.
Dorottya Demszky, Kelvin Guu, and Percy Liang. Transforming question answering datasets into natural language inference datasets. arXiv preprint 1809:/8009002920 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, 2009.
Lei Deng, Guoqi Li, Song Han, Luping Shi, and Yuan Xie. Model compression and hardware acceleration for neural networks: A comprehensive survey. Proceedings of the IEEE, 108(4): 485-532, 2020.
Li Deng, Geoffrey Hinton, and Brian Kingsbury. New types of deep neural network learning for speech recognition and related applications: An overview. In 2013 IEEE international conference on acoustics, speech and signal processing, pp. 8599-8603. IEEE, 2013.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv: 1706.026777 2017.
Haoze He, Juncheng B Li, Xuan Jiang, and Heather Miller. Smt: with sparse matrices. In The Thirteenth International Conference on Learning Representations, 2025.
Fine-tuning large language models
10

---

Published as a conference paper at ICLR 2026
Geoffrey Hinton. The forward-forward algorithm: Some preliminary investigations. arXiv preprint arXiv:2212.13345, 2022.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3 2022.
Ozlem Durmaz Incel and Sevda Ozge Bursa. On-device deep learning for mobile and wearable sensing applications: A review. IEEE Sensors Journal, 2023.
Ziyu Jiang, Xuxi Chen, Xueqin Huang, Xianzhi Du, Denny Zhou, and Zhangyang Wang. Back razor: Memory-efficient transfer learning by self-sparsifidd backpropagation. Advances in neural information processing systems, 35:29248-29222 2022.
Gal Kaplun, Andrey Gurevich, Tal Swisa, Mazor David, Shai Shalev-Shwartz, and Eran Malach. Less is more: Selective layer finetuning with subtuning, 2023.
Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolu- tional neural networks. Advances in neural information processing systems, 25, 2012.
Young D. Kwon, Rui Li, Styliamos I. Venieris, Jagmohan Chauhan, Nicholas D. Lane, and Cecilia Mascolo. Tinytrain: Resource-aware task adaptive sparse training of dnns at the data-scarce edge, 2024. htt.:tts::/aaxtttt
wrg. /200 09988
Yoonho Lee, Annie S Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Percy Liang, and Chelsea Finn. Surgical fine-tuning improves adaptation to distribution shifts. arXiv preprint arXiv:2210.11466, 2022.
Ziyu Li, Enzo Tartaglione, and Van-Tam Nguyen. Scotti: Save computation at training time with an adaptive framework. Proceedings of the IEFE/CVF International Conference on Computer Vision, pp. 1443-1452, 2023.
Lin, Wei-Ming Chen, Yujun Lin, Chuang Gan, Song Han, et al. Mcunet: Tiny deep learning on iot devices. Advances in Neural Information Processing Systems, 33, 2020.
Ji Lin, Ligeng Zhu, Wei-Ming Chen, Wei-Chen Wang, Chuang Gan, and Song Han. On-device training under 256kb memory. Advances in Neural Information Processing Systems, 35, 2022.
Shih- Yang Shih-Yang Liu, Chien-Y. Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora:
low-rank adaptation. In Forty-first
International Conference Machine Learning, 2024.
Yinhan Liu. Roberta: robustly optimized bert pretraining approach. arXiv:190..1169,, 364, 2019.
arXiv preprint
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEFF/CVF international conference on computer vision, pp. 10012-10022, 2021.
Bradley McDanel, Helia Dinh, and John Magallanes. Accelerating dnn training with structured data gradient pruning. In 2022 26th International Conference on Pattern Recognition (ICPR), pp. 2293-2299. IEEF, 2022.
Shervin Minaee, Yuri Boykov, Fatih Porikli, Antonio Plaza, Nasser Kehtarnavaz, and Demetri Terzopoulos. Image segmentation using deep learning: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(7):3523-3544 2021.
Mohammad Mohammadi, Adel Mohammadpour, and Hiroaki Ogata. On estimating the tail index and the spectral measure of multivariate o-stable distributions. Metrika, 78(5):549-5611 2015.

---

Published as a conference paper at ICLR 2026
Ali Bou Nassif, Ismail Shahin, Imtinan Attili, Mohammad Azzeh, and Khaled Shaalan. Speech recognition using deep neural networks: systematic review. IEEE access, 7:19143-19165, 2019.
Le- Re-Truun Nguyen, Ael Quélennec, Enzo Tartaglione, Samuel Tardieu, and Van-Tam Nguyen. Activation map compression through tensor decomposition for deep learning. arXiv preprint arXiv:2411.06344 2024.
Le- Trung Nguyen, Aël Quélennec, Van-Tam Nguyen, and Enzo Tartaglione. Beyond low-rank decom- position: A shortcut approach for efficient on-device learning. arXiv preprint arXiv:2505.0500 2025.
Maria-Elens Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pp. 722-729. IEEE, 2008
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEFE conference on computer vision and pattern recognition, pp. 3498-3505. IEFE, 2012.
Danilo Pietro Pau and Fabrizio Maria Aymone. Suitability of forward- and pepita learning to mlcommons-tiny benchmarks. In 2023 IEEE International Conference on Omni-layer Intelligent Systems (COINS), 2023.
Adam Poliak. A survey on recognizing textual entailment as an nlp evaluation. arXiv preprint 1rXiv:2010.03061, 2020.
Aël Quélennec, Enzo Tartaglione, Pavlo Mozharovskyi, and Van-Tam Nguyen. Towards on-device learning on the edge: Ways to select neurons to update under budget constraint. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 685-694, 2024.
Berkman Sahiner. Weijie Chen, Ravi Samala, and Nicholas Petrick. Data drift in medical machine learning: implications and potential remedies. The British Journal of Radiology, 96(1 150): 20220878, 2023.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Biang-Chiel Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018
Jaime Sevilla, Lennart Heim, Anson Ho, Tamay Besiroglu, Marius Hobbhahn, and Pablo Villalobos. Compute trends across three eras of machine learning. In 2022 International Joint Conference on Neural Networks (JJCN), pp. 1-8. IEEE, 2022.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Umut Simsekli, Mert Gürbüüzalaaan, Thanh Huy Nguyen, Gaël Richard, and Levent Sagun. On the haive- theory of stochastic gradient descent for deep neural networks. arXiv preprint arXiv: 00018, 222, 2019.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1642, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL httph:ttp///i.... hwwp::://wwww... acl.elllwle wwt.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. Bert rediscovers the classical nip pipeline. arXiv 2019.
preprint arXivv
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems. 30, 2017.
Yijun Wan, Melih Barsbey, Abdellatif Zaidi, and Umut Simsekli. Implicit compressibility of overparametrized neural networks trained with heavy-tailed sgd. arXiv preprint 0881v:2306.08125, 2023.
12

---

Published as a conference paper at ICLR 2026
Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Be- longie, and Pictro Perona. Caltech-ucsd birds 200. Technical Report CNS-TR- TR-201, Caltech, 09 /wwp///llaee http/eeeiiddetaall.... CUB-200 pdf, http: wttp./visio...alaeeeee........ee wtt.ao/nn.....tcceeeee.........e edu/viss pedi a CUB 200000--000.0 htmll
2010. URL /sss://mccootttttt/ppoosss/////p
vi
Sunghyeon Woo, Sol Namkung, Sunwoo Lee, Inho Jeong, Beomseok Kim, and Dongsuk Jeon. Paca: Partial connection adaptation for efficient fine-tuning. arXiv preprint arXiv:2503.01905, 2025.
Junhuan Yang, Sheng, Yuzhou Zhang, Weiwen Jiang, and Lei Yang. On-device unsupervised image segmentation. arXiv preprint (ttt:2333.12755.. 2023a.
Yuedong Yang, Guihong Li, and Radu Marculeseu. Efficient on-device training via gradient filtering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 2023b.
Xucheng Ye. Pengcheng Dai, Junyu Luo, Xin Guo, Yingjie Qi. Jianlei Yang. and Yiran Chen. Accelerating enn training by pruning activation gradients. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXV 16, pp. 322-338. Springer, 2020.
Chiyuan Zhang, Samy Bengio, and Yoram Singer. Are all layers created equal? ArXiv, 2019.
Jianyu Zhang and Léon Bottou. Fine-tuning with very large dropout, 2024.
Zhi Zhang, Qizhe Zhang, Zijun Gao, Renrui Zhang, Ekaterina Shutova, Shiji Zhou, and Shanghang Zhang. Gradient-based parameter selection for efficient fine-tuning, 2024. URL https://
arxiv. wtw.v/oorr/abbs/22..00000
13

---

Published as a conference paper at ICLR 2026
TABLE OF CONTENTS
A Limitations
15
Parameter-Efficient Fine-Tuning
15
Layer Ranking Consistency Detailed Analysis
15
D Additional Experimental Results
16
D.1 Experimental Setup
16
D.2 Proposition Validation on CNN Architectures
D.3 Reweighted Gradient Norm Metric Validation
D.4 Layers RGN Behavior
18
D.5 TopK Layers Selection
19
D.6 Extension of Main Results
20
D.7 Results Transformers Architectures
20
D.8 Static Strategies Results
24
E LLM Usage
24
14

---

Published as a conference paper at ICLR 2026
A LIMITATIONS
Related Works. In Sec. we discuss three key approaches to efficient subnetwork selection for on-device learning, yet our experimental comparisons focus only on Lin et al. 's SU method. Que- lennec et al. 's implementation excludes activation memory from their budget calculations, making direct comparisons methodologically inconsistent with our approach which accounts for both weight and activation memory. Similarly, while Kwon et al. 's work offers improvements upon SU, the absence of publicly available code at the time of our research prevented us from implementing and benchmarking against their method.
On-device Implementation. Although our work aims to enable efficient on-device learning, we do not present metries on actual hardware performance (latency, energy consumption, etc.). This limitation stems from our method's 's reliance on dynamic channel reselection between epochs, which requires specialized implementation for efficient execution on edge devices. Our current implementa- tion serves as a simulation to demonstrate the potential algorithmic benefits, but further engineering work is needed to translate these theoretical gains into optimized on-device performance.
Backpropagation Cost. In our work, we report the FL.OPs gained regarding the computation of weight derivatives. We however acknowledge that total backpropagation cost includes both weight and activation derivative calculations. The latter depends the deepest layer requiring updates, as gradients must propagate from the output through all intermediate layers. Our approach typically selects relevant layers at greater depths than SU schemes, potentially increasing overall backpropa- gation latency despite weight derivative savings. In future work, we plan to explore techniques for exploiting the natural sparsity in activation gradients to enable compressed backpropagation, which would allow efficient updating of deeper layers with minimal accuracy degradation and reduced computational overhead.
B LINK WITH PARAMETER- FINE-TUNING
Garameter-Efficienn Fine-Tuning (PEFT) methods have emerged popular approach for adapt- ing pre-trained models to downstream tasks while minimizing trainable parameters. Prominent adapter- based methods like LoRA Hu al. (2022) and DoRA Liu et al. (2024) introduce low-rank decomposition matrices in parallel with frozen pre-trained weights, achieving impressive parameter efficiency by updating less than 1% of total parameters. LoRA decomposes weight updates into low-rank matrices that are trained alongside frozen weights, while DoRA further decomposes weights into magnitude and direction components, applying low-rank adaptation only to the directional component. These methods have demonstrated effectiveness across diverse architectures and tasks, establishing PEFT as a standard paradigm for efficient model adaptation.
However, adapter- based PEFT methods are fundamentally incompatible with extreme memory- constrained scenarios. These approaches introduce parallel computation paths that require computing forward pass through both weights and adapters paths during inference (thus increasing computational cost), and storing the full activation maps to update the adapters modules during backpropagation. Additionally, adapter modules introduce parameter storage overhead during training, each forward pass must execute through both the frozen backbone and adapter pathways. These limitations render such methods impractical for on-device learning where activation memory constitutes the primary bottleneck.
Among PEFT methods, PaCA Woo et al. (2025) represents the closest approach to our setting, it addresses both parameter and activation memory by randomly selecting channels for update within existing layers rather than introducing adapters. However, PaCA performs uniform random selection across all network layers without considering layer-wise gradient importance or dynamic resampling across epochs. In our experimental framework, this approach corresponds directly to our S-Full Random baseline, which we demonstrate to be the worst-performing selection strategy (Fig. 4 in our ablation study. practice PaCa outperforms adapter- based strategies and by transitivity, Traly provides further improvements in performance due to its innovations.
C LAYER RANKING CONSISTENCY DETAILED ANALYSIS
Let us consider the simple case of R convolutional layers having the same size, intercepted by ReLU activations, where a skip connection re-injects the input of the first in the final output ), reading
15

---

Published as a conference paper at ICLR 2026
Y' A++R-1 + A, Cw.+n-1 o... (A,) + A,. We also note 2, the i-th layer pre-activation, 1 the indicator function. and @ the Hadamard product operator. According to (2). the weights derivatives could be further expressed as:
R-++R-1 conv (ALtR-1
OC
Jc C 12.mm1120)
(10)
ay
vk E ( R - 2], we define I with the following recursive expression:
{
J(i + 1) = - ④ [2
(11)
J(k) (conv (J(k + 1), W + 12,00)
Subsequently, vk E [ + R 28-2 2],
ac conv (Ak: J(k)).
(12)
aW
Besides, the input derivative written as:
ac
ac
[ 1+ a(CWitR-1 o... CWi+i Cvr.)
(13)
0A:
oy
0A:
with I being the identity tensor. Given that we're working with pre-trained networks, we can leverage specific properties established during their initial training phase. Pre-trained deep neural networks typically undergo regularization via weight decay and gradient clipping, which constrains weight norms to generally remain below one. Simultaneously, the inclusion of batch normalization layers during pre-training ensures that activation norms are similarly bounded. When fine-tuning on downstream tasks that share reasonable similarity with the pre-training domain, these weight and activation properties tend to be preserved, as the magnitude of weight adjustments remains relatively small.
For instance, when [[e++1]] l2 S 1, Ak E [i, + R 2) and [Ail|2 S 1, we have that
OC
dC
<
< ... <
(14)
awM
aW
OW
While exceptions may occur-certain layers occasionally exhibit weight or activation norms exceed- ing one-these instances reflect inherent properties of the pre-trained network rather than task-specific adaptations. The fundamental insight is that layers consistently maintain their relative gradient norm proportions across diverse downstream tasks. This pattern becomes even more pronounced when using our reweighted gradient norm metric, as both channel weight and activation memory costs mrchteectueedependerr constants within each layer.
D ADDITIONAL EXPERIMENTAL RESULTS
D.1 EXPERIMENTAL SETUP
Training. Following Lin et al. (002)) we employ the same architectures pre-trained on Ima- geNet[Deng et al. (2009) MobileNet et V Sandler et al. (2018), ProxylesSNAS(Car et al. (2019), and MCUNet et al. (2020) (we load the weights provided in their code implementation). We perform training on Nvidia Tesla V100 SXM2 and systematically train the classifier layer, independently of the freezing strategy. Algorithms are implemented in Python using Py Torch 2.0.0. We also provide results transformer architectures in the appendix.
Datasets. We collect channel freezing metrics and transfer learning accuracy on multiple downstream
datasets: CIFAR- 10 Krizhevsky et al. (2009). CIFAR-100 Krizhevsky et al. (2009), CUB Welinder et 2000))) Flowers Nilshack & Zisserman (2008), Food Bossard et al. (2044). Pets PPrrkh et al. (2012) and VWW Chowdhery al. (2019)
The learning policy consists of cosine learning rate
decay with 5 warm-up epochs Goyal el al. 2017). 50 epochs for larger datasets (CIFAR- 100,, Food,
Pets: https//creetivecommaen htt.cc/omooooooo/iiieee////////e org/licesseshyy-saaaa CC BY-SA 4.0 license; ImageNet: https://image- net.org/dowmlood.phhg the ImageNet license; others are listed.
16

---

Published as a conference paper at ICLR 2026
and VWW), 100 epochs for CIFAR-IO, and 200 epochs for smaller datasets (CUB, Pets, and Flowers). Learning rates range from 0.125 to 0, and we do not use weight decay or dropout. Our optimizer is Stochastic Gradient Descent (SGD) with no momentum (as keeping states in memory would go against our memory constraint). Each training is performed over three random seeds, and report average results with standard deviation.
Experiment Design. For experimental consistency, we adopt the same memory budgets Bmem used in the original SU work, implementing three distinct memory constraint levels for each network architecture. These budgets represent the maximum allowable memory consumption for both parameter storage and activations during the update process. Note that Velocity considers only weight memory in its budget calculations, so we evaluate it taking the weight component of the budget (excluding the activation part) for fair comparison. This comparative framework allows us to assess whether our theoretical insights translate into practical performance advantages while maintaining strictly equivalent memory constraints. For each channel selection strategy, we perform experiments the cross- pross-product of three networks, seven datasets, three memory budgets, and three seeds, producing 189 individual trainings per strategy, thus ensuring the statistical significance of the obtained results.
D.2 PROPOSITIONS VALIDATION ON CNN ARCHITECTURES
monnetii
CFFM--
CIEA--I
CLu
cuR
Cui
Sed
Food
Poo
Fod
www
veW
VWW
Pea
C20
Rod
1911
Fowms
cane 20
comt
Co Poot
count
coo
Figure 6: Spearman correlation of layer gradient norm across seeds and datasets.
Figure T-test of channel gradient norm across seeds and datasets.
Fig. 6 Fig. respectively correspond to the validation of Proposition 3.1 (layer gradient norm consistency) and Proposition 3.2 (channel gradient norm variability) as presented in Sec. with MCUNet and ProxylessNAS architectures. both cases, the observed results are very similar to those obtained with MobileNetV2 (Fig. 3), thus providing further confirmation of the theoretical insights proposed in Sec.
D.3 REWEIGHTED GRADIENT NORM METRIC VALIDATION
Here, we experimentally validate the efficacy of our RGN metric introduced in Sec. 3.2. We conduct a series of experiments where channels are selectively frozen during training based on whether their gradient norm (either raw reweighted) falls below a predefined threshold Even though memory is not a channel freezing criterion this setup, by logging metrics such as final accuracy, per-epoch memory, and FLOPS, freezing. We consider a pre-trained MobileNetV2 that we fine-tune on CIFAR-10 10 and Flowers. The results of this study are shown in Fig. g
observe how well network converges given different levels of partial
Each plot in Fig. [ illustrates a progression of freezing strategies: the top-right corner represents fully permissive threshold where all channels remain active during fine-tuning, while the bottom-left represents the most restrictive case where all channels (except the classifier) are frozen. Moving from right left along each curve corresponds to increasingly stringent thresholds that progressively freeze more channels.
key observation is that gradient norms naturally decrease during training, causing a fixed threshold to freeze an increasing number of channels training progresses. To capture this dynamic behavior across the entire training process, we present cumulative metrics for channel updates.
17

---

Published as a conference paper at ICLR 2026
001
30000
5G-
Tocal uptated channels
253030
20000
23
853000
21
CIEAR-In
CUPAR-10
800000
CIEAR-IO
Flenees
55
spaliae somn
Boor pradient MAE
Rar
BGN
BRN
50
1b
25 K1P
0.25 Total computed
Total minorry requited [LB]
X
Total memory required [KB]-110
(b)
(c)
Figure 8: Channel thresholding results based on gradient norm. Each point represents a complete training with respect pre-defined threshold Plots show: (a) final accuracy vs. total memory usage, (b) final accuracy vs. total computational cost, and (c) total updated channel count vs. total memory usage.
Fig. 8a reveals significant difference between raw and reweighted norm- based pruning: with raw gradient thresh- olding, accuracy begins to deteriorate as soon as any mem- ory reduction occurs. In contrast, the reweighted approach maintains full accuracy even when eliminating over half the total training memory. Fig. 8b shows analogous pat- terns for computational savings.
norm
8.8
0.4
Total
0.2
Fig. 8c helps understand such phenomenon as, for the same amount of total memory, raw norm thresholding re- moves substantially more channels than the reweighted approach. This occurs because reweighting prioritizes freezing memory-intensive channels with relatively low gradient-to-ment memoory ratios. In the raw scenario, channels with high gradient norms often coincide with high mem- ory costs due to their larger parameter counts, yet these channels may have lower per- per-parameter importance, Our reweighting mechanism effectively identifies this ineffi- ciency, allowing channels with high per-parameter gradi- ent impact to be preserved while eliminating those with disproportionate memory requirements.
Channel menorry cood [B]
DIP
21P
(b)
Tobal IGN
D.4 LAYERS RGN BEHAVIOR
B.00C
Fig. 9 iilustrates how reweighting transforms the impor- tance profile across network layers. Fig. 9a shows the raw gradient norm cumulated over training epochs, while Fig. presents the corresponding reweighted values after accounting for channel memory costs (shown on a loga- rithmic scale in Fig. (9). In Fig. 9c observe that some layers stand out in terms of cumulated RGN compared to others, namely the depthwise and the second point-wiig wise layers of the blocks closer to the output.
DU00C
Lagets
(c)
Figure 9: Layers gradient norm and reweighting analysis in the case of a MobileNetV2 fine-tuned on CIFAR-10. Fig. [aarrpreseets the raw cumulated gra- dient norm of layers, Fig. 9b the per- layer channel memory cost, and Fig. 9c the cumulated RGN of layers.
With this observation and the knowledge that such topol- ogy is shared between downstream tasks and over time, we deduce that we can freeze a-priori a certain subset of layers as they provide negligible contribution to the con- vergence of the network. To illustrate this point, we plot in Fig. 10 the evolution of cumulative RGN (expressed as percentage of total network RGN) with respect to the number of layers considered (layers ranked descending order of RGN).
18

---

Published as a conference paper at ICLR 2026
We observe that for each network, half of the total network RGN is contained within less than a quarter of the layers and half of the layers correspond to more than 90% of the total RGN.
mbv2-w0.33
100
Cunnlited RGN [X]]
80
D.5 TOPK LAYERS SELECTION
Our proposed TraDy algorithm requires pre-selecting a subset of relevant layers for channel sampling. As established in Sec. S.c.B.3 and experimentally confirmed in Fig. [6] the relative ranking of layers according to our RGN metric remains consistent across downstream tasks. This enables offline determination of layer importance by fine-tuning the target network on any available rel- evant downstream task and recording RGN values during training (even a few epochs suffice to establish reliable rankings).
404
20-
Layers conniaatdd RON
Top layers
mounet-inl
The critical question becomes determining the optimal num- ber K of top-ranked layers to include in our selection pool. To investigate this parameter's 's impact, we conduct an experimen- tal study using transfer learning with the smallest memory budget Bmem on CIFAR-10. We use the gradients norm information to rank a-priori the layers along their total RGN and select different
Commaltted RGN [X]]
80x
60x
40
levels of top K layers to perform sampling within (K being the variable denoting the number of layers considered). We compare three dynamic channel selection strategies within the selected layer subsets:
20-
Lagers cuncllted
20
30
Top layers
1. Random selection of channels until memory budget Bmem is met (Random).
proxyless-wOL w0.3
2. Deterministic selection of channels with highest RGN values using complete gradient knowledge until Buem is met (Det RGN).
100
Connlated RCN [8]]
80
3. Deterministic selection based on raw gradient norm val- ues using complete gradient knowledge until Bmem is met (Det Raw Norm).
60
40-
Fig. presents the results of this analysis. For random selec- tion, progressively excluding the least important layers initially improves training accuracy by constraining the sampling pool to more relevant channels. Performance peaks in the range around 35 to 40 layers before declining as essential layers are eliminated, highlighting their critical role convergence. Notably, these top 35 layers capture 97% of the network's 's total RGN as can be observed in Fig. 10 We apply this 97% criterion to our other architectures, yiclding 27 layers for MCUNet and 43 layers for ProXylessNAS.
20-
Tagers comellated RGN
20
40
Top la layars
Figure 10: Layers RGN cumula- tive contribution for 3 networks, fine-tuning on CIFAR-10.
We acknowledge that layer selection approach is relatively straightforward and presents opportu- nities for further refinement. For instance, K could be adaptively determined based on the available memory budget, maintaining a constant ratio between the budget and the total memory requirements of the selected layers. However, developing such an adaptive scheme would require substantial theoretical analysis and extensive empirical validation, which is outside of the scope of this work and will be explored in future research. As demonstrated in Fig. 5, our current fixed- -hre-shdr approach for determining K enables TraDy to achieve competitive performance against alternative strategies within the scope of this work.
The RGN-based deterministic approach exhibits remarkable stability across different values of K, maintaining consistent performance until decline occurs when reducing from the top 10 to top 5 layers. This behavior aligns with our expectation that gradient importance concentrates heavily within a small subset of layers expressed in Sec. 3.2
The Raw Norm approach demonstrates more complex dynamics. When applied to the entire network, it yields relatively modest performance, but shows substantial improvement as the least important
19

---

Published as a conference paper at ICLR 2026
layers progressively excluded. This pattern suggests that these lower-ranked layers contain channels with high absolute gradient magnitudes but poor gradient-to- -ommmorr efficiency ratios, which our reweighting scheme effectively identifies and deprioritizes.
Intriguingly, Raw Norm selection achieves superior accuracy compared to either RGN or Random selection within the 20-40 layer range, potentially indicating alternative ways balance gradient magnitude and memory efficiency beyond our current formulation. However, as stated in Sec. accessing the gradient norm to perform channel selection is energy inefficient compared a Random selection approach. Beyond the 20- 20-layer threshold, both methods converge toward similar perfor- mance levels, likely because the reweighting has diminishing impact on channel ordering when focusing on the most gradient-rich layers where memory costs become more uniform.
While Lin et al. (2022) observed that depthwise layers contribute minimally to accuracy when up- dated in isolation, iZhang et al. (2019) demonstrated that layer contributions cannot be evaluated independently as they depend critically on which other layers remain frozen or active. Our findings suggest that the coordinated updating of depthwise layers alongside their corresponding second pointwise layers within each block creates synergistic effects that promote efficient convergence. This hypothesis is supported by the consistently high ranking we observe for these layer combinations, indicating their collective importance for gradient- based optimization under memory constraints.
D.6 EXTENSION OF MAIN RESULTS
Here we provide additional results regarding our main experimental setup described in Sec. Sec.D.6 Tab presents report of final test top-1 accuracies across our full experimental matrix spanning multiple architectures, datasets, and memory budgets when comparing SU and the dynamic selection strategies. The results for the static variants are displayed appart for readability in Sec. Sec.D.8 (Tab.
50.6
1oat tgg1 aosumur
69.5
88.0
Bandors
We also provide comprehensive training metrics the following GitHub repository in the train- ing me_mmeiris folder. This supplementary data includes detailed figures tracking multiple performance indi- cators across all fine-tuning experiments: training and test top-1 accuracies and losses, weight and activation sparsity percentages induced by each channel selection strategy, computational costs for weight derivative calculations (measured in FLOPPs,, relative FLOP savings compared to full fine-tuning, and additional memory-related control metrics. These extensive logs provide deeper insights into the behavior and efficiency characteristics of each evaluated approach. The anonymous repository also contains the complete source code required to reproduce our experimental results, accompanied by detailed execution instructions in the README file and Jupyter notebook for generating all figures presented in this paper.
Nict
67.5
Top X ligets
Figure 11: Final test top1 accuracies depend- ing on the number of top K layers for differ- ent dynamic channel selection strategies.
In Tab. [I] memory budgets Brux expressed memory units, where each unit represents an individual memory slot. Actual memory consumption is calculated by multiplying these units by the number of bits per slot. The results reveal consistently low variance across repeated experiments for each combination of memory budget, architecture, dataset, and selection strategy. While accuracy differences between methods appear modest in individual comparisons, the extensive experimental validation across multiple dimensions provides strong statistical evidence for TraDy's superior performance. Additionally. TraDy offers practical advantages through its straightforward implementation compared to alternative approaches.
D.7 RESULTS ON TRANSFORMERS ARCHITECTURES
Although TraDy was conceived with the goal of enabling on-device learning, it can easily be adapted to tne-tune larger architectures with limited memory or energy resources. In this section, we chose to consider SwinT model Liu et al. (2021) pre-trained on ImageNet and fine-tuned on the seven downstream tasks introduced in the main paper. Regarding natural language processing (NLP), we
consider both BERT Kenton & Toutanova 2019) and RoBERTa Liu (2019), standing as traditional
20

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of final top1 test accuracies between SU and dynamic channel selection strategies over various pretrained CNN models, datasets, and budgets.
Model
Bass
Method
CIFAR-10
CIFAR-100
CUB
Flowers
Food
Pets
VWW
Average
SU
89.10:0.26
67.3420.18
56,8560222
N0.3360.66
6166260.3
76.5346.26
87.7320.06
74.22:6.74
Velocity
89,8:20.19
68, 14.1471117
57.5120.30
79.4620.22
61.79a0.12
76.2486.23
88.2920.18
74,4720.00
27 946
D Random
89.3220.15
67.85:0030
57.426012
79.1930.66
60.6950.16
76.6350.19
88.5620.12
74.2430.52
D Det RGN
89,29:0005
67.182005
57,7020.36
79.4420.54
61.8820.17
76.8030.04
88.3620.21
74,4920.71
TRaDy
89.88:0.19
68.6810.17
57,905000
79.5740522
62.6160.15
76.9956.17
88.7620.15
74.912064
SU
90,42:0:12
68,732022
57.97a0.25
81.1550.51
64.5650.17
77.0420.28
87, $7.7620.16
75,37a0.7
Velocity
90,94:0031
69.74.0.45
58445009
80.1330.52
65.435033
77.146037
68.31:0:22
78.73a1.3
w0.3
592
D Full Random
90,06:0008
68.93:0:28
58,440015
79.59.0.45
62.9620.33
76.8830.33
88.76 38.7640.34
73,09,0.70
D-Det RGN
90.25::005
68 68.82:0.13
58.73a0000
80.5840.40
64:22a00:0
76.6510.22
88.25
73.3630.72
TRaDy
90,79:0211
69. 39.57:40.77
59,09a0115
80.09
64.9
64.9620.32
76.6430.11
88 28.22
75.62a075
SU
90
69. 17:000
57.90
81.0910.WW
65.33a003
77.1220.16
87 87.30
75.52a0.70
Velecity
91,37
70.62:0011
59.03
80.5740.26
66.69:003
76.6750.35
88.11:03
76.15:0070
93 606
D-Full Randon
90.69
60 20:32
58.7450008
79.9910.51
63.90s0:12
76.5140:4
68.85:0:22
75.44:0.77
D-Drt RGN
90
60:41-0:88
58.90 30
80,9340.43
65.48
76.9610.25
67.84:0066
75.74:0.2
TRaDY
90.05=0
700040004
58:0140015
80.7640.37
65:80:0004
77.2140.32
58.01:0:15
75.97:0.70
252 320
Reseline
02.72:001
72:09:0
60.09:0018
81 81.88-1034
70.79:0020
76668:0.33
88. 68.58-0019
77.62:0.60
80.51:021
68.41:0.37
60.68:027
82,9240.33
65:5750.06
81.1550.29
19.14:0.10
76.7750.69
Velocity
80.93=0.11
60.20:0015
60.0960550
82.0540.34
64.42:0.11
81.21+6.25
89.53:0006
76.72:0.72
15 9366
D-Pull Randon
90.22:0066
60:0800.34
61,215022
X2.3740.26
65.7180.16
81,20:0.16
89.96:00.0
77.1150.48
D-Det RGNN
90.29:002
69.06:10.28
61.03:0.00
N2.3450.37
65.95:0.07
81.0720.13
89:90:20.19
77.19:0.99
TRaDY
10.38:0
69.72:0.14
61,3050.00
N2.5410.59
66:7820.17
N1,1050.11
89.7920.27
77.37:0.74
SU
91,05:0266
70.6:10.23
620050032
N3.7930.53
69.77:0.03
81.5280.11
88.67:0.34
78.34:0.73
Velocity
92,24:0003
72.31:0.34
62.56:0.42
N22010.51
70.4260.17
N1.4030.59
19:37.0.23
78.73:0.96
MCUNetinn
832
13 Full Random
91.70:0.13
71.5M:0.18
62.4320.00
N2.3320.31
69.07.0.88
81.20:3000
89.75.0.16
78.30:10.52
DeD RGN
91,60:0119
71.11:0.15
61,8630366
N2.9910.56
69.3320.19
8097260.22
89.3220.04
78.20:1.15
TRaDy
92.16:0225
72:11:10:40
62. 62.20
83.0220.52
70.5720.17
81.1120.28
89:30:0.27
78.6420.83
S11
92,07:0
71,58:0.15
61.44:041
83.7450.47
71,02:0:55
81.0756.34
88.7700:11
78.5380.76
Volocity
02.90-0
73.66:0039
62.53+051
82.0910.54
72.3240:14
808875045
89.36:0044
79.23-0.93
640
D-Pull Randon
92.20:00155 18
72.11:0.16
62.85+0011
82.84+0.08
70.70:0.05
81.30:0.07
89:54:0.17
78.88:00.33
D-Det RGN
02.01:007
72.30:0.13
62,3640556
83.0240.37
71.1650.22
807666627
89 15:15:
78.68:0.82
TRaDy
92.53:0021
7295:20.2
62,1250114
83.25
71.88:0.12
81.29H0.25
89.39:0.31
79.66:0.66
309
Boseline
93.87:0.10
7603:0.18
61/625062
83.4550.42
75.74:0.14
79:49:6.000
90.06:00166
$0.421..0
SU
91,00:1125
68.9420166
57.04a0.66
82,3630.25
63.3030.11
78.9630.43
88 88.26:90.6
75.69.0.74
Velocity
90.69:0100
69
12:000
55.98
818851034
61.463913
78.5830.00
88.67.0.00
79.192007
D-Full Kandom
20.76:022
69.20:1024
56.3550013
N1.5420.64
62.0950112
78.0420.29
88.050011
75.4720.00
D-Det RUN
91.06:004
69:201016
57,705034
81,0050.44
64.2250.16
78.726645
88.7150.12
75.9250.89
TRaDy
91.34:0.14
69833004
57.025026
N2.1320.34
64.3050.11
78.7320.88
88.8620.21
76.12:0.66
SU
91885027
70.340.19
58,336036
N3.1550.28
66.495029
78.9950.74
87.8220.22
76.7130.95
Velocity
92,37:00
718842008
58/72.072
82.3520,20
66.6320.33
78.920.4
88.3320.18
77.0520.00
Proxyless w0.3
960
Random
91.975036
71.0420.1
58.226043
81.6040.78
65.62a0.36
79.0030.28
88.8620225
76.62a1.10
D- Det RGN
91.92:026
70.6720.6
58.72.023
82.5130.55
66.832000
79.20:106
88.0820.18
76.852092
TRaDy
92,27a030
71.392027
58.805047
82,0950.18
67.1750.0
79.1030.14
88.3130233
77.0629733
SU
92.42:1116
71.322012
58.5250055
83.2440025
67, 67.185009
79 79.00:0:4
87.9220.17
77.09.0:1
Velocity
92.82:012
72.51
59.718040
82.6119.16
68 68.1859006
79.01:0.22
68.40:00115
77.612977
376
D-Full Random
92.21=um
71.54:0211
58. 58.8650422
82.4150.10
66 66.6930002
78 78.80.0..9
89 19.04.0019
77.08
D-Det RGN
92.37
71.064002
59.27400
82,734031
67.9759.19
79.061903
Lao0s
77.22.1.04
TRaDy
92,50-034
72, 18:0035
59.3450035
82,50.0.45
68.054031
79 39:29.0..0
88 06:40:24
77. 77.46:1078
162
Roseline
13
74.11.015
61.7580122
84.4450.50
72.98
78 53:0100.1
88.05:0004
79. 20.31
21

---

Published as a conference paper at ICLR 2026
Ther
qeo
0M0
Spearman correlation of layer gradi- ent norm across seeds and datasets.
Figure 12:
Figure 13: T-test of channel gradient norm across seeds and datasets.
Table 2: Comparison of final top1 test accuracies between dynamic channel selection strategies with pretrained Swin'T model fine-tuned various datasets and budgets.
Model
Bas
CIFAR-10
CIFAR-100
Flowers
VWW
Average
D-Full Randorn
96,3410099
82,7720.100
71,83:4.14
88, 64.64400... 36
80 10.66
90.76.10.33 33
93.79.10.07
86.40.14.77
27 946
D-Dei RGN
96.60:0004
83.18:004
74.56.10.11
08.52:0.25
61,25 10
90.91:10.9
93.25 17
86.90:10.55
TRaDy
96.30:0066
82.85:0.16
74.40
88.6110.51
80.75..0.05
91.1510.20
93.73.40.09
86.8310.000
D-Pull Random
96,59+020
83.44-0.09
72,76+0032
82.26++655
80.88+0445
9061-008
93.92+0.10 10
85.78+6664
112 640
D-Det RGN
9682+007
83.7780005
74.67=0.40
89.51+0004
82,49:0000
90,78+00111
92.96-40.14
87.28:0046
TRaDY
06.7411077
83.5520.13
74.30.0.14
88.60.0.44
81.5610.100
91,11:10.44
87.10.00.5
SwinT
D-lull Random
97.06=0.12
84.65:0.24
75.11
19.10
83.5920.12
90.95a0.22
93.25a00:4
87,67:0.56
633 8.59
D RGN
97.37
85.11
75.20::0.0s
90.45.0.5
83.96.0.u5
91.3910.07
93.32.0.24
88.11:0.59
TRaDy
97,25=au3
84.68+0.24
75.11
89.10=0.17
83.5940.12
90.9540.22
98.2540.04
87,70+0.55
D-Full Random
97.40=au17
85.7720.09
75.89
90.00::0.49
84.7620.13
91,55
93.74a0.17
88,44:0.73
2 767 686
D-Dei RCN
97.642006
85.88
76.26:10.3
91 41.46:0553
84.95:0.05
91:20:10.00
93.8830.17
88.75:0.73
TRaDy
97.6240009
85.7720.09
75.89.0..29
90.00:10.49
84.76.1.133
91.55
93.74.0.177
88.48:0.73
31 $89 952
Baseline
97,78:0:16
86 0,000
74.89
90,57.10.43
86.07.0.23
90.18:10.00
93.72.0.10
88.50.10.31
NLP architectures and which we fine-tune on three tasks: QNLI Demszky et al. (2018). RTE Poliak
(2020) and SST2 (Socher et al. (2013).
In Fig. 12 and Fig. 13 we reproduce for these three architectures, the layer gradient norm Spearman correatitno and t-test of channel gradient norm as introduced in Sec. Sec. [D.3] Regarding the Spearman correlation, Fig. [2] provides confirmation that the transformer architectures considered also follow the layer invariance with respect downstream tasks as introduced in Proposition 3.] Similarly, Fig. [3, showcases the rejection of the hypothesis of channel topology preservation between datasets. Interestingly, in the case of the simpler convolutional architectures, we observed in Fig. 7 that the different seeds of the same dataset resulted in a similar channel topology. In the case of the transformer architectures, we observe that most cases, different seeds for the same dataset does not necessarly result in similar channel topology. We suppose that this is due to the higher expressivity of these complex architectures, allowing for differennt subnetworks to perform the same task.
Tab, 2 and Tab. present test accuracies for SwinT and BERT-family models respectively, evaluated using our three dynamic selection approaches (static approaches result are provided separately in Sec. D.X for improved readability). Given that these transformer architectures substantially larger than the CNNs in our main experiments (approximately 26-30x larger for SwinT and 77-90x larger for BERT/RoBERTa in terms of combined weight and activation memory), we explore two distinct memory constraint scenarios:
1. Absolute Budget Matching: We apply identical memory budgets to those used for CNN experiments. For transformer architectures, these budgets represent dramatically smaller proportions of the total network, simulating extreme resource constraints where users seek exploit large model capabilities with severely limited computational resources.
2. Proportional Budget Matching: We scale memory budgets to maintain equivalent proportions of total model memory as in the CNN experiments, enabling more substantial network portions participate in updates during each epoch.
This dual-budget approach allows us to evaluate our method's 's effectiveness across different constraint severity levels while providing insights into transformer fine-tuning behavior under varying resource limitations.
When comparing SwinT against CNN architectures, all three channel selection methods achieve superior accuracy even under the most restrictive memory constraints (less than 0.1 % of total network
22

---

Published as a conference paper at ICLR 2026
S-Full Random
S-Full Random
0.8
S-Det RGN
S-Det RGN
S-Topk Random
0.6
S-Topk Random
0.6
D-Full Random
D-Full Random
D-Det RGN
D-Det RGN-
0.2
0.2
TraDy
TraDy
S-Fall Random
S-Det RGN
S TToph Random
D-Full Random
D-Det RGN
ThaD
S-Full Randon
S-Det RGN
8 Top Randow
D-Full Randoa
D-Det RGN
TraDy
(a) SwinT model
(b) BERTs models
Figure 14: T-test comparisons of average final test accuracies across multiple experimental dimensions for each group of transformer architectures.
memory). Furthermore, both vision and NLP transformers exhibit smaller accuracy degradation between the most constrained budgets and full fine-tuning baselines compared to CNNs, despite these budgets constituting much smaller network fractions. This resilience underscores transformers' capacity to learn rich, transferable representations during pre-training that remain effective with minimal parameter updates during downstream adaptation.
Fig. 14 replicates the statistical analysis from Fig. for trans- former architectures. For SwinT (Fig. 14a), we observe an overall consistent strategy ranking with D-Det RGN achieving the best performance, followed by TraDy. However, for BERT-family mod- els, both approaches appear be outperformed by S-Det RGN, though this result carries greater uncertainty due to the smaller experimental sample size. Ad- ditionally, the substantial scale of these architectures suggests that our top K layer selection methodology, while effective for CNNs, may require more phisticated calibration for trans- former models of this magnitude.
Table 3: Comparison of final topl test accuracies between dy- namic channel selection strategies with pretrained BERT and RoBERTa models, fine-tuned on various datasets and budgets.
Model
Bewn
Methol
QNLI
RTE
SST2
Average
D Full Random
81.0010.25
$6.3210.36
39.4111.46
76.74 76.74
D Det RGN
$7.7810.45
57.28 38.2881778
91.17.0.00
78.74
TRaDy
84. 38.211021
76266196
89.5320.13
07.23
89.41
112 640
RGN
89 09.00.0.222
60.0812666
91.2510.57
20.10
BERT
TRaDy
84.5620.28
57 882091
89.0030.26
17,35
D. Full Random
65 15.45
54 19.9
89. 19.76.10.88
76.85
76.8711.
D Det RGN
89.8300.09
60.510355
91,88102244
80.61.0.01
TRaDy
05:0000.30
90,
D-Full
88.6880.14
74.4411.54
8 8.311 308
RGN
90,47100166
10.307
90.57
80.77 3,12
TRaDy
88.7210.20
57. 1.:31.3
90.96100188
79.00000.88
225 792
Baseline
90.0110.27
62.45 62.4511111
91.7460130
$1.67=1.30
Full Random
89.69
$7.00 57.40.10.72
93.231.0.4
10.0000..00
D RGN
29..5
92.51.0.40
89.7120.13
16:0075
Full Random
89 19.9910.25
121
93.12.0.20
80.4 1.29
112 640
D Det RGN
90.78.10.25
07.221021
93.0010.11
86. 0,3330
Limitations and Future Direc- tions for Transformer Architec- tures. While our experimental results on transformer architec- tures demonstrate the applicabil- ity of TraDy's core principles, the performance gap compared to CNN architectures suggests that deeper understanding of fine-tuning dynamics in transformers is necessary to achieve opti- mal results. Recent work on sparse matrix fine-tuning for LLMs He et al. (2025) provides valuable insights that could inform more effective adaptations of our approach. Specifically, their analysis reveals that attention layer V vectors require the most trainable parameters while MLPs need min- imal updates- iodates-insights that could guide more informed layer selection strategies for transformer
TRaDy
90.05:20.12
59.572277
93.31.90.33
50.9511.06
D-Fill
91.23=0.18
93.85
912
RGN
91.2310.26
93,0410.266
21.2320.20
68.8511.78
93.4341.36
$4.50
D-Pull Randon
91.54-0.11
73 73-72222
93.27
N6. 2.99
S:251 308
D Dei RGN
91,90 91.9010.113
70 10.28115.25
93.58.10.11
85. 25.25
TRaDy
91.36210.33
73.59411.78
93.31.20.35
16
96 792
92.31
76.1:0055
93.1649092
23

---

Published as a conference paper at ICLR 2026
Table 4: Comparison of final topl test accuracies between static channel selection strategies over various pretrained vision models, datasets, and budgets.
Model
Ben
Method
CIEAR-10
CIFAR-100
CUB
Flowers
Food
Pets
VWW
Average
Random
57.79
66:52.
55.639.55
79. 79.154044
58:21:021
76.463044
88.175000
73.13-076
27 946
S-Dee RGN
88.78:0.0
67 25:112:
57.1140.0
79.99:0.41
6000500111
76.70:0.2
88.4450.15
7399-477
S Top Random
68 52:001
66.92
56.5240.55
79.684002
59.5240.39
76.713055
88, 12.00
23
S- S-Fall Random
63 18.0211112
66.81:0012
56/7530009
79.289063
59.75
76.30305
87.8720000
73:65-114
MhV2-w00.35
66
S-Det RGIN
89.88:0066
68. 14:004:0
57.8420.17
80.31:0:255
62.70::011
766160554
88, 88.11:0.33
0.7000070
R Random
76766
68. 10:411
57,4940366
80.39:0:46
62.08:4229
76644424
87,7550.13
74.60-078
S-Pall Rasdom
89.01:0:22
67 07:11:
56.57:0.51
79.78:0.79
60.40::0.66
76.58:0.35
88.2260488
74:01=127
93
S-Det RGIN
90,45:0.10
68:84:0004
57,0030.09
N0.59.0.11
64.11:0.199
76.758026
87.5450.06
75.15-0.7
8- R Random
90.25
68 :1:30
57.9420011
80.42:033
63.14:0:1
76667002
87.612015
74.92-1101
S Full Random
65 78
67 78:0033
60. 14:014
82:00:0045
62:68:0034
81.09:0.19
89 20.50.023
76.02-174
15 936
SeD
89.12:0.14
67:97:0.10
60:06:0.32
M2.14:0.37
63.77:0.15
N0.79.0.34
89.55600011
76.20:00.9
S- R Random
80 00:08:1111
67 86:00.0
60.26460.17
82.34:082
63:60:00.20
81.09:0.30
89, 11
76.26 uy7
S.Pall Random
90.02:0.22
69.70:30.22
6019660.11
M2.7:00.0
64.87:1229
89.4420.06
7635-115:
MCUNes-inl
64 832
S. Det RGN
89:9720.18
67.78:0.19
61,8010.24
80.80:10.66
05.16:20:44
N1.7620.56
89 89.2720.12
76.06=1120
T- Random
00:82:00.22
70 10:34:0:15
60,0010.14
82:92:00:44
67.29.1024
81.34:0.29
80: 10:10:
77.53 0.76
S Random
90882:0.13
70.75:0.41
61.2246.11
H2.77:0.35
66.92:0.77
808000.11
59:0150:12
77.47:0.9
112 640
S-Dhe RGN
91. 01.28:0.13
71 23:53:00:0
61.00.60.16
82.60:0055
69.1 69.17:0223
80:64:0116
68.8910.17
77.89
S-Top Random
18:11
11.55-0.:4
60.9510.71
82.50:10.66
69.28:10117
68.7366335
77.90
S Full Random
89.15:0.33
67.90:2021
55.22:40.33
81.6420,54
58.72:026
78.32:0.14
$8.3920.08
74.19:0077
25 984
S Det RGN
90. 20.19:0.27
68. 20:0002000
57.1380.25
81,89:0037
61.69:0.19
7890030.4
68.5120.10
75.26:0061
T-Top Random
10.98:0.18
68. 33:00.2
56.17:0.01
81.89:0030
60:00:0112
78:17:025
18.99:0.19
74
S Full Rundom
90,34:0.00
68.78:40144
36.35
81 81.98
61,32a40377
78.79.0.43
58 38,40200.22
75.13=1116
Proxyless w0.3
72 960
S-Des
30:012
70 15:
58.2610.45
82.62:00:46
65:09:00:
7867:0.29
87.94:0.54
76.32
S- T-TopK Random
91.09:0.14
70 00:10::0.3
57.32:0.45
82. 12.15:0.10
63.75:016
78.410.32
87.861:040
75
S Full Random
90.6::0.15
69.37:0014
57,1840.33
82.15
62:60:002
78.513020
68.0650332
75:50=104
101 376
S-Des
91.7610.15
18:333333
58660.18
82.92:0:42
66.45:025
78:703049
87.84:0:27
76.78 0.8
S Top Random
9161=033
70.73a4046
$7,88
82.51
64. 64.92
78.463005
57.5850.27
76.24:077
Random
96.312001
62.94am017
73.80H0.11
88.340199
80.55
91.08a011
90.63s000
86.66.033
27 946
S-Det RGN
96. 36:0000
83.05:00.12
74.36.0.12
88.7630.20
8162:0011
91.03:0.11
93.6050.15
86.03 0.41
S-Top Random
28.0000
82:97:0009
74.0820.07
W8.54.0.11
W0.58:025
908779021
98,7010.04
W6.72:0.33
S-Fall Random
96. 96.5610010
83:25:10306
74,00240.23
88.743019
81.09:300
912130011
93.64:0.11
86.92-11
112 640
S-Det RGIN
96 70:00066
83. 16.599n..16
74.68.0.20
39.36:10.22
81.97:0000
90.9750244
92.9950006
07.18::0.49
S- Random
96:6011111
63:29:40:0
74.19:0.34
86.83a016
81.19:0.16
90.50s0017
86.94.0100
SwinT
S-Full Rundom
96 06.9112112
84 24:24::200
74.55.0.44
89:33:a115
82:99.0013
91 91.26:uu17
90.1:nm
87, 53
633
S-Det
27.26600.05
84.7M:n.133
75.0910.33
90.29:0.15
N5:72::0.15
91.28an.25
79.72:23.27
26.31:22.27
T-Top Random
97. 77.06:0000
84 24:24:30:0
74.55.0..4
89.33a00s
82:99.0012
91,26a00
90. 10:0000000
87.50
S S-Full Random
97. 50:00.000
85 53:1018
75,75:0..1
89 84:008
84.46-1013
91. 21.30:30:3
93,7360.17
68. 30.3300..
767 686
SeDet
97.501:.06
53:3..
75.73
39.5420005
84.46:0.12
91,30,0.23
93.7360.17
08.30
S- T-Top Random
97. 11
85 53:100
75.75.60.14
89 84:00s
84 84-46-1013
91. 01.30:003
93 93.7366017
68 30.00.0. 34
architectures. However, a key challenge remains: such analyses typically require a warm-up phase with full fine-tuning on downstream task data to compute Fisher information, whereas our method assumes no downstream task data is available a priori and enforces striet memory constraints at all times. Merging insights from layer-importance analysis in transformers with our dynamic channel selection framework represents a promising direction for future work, potentially enabling more effective memory-consttriined fine-tuning of large language models.
D.8 STATIC STRATEGIES RESULTS
This section presents experimental results for the static selection strategies evaluated in our study. Tab. Adisplays results for all vision architectures, while Tab. presents findings for NLP models.
E
LLM USAGE
24

---

Published as a conference paper at ICLR 2026
The redaction of this paper received sup- port from LLM to help improve grammar and readability. No scientific or technical content was generated through LLM. All nu- merical results, tables and figures are our own production.
Table Comparison of final topl test accuracies between static channel
selection strategies with pretrained BERT and RoBERTa models, fine-tuned on various datasets and budgets.
Model
Bacn
Method
QNLI
RTE
SS12
Average
S-Full Random
84.48:10.22
57.28:00.83
89.68-00.11
77.15:-0.87
946
S-Dee RGN
86.4240.13
18.72:4.18
90.86:+0.35
78.67:-2.21
S-Top Random
84.5340.34
58.24#2.32
89.37=0.133
77.38:2.35
S-Full Random
84.511:0.08
58.72=2.40
89.72:20.48
77.65:2.45
112 640
5-Dee RGN
88.30+00.43
59.09:#1.46
91.214-0.46
79.53
BERT
Top
77.41-2.23
S-Full Random
86.0810.49
17.04:1.444
89.91220.44
77.6811.56
912 629
5-Dee RGN
89.22440.16
59.81=22.05
91. 01.55-00.99
80.19
5- Top
8680++0.3
90.29:00.07
S-Full Random
88.5520.17
56.80:20.21
90.29::0.18
78.55:0032
8 351 308
S-Det RGN
89.87#0.22
61,01#2.53
91.55:0.229
80.81=2.57
T-Top
88.7740.09
91.28-40.57
79.27
S-Full Random
89.6810.17
56.56:20.75
93.31:00118
79.85:0079
27 946
S-Det RGN
90.8120.12
76.90=00.2
03,43:0.35
87.01
To
89.66+0.07
56.68-00:22
93.31-40.07
79.88:0073
S-Fill Random
89.69+0.10 10
59.09
93.39-40.29
80.72-3.63
112.640 6410
S-Det RGN
90.9120611
76.77=2.61
92.85:00335
86.85:2.70
RoBERTa
S-TopK Random
89.6010.02
03.27:10.18
79.8920.58
S-Fill Random
90.95++0.26
62.33-19.49
93,58-40.34
82.29-19.50
1 912 629
S-Dee RGN
91,10+00.24
77.26:t:166 1.68
92.89-+0.34
87.08=1.77
S-TopK Random
91,1410.22
62.33:26114
03.2310.46
82.2326.18
S-Full Random
91.28:10.26
0..80::0.75
02.78:10.00
85.62:0079
8 351 308
S-Dee RGN
91.20+0031
75.09=00.00
93.20+0066
86.49:00.33
S-TopK
91,0910.17
93.08:10226
85.34:3.222
25


Reviews:
[{"review_id": "m3hiedPz3L", "reviewer": "Reviewer_izYB", "summary": "This paper introduces TraDy, a novel hybrid approach for memory-constrained fine-tuning of pre-trained models. The core idea is to combine (1) a static, a priori layer selection process, which the authors argue is architecture-dependent, with (2) a dynamic, stochastic channel selection process, which is argued to be task-dependent.\n\nThe methodology is justified by a solid analysis of training dynamics, identifying three key insights: the heavy-tailed nature of gradients, the task-invariance of layer importance, and the task-dependence of channel importance. Experimentally, TraDy demonstrates state-of-the-art performance against static sparse update baselines (like SU) across various vision tasks and architectures, achieving significant efficiency gains, including up to 99.5% activation sparsity and 97% reduction in weight gradient FLOPs.", "questions": "1.  **Scalability to Large-Scale Models (LLMs):** The paper successfully demonstrates TraDy on CNNs and, in the appendix, on ViT and smaller BERT models (though with mixed results for NLP). A key question is whether this framework can be effectively scaled to the fine-tuning of modern, massive-scale models, such as LLMs with >7B parameters? How does the static layer selection and dynamic channel selection interplay in such homogeneous, transformer-heavy architectures?\n\n2.  **Runtime Overhead of Dynamic Selection:** The paper clearly demonstrates the *memory* and *FLOPs* advantages of TraDy. However, it does not discuss the potential *wall-clock time* overhead. Does the dynamic resampling of channels at every epoch (including random number generation, index selection, and mask creation) introduce a non-negligible runtime cost compared to static methods (like SU) that perform this selection only once offline?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1.  **Strong Theoretical and Experimental Grounding:** The paper's primary strength lies in its thorough justification. The authors do not just propose a method but provide a robust analysis of *why* it should work. The experimental validation of the three key insights (heavy-tailed gradients, task-invariant layer ranks, task-dependent channel ranks) is convincing and provides a solid foundation for the proposed hybrid design.\n\n2.  **High Efficiency and Strong Performance:** The method achieves impressive results, demonstrating SOTA accuracy while operating under severe memory constraints. The reported efficiency metrics (e..g., >99% activation sparsity) are highly significant for the target application of on-device learning and data-drift adaptation.\n\n3.  **Rich Ablation Studies:** The paper's claims are well-supported by a comprehensive set of ablation studies. The comparison of `TraDy` (TopK Random) against alternatives like `Full Random` and static methods (like `S-Det RGN`) clearly isolates the benefits of both the static layer selection and the dynamic channel selection components, strengthening the paper's overall argument.", "weaknesses": "1.  **Evaluation on Simple Tasks:** The empirical evaluation, while broad in terms of datasets (CIFAR-10/100, CUB, Flowers, etc.), is primarily limited to relatively simple, small-scale classification tasks. To truly validate the robustness and scalability of TraDy, an evaluation on more complex, large-scale benchmarks (e.g., ImageNet-1K) is necessary.\n\n2.  **Missing Comparison to Key PEFT Methods:** The paper's related work and experimental comparisons focus almost exclusively on *sparse update* methods (like SU). However, it overlooks a major and highly relevant category of Parameter-Efficient Fine-Tuning (PEFT) methods. A discussion and, ideally, a comparison against popular *adapter-based* methods that also target memory and compute efficiency would be crucial. Key missing comparisons include:\n    * **LoRA** (Hu et al., 2021)\n    * **DoRA** (Liu et al., 2024)\n    * **PaCA** (Woo et al., 2025)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dnHZUn53Xy", "reviewer": "Reviewer_eoHC", "summary": "The paper presents TraDy, a novel approach for fine-tuning pre-trained deep neural networks to downstream tasks under memory-constrained budgets. TraDy proposes a dynamic channel selection scheme that exploits the gradient sparsity of input channels to achieve both weight and activation sparsity during fine-tuning for downstream tasks.", "questions": "- Could you maybe compare the fine-tuning to downstream tasks accuracy of TraDy against the state of the art memory constrained fine-tuning methods cited in the related works Section 2? It would be interesting to compare the computational benefits vs fine-tuned accuracy of TraDy with those of other state-of-the-art fine-tuning methods. \n- Could you maybe add the results for vision attention-based models (i.e., ViTBase), BERT and RoBERTA in the main paper? \n- How does TraDy perform when compared to low-rank fine-tuning methods (i.e., LoRA)?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The analysis of the stochastic gradients' heavy-tailed behaviour during fine-tuning of a pre-trained network, relative importance of the network layers, consistency across downstream tasks and channel importance distribution is rigorously presented and well explained.", "weaknesses": "- The paper is currently lacking comparison with the state-of-the-art fine-tuning reported in section 2 (i.e., Lin et. al (2022), Kwon et al. (2024) and Quèllenec et al. (2024)). \n- TraDy performances are reported in the main paper only for CNN models.  \n-  Unfortunately, the plots reported in Fig. 3, Fig.4 and Fig. 6 are not easy to read and to position within the main contributions of TraDy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oN5Dkn9JDA", "reviewer": "Reviewer_FTbs", "summary": "This paper propose a dynamic training method (TraDy) for memory-constrained scenarios. . By stochastically resampling channels between epochs within architecturally important layers, the experiments demonstrate the effectiveness of the proposed method across various transfer learning scenarios. This article is well-written and well-organized.", "questions": "1.Although the author claims that lennec et al.’s implementation excludes activation memory from their budget calculations, it doesn't seem to affect adding it as an additional baseline to Table 1-5. Why wasn't this done?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. This paper is grounded in a solid theoretical foundation, featuring detailed derivations and rigorous argumentation.\n2. This article is well-written and well-organized.", "weaknesses": "1.Fig. 2 is positioned too far from the corresponding text section. It is recommended to optimize the image layout.\n2.The baseline for existing studies compared in this paper is limited, with only SU available.\n3.It is recommended to highlight the best-performing results in Tables 1-5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sDLylnJFoa", "reviewer": "Reviewer_LAwL", "summary": "This paper proposes TraDy (Training Dynamics), a memory-efficient fine-tuning approach for deep neural networks under strict resource constraints. The method leverages two key insights: (1) layer importance for updates is architecture-dependent and can be determined a priori, and (2) dynamic stochastic channel selection provides superior gradient approximation compared to static approaches. The authors introduce a Reweighted Gradient Norm (RGN) metric that accounts for both gradient magnitude and memory costs. TraDy operates by pre-selecting important layers based on architectural properties, then dynamically resampling input channels between epochs within these layers. Experiments on CNNs and transformers across multiple vision and NLP tasks demonstrate that TraDy achieves competitive performance while maintaining up to 99% activation sparsity, 95% weight derivative sparsity, and 97% reduction in FLOPs for weight derivative computation.", "questions": "see weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The core idea is both creative and elegant. The central challenge of memory-constrained training is that one cannot afford to compute the very importance metrics needed to decide what to update. TraDy's solution—decoupling the problem into a static, a priori layer selection and a dynamic, stochastic channel selection—is a novel and highly effective way to break this circular dependency.\n\n2. The authors compare TraDy against a comprehensive set of baselines, including static/dynamic and random/deterministic variants, across 3 CNN architectures, 7 vision datasets, 3 memory budgets, and 3 random seeds.\n\n3. Well-structured paper with clear problem formulation and notation.", "weaknesses": "1. The main comparison is against Sparse Update (SU). More recent and relevant works are mentioned but not included in the main experiments. For example, how is the proposed method compared with \"SMT: Fine-Tuing Large Language Models with Sparse Matrices\".\n\n2. LoRA and other parameter-efficient fine-tuning methods are not discussed or compared.\n\n3. The authors acknowledge this limitation explicitly. The paper lacks actual on-device experiments showing latency, energy consumption, or throughput on target edge devices.\n\n4. The paper is compactly formatted and the template style is changed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)