TrAct: Making First-layer Pre-Activations Trainable
Authors: Felix Petersen
Christian Borgelt
Stefano Ermon
Venue: NeurIPS 2024 poster (2024)
Keywords: computer vision
convolution
second-order
optimization
Primary area: machine_vision
TL;DR: Making the training dynamics of the first layer of vision models similar to those of the embedding layer of language models.

Abstract: We consider the training of the first layer of vision models and notice the clear relationship between pixel values and gradient update magnitudes: the gradients arriving at the weights of a first layer are by definition directly proportional to (normalized) input pixel values. Thus, an image with low contrast has a smaller impact on learning than an image with higher contrast, and a very bright or very dark image has a stronger impact on the weights than an image with moderate brightness. In this work, we propose performing gradient descent on the embeddings produced by the first layer of the model. However, switching to discrete inputs with an embedding layer is not a reasonable option for vision models. Thus, we propose the conceptual procedure of (i) a gradient descent step on first layer activations to construct an activation proposal, and (ii) finding the optimal weights of the first layer, i.e., those weights which minimize the squared distance to the activation proposal. We provide a  closed form solution of the procedure and adjust it for robust stochastic training while computing everything efficiently. Empirically, we find that TrAct (Training Activations) speeds up training by factors between 1.25x and 4x while requiring only a small computational overhead. We demonstrate the utility of TrAct with different optimizers for a range of different vision models including convolutional and transformer architectures.

TrAct: Making First-layer Pre-Activations Trainable
Felix Petersen Stanford University wettffeelxx-eeeeeen..
Christian Borgelt
Stefano Ermon Stanford University ermonêcs. stanford edu
University of Salzburg
christiandhorgelt.tem
Abstract
We consider the training of the first layer of vision models and notice the clear relationship between pixel values and gradient update magnitudes: the gradients arriving the weights of first layer are by definition directly proportional to (normalized) input pixel values. Thus, an image with low contrast has a smaller impact on learning than image with higher contrast, and very bright or very dark image has a stronger impact on the weights than an image with moderate brightness. In this work, we propose performing gradient descent on the embeddings produced by the first layer of the model. However, switching to discrete inputs with an embedding layer is not reasonable option for vision models. Thus, we propose the conceptual procedure of (i) a gradient descent step on first layer activations to construct activation proposal, and (ii) finding the optimal weights of the first layer, i.e., those weights which minimize the squared distance to the activation proposal. We provide closed form solution of the procedure and adjust it for robust stochastic training while computing everything efficiently. Empirically, we find that TrAct (Training Activations) speeds up training by factors between 1.25 and 4x while requiring only a small computational overhead. We demonstrate the utility of TrAct with different optimizers for range of different vision models including convolutional and transformer architectures.
Introduction
We consider the learning of first-layer embeddings / pre-activations in vision models, and in particular learning the weights with which the input images are transformed in order to obtain these embeddings.
In gradient descent, the updates to first-layer weights are directly proportional to the (normalized) pixel values of the input images. As a consequence (assuming that input images are standardized), high contrast, very dark, or very bright images have a greater impact on the trained first-layer weights, while low contrast images with medium brightness have only smaller impact training.
While, in the past, mainly transformations of the input images, especially various forms of normaliza- tion have been considered, either as a preprocessing step or part of the neural network architecture, our approach targets the training process directly without modifying the model architecture or any preprocessing. The goal of our approach is to achieve training behavior that is equivalent to training the pre-activations or embedding values themselves. For example, in language models [1], the first layer is "Embedding" layer that maps token id to embedding vector (via a lookup). When training language models, this embedding vector is trained directly, i.e., the update to the embedding directly corresponds to the gradient of the rreaccivationn of the first layer. As discussed above, this is not the case in vision models as, here, the updates to the first-layer weight matrix correspond to the outer product between the input pixel values and the gradient of the pre-activation of the first layer. Bridging this gap between the "Embedding" layer in language models, and "Conv2l>" / "Linear"" "Dense" layers in vision models, propose novel technique for training the pre-activations of the latter, effectively mimicking training behavior of the "Embedding" layer in language models. As vision models rely pixel values rather than tokens, and any discretization of image patches,
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Language
Vision
TrAct
-1.0 1.10
0:53 -100
(LS0) +1.00
2290
-1:00 1.30
4000 LM
2:00
4000
(94)
W
W
W
06 0.
0.0 0
4.5
<BOSS
0.0 [uc
(1. 0.5
AW
AW
002 0.4
0.5
0.2 0.4
+-A-
0.4
AW
A.V
AW
A.
A2
0.5
Figure 1: TrAct learns the first layer of vision model but with the training dynamics of embedding layer. We illustrate this an example with two d-miennsional inputs x, a weight matrix W of size x 3, and resulting pre-activations z size x 3. For language models (left), the input x is two tokens from a dictionary size 4. For vision models (center + right), the input is two patches of the image, each totaling 4 pixels. During backpropagation,
obtain the gradient wrt. pre-activations 22, from which the gradient and update to the weights W is computed (AW). The resulting update to the pre-activations Az equals x AW. For language models (left), Az = Vz. ie.. the training dynamics of the embeddings layer corresponds to updating the embeddings directly wrt. the gradient. Specifically, the update in a language model, for a token identifier i, is W. + W. - 7 V. where 2 = W; is the activation of the first layer and at the same time the ith row of the embedding (weight) matrix W. Equivalently,
write f- 2 17- V. C(2). However, in vision models (center), the update A strongly deviates from the respective gradients vz. TrAct corrects for this by adjusting AW via corrective term (xr + + (orange box), such that the update to closely approximates Vz.
e.g.. via clustering is not a reasonable option, we approach the problem via a modification of the gradient (and therefore a modification of the training behavior) without modifying the original model architectures. We illustratively compare the updates in language and vision models and demonstrate the modification that TrAct introduces in Figure
The proposed method is general and applicable to a variety of vision model architecture types, from convolutional vision transformer models. In a wide range of experiments, demonstrate the utility of the proposed approach, effectively speeding up training by factors ranging from 1.25x to 4x, or, within given training budget, improving model performance consistently. The approach requires only one hyperparameter X, which is easy to select, and our default value works consistently well across all 50 considered model architecture + data set optimizer settings.
The remainder of this paper organized as follows: Section 2, we introduce related work, in Section 3, we introduce and derive TrAct from theoretical perspective, and in Section 3.1 we discuss implementation considerations of TrAct. In Section 4, we empirically evaluate our method in variety of experiments, spanning range of models, data sets, and training strategies, including analysis of the mild behavior of the hyperparameter, an ablation study, and a runtime analysis. We conclude the paper with discussion in Section 5. The code is publicly available github.com//elle Pet.eeiiii-eeeese/rrtttttt
2 Related Work
It is not surprising that the performance of image classification and object recognition models depends heavily on the quality of the input images, especially on their brightness range and contrast. For example, image augmentation techniques generate modified versions of the original images as additional training examples. Some of these techniques work by geometric transformations (rotation, mirroring, cropping), others by adding noise, changing contrast modifying the image in
2

---

the color space [2]. In the area of vision transformers [3], [4] so-called 3-augmentation (Gaussian blur, reduction to grayscale, and solarization) has been shown to be essential to performance [5]. Augmentation approaches are similar to image enhancement as preprocessing step, because they generate possibly enhanced versions of the images as additional training examples, even though they leave the original images unchanged, which are also still used as training examples.
Another direction related the problem we deal with in this paper are various normalizations and standardizations, starting with the most common one of standardizing the data to mean 0 and standard deviation 1 (over the training set), and continuing through batch normalization [6], weight normalization [7], layer normalization [8], which are usually applied not just for the first layer, but throughout the network, and in particular patch- wise normalization of the input images [9], which we will draw on for comparisons. We note that, e.g., Dual PatchNorm [9], in contrast to our approach, modifies the actual model architecture, but not the gradient backpropagation procedure.
However, none of these approaches directly addresses the actual concern that weight changes in the first layer are proportional the inputs, but instead only modify the inputs and architectures make training easier or faster. In contrast to these approaches, we address the training problem itself and propose a different way of optimizing first-layer weights for unchanged inputs. Of course, this does not mean that input enhancement techniques are superfluous with our method, but only that additional performance gains can be obtained by including TrAct during training.
In the context of deviating from standard gradient descent- descent-based optimization [10], there are dif- ferent lines of work in the space of second-order optimization [11], e.g., K-FAC [12], ViVIT [13), ISAAC [14], Backpack [15], and Newton Losses [16], which have inspired our methodology for mod- ifying the gradient computation. In particular, the proposed approach integrates second-order ideas for solving a (later introduced) sub-optimizaaion-rrobbee in closed-forn [17], and has similarities to
a special case of ISAAC [14].
3 Method
First, let us consider a regular gradient descent of a vision model. Let 2 = f(x; W) be the first layer embeddings excluding an activation function and W be the weights of this first layer, i.e., for a fully-connected layer fæ; W)- = W x. Here, we have x E R=xb = E Rm=b and W E mxx for a batch size of b. We remark that our input x may be unfolded, supporting convolutional and vision transformer networks. Further, let ÿ = I(;; 0,w) = ( W); (w; be the prediction of the entire model. Moreover, let L(g. y) be the loss function for a label y and wlog. let us assume it is an averaging loss (i.e., reduction over batch dimension via mean). During backpropagation, the gradient of the loss wrt. 2, i.e., V. L(g(2; O (w),) or Vz[(2) for short, will be computed. Conventionally, the gradient wrt. W, i.e., Cw (z; ():Ow)).)) or VwC(W) for short, is computed during backpropagation as
CwC(WW) V=
leading to the gradient descent update step of
W - W - Vw WWW).
(2)
Equation 1 clearly shows the direct proportionality between the gradient wrt. the first layer weights and the input (magnitudes), showing that larger input magnitudes produce proportionally larger changes in first layer network weights. We remark that corresponding relationships also holds in later layers of the neural network, but emphasize that, in later layers, the relationship shows a proportionality to activation magnitude, which is desirable.
To resolve this dependency on the inputs and make training more efficient, we propose to conceptually optimize in the space of first layer embeddings z. In particular, we could perform a gradient descent step on z, i.c.,
レ- 2 17- b. VzL(z).
Here, b is a multiplier because L(z) is (per convention) the empirical expectation over the batch dim.
However, now, =* depends the inputs and is not part of the actual model parameters. We can resolve this problem by determining how to update W such that f(z; W) is as close to z" as possible. Conceptually, we compute the optimal update AW* by solving the optimization problem
arg min || - (W + AW) =1
subject to ||W||2 Se
3

---

where we (1) want to minimize the distance between 2" the embeddings implied by the change of W by AW, and (2) want keep the change AW small.
We enforce that weight matrix changes AW are small (IAW by taking the Lagrangian of the problem, i.e.,
arg min 2 (W + AW) =11 + Ab 1All2
(5)
with heuristically selected Lagrangian multiplier X (parameterized with b because the first part also proportional to b). We simplify Equation 5 to
arg min || - myb v.C(=) AW xx12 + Ab 10-12
(6)
AW
and ease the presentation by considering from a row-wise perspective, i.e., for AW; E RIx=;
arg min AW;
nb v,, [(2) AW; =112 + Ab - 11
The problem is separable into row-wise perspective because the norm (l (12) is the squared Frobenius norm and the rows have independent solutions.
In the following, we provide foosed-form solution for optimization problem (7), which is related to [17), [18).
Lemma 1, The solution AW; of Equation 7
xT xx
AW; = - V2, (()
+ +
(8)
Proof deferred to Supplementary Material
Extending the solution to AW, we have
[xx" (me) +1 + -
AW =
(9)
and can accordingly use it for an update step for W, i.e.,
W - W + AW" or
- V.L(2) xx TT.
( + x In
xx
(10)
The update in Equation 10 directly inserts the solution of the problem formulated in Equation 4. This computation efficient as only requires inversion of an of convolutions correspond to 3 (RGB) times the squared first layer's 's kernel size, and for vision transformers corresponds to the number of pixels per patch. The values of n typically range from 27 (CIFAR ResNet) to 768 (ImageNet sarge-scale vision transformer).
matrix, where in the case
Lemma 2. Using TrAct does not change the set of possible convergence points compared to vanilla (full batch) gradient descent. Herein, we use the standard definition of convergence points as those points where no update is performed because the gradient is zero.
Proof sketch: First, we remark that only the training of the first layer is affected by TrAct. To show the statement, we show that (i) zero gradient for GD implies that TrAct also performs no update and that (ii) TrAct performing no update implies zero gradients
GD. Proof deferred to SM
The statement formalizes that TrAct does not change the set of attainable models, but instead only affects the behavior of the optimization itself.
For an illustration of how TrAct affects updates W and 2, with a comparison language models and conventional vision models, see Figure 1.
3.1 Implementation Considerations
To implement the proposed update in Equation 10 in modern automatic differentiation frame- works [19], [20], we can make use of custom backward or backpropagation for the first layer.

---

Standard gradient for the weights:
For TrAct, we perform an in-place replacement by:
Vw < V=f(z) -x x
(11)
W t V=L(z) . x ( 2z
A I In
(12)
implemented via backward function: def backward (grad x, W): grad_W W grad_z.T a.d_ ar@ return grad_W
i.c., we replace the backward of the first layer by:
def backward x, 1-0.1): b, n shape
grad_ V grad_ 2.
inverse( x.T 0 x / b + eye(n))
return grad_W
Figure 2: Implementation of TrAct, where corresponds the hyperparameter A.
Details are shown in Figure 2. This applies the TrAct update from Equation 10 when using the SGD optimizer. Moreover, extensions of the update corresponding to optimizers like ADAM [21] (including, e.g., momentum, learning rate scheduler, regularizations, etc.) be attained by using respective optimizer and pretending (towards the optimizer) that the TrAct update corresponds to the gradient. As it only requires modification of the gradient computation of the first layer, the proposed method allows for easy adoption in existing code. All other layers / weights (0, () are trained conventionally without modification. Convolutions can be easily expressed via an unfolding of the input; accordingly, we unfold the inputs respectively in the case of convolution.
matrix multiplication
Moreover, we would like show a second method of applying TrAct that is exactly equivalent. Typically, we have some batch of data and first embedding layer embed, as well as remaining network net, loss, and targets gt. In the following, we show how the forward and backward usually written (left) and how it can be modified to incorporate TrAct (right):
(x) first layer pre-act y (z) $ remainder of the net loss(y, gt) backward # backprop
embed inverse x/b-1+eye(n))) data embed overwrs tes the values in x but leaves the gradient as before y = net(z)
loss(y, gt) .backward))
This modifies the input of embed for the gradient computation, but replaces the actual values propagated through the remaining network data by the original values, therefore not affecting downstream layers. which modifies the imput of embed for the gradient computation, but replaces the actual values propagated through the remaining network z. data by the original values, therefore not affecting downstream layers. This illustrates interesting relationships: TrAct is minimally invasive, can be removed or included any time without breaking the network, and does not have learnable parameters. TrAct can be seen as in some sense related to normalizing / whitening inverting the input for the purpose of gradient computation, but then switching the embeddings back to the original first layer embeddings/ activations for propagation through the remainder of the network.
We provide easy-to-use wrapper module that be applied the first layer, and automatically provides the TrAct gradient computation replacement procedure. For example, for PyToorch [19], the TrAct module can be applied to nn Linear and nn Conv2d layers by wrapping them as
TrAct (nn. Linear (...))
TrAct (nn. Conv2d (...))
and for existing implementations, can apply TrAct, e.g., for vision transformers via:
net www.pptchheeeee benccheembed..m proj TrAct (net. seaet.pacch_emeeeee )
4 Experimental Evaluation
4.1 CIFAR-10
Setup For the evaluation on the CIFAR-10 data set [22], we consider the ResNet-18 [23] well as a small VIT model. We consider training from scratch as the method is particularly designed for this case. We perform training for 100, 200, 400, and 800 epochs. For the ResNet models, we use the Adam and SGD with momentum (0.9) optimizers, both with cosine learning rate schedules; learning rates, due to their significance, will be discussed alongside respective experiments. Further, use the standard sofumax cross-entropy loss. For the VIT, use Adam with cosine learning
5

---

Testt TOLUCCUPSO
82
90
960
TrAct (A- 0.053
Adami TrAct JA 0:05)
(A
|ventla
Figure 3: Training ResNet-18 CIFAR-10. We train for {100, 200, 400, 800} epochs using cosine learning schedule and with SGD (left) and Adam (right). Learning rates have been selected optimal for each baseline. Averaged over seeds. TrAct (solid lines) consistently outperforms the baselines (dashed) many cases already with a quarter of the number of the epochs of the baseline.
rate scheduler as well as a softmax cross-entropy loss with label smoothing (0.1). The selected ViT is particularly designed for effective training CIFAR scales and has 7 layers, 12 heads, and hidden sizes of 384. Each model trained with a batch size of 128 on an Nvidia RTX 4090 GPU with PyTorch [19].
As mentioned above, the learning rate significant factor the evaluation. Therefore, throughout this paper, remove any bias towards the proposed method (and even give an advantage to the baseline), we utilize the optimal learning rate of the baseline also for the proposed method. For the Adam optimizer, consider learning rate grid of {10-2 10 10-3 10 10-3.5, 3.5]; for SGD with momentum, a learning rate grid of {0.1, 0.09, 0.08, 0.07]. The optimal learning rate is determined for each number of epochs using regular training; in particular, for Adam, have {100->10-3 200-+10-2 1,,00~>10 +10 800-+10 -10 ). and, for SGD with momentum, we find that a learning rate of 0.08 is optimal in each case. For the ViT, considered learning rate grid of {10-3 10 10-1.1, 10 3.2 10 3.3 10 3.4 10 3.5 10-3--0 3.6 10
3.7 -07 3.8 10 3.9 10 10-4). Here, the optimal learning (based the baseline) are (100~10- 200 +10 3.2 4000+10 10--3, 800~~10 10.3.3.
Results In Figure 3 we show the results for ResNet-15 18 trained on CIFAR-10. We can observe that TrAct improves the test accuracy in every setting, in particular, for both optimizers, for all four numbers of epochs, and for all three choices of the hyperparameter À E {0.05, 0.1, 0.2). Moreover, observe that, for SGD, the accuracy after 100 epochs is already better than for the baseline after 800 epochs. For Adam, we can see that TrAct after 100 epochs performs similar to the baseline after 400 epochs, and TrAct after 200 epochs performs similar to the baseline after 800 epochs. Comparing the different choices of X, X = 0.05 performs best in most cases.
The results for the ViT model displayed in Figure Again, we can observe that TrAct con- sistently outperforms the baselines for all X. Fur- ther, we can observe that TrAct with 200 epochs performs comparable to the baseline with 400 epochs. We emphasize that, again, the optimal learning rate has been selected based the base- line. Overall, here, À 0.1 performed best.
Test accurncy
91
TrAct
4.2
C1F0R-100
88
Adem Trvct (t -013
Adam trAct w0.2)
Setup For 100, we consider two ex- perimental settings. First, consider the train- ing of 36 different convolutional model architec- tures based strong and popular repository² for 100 We use the same hyperparame-
1.00
200
300
Epechs
400
700
900
Figure 4: Training VIT CIEAR-10. We train for (100, 200, 400, 800) epochs using a cosine learning rate schedule and with Adam. Learning rates have been selected as optimal for each baseline. Avg. over 5 seeds.
I Based on github..com/mmiinun77/7/1--CEE
Based on ggi.ab..om/veeaccccaaa/yacccccee

---

ters as the reference, i.e. SGD with mo- mentum (0.9), weight decay (0.0005).
Baseline
TrAct (A=0.1)
Model
Top-1
Top-5
Top-1
Top-5
and learning rate schedule with 60
epochs at 0.1, 60 epochs at 0.02,
SqueezeNet [24]]
69.45%
91.09%
70.48%
91.50%
40 epochs at 0.004, 40 epochs at 0.0008, and a warmup schedule dur- ing the first epoch, for total of 200 epochs. We reproduced each baseline on set of 5 separate seeds, and dis- carded the models that produced NaNs on any of the 5 seeds of the base- line. To make the evaluation feasi- ble, we limit the hyperparameter for TrAct to À= = 0.1. Second, we also re- produce the R8 CIFAR- 10 periment but with CIFAR-100. The results for this are displayed in Fig- ure 10 in the Supplementary Material and demonstrate similar relations as the corresponding Figure 3. Again, all mod- els are trained with a batch size of 128 on a single NVIDIA RTX 4090 GPU.
MobileNet [25]
66.99%
88.95%
87.06%
89.12%
MobileNetV2 [26]
67.76%
90.80%
67.89%
90.91%
ShuffleNet [27]
69.98%
91.18%
69.97%
91.45%
ShuffleNetV2 [28]
69.31%
90.91%
69.88%
91.02%
VGG-11 [29]
68.
88.02%
69.66%
88 98%99%
VGG-13 [29]
71.96%
90.27%
72.98%
90. 90.78%
VGG-16 [29]
72.12%
89.81%
72.73%
90.11%
VGG-19 [29]
71.13%
88.10%
71.45%
88.42%
DenseNet121 [30]
78.93%
94.83%
79.55%
94.92%
DenseNet161 [30]
79.95%
95.25%
80.47%
95.37%
DenseNet201 [30]
79.39%
95.07%
79.94%
95.17%
GoogLeNet [31]
76.85%
93.53%
77.18%
93.86%
Inception-v3 [32]
79.40%
94.94%
79.24%
95. 95.04%
Inception-v4 [33]
77.32%
93.80%
77.14%
93 93.90%
Inception-RN-v2 [33]
75.59%
93.00%
75.73%
93.32%
Xception [34]
77.57%
93.92%
77.71%
93.97%
ResNet18 [23]
76.13%
93.01%
76.67%
99%
ResNer34 [23]
77.34%
98.78%
87.87%
93.75%
ResNet50 [23]
78.20%
94.28%
79.07%
94.67%
ResNet101 [23]
79.07%
94.71%
79.51%
94.87%
ResNer152 [23]
78.86%
94.65%
79.83%
94.96%
ResNeX:50 [35]
78.55%
94.61%
78.92%
94.80%
Results We display the results for the 36 CIFAR-100 models in Table 1. We can observe that TrAct outperforms the baseline wrt. t and top-5 accuracy for 33 and 34 out of 36 models, re- spectively. Further, except for those 5 models, for which TrAct and the base- line perform comparably (each better on one metric), TrAct is better than vanilla training. Specifically, for 31 models, TrAct outperforms the baseline on both metrics, and the overall best result is also achieved by TrAct. Further, TrAct improves the accuracy on average by 0.49% on top-1 accuracy and by 0.23% on top-5 accuracy, a statistically very significant improvement over the base- line. The average standard deviations are 0.25% and 15% for top-1 and top-5 accuracy, respectively.
ResNeX:101 [35]
79.13%
94.85%
79.54%
94.84%
ResNeXu152 [35]
79.26%
94.69%
79.48%
94.89%
SE-ResNet18 [36]
76.25%
93.09%
76.77%
93.36%
SE-ResNe(34 [36]
77.85%
93.88%
78. 20%
94.13%
SE-ResNet50 [36]
77.78%
94.33%
78.79%
94.53%
SE- RE-RReNNe1101 [36]
77.94%
94.22%
79.19%
94. 70%
SE- JE-ResNer1521 [36]
78.10%
94.46%
79.35%
94. 73%
NASNet [37]
77.76%
94.26%
78.17%
94.35%
Wide-RN-40-10 [38]
78.93%
94.12%
79.60%
94.80%
StochD-RN-18 [39]
75.39%
94.09%
75.44%
94.13%
StochD-RN-33 [39]
78.03%
94.81%
78.16%
94.97%
StochD-RN-S5 RN50 [39]
77.02%
94.61%
77. 77.40%
94.78%
StochD-RN-101 [39]
78.72%
94.67%
78. 98.96%
94.75% %
Average
75.90%
93.
76.39% 42%
Table I: Results CIFAR-100 trained for 200 epochs, averaged over seeds. The standard deviations and results for TrAct with only 133 epochs are depicted in Tables 6 and 7 the SM.
In addition, we also considered training the models with TrAct for only 133 epochs, i.c., 2/3 of the training time, Here, we found that, on average, regular training for 200 epochs is comparable with TrAct for 133 epochs with a small advantage for TrAct. In particular, the average accuracy of TrAct with 133 epochs 75.94% (top-1) and 93.34% (top-5), which is small improvement over regular training for 200 epochs. The individual results are reported in Table 7 in the Supplementary Material.
4.3 ImageNet
Finally, we consider training on the ImageNet data set [40]. We train (esNNet-(18,44 50). ViT-S and ViT-B models.
ResNet Setup For the ResNet- 34, 50) models, we train for (30, 60, 90) epochs and consider base learning rates in the grid {0.2, 0.141, 0.1, 0.071, 0.05) and determine the choice for each model / training length combination with standard baseline training. We find that for each model, when training for 30 epochs, 0.141 performs best, and, when training for (60,90) epochs, 0.1 performs best as the base learning rate. We use SGD with momentum (0.9), weight decay (0.0001), and the typical lcarning rate schedule, which decays the learning rate after 1/3 and 2/3 of training by 0.1
7

---

each. For TrAct, we (again) use the same learning rate as optimal for the baseline, and consider {0.05, 0.1, 0.23. Each ResNet model is trained with batch size of 256 on single NVIDIA RTX 4090 GPU.
ResNet Results We start by discussing the ResNet results and then proceed with the vi- sion transformers. We present training plots forResNet-50 in Figure 5. Here, we can ob- serve effective speedup of factor of 1.5 during training, which we also demonstrate in Table 2. In particular, the difference in ac-
Baseline
TrAct (A=0.1)
Num. epochs
Top-1
Top-5
Top-1
Top-5
30
71.96%
90.70%
73 48%
91.61%
60
74.98%
92.36%
75. 75.68%
92. 78%
90
75.70%
92.74%
76. 20%
93.12%
Table 2: Final test accuracies (ImageNet valid set) for train- ing ResNet-50 [233] on ImageNet. TrAct with only 60 epochs performs comparable to the baseline with 90 epochs.
curacy for TrAct (A = 0.1) with 60 compared to the baseline with full 90 epoch training is -0.02% and +0.04% for top-1 and top-5.
94
92
Top-3 acurccyy
90
88
86
Adem Telct (A
0.05)
Adem Teleet 0.05)
Adem
Telct
0.11
Adam
To 0.3)
Adam
Adam
TToceeA 02)
100
20
40
50
70
80
90
30
40
60
70
Epochs
Figure 5: Test accuracy of ResNet-50 trained ImageNet for 430, 60, 90} epochs. When training for 60
epochs with TrAct, achieve comparable accuracy standard training for epochs, showing a 1.5x speedup. Plots for ResNet-1880 18/34 are the SM.
VIT Setup For training the ViTs, we reproduce the "DeiT III" [5], which provides the strongest baseline that is reproducible on single 8-GPU node. We train each model with the same hyper- parameters as in the official source code?. We note that the ViT-S and ViT-B are both trained at batch size of 048 and are pre-trained on resolutions of 224 and 192, respectively, and both models are finetuned on a resolution of 224. We consider pre-training for 400 and 800 epochs. Finetuning for each model is performed for 50 epochs. For the 400 epoch pre-training with TrAct, we use the stronger À = 0.1, while for the longer 800 epoch pre-training we use the weaker X = 0.2. We train the ViT-S models on 4 NVIDIA A40 GPUs and the ViT-B models on 8 NVIDIA V100 (32GB) GPUs.
VIT Results In Table we present the re- sults for training vision transformers. First, we observe that our reproductions follow- ing the official code and hyperparameters improved over the originally reported base- lines, potentially due contemporary im-
DeiT-IlI Model
Epochs
Top-1
Top-5
ViT-S [S] VIT-S
400ep
80.4%
400ep
81.23%
95.70%
VIT-S TrAct (A=0.1)
400ep
81.50%
95.73%
provements in the underlying libraries (our
VIT-S [5]
800ep
81.4%
ViT-$
800ep
81.97%
95.90%
hardware only supported more recent ver- sions). Notably, TrAct consistently im- proves upon our improved baselines. We
VIT-S TrAct (A=0.2)
800ep
82. 42.18%
95.98%
VIT-B [5]
400ep
83.5%
note that we did not change any hyperpa- rameters for training with TrAct. For ViT-S, using TrAct leads to 36% of the improve- ment that can be achieved by training the baseline twice as long. These improve- ments can be considered quite substantial modified only the training of the first layer. Notably, here, the runtime overheads were particularly small, ranging from 0.08% to 0.25%. Finally, we consider the quality of the pre-trained model outside
ViT-B f
400ep
83.34%
96.44%
VIT-B + TrAct (A=0.1)
400ep
83.58%
96.52%
Table 3: Results for training VIT' (DeiT-III) on ImageNet-1k, t denotes our reproduction.
considering that these are very large models and we
Based on giitub.cco//ncccoooesseccccc
8

---

of ImageNet. We fine-tune the ViT-S (800 epoch pre-training) model on the data sets CIFAR-10 10 and CIFAR- 100 [22] (200 epochs), Flowers- [41] (5000 epochs), and Stanford Cars [42] (1000 epochs). For the baseline, both pre-training and fine- tuning were performed with the vanilla method, and, for TrAct, both pre-training and fine-tuning were performed with TrAct. Table 4, we can observe consistent improvements for training with TrAct.
Model/DDaaset CIFAR-I0 CIFAR-100 Flowers
S. Cars
VIT-S
98.94%
90.70%
90.44% 95. 58% 91.07%
94.39%
ViT-S + TrAct 99.02%
90.85%
Table 4: Transfer learning results for ViT-S on CIFAR-10 and CIFAR-100 [22], Flowers- 102 [41], and Stanford Cars [42].
4.4
Effect of A
is the only hyperparameter introduced by TrAct. Often, with an additional hyperparam- eter, the hyperparameter space becomes more difficult to manage. However, for TrAct, the selection of À is simple and compatible with ex- isting hyperparameters. Therefore, throughout all experiments in this paper, we kept all other hyperparameters equal to the optimal choice for the respective baselines, and only considered A E {0.05, 0.1, 0.2). A general trend is that with smaller As, TrAct becomes more aggressive, which tends to be more favorable in shorter training, and for larger Äs, TrAct is more moderate, which is ideal for longer trainings. However, in many cases, the particular choice of E (0.05, 0.1, 0.2] has only a subtle impact on accuracy as can be seen throughout the figures in this work. Further, going beyond this range of \s, in Fig. 6, we can observe that TrAct is robust against changes in this parameter. In all experiments, the data was as-per-convention standardized to mean 0 and standard deviation 1: deviating from this convention could change the space of As. For significantly different tasks and drastically different kernel sizes or number of input channels. we expect that the space of As could change. Overall, we recommend A = 0.1 as a starting point and, for long training, we recommend À = 0.2.
91.5
Vanilla
Test accuracy
91.0
90.5
90.0
.00225
0125
.025
.05
-2
.8
1.6
Figure 6: Effect for training a ViT on CIFAR-10 10. Training for 200 ep., setup as Fig. 4, avg. over seeds.
4.5
Ablation Study
As ablation study, we first compare TrAct patch-wise layer normalization for ViTs. For this, we normalize the pixel values of each input patch to mean 0 and standard deviation 1. This is an alternate solution to the conceptual problem of low contrast image regions having lesser effect on the first layer optimization compared to higher contrast image regions. However, here, we also note that, in contrast to TrAct, the actual neural network inputs are changed through the normalization. Further, we consider DualPatchNorm [9] as a comparison, which additionally includes a second patch normalization layer after the first linear layer, and introduces additional trainable weight parameters for affine transformations into both patch normalization layers.
We use the same setup for the CIFAR-10 ViT and run each setting for seeds. The re- sults displayed in Figure/ 7. Here, ob- serve that patch normalization improves training for up 400 epochs compared to the baseline; however, not as much as TrAct does. Further, find that DualPatchNorm performs equiva- lently compared to input patch normalization and worse than TrAct, except for the case of 200 epochs where it performs insignificantly better than TrAct. For training for 800 epochs, patch normalization and DualPatchNorm do not im- prove the baseline and perform insignificantly worse, whereas TrAct still shows accuracy im- provements. This effect may be explained by the fact that patch normalization is a scalar form of whitening, and whitening can hurt generalization capabilities due to a loss of information [43]. In particular, what may be problematic that patch normalization also affects the model behavior during inference, which contrasts TrAct.
920
91.5
90.5
90.0
Adam T (b.b)
Aam
Trict (A @0.D
Adam
TTlck -0.2
89.0
Adom
Vin
885
Adam
PatchNorm
Adwm Dux Pasttrrtterr
88.0
300
400
500
600
700
800
Epochs
Figure 7: Ablation Study: training ViT on CIFAR- 10, including patch normalization (black, dashed) and DualPatchNorm (cyan, dashed). Setups as in Figure 4, averaged over 5 seeds.

---

As second ablation study, we examine what happens if we (against convention) do not perform standardization of the data set. We train the same ViTs above on CIFAR-10 for 200 epochs. averaged over 5 seeds. We consider two cases: first, an input value range of [0. and quite extreme input value range of [0, 255].
We display the results in Figure 8. Here, we ob- serve that TrAct more robust against a lack of standardization. Interestingly, we observe that
90
Tea sccuree
Vanilla [0.
TrAct performs better for the range of [0, 255) than [0, 1]. The reason for this is that TrAct suf- fers from obtaining only positive inputs, which affects the ræ matrix in Equation 10; however, we note that regular training suffers even more from the lack of standardization. When consider- ing the range [0, 255], we observe that TrAct is virtually agnostic to A, which is caused by the ae matrix becoming very large. The reason why TrAct performs so well here (compared to the baseline) is that, due to the large zr the updates AW become very small. This is more desirable compared to the standard gradient, which explodes due to its proportionality to the input values, and therefore drastically degrades training.
2551
80
TRACE [5.
.06225
.0125
.025
Figure 8: Ablation Study: training a VIT on CIFAR-10 without data standardization and with input value ranges of [0, 1] vs. [0, 255]. Setups in Figure 4, 200 epochs. and avg. over 5 seeds. All other experiments in this work are trained with data standardization.
In each experiment, we used only single opti- mizer for the entire model; however, our theory assumes that TrAct is used with SGD. This moti- vates the question of whether is advantageous to train the first layer with SGD, while training the remainder the model, e.g., with Adam.
Teat sccurity
81
Aami TrAct 0.05)
Adam Tt 0.11
Thus, as final ablation study, we extend the experiment from Figure (right) by training the first layer with SGD while training the remain- ing model with Adam. We display the results in Figure 9 where we can observe small improve- ments when using SGD for the TrAct layer.
0
1000
200
300
400
500
600
700
800
Epochs
Figure 9: Ablation Study: extending Figure 3 (right) by training the first layer with TrAct and SGD (pink) and the remainder the model with Adam.
4.6 Runtime Analysis
In this section, we provide a training runtime analysis. Overall, the trend is that, for large models, TrAct adds on tiny runtime overhead, while it can become more expensive for smaller models. In particular, for the CIFAR-10 10 ViT, the average training time per 100 epochs increased by 9.7% from 1091s to 1197s. Much of this can be attributed to the required additional CUDA calls and non-fused operations, which can be expensive for cheaper tasks. However, when considering larger models, this overhead almost entirely amortizes. In particular the VIT-S (800 epochs) pre-training cost increased by only 0.08% from 133:52 hours to 133:58 hours. The pre-training cost of the ViT-B (400 epochs) increased by 0.25% from 98:28 hours to 98:43 hours. We cost overhead is clearly more than worth the reduced requirement of epochs already. Further, fused kernels could drastically reduce the computational overhead; in particular, replaces an existing fused operation by multiple calls from the Python space. As TrAct only affects training, and the modification isn't present during forwarding, TrAct has effect inference time.
that, in each case, the training
current implementation
Discussion & Conclusion
In this work, we introduced TrAct, novel training strategy that modifies the optimization behavior of the first layer, leading to significant performance improvements across a range of experimental setups. The approach is efficient and effectively speeds up training by factors between 1.25x and 4 depending on the model size. We hope that the simplicity of integration into existing training schemes well as the robust performance improvements motivate the community adopt TrAct.
10

---

Acknowledgments and Disclosure of Funding
This work was in part supported by the Federal Agency for Disruptive Innovation SPRIN-D, the Land Salzburg within the WISS 2025 project IDA-Lab (20102-F1901 119901166-K2 (6--666PP and 2724--12S//25/197 2019), the ARO (W911NF-21-1-0122 the ONR (N00014-23-1-2159) and the CZ Biohub.
References
[]] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, I., Jones, A. N. Gomez, F.. Kaiser, and I. Polosukhin, "Attention is all you need," R in Proc. Neural Information Processing Systems (NeurIPS), 2017.
[2] C. Shorten and T. M. Khoshgoftaar, "A survey image data augmentation for deep learning." Journal of Big Data, vol. 6, 2019.
[3] A. Dosovitskiy. L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer,
G. Heigold, and G. et al., "An image is worth 16x16 words: Transformers for image recognition at scale," Proc. International Conference on Learning Representations (ICLR), 2020.
[4] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou, "Training data-efficient image transformers & distillation through attention," in Proc. International Conference on Machine Learning (ICML), 2021.
[5] H. Touvron, M. Cord, and H. Jégou, "DeiT III: Revenge of the ViT," in Proc. European Conference on Computer Vision (ECCV), 2022.
[6] S. loffe and C. Szegedy, "Batch normalization: Accelerating deep network training by reducing internal covariate shift." in Proc. International Conference on Machine Learning (ICML), 2015.
[7] T. Salimans and D. P. Kingma, "Weight normalization: A simple reparameterization accelerate training of
deep neural networks." in Proc. Conf. Neural Information Processing Systems (NeurIPS 2016, Barcelona, Spain), Neural Information Processing Systems Foundation, 2016.
[8] J. L. Ba, J. R. Kiros, and G. E. Hinton, "Layer normalization." " Computing Research Repository (CoRR) in arXiv,
2016.
[9] M. Kumar, M. Dehghani, and N. Houlsby, "Dual patchnorm," "
Trans. on Machine Learning Research, 2023.
[10] F. Dangel, "Backpropagation Beyond the Gradient," Ph.D. dissertation, University of Tübingen, 2023.
[11] S. J. Wright, Numerical optimization. 2006.
[12] J. Martens and R. Grosse, "Optimizing neural networks with kronecker-factored approximate curvature," in Proc. International Conference on Muchine Learning (ICML). 2015.
[13] F. Dangel, I.. Tatzel, and P. Hennig, "ViViT: Curvature access through the generalized Gauss-Newton's 's low-rank structure," " Transactions Machine Learning Rexearch, 2022.
[14] F. Petersen, T. Sutter, C. Borgelt, D. Huh, H. Kuehne, Y. Sun, and O. Deussen, "ISAAC Newton: Input-
approximate curvature for Newton's method," in Proc. International Conference on Learning Representations (ICI.R), 2023.
[15] F. Dangel, F. Kunstner, and Hennig, "Backpack: Packing more into backprop." Proc. International Conference on Learning Representations (ICLR), 2020.
[16] F. Petersen, C. Borgelt, T. Sutter. H. Kuehne, Deussen, and S. Ermon, "Newton losses: Using curvature information for learning with differentiable algorithms," in Conference on Neural Information Processing Systems (NewrIPS), 2024.
[17] A. E. Hocri and R. W. Kennard, "Ridge regression: Biased estimation for nonorthogonal problems," Technometrics, vol. 12, no. I, pp. 55-67, 1970.
[18] D. Calvetti and L. Reichel, "Tikhonov regularization with a solution constraint," SIAM Journal on Scientific Computing, vol. 26, no. 1, pp. 224-239, 2004.
[19] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. "Pytorch: Au imperative style, performefee deep leaminng library." Proc. Neural Information Processing Systems (NeurIPS), 2019.
[20] J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas,
S. Wanderman-Milnn Miine, and Q. Zhang, JAX: Composable transformations of Python Python+Numt +NumPy programs, 2018. [Online]. Available: ht //gt::://iue thub. com/ com/google/ googgell/////o jax.
[21] D. Kingma and J. Ba, " "Adam: A method for stochastic optimization," in Proc. International Conference on Learning Representations (ICLR), 2015.
[22] A. Krizhevsky, V. Nair, and G. Hinton, "Cifar- 10 (canadian institute for advanced rescarch)," 2009. [Online]. Available: http:/ http:///ww.c.e cs. w.s.toronto.co www.tto.eedouotntto.oo aadu/"kri//ea ciiaa./ciar... html.
[23] K. He, X. Zhang. S. Ren, and Sun, "Deep residual learning for image recognition." in Proc. International Conference on Computer Vision and Puttern Recognition (CVPR), 2016.

---

[24] F. N. landola, S. Han, M. W. Moskewicz. K. Ashraf. W. J. Dally, and K. Keutzer, "Squeezenet: Alexnet-level accuracy with 50x fewer parameters andk 0.5 mb model size," Computing Research Repository (CoRR) in arXiv, 2016.
[25] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, "Mobilenets: Efficient convolutional neural networks for mobile vision applications." Computing Research Repository (CoRR) in arXiv, 2017.
[26] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.,-C. Chen, "Mobilenetv2: Inverted residuals and linear bottlenecks," in Proc. International Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[27] X. Zhang, X. Zhou, M. Lin, and J. Sun, "Shuffienet: An extremely efficient convolutional neural network for mobile devices," in Proc. International Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[28] N. Ma, X. Zhang, H.-T. Zheng, and J. Sun, "Shufflenet v2: Practical guidelines for efficient can architecture design," in Proc. European Conference Computer Vision (ECCV), 2018.
[29] K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition," Computing Research Repository (CoRR) in arXiv, 2014.
[30] G. Huang. Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, "Densely connected convolutional networks." in Computer Vision and Pattern Recognition (CVPR), 2017.
Proc. International Conference
[31] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, "Going deeper with convolutions (2014)," Computing Research Repository (CoRR) in arXiv, 2014.
[32] C. Szegedy, V. Vanhoucke, S. loffe, J. Shlens, and Z. Wojna, "Rethinking the inception architecture for computer vision," " in Proc. International Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[33] C. Szegedy, S. loffe, V. Vanhoucke, and A. Alemi, "Inception-v4, inception-resnet and the impact of residual conncctions on learning," in AAAI Conference on Artificial Intelligence, 2017.
[34] F. Chollet, "Xception: Deep learning with depthwise separable convolutions," in Proc. International Conference on Computer Vision and Puttern Recognition (CVPR). 2017.
[35]
S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He, "Aggregated residual transformations for deep neural networks," in Proc. International Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[36] J. Hu, L. Shen, and G. Sun, "Squeeze-ane and---nd----eee excitatiin networks," in Proc. International Conference on Computer Vision and Pattern Recognition ( (CVPR), 2018.
[37] B. Zoph, V. Vasudevan. Shlens, and Q. Le, "Learning transferable architectures for scalable image recognition,"
in Proc. International Conference on Computer Vision and Pattern Recogmition (CVPR), 2018.
[38] S. Zagoruyko and N. Komodakis, "Wide residual networks," " Compuring Research Repository (CoRR) in arXiv,
2016.
[39]
G. Huang, Y. Sun, Z. Liu, D. Sedra, and K. Q. Weinberger. "Deep networks with stochastic depth." in Proc. Euro- pean Conference on Computer Vision (ECCV), 2016.
[40] J. Deng, Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, "Imagenet: A large-scale hierarchical image database." in Proc. International Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
[41] M.-E. Nilsback and A. Zisserman, "Automated flower classification over a large number of classes," in 2008 Sixth Indian conference on computer vision, graphics & image processingg IEEE, 2008
[42] J. Krause, J. Deng, M. Stark, and L.. Fei-Fei, "Collecting a large-scale dataset of fine-grained cars," 2013.
[43] N. Wadia, D. Duckworth, S. S. Schoenholz, F. Dyer, and J. Sohl-Dickstein, "Whitening and second order opti- mization both make information in the dataset unusable during training. and can reduce or prevent generalization." in Proc. International Conference on Machine Learning (ICML). 2021.
[44] S. Ren, K. He, R. Girshick, and J. Sun, "Faster R-CNN: Towards real-time object detection with region proposal networks." in Proc. Neural Information Processing Systems (NeurIPS), 2015.
[45] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman, The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results. [Online]. Available: http://www pascal network. arg/challenges, V0C/sooc2007/xorrkhhpp wwwprrrewwooooooiieoeeeeeeeeeeee index,idexiiiie html
12

---

A Theory
Lemma 1. The solution AW; of Equation 7
- T
-1
AW; = - V
+ In
(13)
Proof. We would like to solve the optimization problem
arg min AW: || - ryb [z, AWæl + Ab 14W
necessary condition for a minimum of the functional
F(AW)) = (-nb Vz) - AWT) + (((A))(AW))
is that VAw, F(AW,) vanishes:
VAW: F(WW)
VAw; V- AW;a) Ab (AW
= 2(-mb 2,, [(2) (Ww, v, ((2) ((2))AW;a)) 2A6 AW;
= 2(-mp ((2) - (-xx + 2Nd AW;
2(nb V., (()) + AWxx) T AW _ 0.
follows for the optimal A AW; that minimizes F(AW))
nb V T(=)) + AWW II Ab =0
-nb (V,, AW, (xx + \bIn)
AW," nb (V,) (Ix \bIn)
x... T
AW,' (V, x
+ Aln) (
Using TrAct does not change the set of possible convergence points compared to vanilla (full batch) gradient descent. Herein, we the standard definition of convergence points those points where no update is performed because the gradient is zero.
Lemma 2.
Proof. First, we remark that only the training of the first layer is affected by TrAct. To show the statement, we show that (i) zero gradient for GD implies that TrAct also performs no update and that (ii) TrAct performing no update implies zero gradients for GD.
(i) In the first case, we assume that gradient descent has converged, i.e., the gradient wrt. first layer weights is zero Ww ((W) We want to show that, in this case, our proposed update is also zero, i.., AW* = 0. Using the definition of AW* from Equation 9, we have
AW* = 71 V.((Z) xT
+ A
In
(14)
XX T
-1
= = - VwW(WW
(b
+ A In
(15)
(xxT T
-1
= = 0. +2. = 0,
(16)
which shows this direction.
(iz) the second case, we have AW* = and need to show that this implies C(W)
0. For this, we can observe that (xx A In) -1 is PD (positive definite) by definition and
13

---

[x / X- In) also exists. If AWW 0, then
xx
0 AW* AW
+ x
(17)
xx
- V=
+ A- In
+ +
b
= - V=f(z) x = VwL(W),
(18)
which also shows this direction.
Overall, showed that if gradient descent has converged according to the standard notion of gradient, then our update has also converged and vice versa.
B Additional Results
We display additional results Figures 10, 11, and 12 as well as in Tables 6 and 7.
As an additional experiment, in order to verify the applicability of TrAct beyond training / pre-training, we train Faster R-CNN models [44] on PASCAL VOC2007 [45] using VGG-16 backbone [29]. However, Faster R-CNN uses pretrained vision encoder where the first 4 layers are frozen. order to enable TrAct, TrAct only affeets the training of the first layer, unfreeze these first layers when training the object detection head. The mean average precision (mAP) on test data for the vanilla model versus TrAct training shown in Table 5.
vanilla
TrAct
0.659 = 0.005 0.671 + 0.004
Table 5: Mean average precision (mAP) test data for Faster R-CNN [44] with VGG-16 backhone PASCAL VOC2007 [45], averaged over seeds.
We observe that TrAct performs better than the vanilla method by about 1.1%. We would like to point out that, while TrAct especially designed for speeding up pretraining or training from scratch, ie., when actually learning the first layer, we find that it also helps in finetuning pretrained models. Here, a limitation is of course that TrAct requires actually training the first layer.
14

---

63
64
Text
922 5k- 0.2)
Adami Thier 0.21
963
Atam varils
100
20
Epecho
Figure 10: Training ResNet-18 CIFAR-100 with the CIFAR 10 setup from Section 4.1. Displayed is top-1 accuracy. We train {100, 200, 400, 800} epochs using cosine learning rate schedule and with SGD (left) Adam (right). Learning rates have heen selected as optimal for each haseline. Averaged over seeds. TrAct (solid lines) consistently outperforms the baselines (dashed lines).
72
70
Top-> arcumsyy
66
48
62
Acam vartil
60
Epoccts
Figure 11: Test accuracy of ResNet-18 trained on ImageNet for (30, 60, 90} epochs. Displayed is the top-1 (left) and top-5 (right) accuracy.
74
Top-5 arcuney
50
70
Adam (hécz (.25)))
Adami TiAct 0.005
Acem (Twee (i 3.30
Advmi Trkct O 0.11
64
Alam (1Ac) (uw
Adam varths
Adam 11400
1.0
70
Epechs
Tpochs
Figure 12: Test accuracy ResNet-34 trained on ImageNet for {30, 60, 90} epochs. Displayed the top-1
(left) and top-5 (right) accuracy.
15

---

Baseline
TrAct (A=0.1)
Model
Top-1
Top-5
Top-1
Top-5
SqueezeNet [24]
69.45% + 0.30%
91.09%
0.20%
70.48% + 0.17%
91.50% + 0.13%
MobileNet [25]
66.99% + 0.16%
88.95%
0.07%
67.06% 0.41%
89.12% 1 0.16%
MobileNetV2 [26]
67.76% +: 0.20%
0.10% 90.80%
67.89% 0.22%
90.91% 0.11%
ShuffleNet [27]
69.98% +: 0.22%
91.18%
0.12% 90.91%
+..97%
91.45% +: 0.29%
Shuffle:NttV2 [28]
69.31% + 0.13%
0.15%
0.26% 69.88%
91.02% + 0.08%
VGG-11 [29]
68,44% + 0.24%
88.02%
0.10%
69.66%
+.20%
88.99% : 0.21%
VGG-13 [29]
71.96% 0.26%
90.27%
0.17%
72.98%
0.18%
90.78% = 0.15%
VGG-16 [29]
72.12% 01.24%
89.81%
0.19%
72.73% 0.16%
90.11% + 0.15%
VGG-19 [29]
71.13% 0.46%
88.10%
0.36%
71.45% 0.34%
88.42% + 0.46%
DenseNet121 [30]
78.93% + 0.28%
94.83% 0.13%
79.55% + 0.25%
94.92% + 0.11%
DenseNet161 [30]
19.95%11..22
95.25%
0.19%
80.47% + 0.25%
95.37% + 0.12%
DenseNe1201 [30]
1..39%1110.00
95.07%
0.12%
79.94% 0.19%
95.17% 1. 0.100%
GoogLeNet [31]
76.85% 1 0.14%
0.16% 93.53%
77.18%
0.11%
93.86% 1 0.10%
Inception-v3 [32]
0.21% 94.94%
0.33% 79.24%
95.04% = 0.06%
Inception-v4 [33]
77.32% + 0.36%
0.33% 93.80%
77.14%
0.28%
93.90% + 0.20%
Inception-RN-v2
75.59% 0.45%
93.00%
0.18%
75.73%
0.30%
93.32% = 0.19%
Xception [34]
77.57% 0.31%
93.92%
0.17%
77.71% 0.17%
93.97% + 0.100
ResNet18 [23]
76.13%4:0 0.27%
93.01%
0.06%
76.67%
0.26%
93.29% + 0..2%
ResNer34 [23]
77.31% + + 0.33%
93.78%
0.16%
77.87% + 0.25%
93.75% + 0.10%
ResNet50 [23]
78.20% + 0.35%
0.09% 94.28%
79.07% + 0.18%
94.67% + 0.07%
ResNet101 [23]
79.07% 0.22%
0.20% 94.71%
79.51% = 0.43%
94.87% + 0.06%
ResNet152 [23]
78.86% 0.28%
0.22% 94.65%
79.83% 0.22%
94.96% + 0.09%
RexNeXt50 [35]
78.55% 0.22%
0.16% 94.61%
0.14% 78.92%
94.80% + 0.12%
ResNeX:101 [35]
+ 0.33%
+
94.85%
0.14%
79.54%
0.25%
94.84% + 0.10%
ResNeXr152 [35]
79.26% 0.29%
94.69%
0.11%
79.48%
0.16%
94.89% + 0.17%
SE-ResNet18 [36]
76.25% 0.18%
93.09%
0.19%
0.10% 76.77%
93.36% 1 0.09%
SE-ResNNe33 [36]
77.85% 0.19%
93.88%
0.15%
0.16% 78.20%
94.13% 1 0.21%
SE-ResNee50 [36]
77.78% 1 0.26%
0.12% 94.33%
78.79%
0.11%
94.53% 1 0.24%
SE-ResNer101 [36]
77.94% + 0.49%
0.10% 94.22%
+ 0.37% 79.19%
94.70% + 0.13%
SE-RcsNec155 [36]
78.10% 0.47%
94.46%
0.13%
79.35%
0.27%
94.73% = 0.15%
NASNet [37]
77.76% 0.19%
0.28% 94.26%
78.17% 0.11%
94.35% + 0.21%
Wide-RN-40-10 [38]
7
94.42%
0.09%
79.60%
0.18%
94.80% + 0.12%
StochD-RRN18 [39]
75.39% 0.14%
94.09%
0.10%
75.44%
+ 0.33%
91.13% + 0.17%
StochD-RR-34 [39]
78.03% 0.33%
94.81%
0.08%
78.
0.39%
94.97% + 0.100%
StochD-RN-50 [39]
77.02% + 0.18%
94.61%
0.13% 94.67%
77.40%
0.24%
94.78% + 0.10%
StochD-RN-UP [39]
78.72% 1 0.12%
0.05%
0.27% 78.96%
94.75% + 0.05%
Average (avg. std)
75.90% (0.26%)
93.19% (0.15%)
76.39% (0.24%)
93.42% (0.14%)
Table 6: Results on CIFAR-100, trained for 200 epochs, averaged over seeds including standard deviations.
16

---

TrAct (A 0.11, 133 133 ep) Top-5
Model
Top-1
SqueezeNet [24]
70.36% 0.30%
91.69% + 0.16%
MobileNet [25]
67.45% 0.38%
89.41% 1 0.13%
MobileNetV2 [26]
68.01% 0.32%
90.90% + 0.13%
ShuffleNet [27]
70.31% 0.32%
91.67% +: 0.25%
ShuffleNetV2 [28]
70.09% 0.34%
91.20% + 0.20%
VGG-11 [29]
69.14% 0.13%
88.92% + 0.18%
VGG-13 [29]
72.53% 0.26%
90.81% 0.12%
VGG-16 [29]
72.11% 0.10%
90.28% + 0.10%
VGG-19 [29]
70.54% 0..66%
88.48% + 0.20%
DenseNet121 [30]
79.09% 0.21%
94.79% + 0.11%
DenseNet161 [30]
80.20% 0.12%
95.30% 1 0.11%
DenseNet201 [30]
79.99% 0.20%
95.12% 1 0.16%
GoogLeNet [31]
76.59% 0.35%
93.83% 0.18%
Inception-v3
78.70% 0.22%
94.76% 0.16%
Inception-v4 [33]
76.50% 0.46%
98.56% 0.21%
Inception-RN-v2211 [33]
75.15% 0.24%
92.99% + 0.29%
Xception [34]
77.55% 0.34%
4
ResNet18 [23]
75.86% 0..00%
93.07%
0.07%
ResNet34 [23]
77.29% 0.23%
93.72% + 0.17%
ResNer50 [23]
78.44% 0.27%
94.47% 0.11%
ResNet101 [23]
79.20% 0.17%
94.77% 0.11%
ResNet152 [23]
79.34% 0.21%
04.92%00.087
ResNeXt50 [35]
78.90% 0.16%
94.75%
0.06%
ResNeXt101 [35]
79.09% 0.15%
94.78% 0.08%
ResNeXt152 [35]
78.91% 0.18%
94.67% 0.12%
SE-ResNec18 [36]
76.51% 0.43%
93.29% 0.16%
SE-ResNer34 [36]
77.81% 0.15%
94.02% 0.18%
SE-ResNee50 [36]
78.32% 0.22%
94.77% 1 0.14%
SE-ResNet101 [36]
79.07% 0.12%
94.79% + 0.31%
SE-ResNer152 [36]
79.03% 0.49%
94.74% 0.10%
NASNet [37]
77.85% 0.22%
94.34% 0.16%
Wide-RN-40-100 [38]
79.37% 0..55%
94.72% 0.07%
StochD-RRN-18 [39]
71.11%
0.16%
93.75% + 0.13%
StochD-RRN33 [39]
76.83%
0.31%
94.61% + 0.19%
StochD-RN-50 [39]
75.87%
0.29%
94.28% + 0.18%
StochD-RN-101 [39]
0.20% 77.73%
94.55% + 0.02%
Average (avg. std)
75.94% (0.25%)
93.34% (0.15%)
Table 7: Results on CIFAR-100, trained for 133 epochs, averaged over seeds including standard deviations.
17

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We address all claims made in the abstract and introduction in the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: All assumptions are pointed out in the work. Limitations are discussed.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: All assumptions are provided. Proofs are provided in the SM.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed reproduce the main experimen- tal results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We discuss all experimental parameters necessary for reproduction.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The code will be made publicly available gittub..ommfeelix-Peeeeeeeerer
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All training details are either explicitly discussed the main paper or inherited from the references.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Standard deviations are reported and correctly defined, described such, and utilize Bessel's 's correction. In some experiments, we display each seed's 's accuracy. In the ImageNet VIT experiments, running multiple seeds was not feasible due compute cost.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experi- ments?
Answer: [Yes]
Justification: We describe the hardware for each experiment and provide runtime analysis section.
9. Code Of Ethics
18

---

Question:
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips. httppppcccc/cccccccciceeeeeeeee.
Answer: [Yes]
Justification: The research conducted in this paper is conforming with the NeurIPS Code of Ethics. No animal were harmed in the execution this research.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The proposed method reduces training cost of vision models improves vision models under the same computational budget.
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All utilized assets (data sets, models, and code bases) are cited.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, well details about compensation (if any)?
Answer: [NA]
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper deseribe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based the requirements of your country institution) were obtained?
Question:
Answer: [NA]
19


Reviews:
[{"review_id": "detuRve6Go", "reviewer": "Reviewer_v9Bj", "summary": "When training vision models, the update of the weights of the first layer is proportional to the input pixel values. This can make the model learn images with high contrast faster and damage learning efficiency. To reduce this dependency, the paper proposed to optimize the first layer embedding (before activation) directly. This perspective leads to an optimization on a new way to update the weights and results in a light weight modification to a variety of vision models. The method was demonstrated on image classification problems.", "questions": "I believe the proposed method is general enough for vision tasks other than image classification. It would be interesting to see the performance on segmentation, detection, and even image generation. Specifically, I would like to see if the modification on the embedding hurts the performance of the tasks requiring more pixel-by-pixel understanding.", "limitations": "The authors stated that limitations are discussed as all assumptions are pointed out in the work. However, the assumptions listed are fairly broad and not specific to the proposed work. I’d like to see more discussion on the limitations imposed when modifying the first layer embedding.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors identified the fundamental problem of training vision models. Compared to modifying the input or the model architecture, targeting the update of the first layer weight is a more direct approach and easy to understand the impact. They formulated an optimization problem and clearly listed out the derivation for the solution to the update of the weights.\nThe proposed method is easy to implement.", "weaknesses": "There should be more elaboration on the intuition of the first layer embedding optimization. I can understand the purpose and the conceptual procedure can be viewed as activation applied to the first layer, but how this formulation reduces the dependency on the inputs without hurting the training is not obvious.\nIn the introduction, the authors argued that they bridged the gap between the “embedding” layer in language models and the first layers in vision models. However, it is not clear to me the connection of the proposed work to the “embedding” layer update in the language models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for your time reviewing and for helping us to improve our paper.\n\n> The authors identified the fundamental problem of training vision models. Compared to modifying the input or the model architecture, targeting the update of the first layer weight is a more direct approach and easy to understand the impact. They formulated an optimization problem and clearly listed out the derivation for the solution to the update of the weights. The proposed method is easy to implement.\n\nThank you very much for appreciating the clarity of the derivation of our method, and the simplicity to implement. We hope that these properties, combined with the training speedups lead to a wide adoption in the community.\n\n> There should be more elaboration on the intuition of the first layer embedding optimization. I can understand the purpose and the conceptual procedure can be viewed as activation applied to the first layer, but how this formulation reduces the dependency on the inputs without hurting the training is not obvious. In the introduction, the authors argued that they bridged the gap between the “embedding” layer in language models and the first layers in vision models. However, it is not clear to me the connection of the proposed work to the “embedding” layer update in the language models.\n\nIn language models, the embedding layer is a lookup table, with an embedding / activation for each token.\nThus, given an input token (integer), one embedding is selected (a row of the embedding matrix), and fed forward into the next layer.\nThe training dynamics of the embeddings layer corresponds to updating the embeddings directly wrt. the gradient.\nThe update in a language model, for a token identifier $i$, is\n$W_i \\leftarrow W_i - \\eta \\cdot \\nabla_{z} \\mathcal{L}(z)$ where $z = W_i$ is the activation of the first layer and at the same time the $i$th row of the embedding (weight) matrix $W$. Equivalently, we can write $z \\leftarrow z - \\eta \\cdot \\nabla_{z} \\mathcal{L}(z)$.\n\nIn contrast, in vision models, the update is\n$W \\leftarrow W - \\eta \\cdot \\nabla_{z} \\mathcal{L}(z) \\cdot x^\\top$\nand our goal is to achieve an update that is close to\n$z^* \\leftarrow z - \\eta \\cdot \\nabla_{z} \\mathcal{L}(z)$, which we achieve via our closed form solution.\n\nWe will include the extended discussion into the camera-ready.\n\n> I believe the proposed method is general enough for vision tasks other than image classification. It would be interesting to see the performance on segmentation, detection, and even image generation. Specifically, I would like to see if the modification on the embedding hurts the performance of the tasks requiring more pixel-by-pixel understanding.\n\nThank you for this suggestion, which we combined with Reviewer yoFY's suggestion of training a FasterRCNN object detection model. Please see our discussion and preliminary results table in our response to Reviewer yoFY.\n\n> The authors stated that limitations are discussed as all assumptions are pointed out in the work. However, the assumptions listed are fairly broad and not specific to the proposed work. I’d like to see more discussion on the limitations imposed when modifying the first layer embedding.\n\nThere are no assumptions that are specific to this work as our method is quite generally applicable as long as the first layer is a fully-connected or convolutional layer.\nA potential limitation is that the approach is not applicable in settings where the first layer is frozen, e.g., at a random initialization; however, the technique of the first layer being frozen in some models stems from the first layer's training dynamics problem identified in our work and actually training the first layer with our technique could improve performance.\n\nPlease let us know if your had a different specific assumption in mind that you would like us to point out in the paper or discuss.\n\n\nWe would appreciate if you could let us know whether we have resolved and successfully answered all of your questions and concerns. If new question or concerns come up, please let us know.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QVENKll6Mf", "reviewer": "Reviewer_juWi", "summary": "The paper introduces TrAct, a training strategy that modifies the optimization behaviour of the first layer. The proposed first-layer optimization enables a slightly better results for training the model at lesser number of epochs. TrAct is being demonstrated for a wider setup on image classification and have shown to be consistent.", "questions": "1. What happens when only the last layer of a pre-trained model is fine-tuned? This is a setup used in evaluating many self-supervised  methods. Wouldn't the first layer optimization using TrAct create bias for a new data distrbution?", "limitations": "Not explicitly stated.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. TrAct enables faster convergence or can achieve slightly better performance for the same number of epochs.\n\n2. The paper demonstrates applicability of TrAct in various possible scenarios covering a wide range of settings.", "weaknesses": "1. A small overhead (due to additional parameters) in computational time when training with TrAct.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for your time reviewing and for helping us to improve our paper.\n\n**Strengths:**\n\n> 1. TrAct enables faster convergence or can achieve slightly better performance for the same number of epochs.\n> 2. The paper demonstrates applicability of TrAct in various possible scenarios covering a wide range of settings.\n\nThank you for appreciating especially the faster convergence of our method, as well as the applicability of our method across a wide range of settings.\n\n**Weaknesses:**\n\n> 1. A small overhead (due to additional parameters) in computational time when training with TrAct.\n\nWe would like to clarify that the small overhead is only due to the backward of the first layer being slightly more expensive, whereas the forward is not modified.\nIndeed, we do not introduce additional parameters.\n\n**Questions:**\n\n> What happens when only the last layer of a pre-trained model is fine-tuned? This is a setup used in evaluating many self-supervised methods. Wouldn't the first layer optimization using TrAct create bias for a new data distrbution?\n\nDuring additional fine-tuning experiments, not reported in the paper, when comparing a model pre-trained with TrAct vs. pre-trained without TrAct, and finetuning both models without TrAct, we observe the model that had originally been pre-trained with TrAct performed better. If you would like us to perform this experiment also for only fine-tuning the last layer, we offer to include these results in the camera-ready; however, we strongly assume that the model pre-trained with TrAct still performs better in this case because the performance of the entire model is mostly affected by pretraining, and the first layer being optimized more efficiently, leads to a more efficient optimization of the rest of the network.\n\nWe would appreciate if you could let us know whether we have resolved and successfully answered all of your questions and concerns. If new question or concerns came up, please let us know.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FJbz1tqzPD", "reviewer": "Reviewer_yoFY", "summary": "This paper presents TrAct, a new and novel training strategy to  modify the optimization behavior of the first layer. Basically, it can achieve faster convergence and achieve better classification performance across different models. The effectiveness of TrAct is demonstrated across a range of 50 experimental setups on the various benchmarks, underscoring the method's capability.", "questions": "Shown as above", "limitations": "The paper has concluded this in the checklist", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The technical elaboration of the proposed method is clear. \n2. I really like the motivation for the proposed method, which is straightforward.\n3. The evaluations conducted on the provided benchmarks provide evidence of the effectiveness of the proposed methods. However, there are some concerns regarding the experimental results, which will be further discussed in the weaknesses section.", "weaknesses": "Weaknesses\n\n- What I am curious about is whether Tract is also generally applicable in other visual tasks, such as detection, segmentation, pose estimation, etc. The author could simply do some experiments on FasterCNN to verify the applicability of TrAct in visual dense prediction. If applicable, this will further enhance the impact of the proposed method.\n- Why is there a significant difference in the performance of TrAct between SGD and Adam? For example, in Figure 2, TrAct seems to converge faster on SGD, even surpassing the performance of the baseline 800 epoch with 100 epochs of training time. However, on Adam, performance improvement seems to be rather minor. Is there any theoretical explanation for this?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for your time reviewing and for helping us to improve our paper.\n\n**Strengths:**\n\n> 1. The technical elaboration of the proposed method is clear.\n> 2. I really like the motivation for the proposed method, which is straightforward.\n> 3. The evaluations conducted on the provided benchmarks provide evidence of the effectiveness of the proposed methods. However, there are some concerns regarding the experimental results, which will be further discussed in the weaknesses section.\n\nThank you for appreciating the clarity of our technical elaboration, our straightforward motivation, and the effectiveness of our method. \n\nWe would appreciate if you could let us know whether we have resolved and successfully answered all of your questions and concerns. If new question or concerns come up, please let us know.\n\n\n**Weaknesses:**\n\n> What I am curious about is whether Tract is also generally applicable in other visual tasks, such as detection, segmentation, pose estimation, etc. The author could simply do some experiments on FasterCNN to verify the applicability of TrAct in visual dense prediction. If applicable, this will further enhance the impact of the proposed method.\n\nThank you for this concrete suggestion, which really helps extend our paper. Accordingly, we trained FasterRCNN models.\nHowever, we need to mention that FasterRCNN uses a pretrained vision encoder where the first 4 layers are frozen.\nIn order to enable a fair comparison, we unfroze these first two layers when training the object detection head.\nWe trained the models on PASCAL VOC2007, and have the following preliminary results with 2 seeds (measured in test mAP):\n\n| Seed | vanilla | TrAct |\n|--|--|--|\n| 1 | 0.655 | 0.667 |\n| 2 | 0.664 | 0.674 |\n\nWe can observe that both seeds with TrAct performed better than the best seed of the vanilla method, and the average improvement is 1.1%. We offer to extend these studies for the camera-ready.\n\nWe would like to point out that, while is TrAct especially designed for speeding up pretraining or training from scratch, i.e., when actually learning the first layer, we were excited to find it also helped in finetuning pretrained models. The limitation would be that TrAct is only applicable when the first layer is actually trained.\n\n> Why is there a significant difference in the performance of TrAct between SGD and Adam? For example, in Figure 2, TrAct seems to converge faster on SGD, even surpassing the performance of the baseline 800 epoch with 100 epochs of training time. However, on Adam, performance improvement seems to be rather minor. Is there any theoretical explanation for this?\n\nThis is indeed an interesting question, and there is a theoretical explanation for this:\nThe method is motivated for SGD training as the update that TrAct is intended to perform is returned in form of the gradient, and to execute the update the optimizer should (in theory) be SDG. \nThat it still works very well even for Adam is great, and illustrates the strong robustness and versatility of our method.\nYour comment inspired us to extend Figure 2 by an experiment where we train everything except for the first layer with Adam, and train the first layer with TrAct and the SGD optimizer.\nHere, we observed small improvements over optimizing the first layer with Adam, which are shown in the Author Rebuttal PDF.\nFinally, we want to mention that, while using SGD for the first layer in combination with TrAct can improve performance, for easier adoption as it is often simpler to use the same optimizer for everything. Nevertheless, for advanced users, this is a way to squeeze out even a bit more performance.\nWe will add a respective discussion to the camera-ready.\n\n\nWe would appreciate if you could let us know whether we have resolved and successfully answered all of your questions and concerns. If new question or concerns come up, please let us know.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Additional Figure PDF based on Reviewer yoFY's remark.

Meta-review: This paper is reviewed by three reviewers. All of them are positive to this paper, while they also pointed out a couple of points that could be added/addressed/clarified in the final version. The recommendation is acceptance.
Decision: Accept (poster)