Replacement Learning: Training Vision Tasks with Fewer Learnable Parameters

## Abstract

Abstract Traditional end-to-end deep learning models often enhance feature representation and overall performance by increasing the depth and complexity of the network during training. However, this approach inevitably introduces issues of parameter redundancy and resource inefficiency, especially in deeper networks. While existing works attempt to skip certain redundant layers to alleviate these problems, challenges related to poor performance, computational complexity, and inefficient memory usage remain. To address these issues, we propose an innovative training approach called Replacement Learning, which mitigates these limitations by completely replacing all the parameters of the frozen layers with only two learnable parameters. Specifically, Replacement Learning selectively freezes the parameters of certain layers, and the frozen layers utilize parameters from adjacent layers, updating them through a parameter integration mechanism controlled by two learnable parameters. This method leverages information from surrounding structures, reduces computation, conserves GPU memory, and maintains a balance between historical context and new inputs, ultimately enhancing overall model performance. We conducted experiments across four benchmark datasets, including CIFAR-10, STL-10, SVHN, and ImageNet, utilizing various architectures such as CNNs and ViTs to validate the effectiveness of Replacement Learning. Experimental results demonstrate that our approach reduces the number of parameters, training time, and memory consumption while completely surpassing the performance of end-to-end training.

## 1 Introduction

Updating learnable parameters is a core component of training deep learning models Yang et al. (). Currently, the primary mechanism for updating parameters in these frameworks is global backpropagation Mostafa et al. (), a technique widely applied in various fields, including computer vision Yoo (); Voulodimos et al. (), natural language processing Goldberg (; ), and speech processing Ahmad et al. (); Chauvin & Rumelhart (). However, the increase in network depth and model complexity during training leads to a rapid expansion in the computation time and parameter demands required by global backpropagation Nawi et al. (). This rise in computational and memory costs inevitably poses significant challenges to GPU processing capabilities and memory capacity Bragagnolo et al. (). Furthermore, the high similarity in learning patterns between adjacent layers in deep learning models Kleinman et al. () results in parameter redundancy and inefficient resource usage throughout the computation process. With the growing popularity of large models, there is an urgent need to find training methods that can shorten computation time, reduce GPU memory usage, and still ensure model performance.

To address these challenges, researchers have explored alternatives to traditional backpropagation, such as feedback alignment Lillicrap et al. (); Nøkland (), forward gradient learning Dellaferrera & Kreiman (); Ren et al. (), and local learning Su et al. (; ). These methods aim to reduce computational overhead by updating weights without relying entirely on backpropagation. However, these improvements do not completely address the inherent short-sightedness: the separation into blocks can make each part of the network only focus on its local objectives, possibly ignoring the overall objectives of the network. This can lead to the discarding of globally beneficial information due to the lack of inter-block communication. Additionally, self-attention layers in Vision Transformers (ViT) Dosovitskiy et al. () exhibit high correlations between adjacent layers, leading to the development of the skip attention method Venkataramanan et al. (). This method reuses attention calculations to reduce computation but risks propagating errors, potentially degrading performance, and causing overfitting. Thus, both alternative backpropagation techniques and skip attention strategies struggle to maintain model performance while improving efficiency.

In this paper, we introduce a novel learning approach called Replacement Learning, designed to address the challenge of maintaining model performance while reducing computational overhead and resource consumption. Replacement Learning freezes specific layers, and during backpropagation, these frozen layers utilize parameters from adjacent layers, updating them through a parameter integration mechanism controlled by two learnable parameters, further optimizing efficiency. Considering that parameters from adjacent layers, if solely derived from either shallow or deep layers, often fail to simultaneously enable frozen layers to excel in learning both local and global features, the frozen layers are designed to leverage parameters from both preceding and succeeding layers, which facilitates a more effective fusion of low-level and high-level information. Moreover, to prevent the continuity of feature extraction from being disrupted, we introduce optimized interval settings for frozen layers in Replacement Learning, striking an effective balance between computational efficiency and performance. Replacement Learning significantly reduces the number of parameters while allowing frozen layers to incorporate information from adjacent layers. Balancing historical context with new inputs through the two learnable parameters, Replacement Learning enhances the model’s overall performance. The effectiveness of Replacement Learning has been rigorously validated on multiple benchmark image classification datasets, including CIFAR-10 Krizhevsky et al. (), STL-10 Coates et al. (), SVHN Netzer et al. (), and ImageNet Deng et al. (), across various architectures such as CNNs and ViTs. Experimental results demonstrate that Replacement Learning not only reduces the number of parameters, training time, and memory usage but also outperforms end-to-end Rumelhart et al. () training approaches. We summarize our contributions as follows:

Figure: Figure 1: Comparison between different backbones with the training of Replacement Learning and end-to-end training regarding GPU Memory and Accuracy.
Refer to caption: extracted/5894751/all.png

- •
We propose a novel learning approach, Replacement Learning, which reduces the number of parameters, training time, and memory consumption while obtaining better performance than end-to-end training Rumelhart et al. ().
- •
Replacement Learning has strong versatility, and as a universal training method, it can be seamlessly applied across architectures of varying depths and performs robustly on diverse datasets.
- •
The effectiveness of Replacement Learning has been validated on commonly used datasets such as CIFAR-10 Krizhevsky et al. (), STL-10 Coates et al. (), SVHN Netzer et al. (), and ImageNet Deng et al. () on both CNNs and ViTs structures, and its performance has fully surpassed that of end-to-end training Rumelhart et al. ().

Figure: Figure 2: Comparison of (a) end-to-end backpropagation and (b) our proposed Replacement Learning.
Refer to caption: extracted/5894751/func_e2e.png

## 2 Related work

### 2.1 Alternatives to Backpropagation

To address the limitations of backpropagation, such as high computational cost, various alternative methods have been proposed, including target propagation Lee et al. (); Bartunov et al. (), feedback alignment Lillicrap et al. (); Nøkland (), and decoupled neural interfaces (DNI) Jaderberg et al. (). These approaches bypass traditional global backpropagation by directly propagating errors to individual layers, reducing memory usage and enhancing efficiency. Forward gradient learning Dellaferrera & Kreiman (); Ren et al. () offers a new paradigm for training deep networks more effectively. Local learning Zhang et al. (); Zhu et al. () segments the network into smaller, independently trained modules, optimizing local objectives to lower computational demands while preserving some global features Su et al. (; ). However, excessive segmentation can lead to coordination issues, harming overall performance, especially on complex datasets like ImageNet.

### 2.2 Utilizing Surrounding Layers

Leveraging the high similarity in learning conditions of surrounding layers, researchers have solved many problems in deep learning. Some studies have applied Residual Networks (ResNets) He et al. (), by adding a shortcut connection to the activation function of the next layer, this identity mapping enables ResNet to address the issues of degradation Philipp et al. (); Borawar & Kaur (), enhancing both the convergence speed and accuracy of the network Zhang et al. (); Allen-Zhu & Li (). Additionally, some researchers have proposed skipping attention, reusing the self-attention calculations from one layer in the approximations for attention in subsequent layers, achieving higher throughput Venkataramanan et al. (). However, due to the repeated use of prior layers, this method carries the risk of error propagation and could potentially cause losses during the learning process, impacting the model’s generalization ability.

## 3 Method

### 3.1 Preparations

To begin, we briefly introduce the forward and backward propagation processes of the traditional end-to-end training model Rumelhart et al. () to clarify the background. Let us assume that the depth of a network is $n$. For an input image $x$, the forward propagation process through $n$ layers of the neural network is as follows:

$$ $h_{0}=x$ (1) $$

$$ $h_{i}=f_{i}(h_{i-1};\theta_{i}),i=1,2,\cdots,n$ (2) $$

here, $h_{i}$ represents the activation value of the $i$-th layer, $f_{i}$ is the forward computation function of the $l$-th layer, and $\theta_{i}$ are the learnable parameters of the $i$-th layer.

Once the entire forward propagation process of the network is completed, we can calculate the loss $\mathcal{L}$ based on the label $y$:

$$ $\mathcal{L}=\mathcal{L}(h_{n},y)$ (3) $$

After the loss $\mathcal{L}$ is calculated, the network can perform backward propagation to update the parameters for each layer. The gradient computation and parameter updates for each layer are as follows:

$$ $\delta_{n}=\frac{\partial\mathcal{L}}{\partial h_{n}}$ (4) $$

$$ $\delta_{i}=\delta_{i+1}\times\frac{\partial h_{i+1}}{\partial h_{i}},i=n-1,n-2 ,\cdots,1$ (5) $$

$$ $\frac{\partial\mathcal{L}}{\partial\theta_{i}}=\delta_{i}\times\frac{\partial h _{i}}{\partial\theta_{i}}$ (6) $$

$$ $\delta_{i}=\delta_{i}-\eta\times\frac{\partial\mathcal{L}}{\partial\theta_{i}}$ (7) $$

where $\eta$ denotes the learning rate of the network, $\delta_{i}$ and $\delta_{n}$ are the gradients of the i-th and n-th layers, respectively.

### 3.2 Replacement Learning

Traditional end-to-end training Rumelhart et al. () is the mainstream method for training models. However, as the network depth and model complexity increase during training, all layers are involved in the optimization process. Combined with the high computational complexity of chain rule-based gradient calculations in both forward and backward propagation, this results in a large number of parameters and high demands on computation time and resources. Furthermore, given the high similarity of features between adjacent layers, it becomes unnecessary for every layer to participate in parameter updates during backpropagation Rumelhart et al. (), which results in parameter redundancy and inefficient training.

To address these issues, we propose Replacement Learning. The innovation of Replacement Learning lies in the mechanism of periodically freezing a layer’s parameters, denoted as $\theta_{i}$, utilizing parameters from adjacent layers, and updating them through a parameter integration mechanism controlled by two learnable parameters. This idea is inspired by Exponential Moving Average (EMA) He et al. (), where two learnable parameters are introduced to balance the historical context with new inputs.

When the adjacent layers used come from the preceding layers of $\theta_{i}$, they tend to perform well in learning local features due to capturing preceding contextual information. Still, they are less effective in acquiring global high-level semantic information. Conversely, when the adjacent layers are from the succeeding layers, the deeper layer parameters can ensure the learning of higher-level semantic and global features but perform poorly in extracting fine-grained features and capturing low-level details. Therefore, we opt to simultaneously utilize both preceding and succeeding layers, $\theta_{i-1}$ and $\theta_{i+1}$, and integrate their parameters, which facilitates a better fusion of low-level and high-level information, thereby enhancing the overall performance of the model.

We incorporate learnable parameters for $\theta_{i-1}$ and $\theta_{i+1}$, $a_{i}$ and $b_{i}$. During the forward propagation, the parameters of $\theta_{i}$ are replaced by parameter integration results based on the parameters of $\theta_{i-1}$ and $\theta_{i+1}$. Among them, $a_{i}$ and $b_{i}$ play a role in dynamically adjusting the contributions of $\theta_{i-1}$ and $\theta_{i+1}$. In the backpropagation process, $\theta_{i}$ does not participate in the parameter updates from gradient descent. We will explain this process again using the forward and backward propagation steps of our method, and the specific implementation process refers to Algorithm 1. In the network, for every $k$ layer there is a frozen layer,
It should be noted that if the $n$-th layer is the final layer and $n$ is an integer multiple of $k$, this layer will not be frozen. The set of indices for the frozen layers is:

$$ $\mathcal{F}=\{\,i\mid i\bmod k=0\,\},\quad i=k,2k,3k,\dotsc$ (8) $$

During the forward propagation process, the propagation through non-frozen layers follows the same process as described in Eq.2, while the parameters and computation process for the frozen layers are as follows:

$$ $\theta_{i}=a_{i}\times\theta_{i-1}+b_{i}\times\theta_{i+1}$ (9) $$

$$ $h_{i}=f_{i}(h_{i-1};\,\theta_{i})$ (10) $$

where, $h_{i}$ represents the activation value of the $i$-th layer, $f_{i}$ is the forward computation function of the $i$-th layer, and $\theta_{i}$ are the learnable parameters of the $i$-th layer.

After completing the forward propagation through all the layers of the network, we can calculate the loss $\mathcal{L}$ as described in Eq.3. Subsequently, layer-by-layer backward propagation can begin. The backward propagation process for non-frozen layers is consistent with what is described in Eq.5, Eq.6, and Eq.7. The backward propagation process for the frozen layers is as follows:

First, we calculate the error term for it:

$$ $\delta_{i}=\delta_{i+1}\times\frac{\partial h_{i+1}}{\partial h_{i}}$ (11) $$

Subsequently, we compute the gradients for $a_{i}$ and $b_{i}$ as follows:

$$ $\frac{\partial\mathcal{L}}{\partial a_{i}}=\delta_{i}\times\frac{\partial h_{i }}{\partial\theta_{i}}\times\theta_{i-1},\frac{\partial\mathcal{L}}{\partial b _{i}}=\delta_{i}\times\frac{\partial h_{i}}{\partial\theta_{i}}\times\theta_{i +1}$ (12) $$

After the gradient calculations are complete, we update the parameters $a_{i}$ and $b_{i}$:

$$ $a_{i}=a_{i}-\eta\times\frac{\partial\mathcal{L}}{\partial a_{i}},b_{i}=b_{i}- \eta\times\frac{\partial\mathcal{L}}{\partial b_{i}}$ (13) $$

Finally, the error propagates to the adjacent layers:

$$ $\delta_{i-1}=\delta_{i}\times\left(\frac{\partial h_{i}}{\partial h_{i-1}}+ \frac{\partial h_{i}}{\partial\theta_{i}}\times a_{i}\times\frac{\partial \theta_{i}}{\partial\theta_{i-1}}\right)$ (14) $$

$$ $\delta_{i+1}=\delta_{i+1}+\delta_{i}\times\frac{\partial h_{i}}{\partial\theta _{i}}\times b_{i}\times\frac{\partial\theta_{i}}{\partial\theta_{i+1}}$ (15) $$

Figure: Algorithm 1 Replace Learning

Essentially, the proposed Replacement Learning replaces the complete set of parameters in certain layers with only two learnable parameters, effectively mitigating the issues of high computational cost, long training time, and GPU memory consumption inherent in traditional end-to-end training Rumelhart et al. (). Moreover, by employing the parameter integration mechanism, Replacement Learning enhances the network’s overall performance. Furthermore, Replacement Learning demonstrates strong versatility, as it can be seamlessly applied across architectures of varying depths and performs robustly on diverse datasets. This versatility is crucial for efficiently training deeper and more complex deep learning models.

## 4 Experiments

### 4.1 Experimental Setup

We conduct experiments using four widely adopted datasets: CIFAR-10 Krizhevsky et al. (), SVHN Netzer et al. (), STL-10 Coates et al. (), and ImageNet Deng et al. (), with Vision Transformer (ViT) Dosovitskiy et al. () and ResNets He et al. () of varying depths, serve as the network architectures.

During the experiment, we do not utilize pre-trained models. Instead, we train from scratch. We set $k=4$ as the interval for the parameter integration mechanism. Apart from the frozen layers, all other layers compute the loss using gradient descent and update the parameters via backpropagation Rumelhart et al. ().

### 4.2 Experiment Implement Details

In our experiments on CIFAR-10 Krizhevsky et al. (), SVHN Netzer et al. (), and STL-10 Coates et al. () datasets, we utilize the AdamW optimizer Loshchilov & Hutter () with a weight decay factor of 1e-4 for ViT-B, ViT-L Dosovitskiy et al. (), ResNet-32, and ResNet-110 He et al. (). We employ batch sizes of 1024 for CIFAR-10 Krizhevsky et al. (), SVHN Netzer et al. (), and STL-10 Coates et al. (). The training duration spans 250 epochs, starting with initial learning rates of 0.01, following a cosine annealing scheduler Loshchilov & Hutter ().

For ImageNet Deng et al. (), We use the AdamW optimizer Loshchilov & Hutter () with a weight decay factor of 1e-4. Different hyperparameters are used for each architecture: batch size is 128 for ViT-B Dosovitskiy et al. () and ResNet-34 He et al. (), and batch size is 32 for ResNet-101 and ResNet-152 He et al. (). Training lasts 100 epochs with initial learning rates of 0.04 for ViT-B Dosovitskiy et al. () and ResNet-34 He et al. (), and 0.01 for ResNet-101 and ResNet-152 He et al. ().

We recognize that in the Transformer Encoder of the ViT Dosovitskiy et al. () architecture, one layer consists of an MLP and a Multi-Head Attention. When freezing layers, we freeze only the gradients of the Multi-Head Attention, without altering the gradient descent of the MLP during forward propagation. For the ResNet He et al. () architecture, we refer to each residual block as a layer, where each layer is composed of two convolutions. The entire layer is frozen during gradient freezing, with the parameters derived from the parameter integration mechanism entering the next layer via the residual connection.

### 4.3 Comparison with the SOTA results

#### 4.3.1 Results on Small Image Classification Benchmarks

We start by assessing the accuracy performance of our method using the CIFAR-10 Krizhevsky et al. (), SVHN Netzer et al. (), and STL-10 Coates et al. () datasets. As illustrated in Table 1, the performance of Replacement Learning significantly exceeds the performance trained using end-to-end Rumelhart et al. () training on all structures.

**Table 1: Perfomance of different backbones. The * means the usage of our Replacement Learning.**
| Datasets | Backbone | Test Accuracy | GPU Memory | Time (Each epoch) |
| --- | --- | --- | --- | --- |
| CIFAR-10 | ViT-B | 72.23 | 10.03G | 7.37s |
| ViT-B* | 72.86 (↑ 0.63) | 9.94G (↓ 0.90%) | 7.36s (↓ 0.14%) |  |
| ViT-L | 69.85 | 20.46G | 16.68s |  |
| ViT-L* | 71.23 (↑ 1.38) | 20.29G (↓ 0.83%) | 16.58s (↓ 0.60%) |  |
| ResNet-32 | 79.60 | 5.29G | 3.63s |  |
| ResNet-32* | 81.82 (↑ 2.22) | 5.21G (↓ 1.51%) | 3.46s (↓ 4.68%) |  |
| ResNet-110 | 83.53 | 11.05G | 10.68s |  |
| ResNet-110* | 83.99 (↑ 0.46) | 11.01G(↓ 0.36%) | 9.62s (↓ 9.93%) |  |
| SVHN | ViT-B | 81.74 | 10.03G | 11.02s |
| ViT-B* | 85.15 (↑ 3.41) | 9.94G (↓ 0.90%) | 10.96s (↓ 0.54%) |  |
| ViT-L | 82.27 | 20.46G | 25.41s |  |
| ViT-L* | 84.77 (↑ 2.50) | 20.29G (↓ 0.83%) | 25.19s (↓ 0.87%) |  |
| ResNet-32 | 86.24 | 5.29G | 5.63s |  |
| ResNet-32* | 87.30 (↑ 1.06) | 5.21G (↓ 1.51%) | 5.56s (↓ 1.24%) |  |
| ResNet-110 | 87.86 | 11.05G | 15.96s |  |
| ResNet-110* | 88.18 (↑ 0.32) | 11.01G (↓ 0.36%) | 14.80s (↓ 11.78%) |  |
| STL-10 | ViT-B | 48.27 | 10.03G | 1.50s |
| ViT-B* | 50.81 (↑ 2.54) | 9.94G (↓ 0.90%) | 1.49s (↓ 0.67%) |  |
| ViT-L | 48.35 | 20.46G | 2.94s |  |
| ViT-L* | 49.03 (↑ 0.68) | 20.29G (↓ 0.83%) | 2.90s (↓ 1.36%) |  |
| ResNet-32 | 64.47 | 5.29G | 1.09s |  |
| ResNet-32* | 64.49 (↑ 0.02) | 5.21G (↓ 1.51%) | 1.06s (↓ 2.75%) |  |
| ResNet-110 | 53.25 | 11.05G | 2.24s |  |
| ResNet-110* | 60.08 (↑ 6.83) | 11.01G (↓ 0.36%) | 1.81s (↓ 19.20%) |  |

Replacement Learning, on the CIFAR-10 dataset, considerably improves Test Accuracy across various backbones. In the network of ViT-B and ViT-L Dosovitskiy et al. (), we record an improvement in Test Accuracy, from 72.23, 69.85 to 72.86, 71.23. In the relatively shallower network of ResNet-32 He et al. (), the Test Accuracy rises from 79.60 to 81.82. Even though the performance in the comparatively deeper network, ResNet-110 He et al. (), is somewhat inferior due to the inherent need for more global information in such networks, our method still delivers exceptional performance. It achieves approximately a 0.46 improvement, underscoring the robust effectiveness of Replacement Learning in deeper networks.

When applied to other datasets, Replacement Learning can also increase Test Accuracy by at least 3.41, 2.50, 1.06, and 0.32 on the STL-10 dataset Coates et al. (). On the SVHN Netzer et al. () dataset, our improvements over the four backbones also surpass 2.54, 0.68, 0.02, and 6.83, respectively. As can be seen, the improvement in our Replacement Learning substitution to all backbones is quite remarkable and comparable.

Other significant advantages of Replacement Learning can also be seen in Table 1, integrating Replacement Learning into various backbone architectures demonstrates a consistent reduction in GPU memory usage and training time across multiple datasets while maintaining or improving model performance. On the CIFAR-10 Krizhevsky et al. (), Replacement Learning integration leads to notable reductions in GPU memory consumption and training time. Specifically, GPU memory usage is reduced by 0.90% for ViT-B Dosovitskiy et al. () and 0.83% for ViT-L Dosovitskiy et al. () models, while ResNet-32 and ResNet-110 He et al. () see reductions of 1.51% and 0.36%, respectively. Training time per epoch is also decreased, with ViT-B and ViT-L Dosovitskiy et al. () showing reductions of 0.14% and 0.60%, respectively, and ResNet-32 and ResNet-110 He et al. () benefiting from reductions of 4.68% and 9.93% per epoch. Similar trends are observed on the SVHN Netzer et al. () and STL-10 Coates et al. (), where Replacement Learning consistently reduces GPU memory usage and training time across various backbones, reinforcing its effectiveness in optimizing computational efficiency.

#### 4.3.2 Results on ImageNet

We further validate the effectiveness of Replacement Learning on ImageNet Deng et al. () using four backbones of ViT-B Dosovitskiy et al. (), ResNet-34, ResNet-101, and ResNet-152 He et al. () as depicted in Table 2. When we employ ViT-B Dosovitskiy et al. () as the backbone, it achieves merely a Top1 Accuracy of 49.88 and a Top5 Accuracy of 73.86. However, with the usage of our Replacement Learning, the Top1 Accuracy increases by 0.94, and the Top5 Accuracy rises by 0.87. As illustrated in Table 2, the performance is below when we use ResNet-34, ResNet-101, and ResNet-152 He et al. () as backbones. After using our Replacement Learning, the Top1 Accuracy of these three backbone networks can be increased by 1.57, 1.66, and 1.69, Top5 Accuracy can be increased by 0.95, 0.99, and 1.04, respectively, compared to the original. Not only that, for GPU and training time, Replacement Learning has varying degrees of memory savings on all four models and can save an average training time of 2%-3% each epoch. These results underscore the effectiveness of our Replacement Learning on the large-scale ImageNet dataset, even when using deeper networks.

**Table 2: Results on the validation set of ImageNet**
| Backbone | Top1<br>Accuracy | Top5<br>Accuracy | GPU Memory | Time<br>(Each epoch) |
| --- | --- | --- | --- | --- |
| ViT-B | 49.88 | 73.86 | 19.03G | 4673s |
| ViT-B* | 50.82 (↑ 0.94) | 74.73 (↑ 0.87) | 18.63G (↓ 2.10%) | 4578s (↓ 2.03%) |
| ResNet-34 | 55.49 | 79.82 | 17.46G | 3493s |
| ResNet-34* | 57.06 (↑ 1.57) | 80.77 (↑ 0.95) | 17.03G (↓ 2.46%) | 3391s (↓ 2.92%) |
| ResNet-101 | 53.10 | 77.75 | 17.03G | 10268s |
| ResNet-101* | 54.76 (↑ 1.66) | 78.74 (↑ 0.99) | 16.38G (↓ 3.82%) | 10029s (↓ 2.33%) |
| ResNet-152 | 51.49 | 75.87 | 23.17G | 14478s |
| ResNet-152* | 53.18 (↑ 1.69) | 76.91 (↑ 1.04) | 22.90G (↓ 1.17% ) | 14159s (↓ 2.21%) |

### 4.4 Ablation Study

#### 4.4.1 Comparison of Features in Different Updating Layers

To showcase the advanced capabilities of Replacement Learning, we conduct feature map Selvaraju et al. () analyses with ResNet-32 He et al. () on different configurations, including end-to-end training Rumelhart et al. (), training with parameters updated by the preceding layer, training with parameters updated by the succeeding layer, and our Replacement Learning. The resulting figures detailing these feature maps can be found in Figure 3. Upon analyzing them, we can observe that (a) is concentrated in specific regions, indicating the presence of significant information within those areas. Conversely, after the fusion of (b) and (c), (d) captures more comprehensive global features, including localized edge features. It follows that the outstanding ability of Replacement Learning to capture global features.

Figure: Figure 3: Visualization of feature maps. (a) Feature map of end-to-end training. (b) Feature map of training with parameters updated by the preceding layer. (c) Feature map of training with parameters updated by the succeeding layer. (d) Feature map of Replacement Learning, parameters updated by both the preceding and succeeding layers.
Refer to caption: extracted/5894751/base.png

#### 4.4.2 Representation Similarity Analysis

To further demonstrate Replacement Learning’s effectiveness, we conduct Centered Kernel Alignment (CKA) Kornblith et al. () experiments. On the CIFAR-10 Krizhevsky et al. (), we compute the similarity between layers for both the end-to-end training Rumelhart et al. () and training of Replacement Learning, with ResNet-32 He et al. () serving as a representative case. From Figure 4, we observe that the feature similarity across layers in (b) is generally lower, except between the frozen layers (k=4, indicating that every 4th layer exhibits high feature similarity).

Figure: Figure 4: Visualization of similarity matrixes. (a) Similarity matrix of end-to-end training. (b) Similarity matrix of Replacement Learning.
Refer to caption: extracted/5894751/base_cka.png

Experimental results highlight three key advantages of Replacement Learning. First, in (a), the result of end-to-end training Rumelhart et al. () shows a gradual, smooth decrease in inter-layer similarity, suggesting progressively abstract features with depth. In contrast, (b) reveals more significant fluctuations, indicating that Replacement Learning captures more diverse features at different depths. Second, higher similarity in (a) suggests potential feature redundancy, limiting performance, while (b)’s lower similarity implies more distinct feature extraction, beneficial for complex tasks. Lastly, (a) may overfit the training set due to concentrated features, while (b)’s lower similarity, especially in deeper layers, enhances generalization.

#### 4.4.3 Decoupled Layer Accuracy Analysis

We have demonstrated that Replacement Learning achieves accuracy comparable to the end-to-end training Rumelhart et al. (). To further analyze Replacement Learning’s impact, we train a linear classifier for each layer. Results are shown in Figure 5 using ResNet-32 and ResNet-110 He et al. () as the baselines. The outcomes suggest that the selective freezing of layers combined with the parameter integration mechanism updates effectively enhances the robustness and generalization capability of the model. Specifically, Replacement Learning demonstrates higher accuracy in both ResNet-32 and ResNet-110 He et al. (), with the advantage being more pronounced in the deeper ResNet-110 network. These findings validate the effectiveness of the proposed Replacement Learning in deep neural networks, particularly in managing the interactions between layers. Overall, the experimental results support our hypothesis that improving inter-layer information transmission mechanisms can significantly enhance the performance of deep neural networks without increasing model complexity.

Figure: Figure 5: Comparison of layer-wise linear separability. (a) Linear Separability of RestNet-32 on CIFAR-10. (b) Linear Separability of RestNet-110 on CIFAR-10.
Refer to caption: extracted/5894751/index_32.png

### 4.5 Performance Study

#### 4.5.1 Parameter analysis

To investigate the factors contributing to the learnable parameter reduction in Replacement Learning, we consider a network model with $L$ layers, where the number of learnable parameters in the $i$-th layer is denoted as $P_{i}$. In end-to-end training Rumelhart et al. (), all layers’ parameters are simultaneously optimized, resulting in a total parameter count of $P=\sum_{i=1}^{L}P_{i}$. In contrast, with Replacement Learning, the parameters in frozen layers are not independently updated. Instead, they are adjusted using two learnable parameters that approximate the impact of adjacent layers. Thus, the total parameter count under Replacement Learning becomes $P^{\prime}=P-\sum_{i=1}^{L}P_{i+nk}+2,\;n\geq 0$. Although two additional learnable parameters are introduced, the total number of learnable parameters is reduced by $\sum_{i=1}^{L}P_{i+nk}-2$ compared to end-to-end training, thereby decreasing the computational demand for parameter updates.

To further analyze the range and patterns of learnable parameter reduction, let the number of parameters in all layers $P_{i}$ have a maximum value of $P_{\max}$ and a minimum value of $P_{\min}$ across all layers, $P_{\min}\leq P_{i}\leq P_{\max},\quad\forall i=1,2,\ldots,L$. In the case where all activated layers have the minimum number of parameters, the reduction in parameters is given by $\sum_{i=1}^{L}P_{i+nk}-2\geq\frac{L}{k}\times P_{\min}-2$. While in the case where all activated layers have the maximum number of parameters, the reduction is $\sum_{i=1}^{L}P_{i+nk}-2\leq\frac{L}{k}\times P_{\max}-2$. As the network becomes extremely deep, the ratio $\frac{L}{k}$ increases, making the remaining learnable parameters in the new model significantly lower compared to the original model. The upper and lower limits are:

$$ $\lim_{L\to\infty}\left(\frac{L}{k}\times P_{\max}-2\right)\approx\frac{L}{k} \times P_{\max}-2$ (16) $$

$$ $\lim_{L\rightarrow\infty}\left(\frac{L}{k}\times P_{\min}-2\right)\approx\frac {L}{k}\times P_{\min}-2$ (17) $$

Therefore, when all activated layers have the maximum number of parameters, the reduction is given by: $\sum_{i=1}^{L}P_{i+nk}-2\leq\frac{L}{k}\times P_{\max}-2$, and when all activated layers have the minimum number of parameters, the reduction is $\sum_{i=1}^{L}P_{i+nk}-2\geq\frac{L}{k}\times P_{\min}-2$. As $L$ approaches infinity, the upper and lower bounds converge to a multiple of the parameter values in the activated layers, scaled by $\frac{L}{k}$.

#### 4.5.2 Complexity Analysis

Tables 1 and 2 compare the GPU memory consumption of different network architectures at varying depths using end-to-end training Rumelhart et al. () and Replacement Learning. To explain why Replacement Learning uses less GPU memory, we analyze the computational complexity of the two methods. In E2E training, all parameters participate in forward propagation, resulting in a complexity of $L\times O(F)$. Backward propagation, requiring gradient calculations, has approximately twice this complexity, making the total $3\times L\times O(F)$. In Replacement Learning, forward propagation also has a complexity of $L\times O(F)$. During backward propagation, only $L-\frac{L-1}{k}$ unfrozen layers are optimized, each with a backward complexity of $2\times O(F)$, leading to $(L-\frac{L-1}{k})\times 2\times O(F)$. The frozen layers involve only two learnable parameters, with a negligible backward complexity of $2\times\frac{L-1}{k}\times O(1)$. Thus, the total computational complexity is $L\times O(F)+(L-\frac{L-1}{k})\times 2\times O(F)+2\times\frac{L-1}{k}\times O
(1)\approx(3L-2\times\frac{L-1}{k})\times O(F)$.

Compared to end-to-end training, the complexity of Replacement Learning is reduced by $2\times\frac{L-1}{k}\times O(F)$. Based on the characteristics of deep learning, we analyze the upper and lower bounds, as well as the limit, of the complexity reduction $2\times\frac{L-1}{k}\times O(F)$. The upper bound is achieved when $k=1$, meaning no layers are frozen. In this case, the complexity reduction is:

$$ $2\times\frac{L-1}{k}\times O(F)\bigg{|}_{k=1}=(2L-2)\times O(F)$ (18) $$

While the lower bound is obtained when $k=L-1$, with only one active layer. Here, the complexity reduction is:

$$ $2\times\frac{L-1}{k}\times O(F)\bigg{|}_{k=L-1}=2\times O(F)$ (19) $$

As $L\to\infty$, the limit of the complexity reduction depends on $k$. If $k$ is a constant, the complexity reduction increases linearly with $L$. If $k\approx L$, the reduction converges to $O(F)$, indicating a stable reduction ratio.

### 4.6 Generalization Study

In this section, we aim to investigate the generalization performance of our proposed Replacement Learning. To evaluate its effectiveness, we utilize the checkpoints trained on the CIFAR-10 Krizhevsky et al. () and test them on the STL-10 Coates et al. (), taking inspiration from previous work Qu et al. ().

**Table 3: Generalization study. Checkpoints are trained on the CIFAR-10 and tested on the STL-10. The data in the table represents the test accuracy.**
| Backbone | Test Accuracy | Backbone | Test Accuracy |
| --- | --- | --- | --- |
| ResNet-32 | 36.88 | ViT-B | 28.31 |
| ResNet-32* | 37.95 (↑ 1.07) | ViT-B* | 30.14 (↑ 1.83) |
| ResNet-110 | 39.19 | ViT-L | 26.25 |
| ResNet-110* | 39.76 (↑ 0.57) | ViT-L* | 28.02 (↑ 1.77) |

As shown in Table 3, with the usage of our Replacement Learning, we witness a significant improvement in test accuracy, surpassing all backbones’ end-to-end training Rumelhart et al. (). These findings emphasize the efficacy of our Replacement Learning in improving the generalization capabilities of supervised learning, ultimately leading to enhanced overall performance in the image classification task.

## 5 Conclusion

This paper introduces a novel learning approach called Replacement Learning to address the problem of maintaining model performance while reducing computational overhead and resource consumption. Replacement Learning effectively reduces the parameter count while enabling frozen layers to integrate information from neighboring layers. Utilizing two learnable parameters to balance historical context and new inputs boosts the model’s overall performance. We apply Replacement Learning to various model structures and evaluate its performance on four widely used datasets across different deep network structures. The results demonstrate that our proposed Replacement Learning not only reduces training time and GPU usage but also consistently outperforms end-to-end training in terms of overall performance.

Limitations and future work: Although our proposed Replacement Learning can reduce the number of parameters to be computed, save memory, and decrease training time while outperforming end-to-end training, it has only been applied to image-based tasks and has not yet been extended to other large models, such as those in natural language processing or multimodal settings. In future work, we plan to explore the impact of Replacement Learning on other tasks to achieve a more comprehensive evaluation of the model’s effectiveness.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: I like the paper overall and don't have any major questions to ask.
soundness: 3
strengths: 1. A novel training strategy that replaces frozen layer parameters with a fusion of parameters from neighboring layers, controlled by two learnable parameters, reducing the training load without compromising performance.
2. Replacement Learning shows substantial savings in memory usage and training time and achieves better accuracy than end-to-end training.
3. The method performs well across diverse architectures (CNNs and ViTs) and datasets, suggesting broad applicability.
4. Extensive experiments on benchmark datasets confirm the effectiveness of Replacement Learning in surpassing the performance of standard training approaches.
summary: This paper introduces Replacement Learning, a novel training technique aimed at reducing the number of learnable parameters in deep learning models while maintaining or even enhancing model performance. The approach specifically targets the limitations of traditional end-to-end training, such as high computational demand, memory usage, and parameter redundancy, which are common in deeper architectures. Rather than updating all parameters during backpropagation, Replacement Learning freezes certain layers and uses parameters from adjacent layers, controlled by two learnable parameters, to inform the frozen layers through a parameter integration mechanism. This design enables the frozen layers to leverage both local and global feature representations, balancing historical context with new inputs while reducing memory and computational costs.

The authors conduct experiments across multiple image classification datasets (CIFAR-10, STL-10, SVHN, and ImageNet) using various architectures, including CNNs and Vision Transformers (ViTs). Results demonstrate that Replacement Learning reduces GPU memory usage, training time, and the number of parameters, while achieving higher accuracy than traditional end-to-end training. Furthermore, the method shows versatility, adapting effectively across different architectures and datasets.
weaknesses: I like the paper overall. However, would like to point some weaknesses which the authors have also mentioned in their limitations section:
1. While effective on image-based tasks, the approach has not yet been tested on other domains such as NLP or multimodal tasks, which limits its generalizability.
2. The paper could benefit from a more in-depth discussion of any limitations associated with freezing certain layers and its impact on long-term learning dependencies, especially in deeper networks.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. How does the value of $k$ impact the performance of the models? The author should perform an ablation study on this value.

2. How does the proposed method compare with other parameter-efficient training methods?

3. How does the proposed method compare with other alternative backpropagation methods and methods utilizing surrounding layers during training?

4. Can this method be applied to fine-tuning pre-trained networks?
soundness: 2
strengths: 1. The paper is clearly written and is generally easy to follow.

2. The problem being studied in the paper is becoming increasingly important recently.

3. The idea is simple yet seems to be something that people haven’t tried before.
summary: This paper proposes an efficient training method for deep neural networks, named Replacement Learning, which aims to reduce the number of trainable parameters, training time, and memory consumption. Replacement Learning achieves this by selectively freezing the parameters of certain layers, which then utilize parameters from adjacent layers updated through a parameter integration mechanism controlled by just two learnable parameters. This method leverages information from surrounding structures to enhance overall model performance while reducing computation and conserving GPU memory.
weaknesses: 1. The proposed method only marginally reduces the GPU memory consumption and training time compared to the baseline training method. 

2. The paper did not compare the proposed method with any other parameter-efficient training methods, such as [3].

3. Although the paper discusses related works on alternative backpropagation methods and training utilizing surrounding layers, none of the related works are compared with the proposed methods in the experiments.

4. Parameter-efficient training methods [1, 2] are widely applied in fine-tuning pre-trained networks by selectively updating a small subset of model parameters, streamlining the adaptation process of pre-trained models, and facilitating rapid deployment across various domains. However, this paper only studies the setting for training-from-scratch.

[1] Zhang, Taolin, et al. "Parameter-efficient and memory-efficient tuning for vision transformer: a disentangled approach." ECCV 2024.

[2] He, Xuehai, et al. "Parameter-efficient model adaptation for vision transformers." AAAI 2023.

[3] Mostafa, Hesham, and Xin Wang. "Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization." ICML 2019.

---

confidence: 4
contribution: 1
initial_rating: 1
presentation: 1
questions: I must express my disappointment after spending several hours reviewing such a submission.

I suggest that the authors reconsider the motivation behind the proposed method and conduct all experiments with greater seriousness and care, as many results currently seem unreasonable.
soundness: 1
strengths: I’d say the idea is somewhat interesting, and the paper follows the ICLR submission style.
summary: In this work, the authors aim to improve network efficiency and reduce parameter redundancy by introducing Replacement Learning, a straightforward approach that fixes the parameters in a layer by interpolating between two adjacent layers. The experimental results indicate that this method is effective—although I find these results somewhat difficult to believe.
weaknesses: 1. Figure 1 doesn’t make any sense—using very large models on very small datasets. It would be more meaningful to test on a larger dataset like ImageNet, with different models of different scales for comparison.

2. what is the motivation of such a method?

3. There are several unsupported and arbitrary statements in the paper. For example, in Line 83,  "Considering that parameters from adjacent layers, if solely derived from either shallow or deep layers, often fail to simultaneously enable frozen layers to excel in learning both local and global features" lacks justification. The relationship to local or global features is unclear here, as adjacent layers don’t inherently correspond to shallow or deep feature characteristics.

4. In Eq. 9, the parameters of the i-th layer are a linear combination of those from the previous and next layers. What’s the motivation behind? It’s also unclear why this setup would yield better performance, as shown in Table 1 and Table 2.

5. What is the motivation for providing detailed information about backpropagation? I don’t see any differences or novel contributions in this section.

6. In the experiments, I don’t understand how reducing the number of parameters in a model can consistently improve performance across different datasets and models. This seems completely unreasonable and doesn’t make sense at all, especially I didn't see anything can help with this.

7. BTW, the results on CIFAR-10, SVHN, and STL-10 aren’t reliable, as these datasets are too small to provide meaningful insights.

8. Why is k=4 chosen? There should be an ablation study to justify this choice. Additionally, why is there a frozen layer every k layers throughout the networks? I assume we can select layers.

9. What happens if the frozen layer is the last layer in a ResNet stage? How would parameter interpolation be handled in this case?

10. The experimental settings MUST have issues. All the results for the vanilla models are SIGNIFICANTLY lower than original papers. This calls into question the reliability of the results presented in the paper, potentially not only ImageNet.

11. Typo： feature maps can be found in Figure 4.3.1. 

12. I cannot tell any essiential differences among the four images in Fig. 3.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: See above
soundness: 2
strengths: 1. The writing of the article is quite good and the article is logical.
2. The experiments on the classification task seem to be  adequate.
3. The charts and graphs are more beautiful and properly presented.
4. Supplementary materials are substantial.
summary: This paper proposes a method of replacement Learning, which reduces computational overhead and resource consumption in deep learning. It enhances model performance while surpassing end-to-end training in efficiency and memory usage. The method has been validated on various datasets and architectures, showing its versatility and potential for broader application.
weaknesses: 1. Title: The authors use "Visual Tasks" in the title. "Visual Tasks" include multiple tasks such as classification, detection, segmentation, etc., but it seems that the paper is only validated on the classification task. I suggest adding other tasks to the paper, as has been done in several recent PEFT works [1-3].
[1] 1% vs 100%: Parameter-efficient low rank adapter for dense predictions.
[2] Pro-tuning: Unified prompt tuning for vision tasks.
[3] Adapter is all you need for tuning visual tasks.
[4] Parameter-efficient is not sufficient: Exploring parameter, memory, and time efficient adapter tuning for dense predictions.

2. Related work: the authors perhaps left out some of the most recent work of parameter-efficient fine tuning (PEFT). 
3. Experiments: (1) Experiments are performed only on classification tasks; (2) More parameter-efficient fine-tuning methods are available for comparison.

I would consider increasing the score if the authors could provide more convincing comparative experiments.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: I would improve my rating if experiments are performed for points 1 and 4 in weakness and clearing my questions in points, 3 and 5.

I also have a minor question:
 While it is clear both $\theta_{i-1}$ and $\theta_{i+1}$ are useful for approximating $\theta_i$ why not expand it to $\theta_{i-n}$ and $\theta_{i+n}$, this would increase the global context of the computations with minimal increase in parameters?
soundness: 2
strengths: 1. Replacement Learning introduces a novel approach for training more efficient models with lesser number of parameters. The parameter integration has good future potential, especially when applied to structured pruning and upon supporting transfer learning.

2. The experiments show good performance for image classification tasks for both convolutional and transformer networks for accuracy, memory and time for 1 epoch.
3. The paper presents ablations to determine the robustness of the method and also presents complexity analysis of the method.
4. The paper addresses the most experimental parameters in the appendix, exhibiting good reproducibility.
summary: The paper introduced a neural network learning mechanism - Replacement Learning that replaces the parameters of selected layers with just 2 parameters a,b. The output of that layer is computer by a linear model a* activations of previous layer + b* activations of next layer. Given that similar layers in neural network produce correlated outputs, this linear combination approximates the replaced layer's outputs. The method reduces the parameter count, throughput and increases accuracy for image classification datasets(CIFAR-10,STL-10,SVHN and Imagenet-1k).
weaknesses: 1. The primary concern with the paper is that the experiments are conducted by training networks from scratch. Transfer learning boosts the performance of classification tasks, for instance in the official vit paper[1], VIT B/16 upon transfer learning on CIFAR10 reached an accuracy of 98.13% while the paper achieves only 72.86%, therefore it is important to perform an experiment for examining the effect of replacement learning when transfer learnt. One possible reason why the paper omits trying transfer learning can be initialization of replaced parameters. Parameter Integration $a * \theta_{i-1}+ b* \theta_{i+1}$ is a linear model, this allows approximating a,b values for pre-trained layers in few steps. 

2. The paper only shows results on classification, effect of replacement learning on downstream tasks such as object detection and segmentation would strengthen the approach.

3. The flow of gradients is not in a singular direction during replacment learning as suggested by Eq. 15 where $\delta_{i}$ is used for gradient computation of $\delta_{i+1}$. This could cause issues such as vanishing or exploding gradients and must be given a look at. 

4.  While the experiments show better time per epoch, the effect of replacement learning on convergence must be studied. This is important as time for 1 epoch can be misleading when the number of epochs to converge is significantly higher than backpropagation.

5. While the paper addresses that only MSA layers were chosen for replacement, where $\theta_{i-1}$ and $\theta_{i+1}$ also MSA layers? This is important as [2] does not guarantee correlation between MSA and MLP layers. 

Nitpick: some figure references are wrong, Figures in the ablation display figure numbers 4.3.1, 4.3.2 and 4.3.3 which need to be corrected to figure 3,4,5. Usually it is the placement of caption and label in \begin{figure} that causes this issue.

References:
[1] Dosovitskiy, Alexey. "An image is worth 16x16 words: Transformers for image recognition at scale." arXiv preprint arXiv:2010.11929 (2020).
[2] Venkataramanan, Shashanka, et al. "Skip-attention: Improving vision transformers by paying less attention." arXiv preprint arXiv:2301.02240 (2023).