Detecting and Approximating Redundant Computational Blocks in Neural Networks

Authors: Irene Cannistraci, Emanuele Rodolà, Bastian Rieck
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: latent representations, representation learning, neural network similarities, classification, foundation models, large models

Abstract:
Deep neural networks often learn similar internal representations, both across different models and within their own layers. While inter-network similarities have enabled techniques such as model stitching and merging, intra-network similarities present new opportunities for designing more efficient architectures. In this paper, we investigate the emergence of these internal similarities across different layers in diverse neural architectures, showing that similarity patterns emerge independently of the datataset used. We introduce a simple metric, Block Redundancy (BR), to detect redundant blocks, providing a foundation for future architectural optimization methods. Building on this, we propose Redundant Blocks Approximation (RBA), a general framework that identifies and approximates one or more redundant computational blocks using simpler transformations. We show that the transformation $\mathcal{T}$ between two representations can be efficiently computed in closed-form, and it is enough to replace the redundant blocks from the network. RBA reduces model parameters and time complexity while maintaining good performance. We validate our method on classification tasks in the vision domain, using a variety of pretrained foundational models and datasets.

Decision: reject
Reviewer scores: 5, 6, 3, 3

Reviews and discussion:
<thread>
creation date:
1730060335459

thread_type:
official_review

<thread_object>
creator: reviewer_2ujy

<content>
summary:
This paper introduces a framework to optimize deep neural networks by detecting and approximating redundant computational blocks, aiming to reduce computational complexity without sacrificing performance. The authors propose a metric called Block Redundancy (BR), which identifies components that do not contribute significantly to the network’s final output.  Using this metric, they develop Redundant Blocks Approximation (RBA), a method that approximates these redundant blocks through simple transformations.

strengths:
- The paper is clearly written and easy to follow. 
- The idea of BR and RBA approach is simple and intuitive

weaknesses:
- The paper aims to develop an algorithm for training-free compression of visual transformers. However, the related work appears to be lacking. Several methods such as [1, 2] that achieve compression with minimal training are missing.

- Comparison with state-of-the-art methods is missing. Benchmarking against established compression techniques is essential for positioning RBA’s efficacy relative to existing approaches.  

- The experimental evaluation is limited to small datasets, making it difficult to assess RBA’s robustness for larger, more complex datasets such as ImageNet. Demonstrating generalization on larger datasets would strengthen the claim of RBA’s scalability and efficiency.

References

[1] Zhang, Hanxiao, Yifan Zhou, and Guo-Hua Wang. "Dense Vision Transformer Compression with Few Samples." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.  

[2] Bai, Shipeng, et al. "Unified data-free compression: Pruning and quantization without fine-tuning." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

questions:
Please refer to Weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730124111934

thread_type:
official_review

<thread_object>
creator: reviewer_tlds

<content>
summary:
The paper is interested in finding redundancy within deep neural network models and approximating redundant blocks using a linear transformation. The paper first introduces the Block Redundancy (BR) metric, that computes the MSE between representations of two consecutive blocks on a subset of training data, to evaluate changes in the internal representation of the models. Then, it proposes to approximate highly redundant blocks by finding a good linear transformation between the redundant representations. The linear transformation is found by solving a least square problem between the inputs and outputs of the redundant blocks. Finally, experiments show that using the approximated transformation on redundant blocks preserves performance while reducing number of parameters.

strengths:
- **Significance**: The motivation and research questions of the paper can lead practical impacts. Finding redundancies and approximations for large pretrained models can reduce the number of parameters and make them lighter and more efficient at inference. To this end, the paper shows that we can find such simple approximations while preserving downstream performances.  
- **Quality**: The paper presents experiments on multiple pretrained Transformer models and multiple image datasets.   
- **Clarity**: The organization of the paper and the writing are clear. I appreciate the "Takeaway" part after each experimental subsection.

weaknesses:
**Originality**: 
  - The first part of the paper, regarding the similarities between inner representations of pretrained foundation models, echoes previous studies on the same topic such as Nguyen et al. (2020), as mentioned in the paper. The main difference in this first part, is that the study is done on larger transformer-based foundation models as opposed to convolutional architectures, but the same insights are found, so this is a small contribution in my opinion.
  - It is not clear to what aim the paper introduces this "BR" metric. Why not consider other established similarity metrics such as CKA (Kornblith et al., 2019), or directly the cosine similarity of consecutive representations, for instance ? Why should we use the BR metric instead of other metrics ? In other words, what is measured by BR that is not measured by other metrics ? Furthermore, the similarities between blocks are shown with pairwise cosine similarities in Figure 2 and Section 4.1. Why not use the BR metric in that case ? 
 
**Quality**:
  - There are two weaknesses with the BR metric that are not discussed in the paper: i) it can only be computed between representations of same dimensions, this is usually the case in transformer-based architectures, but not for different kind of architectures, such as CNNs for instance ; ii) the metric is sensitive to different scaling for the representations. That means, for instance, that if the transformation between the two representations is only a linear rescaling by a matrix $A$, such as $h^{(b)}(x) = A h^{(b-1)}(x)$, then $BR(b) = -\frac{||A - I||\_2}{|D\_{sub}|} \sum\_{x \in D\_{sub}} ||h^{(b-1)}(x)||\_2$ which can be very low, even though the overall transformation can be easily approximated linearly. So I'm not sure the BR metric is a good proxy metric to evaluate if the blocks can be linearly approximated or not.
  - The link between the BR metric and the RBA approach is actually not that clear in the paper. Do we observe that redundant blocks found by BR are actually the ones we can "remove" by RBA ? In experiments in Table 1, results when applying RBA to a lot of different blocks are shown, but I don't see a link with actual values of BR for these blocks.

**Significance**:
  - The experiments are only conducted on small scale datasets, the study would be more meaningful and significant with bigger datasets like ImageNet, at least, to evaluate scalability.
  - While the idea of approximating whole blocks to reduce number of parameters is conceptually interesting, is it better than pruning ? Do we achieve a better reduction of number of parameters while preserving performance ? 
  - If I understand correctly, the RBA are computed in closed form using 3000 training samples of the *corresponding* dataset the model is then evaluated on. To what extent these approximations transfer from one dataset to another ? If we use 3000 images from cifar10 to approximate some layers, do we preserve performance on ImageNet ?

questions:
I've written specific questions in the Weaknesses part, please refer to that. I've tried to compile some of the questions below:
- Why not consider the BR metric in the pairwise similarity matrices shown in Figure 2 ?
- Is there a link between BR and the preservation of performance after RBA in Table 1 ?
- If we compute RBA with data from one dataset, does it transfer to a different dataset ?
- Are the values shown for "Params" in Table 1 including the RBA parameters of the approximated block(s) ?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730180143196

thread_type:
official_review

<thread_object>
creator: reviewer_tzqp

<content>
summary:
Based on the observation that multiple blocks in neural networks produce similar representations, the paper identifies these similar blocks (basically a vit encoder layer) and approximate the later blocks from previous blocks with similar representations. The paper uses the inverse of MSE between [CLS] tokens of transformers to identify redundant blocks. The redundant blocks are estimated using a linear transformation. The experiments showed that there is a increase in accuracy when approximating the redundant blocks in some cases but, in some cases accuracy decreases as well.

strengths:
The paper studied an important problem of removing redundant blocks from a vision transformer. The study approximated all intermediate redundant blocks which can remove multiple maybe less redundant layers that still ultimately leads to a similar representation as the preceding layers. The paper perform experiments to measure accuracy on removing different redundant blocks. The writing is written in simple grammar making it easy to understand. The observation that redundant blocks are model specific and not data specific is important as it allows building an architecture that is inherently faster than the base model with similar capabilities. The paper provides detailed reproducibility statement in its appendix.

weaknesses:
1. Novelty of Block Redundancy: Block redundancy is just the negative MSE of block i and block i+n outputs, cant an MSE just do the same job albeit a lower mse meaning more redundancy?
2. Tables 1,2 and 3 show that the best architecture needs to be searched and there is no singular dataset agnostic architecture that consistently maintain or improve the performance as shown in figures 2 and 6, although the change in accuracy is less, in cases where the drop in accuracy is important, the complexity of finding the optimal architecture increases.  This could be because of two reasons, a.)The BR is too simple that it isnt a suitable metric to rank redundant blocks. b.) The linear approximation performed might not be optimal.

3. This paper is not the first to try replacing redundant parts of a vision transformer. Venkataramanan et al [1] does linear approximation of redundant attention layers. While both papers are different as replacement learning focuses on replacing complete encoder blocks, the linear approximation and redundancy metric [1] can easily be extended to block level replacement. How does the BR and linear approximation compare to Venkataraman et al's approach when performing block approximation? basically we need to compare the accuracy and throughput (if significant).  

4. No results on downstream tasks like segmentation and object detection. This is important to understand the effectiveness of replacement of redundant blocks for tasks other than classification. The paper could apply the technique on Uformer on ADE20K dataset for Segmentation like in [1] and DETR on Pascal VOC / COCO for object detection. Feel free to use any other models or benchmarks other than specified.

Nitpick: In figure 6 in the row of DeiT-S there is are hidden column names behind the image, that could be cleared in case of preparing camera ready.

[1] Shashanka Venkataramanan, Amir Ghodrati, Yuki M Asano, Fatih Porikli, & Amir Habibian (2024). Skip-Attention: Improving Vision Transformers by Paying Less Attention. In The Twelfth International Conference on Learning Representations.

questions:
I would improve my score if the following questions are answered:
1.  The need for BR and not simply using MSE? I suggest that the paper removes the contribution of BR and just stick with MSE or negative MSE.
2. Some results on segmentation and object detection? Whats important is whether we can get a dataset agnostic architecture of modern transformers with lesser parameters. If time does not permit both segmentation and object detection results performing only segmentation is also acceptable, but, it is important to compare the results with [1].
3. Can you provide ablations on BR and Linear Approximation with [1]? This is important to justify the design choices made in this paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730428219982

thread_type:
official_review

<thread_object>
creator: reviewer_byfy

<content>
summary:
This paper investigates the similarities across different layers in diverse neural architectures and assesses the redundancy of blocks in vision transformer based on the similarities of outputs between two blocks. Ultimately, redundant blocks are replaced with simple linear layers to achieve lightweight models. Experiments on multiple datasets and models demonstrate that the proposed method appears to be effective.

strengths:
1. Code is released.
2. The paper is well-presented and easy to follow.
3. The experimental results and ablation study show the effectiveness.

weaknesses:
1. In Equation 1, when $b=1$, what does $\bf{h} ^ {(0)} (x)$ represent?
2. In Line 147 on Page 3, the authors state that a higher BR indicates a potential redundancy in block $b$. Why is block $b-1$ not considered to be a redundant block? Similarly, in Line 158, why not skip $b _ i$ while retaining any layer or several layers from $b _ {i+1}$ to $b _ {i+n}$?
3. In Table 2, is the retraining step conducted under the 'skip' mode? If not, the accuracy after retraining should be reported.
4. This paper lacks the comparison with other model compression methods, such as pruning techniques.
5. The results on more complex datasets, such as ImageNet, should be reported. Also, can the proposed method be used to other tasks, such as object detection?

questions:
Please see the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732469856165

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response

comment:
We would like to express our sincere gratitude to the reviewers for their thoughtful feedback and constructive comments. We deeply appreciate that all reviewers found the paper to be well-organized and easy to follow. We are particularly encouraged by the recognition that “the experimental results and ablation study show the effectiveness” of our approach (bYFY). Additionally, we thank the reviewers for acknowledging the practical implications of the motivation and research questions addressed in our work (tLds). Lastly, we are grateful for the appreciation of our “simple and intuitive” idea (2UjY).

However, we would like to take this opportunity to clarify a couple of key points that were raised by different reviewers but can be useful to everyone to better clarify the proposed work:

* **Training-Free and Tuning-Free Methodology**: Our approach is entirely training-free and tuning-free. Unlike existing methods, we do not involve any training or fine-tuning of the approximated network. Instead, we employ a closed-form estimation of a linear transformation to approximate one or multiple blocks. After the approximations, as usual, a single linear layer classifier is trained on top of the pretrained model to perform the desired downstream task.

* **Flexibility Across Arbitrary Blocks**: The method is highly flexible and can be applied not only between two consecutive blocks but also across any number of arbitrary, non-consecutive blocks. For example, it is feasible to approximate representations from block 2 to block 4 and subsequently from block 6 to block 7, as shown in Figure 1 in the paper.

* **Broad Applicability**: Our method does not make any assumptions on specific layers or architectures. Therefore, it can be applied to any computational block and architecture, for any downstream task. 


---



Additionally, we address the reviewer's concerns by performing the **following additional experiments**:
* To enhance the evaluation and address concerns about the generalization of RBA to large-scale datasets, we included results on **ImageNet-1K** in Table 1, demonstrating the method's applicability to larger datasets.
* To further support our approach, we have added Figure 6, which illustrates the **correlation between ImageNet-1K results and the BR metric**. The figure shows that approximating blocks with high BR values preserve comparable accuracy while significantly reducing the parameter count and computation time. Moreover, the results show that approximating more high-BR blocks achieves better accuracy compared to approximating fewer low-BR blocks, which are less redundant.
* In Table 3, we present **additional experiments** to evaluate the model's ability to approximate representations based on a transformation derived from a different dataset but applied using the same architecture. Except for MNIST—which may be too simplistic to generalize effectively—our findings consistently **show that a simple linear transformation can be shared across all tokens and datasets**, highlighting the robustness and flexibility of the proposed method.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors propose a method to identify redundant blocks in neural networks, and replace them with a simpler, less costly linear transforms. The Block Redundancy (BR) metric measures the -MSE between output representations of blocks, with a higher value indicating minimal change in outputs. Redundant blocks can be replaced by a linear transform on the outputs of the preceding similar block, thereby increasing the model efficiency at minimal drop in accuracy.

The final reviewer ratings were 3, 3, 5, 6.

The ACs would like to highlight a related area of work that has not been discussed in the submission or the reviews. Note that the models considered in this work are all residual networks (ViTs have skip layers, just like the original ResNets based on convolutions). There is a rich history of prior work showing that ResNets are robust to dropping of entire blocks and that features in ResNets typically change slowly with depth:

1) Residual Networks Behave Like Ensembles of Relatively Shallow Networks (NeurIPS 2016): Shows the entire layers/blocks can be removed in ResNets, without any noticable change to accuracy. Deleting multiple blocks at once increases error smoothly.

2) Convolutional Networks with Adaptive Inference Graphs (ECCV 2018), BlockDrop: Dynamic Inference Paths in Residual Networks (CVPR 2018): Subsets of layers can be completely skipped on a per-input basis.

3) Neural Ordinary Differential Equations (NeurIPS 2018): Explores the parallels between ResNets and ODEs that continuously and smoothly transform the hidden state.

4) Do Residual Neural Networks discretize Neural Ordinary Differential Equations? (NeurIPS 2022): Explores ResNets as discretization of Neural ODEs.

There is a clear parallel between what this submission tries to do and what the above mentioned works show - that layers/blocks in residual networks change features smoothly, have redundancies, and can even be skipped. So the idea of replacing entire layers/blocks with linear transformations should not come as a big surprise. 

Futher, the last 2 papers mentioned above could even provide a theoretical backing for the proposed work. This submission is incomplete without an examination of the relationship between such prior work.

The ACs have taken into account the confidential feedback submitted by the authors. 

Based on all the reviews and discussions, and the points made above, including the omission of very related work, the ACs do not believe this work in its current form is ready for acceptance.