<figure><img src="image_1.png" /></figure>

HAL
open science

# Spectral Norm Pruning: A Stability-Driven Approach to Deep Neural Network Compression

Arjan Matthams, Scarlet Moreno, Sadia Swanson

► To cite this version:

Arjan Matthams, Scarlet Moreno, Sadia Swanson. Spectral Norm Pruning: A Stability-Driven Approach to Deep Neural Network Compression. 2024. hal-04708640v3

HAL Id: hal-04708640

https://hal.science/hal-04708640v3

Preprint submitted on 20 Dec 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Spectral Norm Pruning: A Stability-Driven Approach to Deep Neural Network Compression

Sadia Swanson*, Arjan Matthams*, Scarlet Moreno*

*Department of Computer Science, University of California, Berkeley, CA, USA

Email: sadia.swanson@berkeley.edu, arjan.matthams@berkeley.edu, scarlet.moreno@berkeley.edu

**Abstract**—Channel pruning has emerged as a powerful technique to reduce the computational complexity and memory footprint of convolutional neural networks (CNNs), enabling their deployment on resource-constrained devices. However, identifying an effective criterion for channel importance remains a critical challenge. In this paper, we propose a novel channel pruning method based on the spectral norm of convolutional filters. The spectral norm, which reflects the largest singular value of a filter's weight matrix, provides a principled measure of a channel's contribution to the network. By leveraging the spectral norm, our approach effectively identifies and prunes less critical channels, leading to significant reductions in both parameters and floating-point operations (FLOPs) without compromising accuracy. We validate our method on popular architectures, including VGGNet and ResNet, and benchmark datasets such as CIFAR-10 and ImageNet. Experimental results demonstrate that our method achieves state-of-the-art performance in model compression, outperforming traditional pruning methods like $L_1$ norm-based pruning in terms of both efficiency and accuracy. Furthermore, we discuss the broader applicability of spectral norm-based pruning to other deep learning tasks and architectures, including object detection and Transformers. Our findings highlight the potential of spectral norm-based pruning as a versatile and effective tool for optimizing deep learning models, enabling their practical deployment in real-world, resource-limited scenarios.

**Index Terms**—CNN compression, matrix norm, network pruning

## I. INTRODUCTION

In the last decade, deep learning [1] has transformed a wide range of industries and research fields, driving advances in areas such as computer vision, natural language processing, speech recognition, and autonomous systems. With the introduction of seminal architectures like AlexNet [2], ResNet [3], and Transformer [4], deep neural networks (DNNs) have achieved unprecedented success in tasks such as image classification, object detection, machine translation, and more [5]. The rise of these architectures can be largely attributed to their ability to scale both in depth and width, leveraging millions or even billions of parameters to model highly complex patterns in data. As a result, these models have continually set new benchmarks in predictive accuracy and generalization performance [1].

However, this increase in the parameter count and network complexity has come at a significant cost. While large-scale networks thrive on high-performance computing platforms with abundant computational and memory resources, their training and deployment in real-world applications pose con-

siderable challenges [6]. The sheer size of modern networks leads to higher memory consumption, longer inference times, and increased energy demands. These constraints are particularly problematic for edge devices, mobile platforms, and IoT applications, where hardware resources are inherently limited. In such environments, deploying a large neural network in its original form becomes impractical, as the high computational and memory demands exceed the capabilities of these devices. Moreover, for real-time applications, latency due to slow inference times is unacceptable. As a result, the deep learning community has recognized the need for model compression techniques that reduce the size and complexity of networks without compromising their performance [7].

To address these challenges, several model compression methods have been introduced, with the goal of enabling efficient neural network deployment. The primary objectives of model compression are to reduce the number of parameters, minimize floating-point operations (FLOPs), and lower the memory footprint, all while preserving the model's predictive accuracy [7]. One widely adopted method is quantization, which reduces the bit precision of weights and activations, often from 32-bit floating-point to 8-bit integer representations [8]. This reduction significantly decreases both the memory requirements and computational complexity, making models more suitable for hardware implementations, such as those on edge devices. However, the challenge with quantization lies in managing the trade-off between compression and performance, as reducing precision too aggressively can degrade accuracy, particularly for large-scale tasks.

Another prominent compression method is low-rank factorization [9]–[11], which decomposes large weight matrices into smaller matrices with lower ranks, thereby reducing the number of parameters and computations required by the model [12], [13]. Low-rank approximations are particularly effective in compressing fully connected layers, which are often responsible for a large portion of the model’s overall parameter count. Despite the success of these methods in reducing model size, they may introduce approximation errors that degrade the model’s performance if not properly tuned [14].

Knowledge distillation [15] is another compression technique, where a smaller “student” network is trained to replicate the behavior of a larger, more accurate “teacher” network. The smaller network learns to mimic the teacher’s outputs, enabling it to achieve comparable performance with significantly
fewer parameters. Although knowledge distillation has shown considerable promise in compressing models without major accuracy loss, the training process is often computationally expensive, as it requires both the teacher and student models to be trained in tandem [16], [17].

Among these compression techniques, pruning [18] has emerged as a particularly effective strategy due to its ability to remove redundant weights, neurons, or channels from a neural network, thereby directly reducing the model's complexity. Pruning can be broadly categorized into unstructured and structured pruning. Unstructured pruning [19] targets individual weights within the network, resulting in sparse weight matrices. While this reduces the parameter count, it often does not lead to practical acceleration on common hardware platforms due to irregular memory access patterns. In contrast, structured pruning [20], particularly channel pruning, removes entire filters or channels from convolutional layers, leading to a reduction in both parameter count and computational operations. This form of pruning results in structured sparsity, which is more compatible with modern hardware optimizations, enabling real-world efficiency gains [21]–[24].

Channel pruning techniques are typically guided by heuristics, with magnitude-based pruning being one of the most commonly employed methods [25]. In magnitude-based pruning, channels with smaller weights are considered less important and are pruned, under the assumption that they contribute less to the overall performance of the network. Despite the simplicity and effectiveness of this approach [25], it often lacks a deeper consideration of the network's stability, robustness, and generalization capacity. By focusing solely on weight magnitudes, these methods may inadvertently remove channels that are critical for preserving the network's overall generalization, especially in complex tasks or datasets [26].

To overcome the limitations of magnitude-based pruning, we turn to more theoretically grounded criteria for pruning decisions. One such criterion is the spectral norm, which measures the largest singular value of a weight matrix and has been shown to correlate with the model's stability, capacity, and robustness [27]. Spectral norm has been successfully applied in various contexts, such as controlling the Lipschitz constant of neural networks [28] to improve adversarial robustness, enhancing generalization, and preventing overfitting [29]. However, its use in guiding channel pruning decisions has not been fully explored. The spectral norm provides a direct measure of a layer's sensitivity to input perturbations, making it a valuable criterion for determining which channels contribute most to the network's stability [27].

In this paper, we propose a novel spectral norm-based channel pruning framework that leverages the spectral norm as a criterion for identifying and removing less important channels. Our central hypothesis is that channels with lower spectral norm values contribute less to the overall stability and robustness of the network, and can therefore be pruned with minimal impact on performance. By incorporating spectral norm into the pruning process, we aim to compress the model while maintaining its generalization capabilities. Our experi-

ments, conducted on standard benchmarks such as CIFAR-10 [30] and ImageNet [26], demonstrate that our spectral norm-based pruning approach achieves significant reductions in model size and FLOPs, while maintaining or even improving accuracy compared to traditional magnitude-based pruning methods [25], [31].

The key contributions of this paper are as follows:

The contributions of this paper are as follows:

1) We introduce a novel channel pruning framework based on spectral norm, providing a more principled pruning criterion.

2) We develop an efficient algorithm for integrating spectral norm into the channel pruning process.

3) We present extensive experimental results demonstrating that our method outperforms traditional magnitude-based pruning in terms of both model efficiency and accuracy.

4) We conduct ablation studies to analyze the impact of spectral norm on different layers and architectures, offering insights into its role in network stability and generalization.

## II. RELATED WORK

Deep Neural Networks (DNNs) [32] have seen a rapid rise in popularity due to their remarkable success across a wide range of applications, from computer vision [33], [34] and natural language processing [35] to autonomous systems [36], human-machine interaction [37]–[39], and medical diagnostics [40]–[42]. These models have achieved unprecedented levels of accuracy, enabling breakthroughs in tasks such as image recognition, language translation, and real-time decision-making. The ability of DNNs to model complex, non-linear relationships has positioned them as the backbone of modern artificial intelligence.

However, this superior performance often comes at a high computational cost. DNNs typically require substantial memory and processing power due to their large number of parameters and extensive floating-point operations (FLOPs). This can result in slow inference times and significant energy consumption, making it challenging to deploy these models in resource-constrained environments, such as mobile devices, embedded systems, or real-time applications. As a result, the scalability and practicality of DNNs are often limited, prompting the need for more efficient solutions.

Model compression has emerged as a key solution to address these challenges by reducing the size and computational demands of neural networks without sacrificing performance. Various techniques, including pruning, quantization, and tensor decomposition, have been developed to optimize DNNs for faster inference and lower memory consumption [43]. Among these, pruning [25] has become a particularly prominent research area, offering a structured way to remove redundant or less important weights and filters from the network while maintaining predictive accuracy. This section reviews the various approaches to model compression [44]–[46], with a particular emphasis on pruning methods [47], [48], which have become a prominent area of research. Model compression
encompasses a range of strategies aimed at reducing the size and complexity of neural networks. These techniques can be broadly categorized into three main groups: quantization, low-rank factorization, and pruning.

## A. Quantization

Quantization reduces the precision of the weights and activations in a neural network, effectively decreasing the model size and computational requirements. The most common approach involves converting 32-bit floating-point numbers to lower precision formats, such as 8-bit integers. This transformation not only reduces memory usage but also accelerates inference on hardware optimized for low-precision arithmetic [49].

Recent advancements in quantization techniques include post-training quantization [50] and quantization-aware training [51]. Post-training quantization applies quantization to a pretrained model, which can significantly reduce the model size with minimal accuracy loss [50]. In contrast, quantization-aware training integrates quantization during the training process, allowing the model to learn to compensate for the effects of quantization, often resulting in better performance [51]. Furthermore, techniques such as mixed-precision training utilize different precision levels for different layers, maximizing efficiency while minimizing accuracy loss [52]. Despite its advantages, quantization can introduce quantization noise, leading to degradation in model performance if not carefully managed.

## B. Low-Rank Factorization

Low-rank factorization [53] is another widely used method for model compression, particularly effective for fully connected layers and convolutional filters. The central idea is to approximate a weight matrix with two or more lower-rank matrices, significantly reducing the number of parameters and computations required during inference.

Techniques such as Singular Value Decomposition (SVD) [13] are commonly employed for this purpose. By decomposing a weight matrix into its singular values, the most significant components can be retained while discarding smaller ones, leading to a compact representation of the model. Although low-rank factorization effectively reduces the size of the network, it requires careful tuning to ensure that the approximations do not lead to a significant drop in performance. Additionally, it may not be as effective for highly structured layers, such as those in convolutional networks, where the spatial correlation between weights needs to be preserved [12].

Tucker decomposition is another tensor decomposition technique widely used in neural network compression [12]–[14], [54]–[57]. In Tucker decomposition [45], a tensor (such as a convolutional kernel) is decomposed into a core tensor and several factor matrices that represent the mode-wise factors of the original tensor. By reducing the size of both the core tensor and the factor matrices, Tucker decomposition can lead to substantial reductions in parameter count and computational cost, particularly in deeper convolutional layers. Unlike SVD,

Tucker decomposition preserves multi-dimensional structures in tensors, making it highly effective for compressing convolutional filters where spatial and channel correlations are important.

Canonical Polyadic (CP) decomposition is another tensor factorization method used in deep network compression. In CP decomposition [14], [58], a tensor is factorized into a sum of rank-one tensors, which drastically reduces the number of parameters. CP decomposition is especially effective for very deep architectures where the compression rate must be high. However, it can suffer from approximation errors if the rank-one approximation fails to capture the complexity of the original tensor. While CP decomposition offers a high compression ratio, it may require additional optimization techniques to minimize performance loss in tasks like image classification.

## C. Knowledge Distillation

Knowledge distillation [15] is a model compression technique that involves transferring knowledge from a larger, pretrained model (the teacher) to a smaller model (the student) [57], [59]. The student model is trained to mimic the teacher's outputs, typically using a softened version of the teacher's class probabilities as labels. This allows the smaller model to capture essential features learned by the teacher, achieving competitive performance with significantly fewer parameters.

Knowledge distillation has been shown to be particularly effective for compressing large neural networks without compromising their accuracy. Various extensions of this technique have been explored, including multi-teacher distillation, where multiple teacher models are used to enhance the student model's robustness [60], and feature distillation, which focuses on transferring intermediate feature representations rather than just final outputs [61]. Despite its success, the training process for both the teacher and student models can be resource-intensive, requiring additional computational overhead [62].

## D. Neural Architecture Search (NAS)

Neural Architecture Search (NAS) [63] is a promising approach to automatically designing efficient neural network architectures that are optimized for both performance and resource constraints. NAS aims to automate the process of finding high-performing architectures by exploring a pre-defined search space, leveraging optimization algorithms to balance between accuracy, computational cost, and memory usage. The motivation behind NAS for model compression is to discover architectures that are inherently more efficient, avoiding the need for post-training compression techniques such as pruning or quantization.

Traditionally, NAS methods search for optimal architectures based on performance metrics like accuracy, but recent advancements have incorporated hardware-aware constraints to ensure that the discovered models are efficient in terms of FLOPs, latency, and memory. For instance, some NAS methods incorporate model size or FLOP count as part of the objective function during the search process [64]. This
enables NAS to find architectures that are not only accurate but also well-suited for deployment on resource-constrained devices such as mobile phones and embedded systems.

Several NAS approaches have been proposed for model compression, with notable examples including EfficientNet [65], which uses a compound scaling method to balance network width, depth, and resolution. EfficientNet achieves state-of-the-art performance on tasks like image classification, while significantly reducing parameters and FLOPs compared to manually designed architectures. It has also been successfully applied in transfer learning across domains such as NLP, satellite imagery, and medical image processing [66]–[69]. Another approach, ProxylessNAS [70], introduces a resource-aware NAS method that searches for architectures optimized directly on target hardware without requiring proxy tasks or separate hardware optimization stages. This direct optimization enables real-time consideration of model compression objectives like inference speed and memory usage during the NAS process.

Despite the success of NAS in generating efficient architectures, the search process can be computationally expensive, often requiring hundreds or thousands of GPU hours to converge on an optimal architecture. To mitigate this, techniques such as weight sharing [71] and one-shot NAS [72] have been developed to reduce the computational overhead by reusing weights across different candidate architectures, thus speeding up the search process.

Incorporating NAS into model compression workflows offers a flexible and scalable solution for designing neural networks that meet specific performance and efficiency goals. By automatically exploring large architectural spaces and jointly optimizing for both accuracy and resource usage, NAS can generate highly compact architectures that outperform traditional compression techniques in terms of both inference speed and energy consumption. As NAS continues to evolve, integrating it with other model compression techniques, such as pruning or quantization, holds promise for further advancing the efficiency and performance of deep learning models.

## E. Pruning Methods

Among the various model compression techniques, pruning [19] has emerged as a highly effective strategy to reduce the complexity of neural networks by removing less important weights, neurons, or entire channels. Pruning can be broadly categorized into two main types: unstructured pruning [73] and structured pruning [20].

1) *Unstructured Pruning*: Unstructured pruning involves removing individual weights based on certain criteria, typically their magnitudes. The most common method, magnitude-based pruning, selects weights with the smallest absolute values for removal, based on the assumption that smaller weights contribute less to the model's overall performance [73]. This approach can achieve significant reductions in the number of parameters, but the resulting sparse weight matrices may lead to irregular memory access patterns, which can hinder computational efficiency on standard hardware.

A variation of unstructured pruning includes dynamic pruning, where the pruning decision is made during the training process. This method allows for the identification and removal of unimportant weights iteratively, potentially leading to better performance than static pruning [74]. However, dynamic pruning requires careful management to ensure stability and convergence during training, adding to the complexity of the approach.

2) *Structured Pruning*: In contrast to unstructured pruning, structured pruning [20] removes entire neurons, filters, or channels, resulting in a more regular structure that is easier to optimize on modern hardware. Channel pruning, in particular, focuses on removing entire convolutional filters based on criteria that reflect their contribution to the model's performance. This method not only reduces the number of parameters but also leads to significant reductions in computational complexity, making it more suitable for deployment in resource-constrained environments.

Several strategies [16], [17], [75] have been proposed for channel pruning, including magnitude-based channel pruning [25], [76], where filters with the smallest summed weight magnitudes are removed. However, this approach often fails to consider the overall impact on model stability and generalization capabilities. Alternative methods, such as variational dropout [77] and Bayesian methods [78], provide probabilistic measures of filter importance, allowing for more informed pruning decisions.

In addition to these, similarity-based methods [79], [80] have also been explored, where filters with redundant or highly correlated feature maps are identified and pruned. For example, methods like network slimming [79] exploit the redundancy in filter activations by applying sparsity-inducing regularizers, ensuring that only a minimal set of unique filters is retained. This technique reduces model complexity while maintaining performance by eliminating filters that contribute little unique information [81], [82].

Decomposition-based pruning methods [45], such as those leveraging tensor decomposition, aim to decompose convolutional filters into smaller, low-rank matrices, thus reducing the computational burden. Techniques like CP-decomposition [14] and Tucker decomposition [45] have been widely used to approximate the weight tensors of convolutional layers, significantly lowering the number of parameters and floating-point operations (FLOPs) while preserving model accuracy.

Hybrid methods combine the strengths of multiple pruning strategies to achieve better compression. For instance, some approaches use a combination of magnitude-based pruning and low-rank decomposition to prune less important filters while simultaneously compressing the remaining ones. These hybrid methods allow for a more comprehensive approach to network compression by balancing parameter reduction, model stability, and generalization capabilities [83], [84].

Despite the success of these approaches, many existing pruning methods still rely on heuristic-based criteria that do not fully account for the underlying stability and robustness of the network. This can lead to suboptimal pruning outcomes,
particularly in tasks where maintaining generalization is critical.

The spectral norm of a weight matrix, defined as its largest singular value, has recently emerged as a promising criterion for model evaluation and stability [27]. It has been utilized to control the Lipschitz constant of neural networks, thereby enhancing adversarial robustness and improving generalization [28]. However, its application in guiding pruning decisions has not been extensively explored. By leveraging spectral norm as a pruning criterion, researchers can better identify channels that contribute significantly to model stability, providing a more principled approach to channel pruning. This novel perspective on pruning will be explored further in this paper, as we present our spectral norm-based pruning framework and evaluate its performance against traditional pruning methods.

### III. PROPOSED METHOD

In this section, we present our novel channel pruning framework that leverages spectral norm as a criterion for pruning decisions. Our approach is designed to optimize deep neural networks by effectively identifying and removing less significant channels based on their contributions to the model's overall stability and generalization capabilities. We begin by providing a foundational understanding of spectral norm and its properties, followed by an overview of convolutional neural networks (CNNs), and subsequently, we describe how spectral norm can be applied to the compression of CNNs. Finally, we outline our overall pruning algorithm for the entire network.

#### A. Spectral Norm in Linear Algebra

In linear algebra, understanding the properties of matrices is fundamental to analyzing their behavior in various applications, particularly in machine learning and neural networks. One critical aspect of matrix analysis involves the concept of Singular Value Decomposition (SVD), which facilitates the computation of the spectral norm—a key measure of a matrix's magnitude and stability.

**1) Singular Value Decomposition (SVD):** The Singular Value Decomposition (SVD) is a factorization technique that expresses a matrix in terms of its singular values and singular vectors. For any matrix $A \in \mathbb{R}^{m \times n}$, SVD allows us to represent A as:

$$A = U \Sigma V^*, \quad (1)$$

where:

* $U \in \mathbb{R}^{m \times m}$ is an orthogonal matrix whose columns are the left singular vectors of A.

* $\Sigma \in \mathbb{R}^{m \times n}$ is a diagonal matrix containing the singular values $\sigma_1, \sigma_2, \dots, \sigma_r$ (where r is the rank of A), sorted in non-increasing order, i.e., $\sigma_1 \ge \sigma_2 \ge \dots \ge \sigma_r \ge 0$.

* $V^* \in \mathbb{R}^{n \times n}$ is an orthogonal matrix whose columns are the right singular vectors of A.

The singular values $\sigma_i$ reflect the intrinsic geometric properties of the matrix and indicate how much the matrix stretches or compresses vectors in various directions. The largest singular value, $\sigma_{\max}(A)$, provides a measure of the maximum stretching effect that the matrix can exert on any unit vector.

**2) Spectral Norm:** The spectral norm of a matrix is defined as the largest singular value, representing the maximum amount by which the matrix can stretch a vector. Mathematically, the spectral norm $\|A\|_2$ can be expressed as:

$$\|A\|_2 = \sigma_{\max}(A) = \sqrt{\lambda_{\max}(A^*A)}, \quad (2)$$

where:

* $\lambda_{\max}(A^*A)$ is the largest eigenvalue of the matrix $A^*A$ (the conjugate transpose of A multiplied by A).

* $\sigma_{\max}(A)$ denotes the largest singular value derived from the SVD of A.

The spectral norm serves as an important measure of the matrix's sensitivity to perturbations. Specifically, it is interpreted as the maximum possible amplification of any input vector's magnitude when transformed by the matrix. This property is particularly useful in assessing the stability of linear systems, as it quantifies how changes in input can affect output.

The spectral norm can also be characterized as follows:

$$\|A\|_2 = \sup\{x^*Ay : x \in K^m, y \in K^n \text{ with } \|x\|_2 = \|y\|_2 = 1\}. \quad (3)$$

This expression indicates that the spectral norm is the supremum of the quadratic form $x^*Ay$, where x and y are unit vectors. This relationship underscores the spectral norm's significance in capturing the overall influence of the matrix on input vectors.

Several essential properties of the spectral norm further reinforce its utility:

**Self-Relation:** The spectral norm has a noteworthy relationship with the matrix products of A:

$$\|A^*A\|_2 = \|A\|_2^2 = \|AA^*\|_2.$$

This property, derived from the SVD, indicates that the spectral norm of the product of a matrix and its conjugate transpose reflects the norm of the matrix itself.

**Comparison with Frobenius Norm:** The spectral norm is bounded by the Frobenius norm $\|A\|_F$, defined as:

$$\|A\|_F = \sqrt{\sum_i \sigma_i(A)^2},$$

where $\sigma_i(A)$ are the singular values of A. The spectral norm provides a measure of the maximum singular value, while the Frobenius norm reflects the overall magnitude of all singular values. The relation is given by:

$$\|A\|_2 \le \|A\|_F,$$

with equality holding if and only if A is a rank-one matrix or the zero matrix.

**Relationship to Spectral Radius:** The spectral norm is also connected to the spectral radius of $A^*A$:

$$\|A\|_2 = \sqrt{\rho(A^*A)} \le \sqrt{\|A^*A\|_\infty} \le \sqrt{\|A\|_1 \|A\|_\infty}.$$

This relationship provides insights into the bounds of the spectral norm in relation to other matrix norms.

The spectral norm serves as a vital measure for evaluating the stability, robustness, and overall performance of linear transformations due to several critical characteristics:

**Stability and Sensitivity:** The spectral norm quantifies how sensitive a linear transformation (represented by a matrix)
is to perturbations in its input. Specifically, it indicates the maximum amplification that any input vector can experience when transformed by the matrix. This is crucial in neural networks, where small changes in input data can significantly affect the output. A network with a low spectral norm may demonstrate greater stability, as it suggests that the model's predictions are less sensitive to variations in input, which is desirable for robust performance in real-world applications.

**Capacity and Complexity:** In the context of neural networks, the spectral norm provides insights into the model's capacity. A matrix with a high spectral norm can represent more complex functions and exhibit a higher degree of expressiveness. Conversely, if certain layers or channels exhibit low spectral norms, it suggests that they contribute less to the model's overall capacity. This information can guide decisions about which channels to prune, focusing on those that are less critical for maintaining the network's performance.

**Generalization and Overfitting:** Models with a lower spectral norm are often associated with improved generalization capabilities. This is particularly important in deep learning, where models trained on limited datasets may overfit, memorizing training examples rather than learning generalizable features. By utilizing spectral norm as a pruning criterion, one can identify and remove channels that contribute to overfitting, leading to models that are not only more compact but also potentially more generalizable.

In conclusion, the spectral norm serves as a powerful tool in evaluating the characteristics of linear transformations within neural networks. By providing insights into stability, capacity, and generalization, it lays a strong foundation for informed decisions in channel pruning. This approach not only optimizes the model's efficiency but also enhances its robustness and adaptability in diverse applications.

## B. Convolutional Neural Networks (CNNs)

Convolutional Neural Networks (CNNs) [85] are a specialized class of deep learning models designed to process data with a grid-like topology, most notably image data. CNNs have revolutionized various computer vision tasks, such as image classification, object detection, and image segmentation, due to their ability to effectively capture spatial hierarchies and local patterns within images.

1) **CNN Architecture:** A typical CNN architecture consists of several key components that work together to extract hierarchical features from the input data:

1) **Convolutional Layers:** These layers are the core building blocks of CNNs. Each convolutional layer applies a set of learnable filters (also known as kernels) to the input feature map. Each filter is designed to detect specific patterns, such as edges, textures, or more complex features, at various spatial resolutions. The output of a convolutional layer can be viewed as a new feature map that highlights the presence of learned patterns.

2) **Activation Functions:** After the convolution operation, the output is typically passed through a non-linear activation function, such as the Rectified Linear Unit (ReLU).

This non-linearity allows the network to learn complex representations. Mathematically, the activation function $f$ introduces non-linearities that help the model learn intricate mappings from inputs to outputs.

3) **Pooling Layers:** Pooling layers are employed to downsample the feature maps, reducing their spatial dimensions while retaining the most important information. Common pooling operations include max pooling and average pooling, which help to reduce the computational load and mitigate the risk of overfitting by introducing translational invariance.

4) **Fully Connected Layers:** At the end of the network, fully connected layers serve to combine the features extracted by the previous layers and make the final prediction. These layers take the flattened output from the preceding pooling layer and produce the final output probabilities for each class.

2) **Mathematical Representation:** Mathematically, the output $y$ of a convolutional layer can be expressed as:

$$y = f(W * x + b), \quad (4)$$

where:

* $W$ represents the convolutional weights (filters) of the layer.

* $x$ is the input feature map, typically the output from the previous layer or the original input image.

* $b$ is the bias term, which is added to the output to enable the model to fit the training data more flexibly.

* $f$ denotes the activation function, introducing non-linearity into the network.

The convolution operation $W * x$ effectively slides the filter $W$ across the input $x$ to produce the output $y$, capturing spatial patterns in the input data.

3) **Weights and Channels in CNNs:** In CNNs, the convolutional filters, or weights, are organized into multiple channels corresponding to different feature maps. For a given convolutional layer with $C_{in}$ input channels and $C_{out}$ output channels, the weight matrix $W$ has a size of $(K_h \times K_w \times C_{in}) \times C_{out}$, where $K_h$ and $K_w$ are the height and width of the convolutional filter.

* **Input Channels:** Each input image can have multiple channels (e.g., RGB images have three channels corresponding to red, green, and blue). The filters in the first convolutional layer learn to detect patterns in each of these channels, and the combination of these patterns forms the basis for higher-level features in subsequent layers.

* **Output Channels:** The output of each convolutional layer is a set of feature maps, one for each filter (output channel). Each feature map captures different aspects of the input image, allowing the CNN to learn complex representations of the data.

4) **Implications for Model Complexity:** While the depth and breadth of CNN architectures contribute significantly to their capacity for learning complex feature representations, they also lead to an exponential increase in the number of parameters.
ters. This increase in parameters directly results in higher computational requirements and memory consumption, which can be prohibitive in scenarios where resources are limited, such as in mobile devices or real-time applications. Specifically, as the number of filters and layers increases, the number of learnable weights grows substantially, leading to longer training times and increased memory usage. Consequently, there is a pressing need for techniques that can effectively compress these models without sacrificing their performance, making methods like channel pruning particularly valuable.

### C. Calculating Spectral Norm of Output Channels

To assess the importance of output channels in a Convolutional Neural Network (CNN), we utilize the spectral norm of the weight matrices corresponding to the channels. The spectral norm serves as a measure of the capacity and sensitivity of a channel to input perturbations, making it a suitable criterion for determining channel importance during pruning.

Given a convolutional layer with a weight matrix $W \in \mathbb{R}^{K_h \times K_w \times C_{in} \times C_{out}}$, where $K_h$ and $K_w$ are the height and width of the convolutional kernel, $C_{in}$ is the number of input channels, and $C_{out}$ is the number of output channels, the spectral norm of each output channel can be computed as follows:

1. Weight Matrix Extraction: For each output channel $c$ (where $c = 1, 2, \dots, C_{out}$), we extract the corresponding weight matrix $W_c \in \mathbb{R}^{K_h \times K_w \times C_{in}}$.

2. Computation of the Spectral Norm: The spectral norm $||W_c||_2$ is computed as the largest singular value of the matrix $W_c$. This can be done using singular value decomposition (SVD), where:

$$W_c = U \Sigma V^T,$$

where $U$ and $V$ are orthogonal matrices, and $\Sigma$ is a diagonal matrix containing the singular values. The spectral norm is then given by:

$$||W_c||_2 = \sigma_{\max}(W_c) = \sqrt{\lambda_{\max}(W_c^* W_c)},$$

where $\lambda_{\max}$ denotes the largest eigenvalue of the matrix $W_c^* W_c$.

3. Channel Importance Ranking: The spectral norm values for all output channels are computed and ranked in ascending order. Channels with lower spectral norms are considered less important, as they contribute less to the stability and generalization of the network.

By using the spectral norm as a measure of channel importance, we can make informed pruning decisions, focusing on the channels that have the least impact on the model's performance.

### D. Iterative Channel Pruning Algorithm Based on Spectral Norm

We propose an iterative channel pruning algorithm that leverages the spectral norm to prune less important channels

from the network. The steps of the algorithm are outlined below:

#### Algorithm 1 Iterative Channel Pruning Algorithm Based on Spectral Norm

1. **Input:** Pre-trained CNN model $M$, pruning ratio $r$, number of iterations $T$

2. **Output:** Pruned CNN model $M'$

3. **for** each convolutional layer $L$ in $M$ **do**

4. Initialize pruning mask $M_L$ ← ones($C_{out}$) (indicating all channels are kept)

5. **end for**

6. **for** iteration $t = 1$ to $T$ **do**

7. **for** each convolutional layer $L$ in $M$ **do**

8. Extract weight matrix $W_L$ corresponding to layer $L$

9. Calculate spectral norms for output channels $||W_{L,c}||_2$ for $c = 1, 2, \dots, C_{out}$

10. Rank channels based on spectral norm values

11. Determine number of channels to prune: $C_{prune} = [r \cdot C_{out}]$

12. Identify channels to prune based on lowest spectral norms

13. Update pruning mask: $M_L[c] = 0$ for pruned channels

14. **end for**

15. Fine-tune the pruned model $M'$ using a small learning rate to recover performance

16. **end for**

17. **Return** Pruned model $M'$ with updated pruning masks

The proposed iterative channel pruning algorithm based on the spectral norm is designed to effectively reduce the number of output channels in a Convolutional Neural Network (CNN) while maintaining the model's performance. The algorithm begins with three input parameters: a pre-trained CNN model $M$, a pruning ratio $r$ that specifies the proportion of channels to prune, and the number of iterations $T$. For each convolutional layer in the model, a pruning mask $M_L$ is initialized to retain all output channels, represented as an array of ones. During each iteration $t$, the algorithm computes the spectral norm of each output channel in every convolutional layer $L$ by extracting the weight matrix $W_L$ and calculating the spectral norms $||W_{L,c}||_2$. Channels are then ranked based on their spectral norm values, with lower norms indicating less importance to the model's stability and performance. The number of channels to prune, $C_{prune}$, is determined by multiplying the pruning ratio $r$ with the total number of output channels $C_{out}$ and rounding up to the nearest integer. The channels with the lowest spectral norms are selected for pruning, and the pruning mask $M_L$ is updated accordingly to set the values of the pruned channels to zero. After the pruning process in each iteration, the model $M'$ undergoes fine-tuning with a small learning rate to recover any potential performance loss from channel removal. This iterative process enables the algorithm to achieve a more efficient model that
maintains competitive accuracy while effectively balancing
model compression and performance preservation.

IV. EXPERIMENTS AND ANALYSIS

The effectiveness of the proposed channel pruning method using spectral norm is evaluated comprehensively across three key dimensions: model accuracy, computational complexity measured in Floating Point Operations (FLOPs), and the number of parameters. These metrics serve as the foundation for assessing the trade-offs between model performance and efficiency. FLOPs provide an estimate of the computational cost required for inference, while the number of parameters reflects the storage and memory demands of the network. Accuracy is the primary indicator of how well the pruned models generalize to unseen data after compression. For CIFAR-10, top-1 accuracy is reported, while for ImageNet, both top-1 and top-5 accuracy metrics are considered due to the complexity and scale of the dataset.

Tables I and II summarize the results of our experiments. Each table presents the accuracy, number of parameters, and FLOPs for the baseline and pruned models. Additionally, the percentage reduction in parameters and FLOPs is highlighted, providing insights into the efficiency gains of each pruning method. The baseline model in each case is the unpruned architecture (VGG-16-BN [86] for CIFAR-10 [30] and ResNet-50 [3] for ImageNet [26]). The pruned models, including those generated by our spectral norm approach, are compared against a variety of state-of-the-art pruning methods, such as HRank-1 [31], L1 [25], SSS [87], and others.

The experiments were conducted on a machine equipped
with an NVIDIA GTX 2080Ti GPU, using the PyTorch
deep learning framework [88]. For optimization, we employed
Stochastic Gradient Descent (SGD) [89] with specific hyper-
parameters tailored for fine-tuning after pruning. The CIFAR-
10 experiments involved fine-tuning for 400 epochs, using a
batch size of 256. The momentum was set to 0.9, weight
decay to 0.05, and the initial learning rate to 0.01. These
settings ensure that the model has sufficient capacity to recover
accuracy after the pruning process, as the network undergoes
structural changes due to channel removal. The learning rate
was decayed progressively throughout the training process to
stabilize convergence.

For ImageNet, fine-tuning was performed for 200 epochs with a larger batch size of 512 to handle the increased complexity and scale of the dataset. The momentum was slightly increased to 0.99, weight decay was set to 0.0001 to avoid overfitting, and the initial learning rate was set to 0.1. The learning rate schedule employed a step-wise decay strategy, gradually reducing the learning rate as training progressed. This allowed the pruned model to steadily regain performance, particularly in the later stages of fine-tuning when the model's weights had largely converged.

One of the primary objectives of these experiments was to
ensure a fair comparison between different pruning strategies.
For each method, we aimed to either match or exceed the base-
line model's accuracy while achieving significant reductions

in both parameters and FLOPs. In some cases, we prioritized
reducing the computational and storage overhead by matching
the level of parameter and FLOP reduction seen in other meth-
ods. In other cases, we focused on maintaining or improving
the model's accuracy relative to other pruning approaches,
even if the reduction in parameters and FLOPs was somewhat
lower. This balance is crucial in evaluating the practical impact
of pruning, as excessive compression can degrade the model's
ability to generalize, while insufficient compression may not
provide meaningful computational savings.

Our spectral norm-based pruning method proved highly
effective in this context. By leveraging the spectral norm as
a criterion for channel importance, we were able to identify
and prune less important channels across multiple layers of the
network, leading to a significant reduction in parameters and
FLOPs. As shown in Tables I and II, our approach consistently
achieved a higher compression rate compared to several other
techniques while maintaining or even improving accuracy.
This demonstrates the potential of spectral norm-based pruning
as a robust and scalable solution for model compression in
real-world applications, especially when deployed on large-
scale datasets such as ImageNet.

A. Analysis of Results on VGG-16 with CIFAR-10

Table I summarizes the pruning performance of various methods on VGG-16-BN using the CIFAR-10 dataset. The table presents the Top-1 accuracy, the number of parameters reduced, and the FLOPs reduction for each pruning method compared to the baseline, unpruned VGG-16-BN model. The results demonstrate the effectiveness of our proposed method, Spectral Norm-based pruning, in achieving a high compression ratio while maintaining superior accuracy. Our method, based on spectral norm, achieves a Top-1 accuracy of 94.05%, which is the highest among all methods, surpassing both traditional approaches and recent state-of-the-art methods. Notably, it maintains a significantly lower number of parameters and FLOPs, reducing the number of parameters by 83.2% and FLOPs by 53.6%. These results are comparable to HRank-1 [31], which also achieves an 83.2% reduction in parameters and 53.6% reduction in FLOPs, yet the Spectral Norm method slightly outperforms it in accuracy (94.05% vs. 93.43%). Compared to other magnitude-based pruning methods such as L1 [25], which reduces parameters by 64.0% and FLOPs by 34.3%, the spectral norm method achieves significantly better compression and a higher accuracy (94.05% vs. 93.40%). Similarly, SSS [87] and GAL-0.05 [90] show lower parameter and FLOPs reduction with a more pronounced drop in accuracy (93.02% and 92.03%, respectively). Among the recent methods, CHIP [91] achieves a similar reduction in parameters (81.6%) and FLOPs (58.1%), with a slightly lower accuracy of 93.86%. Although this performance is strong, our method surpasses CHIP in both accuracy and efficiency. The closest competitor in terms of accuracy is DECORE-500 [92], which achieves a Top-1 accuracy of 94.02%. However, DECORE-500 prunes fewer parameters (63.0%) and FLOPs (35.3%) compared to our spectral norm method, making our approach
<table><caption>TABLE I: Pruning results of VGGNet on CIFAR-10. PR and FR refer to Parameter Reduction and FLOPs Reduction, respectively.</caption><thead><tr><th>Model</th><th>Accuracy (%)</th><th>Compression (%)</th><th>Speedup (%)</th></tr></thead><tbody><tr><td>VGG-16-BN</td><td>93.96</td><td>00.0</td><td>00.0</td></tr><tr><td>HRank-1 [31]</td><td>93.43</td><td>83.2</td><td>53.6</td></tr><tr><td>L1 [25]</td><td>93.40</td><td>64.0</td><td>34.3</td></tr><tr><td>SSS [87]</td><td>93.02</td><td>73.8</td><td>41.6</td></tr><tr><td>GAL-0.05 [90]</td><td>92.03</td><td>77.6</td><td>39.6</td></tr><tr><td>VAS [77]</td><td>93.18</td><td>73.3</td><td>39.1</td></tr><tr><td>CHIP [91]</td><td>93.86</td><td>81.6</td><td>58.1</td></tr><tr><td>EZCrop [93]</td><td>93.01</td><td>81.6</td><td>58.1</td></tr><tr><td>DECORE-500 [92]</td><td>94.02</td><td>63.0</td><td>35.3</td></tr><tr><td>Proposed Method</td><td>94.05</td><td>83.2</td><td>53.6</td></tr></tbody></table>

more efficient in terms of compression. Overall, the Spectral Norm method demonstrates a compelling balance between high accuracy and model compression. By utilizing spectral norm to guide pruning decisions, it significantly reduces model complexity while ensuring minimal performance degradation. These results highlight the efficacy of our approach in compressing deep neural networks, particularly in comparison to both traditional and state-of-the-art pruning techniques.

## B. Analysis of Results on ResNet-50 with ImageNet

ResNet-50 is a widely-used convolutional neural network (CNN) architecture known for its innovative use of residual connections, which help mitigate the vanishing gradient problem and enable the training of very deep networks. ResNet-50 consists of 50 layers and has 25.50 million parameters in its full version. The network's depth allows it to learn complex feature hierarchies, making it highly effective on large-scale datasets such as ImageNet. ImageNet is one of the most significant and widely referenced datasets in computer vision, containing over 1.2 million training images across 1,000 object classes. The scale of ImageNet introduces substantial computational demands for both training and inference, with the baseline ResNet-50 model requiring 4.09 billion floating-point operations (FLOPs) per forward pass. While deep networks like ResNet-50 perform well on tasks such as image classification, their large number of parameters and high computational cost pose challenges for deployment on resource-constrained devices, such as mobile phones or embedded systems. Thus, compressing ResNet-50 while maintaining high accuracy is crucial for real-world applications. Table II compares the performance of various pruning methods applied to ResNet-50 on ImageNet, focusing on Top-1 and Top-5 accuracy, as well as the percentage reduction in parameters and FLOPs. The baseline ResNet-50 achieves a Top-1 accuracy of 76.15% and a Top-5 accuracy of 92.87%, serving as the reference point for evaluating the effects of pruning. Our proposed Spectral Norm-based pruning method reduces the number of parameters by 46.0% and FLOPs by 48.7%, resulting in a Top-1 accuracy of 76.05% and a Top-5 accuracy of 92.84%. Despite the significant reduction in both model size and computational complexity, the accuracy is nearly identical to the unpruned model, with only a slight drop of 0.10% in Top-1 accuracy and

<table><caption>TABLE II: Pruning results of ResNet-50 on ImageNet.</caption><thead><tr><th>Model</th><th>Top-1(%)</th><th>Top-5(%)</th><th>Compression (%)</th><th>Speedup (%)</th></tr></thead><tbody><tr><td>ResNet-50</td><td>76.15</td><td>92.87</td><td>00.0</td><td>00.0</td></tr><tr><td>EZCrop [93]</td><td>75.68</td><td>92.70</td><td>40.9</td><td>45.0</td></tr><tr><td>HRank-1 [31]</td><td>74.98</td><td>92.33</td><td>36.7</td><td>43.8</td></tr><tr><td>PFP [94]</td><td>75.21</td><td>92.43</td><td>30.1</td><td>44.0</td></tr><tr><td>SCOP [78]</td><td>75.95</td><td>92.79</td><td>42.8</td><td>45.3</td></tr><tr><td>CHIP [91]</td><td>76.15</td><td>92.91</td><td>44.2</td><td>48.7</td></tr><tr><td>DECORE-6 [92]</td><td>74.58</td><td>92.18</td><td>44.7</td><td>42.3</td></tr><tr><td>Proposed Method</td><td>76.05</td><td>92.84</td><td>46.0</td><td>48.7</td></tr></tbody></table>

0.03% in Top-5 accuracy. This demonstrates that the Spectral Norm method effectively preserves model performance even under aggressive pruning, making it highly suitable for scenarios where computational resources are limited. In comparison to other pruning methods, the Spectral Norm approach exhibits several advantages. HRank-1 [31], for example, reduces parameters by 36.7% and FLOPs by 43.8%, achieving a Top-1 accuracy of 74.98%. While HRank-1 achieves a substantial reduction in both parameters and FLOPs, its Top-1 accuracy is 1.07% lower than that of Spectral Norm, highlighting the superior ability of spectral norm-based pruning to retain accuracy while offering more compression. DECORE-6 [92] reduces parameters by 44.7% and FLOPs by 42.3%, with a Top-1 accuracy of 74.58%. Although DECORE-6 achieves comparable parameter and FLOP reductions to Spectral Norm, its accuracy drop of 1.47% compared to the baseline is significantly higher. This reinforces the effectiveness of Spectral Norm pruning in preserving accuracy, especially when compared to pruning methods that focus purely on magnitude-based criteria. Several other methods, such as PFP [94], EZCrop [93], and SCOP [78], demonstrate varying levels of compression and accuracy retention. For instance, SCOP reduces parameters by 42.8% and FLOPs by 45.3%, with a Top-1 accuracy of 75.95%. While SCOP provides a reasonable balance between compression and performance, it still exhibits a 0.10% lower accuracy than the Spectral Norm method, despite slightly lower reductions in parameters and FLOPs. On the other hand, CHIP [91] achieves the same FLOP reduction as Spectral Norm (48.7%), with a parameter reduction of 44.2%, while maintaining the original Top-1 accuracy of 76.15%. However, the Spectral Norm method offers a higher parameter reduction (46.0%) with a minimal drop in accuracy, making it a competitive choice. In conclusion, the Spectral Norm method proves to be highly effective for compressing ResNet-50 on ImageNet. By leveraging the spectral norm to guide channel pruning, it achieves one of the highest compression rates while maintaining accuracy levels comparable to or exceeding those of other state-of-the-art pruning techniques. This makes it an attractive approach for large-scale datasets and models where both accuracy and computational efficiency are crucial factors.

## V. CONCLUSION AND FUTURE WORKS

This study presents an innovative channel pruning technique leveraging the spectral norm of convolutional filters. By utilizing the spectral norm to assess channel significance,
our method achieves substantial reductions in model parame-
ters and computational overhead, particularly in convolutional
neural network (CNN) architectures such as VGGNet and
ResNet. The approach delivers state-of-the-art compression
results on the CIFAR-10 and ImageNet datasets, reducing both
floating-point operations (FLOPs) and parameter counts while
maintaining or even enhancing model accuracy.

Our extensive experiments reveal that the spectral norm
provides a robust metric for identifying and removing less
critical channels, outperforming traditional norm-based prun-
ing strategies, such as the L₁ norm. The results underscore
that our method not only preserves high accuracy but also
achieves significant resource efficiency, making it well-suited
for deployment in constrained environments, including mobile
and embedded systems.

Despite its effectiveness, there are areas for future improve-
ment. A notable limitation lies in the computational overhead
associated with calculating the spectral norm during pruning.
Optimizing this step could enhance the efficiency of the prun-
ing process, particularly for large-scale models and datasets.
Additionally, exploring dynamic pruning strategies—where
thresholds adapt based on layer complexity or task require-
ments—could further refine the resulting architectures.

While this work focuses on CNNs for image classification,
the potential applications of spectral norm-based pruning ex-
tend far beyond. Future research could explore its utility in
other model types, such as Transformers and Graph Neural
Networks (GNNs), as well as in tasks like object detection,
semantic segmentation, and natural language processing. An-
other promising direction is combining spectral norm-based
pruning with complementary techniques like quantization or
knowledge distillation to maximize model efficiency.

In summary, the spectral norm-based channel pruning
framework offers a compelling solution for deep network
compression. Its ability to achieve substantial computational
savings without sacrificing accuracy represents a meaningful
advancement in the field. With further optimization and ex-
ploration of new applications, this method has the potential to
drive significant progress in deep learning model compression.

REFERENCES

[1] Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning," Nature, vol. 521, no. 7553, pp. 436-444, 2015.

[2] A. Krizhevsky, I. Sutskever, and G. E. Hinton, "Imagenet classification with deep convolutional neural networks," *Advances in neural information processing systems*, vol. 25, 2012.

[3] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in *Proceedings of the IEEE conference on computer vision and pattern recognition*, 2016, pp. 770–778.

[4] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention is all you need," in *Proceedings of the 31st International Conference on Neural Information Processing Systems*, ser. NIPS'17. Red Hook, NY, USA: Curran Associates Inc., 2017, p. 6000–6010.

[5] K. He, G. Gkioxari, P. Dollár, and R. Girshick, "Mask r-cnn," *IEEE Transactions on Pattern Analysis and Machine Intelligence*, vol. 42, no. 2, pp. 386–397, 2020.

[6] G. Menghani, "Efficient deep learning: A survey on making deep learning models smaller, faster, and better," *ACM Computing Surveys*, vol. 55, no. 12, pp. 1–37, 2023.

[7] L. Deng, G. Li, S. Han, L. Shi, and Y. Xie, "Model compression and hardware acceleration for neural networks: A comprehensive survey," *Proceedings of the IEEE*, vol. 108, no. 4, pp. 485–532, 2020.

[8] S. Han, H. Mao, and W. J. Dally, "Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding," in *International Conference on Learning Representations*, 2016.

[9] Y. Xu, Y. Li, S. Zhang, W. Wen, B. Wang, Y. Qi, Y. Chen, W. Lin, and H. Xiong, "Trp: Trained rank pruning for efficient deep neural networks," in *Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20*, 2020, pp. 977–983.

[10] Z. Cheng, B. Li, Y. Fan, and Y. Bao, "A novel rank selection scheme in tensor ring decomposition based on reinforcement learning for deep neural networks," in *ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, 2020, pp. 3292–3296.

[11] S. Li, E. Hanson, H. Li, and Y. Chen, "Penni: Pruned kernel sharing for efficient cnn inference," in *International Conference on Machine Learning*. PMLR, 2020, pp. 5863–5873.

[12] X. Zhang, J. Zou, K. He, and J. Sun, "Accelerating very deep convolutional networks for classification and detection," *IEEE Transactions on Pattern Analysis and Machine Intelligence*, vol. 38, no. 10, pp. 1943–1955, 2016.

[13] E. L. Denton, W. Zaremba, J. Bruna, Y. LeCun, and R. Fergus, "Exploiting linear structure within convolutional networks for efficient evaluation," *Advances in neural information processing systems*, vol. 27, 2014.

[14] V. Lebedev, Y. Ganin, M. Rakhuba, I. V. Oseledets, and V. S. Lempit- sky, "Speeding-up convolutional neural networks using fine-tuned cp- decomposition," in *International Conference on Learning Representa- tions*, 2015.

[15] J. Gou, B. Yu, S. J. Maybank, and D. Tao, "Knowledge distillation: A survey," *International Journal of Computer Vision*, vol. 129, no. 6, pp. 1789–1819, 2021.

[16] X. Lu, W. Dong, X. Li, J. Wu, L. Li, and G. Shi, "Adaptive search-and-training for robust and efficient network pruning," *IEEE Transactions on Pattern Analysis and Machine Intelligence*, vol. 45, no. 8, pp. 9325–9338, 2023.

[17] Y. Liu, J. Cao, B. Li, W. Hu, and S. Maybank, "Learning to explore distillability and sparsability: A joint framework for model compression," *IEEE Transactions on Pattern Analysis and Machine Intelligence*, vol. 45, no. 3, pp. 3378–3395, 2023.

[18] H. Cheng, M. Zhang, and J. Q. Shi, "A survey on deep neural network pruning: Taxonomy, comparison, analysis, and recommendations," *IEEE Transactions on Pattern Analysis and Machine Intelligence*, 2024.

[19] Y. LeCun, J. Denker, and S. Solla, "Optimal brain damage," *Advances in neural information processing systems*, vol. 2, 1989.

[20] Y. He and L. Xiao, "Structured pruning for deep convolutional neural networks: A survey," *IEEE Transactions on Pattern Analysis and Machine Intelligence*, vol. 46, no. 5, pp. 2900–2919, 2024.

[21] Y. Sui, M. Yin, Y. Gong, and B. Yuan, "Co-exploring structured sparsification and low-rank tensor decomposition for compact dnnss," *IEEE Transactions on Neural Networks and Learning Systems*, 2024.

[22] Y. Guo, W. Gao, and G. Li, "Interpretable task-inspired adaptive filter pruning for neural networks under multiple constraints," *International Journal of Computer Vision*, pp. 1–17, 2024.

[23] M. Lin, B. Chen, F. Chao, and R. Ji, "Training compact cnns for image classification using dynamic-coded filter fusion," *IEEE Transactions on Pattern Analysis and Machine Intelligence*, vol. 45, no. 8, pp. 10478–10487, 2023.

[24] Y. Fu, C. Liu, D. Li, Z. Zhong, X. Sun, J. Zeng, and Y. Yao, "Exploring structural sparsity of deep networks via inverse scale spaces," *IEEE Transactions on Pattern Analysis and Machine Intelligence*, vol. 45, no. 2, pp. 1749–1765, 2023.

[25] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf, "Pruning filters for efficient convnets," *International Conference on Learning Representations*, 2016.

[26] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein et al., "Imagenet large scale visual recognition challenge," *International journal of computer vision*, vol. 115, pp. 211–252, 2015.

[27] R. Mathias, "The spectral norm of a nonnegative matrix," *Linear algebra and its applications*, vol. 139, pp. 269–284, 1990.
[28] M. Fazlyab, A. Robey, H. Hassani, M. Morari, and G. Pappas, "Efficient and accurate estimation of lipschitz constants for deep neural networks," *Advances in neural information processing systems*, vol. 32, 2019.

[29] S. Ye, K. Xu, S. Liu, H. Cheng, J.-H. Lambrechts, H. Zhang, A. Zhou, K. Ma, Y. Wang, and X. Lin, "Adversarial robustness vs. model compression, or both?" in *Proceedings of the IEEE/CVF International Conference on Computer Vision*, 2019, pp. 111–120.

[30] A. Krizhevsky and G. Hinton, "Learning multiple layers of features from tiny images," University of Toronto, Tech. Rep. 0, 2009.

[31] M. Lin, R. Ji, Y. Wang, Y. Zhang, B. Zhang, Y. Tian, and L. Shao, "Hrank: Filter pruning using high-rank feature map," in *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, 2020, pp. 1529–1538.

[32] S. Pouyanfar, S. Sadiq, Y. Yan, H. Tian, Y. Tao, M. P. Reyes, M.-L. Shyu, S.-C. Chen, and S. S. Iyengar, "A survey on deep learning: Algorithms, techniques, and applications," *ACM computing surveys (CSUR)*, vol. 51, no. 5, pp. 1–36, 2018.

[33] A. Voulodimos, N. Doulamis, A. Doulamis, and E. Protopapadakis, "Deep learning for computer vision: A brief review," *Computational intelligence and neuroscience*, vol. 2018, no. 1, p. 7068349, 2018.

[34] H. You, Y. Feng, R. Ji, and Y. Gao, "Pvnet: A joint convolutional network of point cloud and multi-view for 3d shape recognition," *Proceedings of the 26th ACM international conference on Multimedia*, 2018. [Online]. Available: https://api.semanticscholar.org/CorpusID:52077468

[35] U. Kamath, J. Liu, and J. Whitaker, *Deep learning for NLP and speech recognition*. Springer, 2019, vol. 84.

[36] K. Arulkumaran, M. P. Deisenroth, M. Brundage, and A. A. Bharath, "Deep reinforcement learning: A brief survey," *IEEE Signal Processing Magazine*, vol. 34, no. 6, pp. 26–38, 2017.

[37] D. Xiong, D. Zhang, X. Zhao, and Y. Zhao, "Deep learning for emg-based human-machine interaction: A review," *IEEE/CAA Journal of Automatica Sinica*, vol. 8, no. 3, pp. 512–533, 2021.

[38] Y. Luo, J. Ji, X. Sun, L. Cao, Y. Wu, F. Huang, C.-W. Lin, and R. Ji, "Dual-level collaborative transformer for image captioning," *ArXiv*, vol. abs/2101.06462, 2021. [Online]. Available: https://api.semanticscholar.org/CorpusID:231632752

[39] D. Cao, R. Ji, D. Lin, and S. Li, "A cross-media public sentiment analysis system for microblog," *Multimedia Systems*, vol. 22, pp. 479–486, 2016. [Online]. Available: https://api.semanticscholar.org/CorpusID:14833222

[40] D. Shen, G. Wu, and H.-I. Suk, "Deep learning in medical image analysis," *Annual review of biomedical engineering*, vol. 19, no. 1, pp. 221–248, 2017.

[41] S. Gengtian, B. Bing, and Z. Guoyou, "Efficientnet-based deep learning approach for breast cancer detection with mammography images," in *2023 8th International Conference on Computer and Communication Systems (ICCCS)*, 2023, pp. 972–977.

[42] V.-T. Pham, T.-H. Tran, and H. Vu, "Detection and tracking hand from fpv: benchmarks and challenges on rehabilitation exercises dataset," in *2021 RIVF International Conference on Computing and Communication Technologies (RIVF)*. IEEE, 2021, pp. 1–6.

[43] V. Sze, Y.-H. Chen, T.-J. Yang, and J. S. Emer, "Efficient processing of deep neural networks: A tutorial and survey," *Proceedings of the IEEE*, vol. 105, no. 12, pp. 2295–2329, 2017.

[44] W. Wang, Y. Sun, B. Eriksson, W. Wang, and V. Aggarwal, "Wide compression: Tensor ring nets," in *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*, 2018, pp. 9329–9338.

[45] Y.-D. Kim, E. Park, S. Yoo, T. Choi, L. Yang, and D. Shin, "Compression of deep convolutional neural networks for fast and low power mobile applications," in *International Conference on Learning Representations*, 2016.

[46] C. Tai, T. Xiao, X. Wang, and W. E, "Convolutional neural networks with low-rank regularization," in *International Conference on Learning Representations*, 2016.

[47] X. Su, S. You, J. Xie, F. Wang, C. Qian, C. Zhang, and C. Xu, "Searching for network width with bilaterally coupled network," *IEEE Transactions on Pattern Analysis and Machine Intelligence*, vol. 45, no. 7, pp. 8936–8953, 2023.

[48] J. Liu, B. Zhuang, Z. Zhuang, Y. Guo, J. Huang, J. Zhu, and M. Tan, "Discrimination-aware network pruning for deep model compression," *IEEE Transactions on Pattern Analysis and Machine Intelligence*, vol. 44, no. 8, pp. 4035–4051, 2022.

[49] B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko, "Quantization and training of neural networks for efficient integer-arithmetic-only inference," in *Proceedings of the IEEE conference on computer vision and pattern recognition*, 2018, pp. 2704–2713.

[50] M. Nagel, R. A. Amjad, M. Van Baalen, C. Louizos, and T. Blankevoort, "Up or down? adaptive rounding for post-training quantization," in *International Conference on Machine Learning*. PMLR, 2020, pp. 7197–7206.

[51] S. A. Tailor, J. Fernandez-Marques, and N. D. Lane, "Degree-quant: Quantization-aware training for graph neural networks," *arXiv preprint arXiv:2008.05000*, 2020.

[52] P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh et al., "Mixed precision training," *arXiv preprint arXiv:1710.03740*, 2017.

[53] T. N. Sainath, B. Kingsbury, V. Sindhwani, E. Arisoy, and B. Ramabhadran, "Low-rank matrix factorization for deep neural network training with high-dimensional output targets," in *2013 IEEE international conference on acoustics, speech and signal processing*. IEEE, 2013, pp. 6655–6659.

[54] A.-H. Phan, K. Sobolev, K. Sozykin, D. Ermilov, J. Gusak, P. Tichavský, V. Glukhov, I. Oseledets, and A. Cichocki, "Stable low-rank tensor decomposition for compression of convolutional neural network," in *Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXIX 16*. Springer, 2020, pp. 522–539.

[55] Y. Idelbayev and M. A. Carreira-Perpinán, "Low-rank compression of neural nets: Learning the rank of each layer," in *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 2020, pp. 8049–8059.

[56] Y. Li, S. Gu, C. Mayer, L. Van Gool, and R. Timofte, "Group sparsity: The hinge between filter pruning and decomposition for network compression," in *2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2020, pp. 8015–8024.

[57] S. Lin, R. Ji, C. Chen, D. Tao, and J. Luo, "Holistic cnn compression via low-rank decomposition with knowledge transfer," *IEEE Transactions on Pattern Analysis and Machine Intelligence*, vol. 41, no. 12, pp. 2889–2905, 2019.

[58] Y. Zniyed and T. P. Nguyen, "Enhanced network compression through tensor decompositions and pruning," *IEEE Transactions on Neural Networks and Learning Systems*, pp. 1–13, 2024.

[59] S. Lee and B. C. Song, "Fast filter pruning via coarse-to-fine neural architecture search and contrastive knowledge transfer," *IEEE Transactions on Neural Networks and Learning Systems*, vol. 35, no. 7, pp. 9674–9685, 2024.

[60] B. Han, Q. Yao, X. Yu, G. Niu, M. Xu, W. Hu, I. Tsang, and M. Sugiyama, "Co-teaching: Robust training of deep neural networks with extremely noisy labels," *Advances in neural information processing systems*, vol. 31, 2018.

[61] B. Heo, J. Kim, S. Yun, H. Park, N. Kwak, and J. Y. Choi, "A comprehensive overhaul of feature distillation," in *Proceedings of the IEEE/CVF international conference on computer vision*, 2019, pp. 1921–1930.

[62] S. Lee and B. C. Song, "Knowledge transfer via decomposing essential information in convolutional neural networks," *IEEE Transactions on Neural Networks and Learning Systems*, vol. 33, no. 1, pp. 366–377, 2022.

[63] T. Elsken, J. H. Metzen, and F. Hutter, "Neural architecture search: A survey," *The JMLR*, vol. 20, no. 1, pp. 1997–2017, 2019.

[64] M. Tan, B. Chen, R. Pang, V. Vasudevan, M. Sandler, A. Howard, and Q. V. Le, "Mnasnet: Platform-aware neural architecture search for mobile," in *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, 2019, pp. 2820–2828.

[65] M. Tan, "Efficientnet: Rethinking model scaling for convolutional neural networks," *arXiv preprint arXiv:1905.11946*, 2019.

[66] J. G. Kotwal, R. Kashyap, and P. M. Shafi, "Artificial driving based efficientnet for automatic plant leaf disease classification," *Multimedia Tools and Applications*, vol. 83, no. 13, pp. 38 209–38 240, 2024.

[67] D. H. Fudholi, Y. Windiatmoko, N. Afrianto, P. E. Susanto, M. Suyuti, A. F. Hidayatullah, and R. Rahmadi, "Image captioning with attention for smart local tourism using efficientnet," in *IOP Conference Series: Materials Science and Engineering*, vol. 1077, no. 1. IOP Publishing, 2021, p. 012038.

[68] H. M. Alshahrani, F. N. Al-Wesabi, M. Al Duhayyim, N. Nemri, S. Kadry, and B. A. Alqaralleh, "An automated deep learning based efficient integer-arithmetic-only inference," in *Proceedings of the IEEE conference on computer vision and pattern recognition*, 2018, pp. 2704–2713.
satellite imagery analysis for ecology management,” *Ecological Informatics*, vol. 66, p. 101452, 2021.

[69] V.-T. Pham, C.-M. Tran, S. Zheng, T.-M. Vu, and S. Nath, “Chest x-ray abnormalities localization via ensemble of deep convolutional neural networks,” in *2021 International Conference on Advanced Technologies for Communications (ATC)*. IEEE, 2021, pp. 125–130.

[70] H. Cai, L. Zhu, and S. Han, “Proxylessnas: Direct neural architecture search on target task and hardware,” *arXiv preprint arXiv:1812.00332*, 2018.

[71] K. Yu, R. Ranftl, and M. Salzmann, “An analysis of super-net heuristics in weight-sharing nas,” *IEEE Transactions on Pattern Analysis and Machine Intelligence*, vol. 44, no. 11, pp. 8110–8124, 2022.

[72] Z. Guo, X. Zhang, H. Mu, W. Heng, Z. Liu, Y. Wei, and J. Sun, “Single path one-shot neural architecture search with uniform sampling,” in *Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVI 16*. Springer, 2020, pp. 544–560.

[73] S. Han, J. Pool, J. Tran, and W. Dally, “Learning both weights and connections for efficient neural network,” *Advances in neural information processing systems*, vol. 28, 2015.

[74] T. Lin, S. U. Stich, L. Barba, D. Dmitriev, and M. Jaggi, “Dynamic model pruning with feedback,” *arXiv preprint arXiv:2006.07253*, 2020.

[75] C. Dai, X. Liu, H. Cheng, L. T. Yang, and M. J. Deen, “Compressing deep model with pruning and tucker decomposition for smart embedded systems,” *IEEE Internet of Things Journal*, 2021.

[76] H. Wang and Y. Fu, “Trainability preserving neural pruning,” in *International Conference on Learning Representations*, 2023.

[77] C. Zhao, B. Ni, J. Zhang, Q. Zhao, W. Zhang, and Q. Tian, “Variational convolutional neural network pruning,” in *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, 2019, pp. 2780–2789.

[78] Y. Tang, Y. Wang, Y. Xu, D. Tao, C. XU, C. Xu, and C. Xu, “Scop: Scientific control for reliable neural network pruning,” in *NeurIPS*, 2020.

[79] Z. Liu, J. Li, Z. Shen, G. Huang, S. Yan, and C. Zhang, “Learning efficient convolutional networks through network slimming,” in *Proceedings of the IEEE international conference on computer vision*, 2017, pp. 2736–2744.

[80] J.-H. Luo, H. Zhang, H.-Y. Zhou, C.-W. Xie, J. Wu, and W. Lin, “Thinet: Pruning cnn filters for a thinner net,” *IEEE Transactions on Pattern Analysis and Machine Intelligence*, vol. 41, no. 10, pp. 2525–2538, 2019.

[81] Y. Li, S. Lin, J. Liu, Q. Ye, M. Wang, F. Chao, F. Yang, J. Ma, Q. Tian, and R. Ji, “Towards compact cnns via collaborative compression,” in *2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2021, pp. 6434–6443.

[82] X. Ruan, Y. Liu, C. Yuan, B. Li, W. Hu, Y. Li, and S. Maybank, “Edp: An efficient decomposition and pruning scheme for convolutional neural network compression,” *IEEE Transactions on Neural Networks and Learning Systems*, vol. 32, no. 10, pp. 4499–4513, 2021.

[83] M. Shao, J. Dai, J. Kuang, and D. Meng, “A dynamic cnn pruning method based on matrix similarity,” *Signal, Image and Video Processing*, vol. 15, pp. 381–389, 2021.

[84] Y. Zniyed and T. P. Nguyen, “Efficient tensor decomposition-based filter pruning,” *Neural Networks*, p. 106393, 2024.

[85] Z. Li, F. Liu, W. Yang, S. Peng, and J. Zhou, “A survey of convolutional neural networks: analysis, applications, and prospects,” *IEEE transactions on neural networks and learning systems*, vol. 33, no. 12, pp. 6999–7019, 2021.

[86] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in *3rd International Conference on Learning Representations*, 2015.

[87] Z. Huang and N. Wang, “Data-driven sparse structure selection for deep neural networks,” in *Proceedings of the European conference on computer vision (ECCV)*, 2018, pp. 304–320.

[88] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga et al., “Pytorch: An imperative style, high-performance deep learning library,” *Advances in neural information processing systems*, vol. 32, 2019.

[89] L. Bottou, “Stochastic gradient descent tricks,” in *Neural Networks: Tricks of the Trade: Second Edition*. Springer, 2012, pp. 421–436.

[90] S. Lin, R. Ji, C. Yan, B. Zhang, L. Cao, Q. Ye, F. Huang, and D. S. Doermann, “Towards optimal structured cnn pruning via generative adversarial learning,” *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 2019.

[91] Y. Sui, M. Yin, Y. Xie, H. Phan, S. Aliari Zonouz, and B. Yuan, “Chip: Channel independence-based pruning for compact neural networks,” *Advances in Neural Information Processing Systems*, vol. 34, pp. 24604–24616, 2021.

[92] M. Alwani, Y. Wang, and V. Madhavan, “Decore: Deep compression with reinforcement learning,” in *2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2022, pp. 12339–12349.

[93] R. Lin, J. Ran, D. Wang, K. Chiu, and N. Wong, “Ezcrop: Energy-zoned channels for robust output pruning,” in *Proceedings of the IEEE/CVF winter conference on applications of computer vision*, 2022.

[94] L. Liebenwein, C. Baykal, H. Lang, D. Feldman, and D. Rus, “Provable filter pruning for efficient neural networks,” in *International Conference on Learning Representations*, 2020.