Towards Practical Control of Singular Values of Convolutional Layers
Authors: Alexandra Senderovich
Ekaterina Bulatova
Anton Obukhov
Maxim Rakhuba
Venue: NeurIPS 2022 Accept (2022)

Abstract: In general, convolutional neural networks (CNNs) are easy to train, but their essential properties, such as generalization error and adversarial robustness, are hard to control. Recent research demonstrated that singular values of convolutional layers significantly affect such elusive properties and offered several methods for controlling them. Nevertheless, these methods present an intractable computational challenge or resort to coarse approximations. In this paper, we offer a principled approach to alleviating constraints of the prior art at the expense of an insignificant reduction in layer expressivity. Our method is based on the tensor-train decomposition; it retains control over the actual singular values of convolutional mappings while providing structurally sparse and hardware-friendly representation. We demonstrate the improved properties of modern CNNs with our method and analyze its impact on the model performance, calibration, and adversarial robustness. The source code is available at: https://github.com/WhiteTeaDragon/practical_svd_conv

Towards Practical Control of Singular Values of Convolutional Layers
Alexandra Senderovich HSE University
Ekaterina Bulatova" HSE University
Anton Obukhov ETH Zürich
Maxim Rakhuba HSE University
Abstract
In general, convolutional neural networks (CNNs) are easy to train, but their es- sential properties, such generalization eπror and adversarial robustness, hard to control. Recent research demonstrated that singular values of con- volutional layers significantly affect such elusive properties and offered eral methods for controlling them. Nevertheless, these methods present an in- tractable computational challenge resort coarse approximations, In this paper, we offer a principled approach to alleviating constraints of the prior art the expense of insignificant reduction in layer expressivity. Our method is based on the tensor-train decomposition; it retains control over the actual singular values of convolutional mappings while providing structurally sparse and hardware- hardiale-friendll representation. We demonstrate the improved properties of modern CNNs with our method and analyze its impact on the model perfor- mance, calibration, and adversarial robustness. The source code is available at: https:/ggtthub.com wwtteeoo//oeeeeeeaa/ooooooooeee Drageoaa////paacccccccll laatccallssd____oeeeeee
Introduction
Over the past decade, empirical advances Deep Learning have made machine learning ubiquitous to researchers and practitioners from various fields of science and industry. However, the theory of Deep Learning lags behind its practical applications, resulting in unexpected outcomes or lack of explainability of the models. These adverse effects are becoming more prominent with many models put into customer-facing products, such as perception systems self-driving cars, chatbots, and other applications. The demand for tighter control over deployed models has given rise several subfields of Deep Learning, such as the study of adversarial robustness and model calibration, to name a few.
Convolutional neural networks (CNNs) and, in particular, residual CNNs have become a benchmark for various computer vision tasks. The key component of CNNs convolutional layers, representing linear transformations that account for the image data structure. The singular values of these linear transformations are key to the properties the whole network, such as the Lipschitz constant and, hence, generalization еттог and robustness adversarial examples. Moreover, bounding the Lipschitz constant convolutional layers can also improve training, as it prevents gradients from exploding.
Nevertheless, finding and controlling singular values of convolutional layers is challenging. Indeed, computing and then clipping exact singular values of a layer [Sedghi al., 2019], given by a kernel tensor of the size & x & Cm X Cout, has time and space complexities of (n (c + logn)) and Onn²c² respectively, where output channels respectively, = max { (Cin, Cout and & k is the filter size. Since typically 122 < computing singular values requires many more operations than computing a single convolution with
the input image size, Cnn and Cout are the numbers input and
"Equal contribution.
Corresponding author: Alexandra Senderovich (AlcxandraSeendervvice) (mw.ailcommmmmmmmmmmmmmmmmmmmmmm
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

its asymptotic complexity )??) When computing singular values, the storage of an arising kernel tensor of the size n n Gin X Cout (padded to image dimensions) can also be an issue for larger networks and inputs. This effect becomes even more pronounced in firee-dimensiona convolutions used for volumetric data processing. Alternative methods that parametrize the convolutional layers are also computationally demanding [Singla and Feizi, 2021b, Li et al., 2019].
In this paper, we propose a practical approach constraining the singular values of the convolutional layers based on intractable techniques from the prior art. Our approach relies on the assumption that modern over-varamnn parameterizee neural networks can be made sparse using tensor decompositions, incurring insignificant degradation of the downstream task performance. We investigate the impact of using our approach on model performance, calibration, and adversarial robustness. More specifically, our contributions are as follows:
1. We propose a new framework for reducing the computational complexity of controlling singular values of convolutional layer by imposing the Tensor-Train (TT) decomposition constraint on a kernel tensor. It allows for substituting the computation of singular values of the original layer by a smaller layer. As result, it reduces the complexity of singular values control by using method of choice.
2. We extend the formula of exact singular values of a convolutional layer [Sedghi et al., 2019] to the case of strided convolutions, which is ubiquitous in CNN architectures.
3. We apply the proposed framework to different methods of controlling singular values and test them several CNN architectures. Contrary to measuring just the downstream task performance as in the prior art, we additionally demonstrate improvements in adversarial robustness and model calibration of the networks.
The paper is organized as follows: Sec. 2 provides an overview of prior art on singular values and Lipschitz constant estimation, related methods employing these techniques, and assumptions used to develop our approach; Sec. 3 revisits computation of singular values of convolutional layers and describes our approach to dealing with computational complexity; Sec. 4 walks through the steps of our algorithm to computing singular values of compressed convolutional layers; Sec. 5 contains the empirical study of our method; Sec. 6 wraps up the paper.
2 Related Work
2.1 Computing Singular Values of Convolutions
Traditional discrete convolutional layers used in signal processing and computer vision [LeCun et al., 1989] can be seen from two perspectives. On the one hand, a d-dimensional convolution linearly maps (potentially overlapping) windows of e-channel input signal with side is into vectors of e output values (window map). This perspective is related to the one implementation of convolutional layers, decomposing the operation into window extraction termed im2col [Chellapilla et al., 2006), followed by matrix multiplication via GEMM [Blackford et al., 2002]. From this perspective, the transformation is defined by the convolution weight tensor unfolded into a matrix of size k2 cm x Cout (Conv2D case); its singular values can be computed and controlled through SVD-based schemes over the said unfolding matrix. For example, Yoshida and Miyato [2017] proposed a method of Spectral Normalization to stabilize GAN training [Goodfellow et al., 2020J. Their method effectively estimates the largest singular value of the map via the power method. In a similar vein, Liu al. [2019], Obukhov et al. [2021] explicitly parametrized the SVD decomposition of the mapping to control multiple singular values during training.
Nevertheless, the above perspective lacks the generality of treating the input signal as a whole, which especially important when chaining maps, such as seen in deep CNNs. To this end, discrete convolutional layer can be viewed as a mapping from and onto the space defined by the signal shape (signal map). The map is defined by an --lly clock-cireulant matrix, where N is the number of spatial dimensions of the convolution. A straightforward way to compute singular values of such a matrix by applying a full SVD algorithm is prohibitively expensive, even for small numbers of channels, as well as signal and window sizes. To overcome this issue, Sedghi et al. [2019] proposed method based on the Fast Fourier Transform (FFT) for computing exact singular values convolutional layers that has much better complexity than the naive approach. Nevertheless, it is still quite demanding in space
2

---

and time complexities, as it deals with a padded kernel of the size n X n x X Alternativly, Singla and Feizi [2021b] directly impose nearly orthogonal constraints on convolutional layers using Taylor matrix, which appears to be more efficient
expansion of a matrix exponential of a than Block Convolution Orthogonal Parameterization approach of Li et al. [2019] proposed earlier. Singla and Feizi [2021a] drew connections between both perspectives and proved that the Lipschitz constant of the window map could serve as a genuine upper bound of the Lipschitz constant of the signal map if multiplied by a certain constant.
2.2 Effects of Singular Values
The study of singular values and the closely related Lipschitzness of convolutional neural networks impacts many domains of Deep Learning research. One essential expectation about neural networks is to generalize input data instead of memorizing it. Bartlett et al. [2017] present the generalization bound that depends on the product of the largest singular values of layers' Jacobians. Gouk et al. [2021] performed a detailed empirical study of the influence of bounding individual Lipschitz constants of each layer and its effect on the generalization error of the whole CNN. Singular values of Jacobians greater or smaller than 1.0 can also be responsible for the growth decay of gradients. Therefore, controlling all singular values can also be helpful to avoid exploding or vanishing gradients. The decay of singular values in layers also plays role in the performance of GAN generators [Liu et al., 2019].
Apart from boosting the accuracy of predictions, researchers have been working on improving the robustness of neural networks to adversarial attacks, which is also affected by the Lipschitz constant [Cisse et al., 2017]. Adversarial attacks aim to find a negligible (to human perception) perturbation of input data that would sabotage the predictions of a model. In this paper, we use the AutoAttack Robust Accuracy module [Croce and Hein, 2020] evaluate results. We additionally use a metric of classification model calibration (Expected Calibration Error, ECF) [Naeini et al., 2015]. Connections between model Lipschitz constraints, calibration, and out-of-distribution (OOD) model performance have been drawn in recent literature [Postels et al., 2022]. Various training techniques can also affect model properties [Kodryan al., 2022] and efficiency of training [Gusak et al., 2022].
2.3 Tensor Decompositions in Deep Learning
Our proposed framework for CNN weights reparameterization relies on the well-studied property of neural network overparameterizzaion [Frankle and Carbin, 2019]. Many network compression approaches [Anwar al., 2017, Lee et al.. 2019] agree that it is often possible to approach the uncompressed model performance by imposing some form of sparsity on the network weights. Tensor decompositions have been used previously for compression [Novikov et al., 2015, Garipov et al., 2016, Wang et al., 2018, 8, Obukhov et al., 2020], multitask learning [Kanakis et al., 2020], neural rendering [Obukhov et al., 2022] and reinforcement learning [Sozykin et al., 2022]. The idea of this paper is to utilize tensor decomposition closely related to the SVD decomposition and to take advantage of its sparsity to reduce the complexity of controlling singular values. For that, we resort to the TT decomposition [Oseledets, 2011], which in the outlined context is also equivalent to the Tucker-2 decomposition [Kim et al., 2016]. As opposed to alternative tensor decompositions, this one allows to naturally access singular values of the convolution layer with any method of choice.
3 Method--ndepeedent Complexity Reduction
This section explains how to reduce the complexity of calculating the singular values of a convolu- tional layer (in the signal map sense explained in Sec. 2.1), regardless of the chosen method. For the sake of generality, we consider layers acting on (d+1))dimensional input arrays (tensors) with d spatial dimensions and 1 channel dimension. Of particular interest are cases d 2 and d 3. corresponding to regular images and volumetric data.
We bootstrap our approach based on the recent research on neural network sparsity by compressing convolutional layers with the TT decomposition. We focus on dealing with the channel-wise com- plexity of the methods employed. In what follows, Sec. 3.1 revisits the convolutional operator in neural networks; Sec. 3.2 introduces our principled approach to compressing convolutional layers and reducing the complexity of the problem by significant margin.
3

---

3.1 Regular Convolutional Layers
To introduce a convolutional layer formally, let K R Xont be a (4+2)-dimenssonn) kernel tensor, where k is a filter size and Cm, Cout are the numbers of input and output channels, respectively. convolutional layer is given by linear map 6x multichannel convolution with the kernel tensor KC' such that Ex: R xnx-x xn Roou X x((x)))x...xxfa and
(6x(
X
Cout:
(I)
1=
where f(n, k) is an integer, depending on the type of the convolution 's' used and convolution parameters, e.g., strides. The key assumption needed for our derivations is the bilinearity of 'x', which covers different convolution types, e.g., linear, periodic, and correlation. For example, widely used correlation-type convolution with strides equal to one, reads
Gia
naij.. tD....ttttt.ttttt
Pi
for all j = .,..., Cout and da 0,................. [n, k) 1, a = I,...,, where (nn - n - k ++1 1. In what follows in this section, we write
y= (x)),
(2)
implying one of the convolution types mentioned above.
Using the linearity of Ex, we can rewrite (2) as a matrix-vector product:
vec())- Tx vec(X"),
where vec is row-major reshaping of a multidimensional array into column vector. In turn, Tk IR xa.."" is matrix with an additional block structure: its nd nd blocks are d-level Toeplitz matrices (see Sedghi et al. [2019] for d = 2). Representation (3) allows us to replace singular values of linear map 6x with singular values of matrix Tx, which is handy for analysis. In turn, access to singular values allows for controlling the Lipschitz constant of &x in terms of the largest singular value of Tk, denoted as oi (Tk). Indeed, since 01 (Tk) [Till2,. we have
([6)(x)) (Z)]l
v2c < ||allla| Z||F 01 ( Zlle.
3.2 Compressed Convolutional Layers
Recall that even if d = 2, finding the SVD of Tk requires On + log FLOPs, which is too much for practical use with large networks. To reduce the computational cost of SVD of Tk, we propose a low-rank compressed layer representation based on the following tensor decomposition:
(2)
PI--.
K3
a=
B=0
where (C(1) E Rri K33) E Rrat and IC(2) E X are some small tensors and integers r1, F2: 1 I S Gin 1 1 SSr2 12 S Cout are called ranks. This decomposition is essentially the TT decomposition [Oseledets, 2011] of the kernel tensor with filter modes pı....: Pa merged into single index. We also note that is equivalent to the so-called Tucker-2 decomposition, successfully used by Kim et al. [2016] to compress large convolutional networks. The proposed decomposition is visualized in Fig. 1. We note that we chose the TT decomposition as gives us access to the SVD decomposition of the convolution (Theorem 1). It is, however, unclear how to obtain similar results with other decompositions, such as tensor-ring ring or canonical tensor decomposition, as they do not admit SVD-like form, see, e.g., [Grasedyck et al., 2013] for more details of these formats.
By substituting (4) into (1), using bilinearity of 's' and omitting summation limits for clarity:
(xx()))) K.C.niihd X X
Σ
Σ
Kga
(2(2)
o K
X X
(1) (3) (K-33) ΣEx( xela Xc (K Kesinntt
Σ
(3)
(2)
Σ xcla Xesemnee
L *
B

---

we obtain that the convolution with kernel given by (4) is equivalent to a sequence of three convolu- tions without nonlinear activation functions between them:
1. 1 1
convolution with Gn input and 71 output channels (kernel tensor KC(1));
2. k k convolution with 71 input and 12 output channels (kernel tensor K((22);;
3. 1 x x.... convolution with r2 input and Cout output channels (kernel tensor K((3));
equivalently,
6 = Tgos o 6xes Keo,
(5)
where af 3 denotes the composition of two maps of and 8; Tro denotes x... map given by convolution kernel (3)))
... x convolution
In the rest of this section, we show that singular values of the original convolutional layer with kernel K. correspond to singular values of the compressed kernel (C(2), As first step, we demonstrate that we can impose orthogonality constraints on KCO)) and (C(3) without loss of expressivity.
Lemma Let K R.x...xxxo XCout be given by (4). Then there exist matrices Q(1) ReaXri Q(3) E Iort satisfying Q(1) Q(1 Ir ( Q(3)
Ir and tensor (
Rex xxxx X12, such that
r- 12-1
Σ
2
(6)
ia
a=0
8=0
Proof. See Sec. A.1 of the Appendix.
Now have all preliminaries in place to formulate the key result that our framework is based
Theorem 1. Let KC Rkx. xxxx--xxxix Xrout be given by (4), Let also K(O) E (gemxr, K(3) Rone
IC
ACO1)
ACO2)
(CG3)
K()TK(1) Ir1
((3) = Irz
k
Then the multiset of nonzero singular values of 'K defined in (1) equals the multiset of nonzero singular values of Eggay.
Figure 1: Tensor diagram representation of (4) for d 3 with the constraints (7) as in Theorem A node with D "legs" repre- sents D-dimensional tensor; individual legs represent indices; connected legs represent summation over the respective indices; half- filled nodes orthogonal matrices.
Proof. See Sec. A.2 of the Appendix.
Theorem suggests that after forcing the orthogonal- ity of XC(1) and XC(3), one only needs to find singular values of the layer corresponding to the smaller ker- nel tensor (2) of the size n X X T2. At the same time, thanks to Lemma 1, one may always set K(I) and K(() to be orthogonal without additionally losing the expressivity of 6x. Fig. demonstrates the resulting decomposition using tensor diagram notation.
rı X instead of the original kernel tensor IC of the size
4 Application of the Proposed Framework
This section discusses how to apply the proposed framework to various methods. We start with the explicit formula for singular values computation and then discuss how to apply the framework when the convolutions are parametrized, for example, as in [Singla and Feizi, 2021b, Li al., 2019].
5

---

Algorithm Singular values of a TT-compressed periodic convolutional layer (d = 2).
Input:
XCO E R left core tensor of the TT-compressed kernel K.
X((2) R middle core tensor of the TT-compressed kernel K,
K(3) E TR right core tensor of the TT-compressed kernel K,
input image size,
stride of the convolution defined by kernel K. s.t. n = (mod s).
Output:
Non-zero singular values of the convolutional layer, defined by the kernel tensor K.
Perform QR decomposition of the left core Perform QR decomposition of the right core R PL. ROOT, Vp1.P2 Absorb non-orthogonal factors in the middle core 4: K + paapppaaeeeaa zeros R10) shape= x x m2)
1: Q RQO) QR (X(()
2: Q RR() t- QR ( (22)
3: Pippa R
Pad middle core to the image size
5: R reshape (112) shape= (8)
Reshape middle core as in (8). See also Sec. B.1.
6: pli)) + fft2 R2 mad Annd)(i/e)) Vi,j
Perform 2-dimensional FFT of slices of R
7: return U
(P()))P2)
Return a union of all singular values of P(-))
PL.PE((
4.1 Explicit Formulas for Strided Convolutions
Sedghi et al. [2019] consider periodic convolutions, allowing them express singular values through discrete Fourier transforms and several SVDs. We extend the main result of their work to the case of strided convolutions and summarize it in the following theorem.
Theorem 2. Let R T XCous be kernel K. Rkxkxem XCont of a periodic convolution, padded with zeros along the filter modes. Let 8 be a stride of the convolution Ex: n = 0 (mod 8) and let the reshaped kernel R E IR x3
X Con be such that:
R.g.abb... = kaatstaas.l mod b'.
(8)
Let consider (2)² matrices p(Pl.2)) E R x cou with entries:
p(mi.s)) = -F" Rí mod s F)pip
(9)
where F is an X n Fourier matrix. Then the multiset of singular values Tx is as follows:
(Tk)=
U
p P2))
(10)
PL.PSE(1,.... }}
Proof. See Sec. B.2 the Appendix.
The algorithm for finding the singular values of TT-compressed convolutional layer based on Theorem 2 summarized in Alg. This algorithm consists of several major parts. We start with layer with the imposed TT structure as in (4). We reduce this decomposition to the form with orthogonality conditions (7) by using the QR decomposition (steps 1-3). The second part is the application of Theorem to the smaller kernel tensor KC)2 (steps 7). For detailed pseudocode of applying Theorem 2, see Sec. B.I.
Computing singular values using (9) and (10) has the complexity O(n2e2(cs² + log ")), c max (Cin. ),ut), which depends cubically on c. Thus, reducing c can significantly reduce the com- putational cost of finding the singular values. Let us now use the TT-representation (4) for kernel tensor with the ranks Ti r2 =. Theorem I suggests that only need to apply Theorem 2 to the core x() xxxxxryxx2, which leads to the complexity 2n²
log ")) For example, r = e/2 provides a theoretical speedup of up to times (see Fig. 2 for practical evaluation). Memory consumption can also be high with the method of Sedghi et al. [2019]. Processing the padded kernel tensor of the size n X n x with our method requires 4 times less memory for c/2 and c/3 (see also Fig, 2).
times less for

---

= 23 o/4
F 073 0/4
speaduup
20
2
15
10
100
500
750 1000 1250 1500 1750 2000 c, number of thannels
C, number of channels
Figure 2: Left: the speedup of computing singular values of TT-compressed layer using Alg. 1 relative to an uncompressed one. Right: Memory (single precision) to store the full n x n x c kernel and its TT-compressed form. Hyperparameter values: 16 and 1.
4.2 Orthogonality Constraints
To ensure that the frame matrices XC()) and KC(3) are orthogonal, we apply QR decomposition the kernel in Alg. 1. However, we can use other methods to maintain the orthogonality of frame matrices; for example, we can impose regularization on the layers. Regularization is quite helpful in case x((2) a standard convolution but has special complex structure, such as the SOC layer from [Singla and Feizi, 2021b]. For that, let ny denote the number of TT-decomposed layers in network, X. and x(()) left and right TT-cores the ith layer, and Fi,i; 22, layer ranks. Then, the orthogonal loss reads as:
/
lossort
Σ 1 ((()
Jmnale +
-Ira
Σ 2=
r. + 22
(11)
In the case of applying this regularization, we sum this loss with cross-entropy:
loss losSCE + loss
(12)
The value of lossurt is the MSE measure of non-orthogonality of all xc(") and K-(3) so the coefficient Aurt tends be large. We set it to le5 for experiments with WeseRss
5
Experiments
To study the effects of the proposed regularization on the performance of CNNs, we combined it with various methods of controlling singular values and applied during training on the image classification task on the CIFAR- 10 and CIFAR- 100 datasets [Krizhevsky et al., 2009]. The code was implemented in PyTorch [Paszke et al., 2019] and executed on NVIDIA Tesla V100, 32GB. In our experiments. we used the LipConvNet architecture [Singla and Feizi, 2021b], WideResNet [Zagoruyko and Komodakis, 2016], and VGG-19 [Simonyan and Zisserman, 2015). Furthermore, the ranks are tested in the range [c/4, e/2), where e is the largest number of channels among all convolutions (except for 1 x 1), which we found to give the better performance/speedup trade-off.
Evaluation Metrics We consider four main metrics: standard accuracy on the test split, accuracy on the CIFAR-C dataset (CC) [Hendrycks and Dietterich, 2019]' accuracy after applying the Auto/Atack module (AA) [Croce and Hein, 2020], and the Expected Calibration Error (ECE) [Guo et al., 2017]. The last columns of tables with the results of our experiments contain compression ratios of layers in the network. In addition, we reported a dedicated metric for the LipConvNe architecture: p.r. provable bound-norm robustness, introduced by Li al. (2019]. This metric shows percentage of input images guaranteed to be predicted correctly despite any perturbation within radius 36/ /555.

---

Table 1: Comparison between SOC [Singla and Feizi, 2021b] and SOC combined with the proposed framework using LipConvNet-N on CIFAR-1O. "Speedup" is the speedup training w.r.t. the SOC baseline. "Comp." " (compression) is the ratio between the number of parameters of convolutional layers in the original and the decomposed networks. Despite speedups not being as substantial as in other experiments, we observe consistent improvement in all metrics, which is discussed in Fig.
RANK
N
Acc.
A
CC
ECE
P.R.
SPEEDUPT
COMP.
5
75,6
0.2
6.1
8.6
59.1
1.0
1.0
128
5
76.9 0.2
0.5
6.2
6.1
6.1
1.4
2.9
256
5
78.3 0.1
3.67
6.2
0.2
6.3
0.9
1.2
20
76.3
33.
6.2
7.1
6.24
128
20
0.2
33.0
68.3
0.2
6.24
1.2
3.3
256
20
78.4 0.2
0.5
70.4
4.7
6.26
1.1
1.4
30
76.3 1.0
3.60
1.0
6.29 1.0
1.0
1.0
128
30
0.1
32.6
6.1
5.0
6.67
1.3
3.5
256
30
77.9 0.1
3.37
0.3
4.1
64.8 0.2
1.1
1.5
5.1
Experiments on LipConvNet
Singla and Feizi [2021b] introduced new architecture called Lipschitz Convolutional Networks, or LipConvNet for short. LipConyNet-N consists of convolutional layers and activations. This architecture provably 1-Lipschitz by contrast popular residual network architectures. In Singla and Feizi [2021b], the authors also propose an orthogonal layer called Skew Orthogonal Convolution (SOC) and applied it in LipConvNet.
modify the SOC layer in correspondence with the proposed framework by adding x 1 X Cin con- volution before applying it, and finally applying the Cout, thus reducing the number of channels to r r in the bottleneck. To maintain the orthogonality of this new layer that call SOC-TT, we add orthogonal loss (Sec. 4.2) to keep these two frame matrices orthogonal.
15
N-30
last convolution of size 1 X
N-20
10:
N=20, TT
N=5,
Tab. demonstrates metrics improvement after replac- ing SOC with SOC-TT. For rank 256, performed a grid search the Aort coefficient. The training protocol is the same in Singla and Feizi [2021b] except for the learning rate of LipConvNet-20 with the SOC-TT layer, which set 0.05 instead of 1. To investigate the observed increase in metrics, we present histograms of empirically estimated Lischitz constants by using adversarial attacks in Fig. (see Sec. C.1) with optimally chosen Aoet from (12). We observed that increasing Aort forces the frame matri- in the TT decomposition nal and, as a result, the histogram converges to that of the original SOC method. At the same time, decreasing Aort may lead to networks with constants larger than but with better metrics.
0.0
0.2
0.4
0.6
0.8
Lestimated
Figure 3: Histograms of empirically esti- mated Lipschitz constants of LipConyNet-N architectures (see Sec. C.1). When applying the TT method to LipConyNet architecture, the peaks are wider and closer 1, giving more flexibility to the model but still keeping constants below 1. Rank parameter: r 256.
be closer to orthogo-
5.2 Experiments with WideResNeet-16-10
Next, consider WRN-16-10 [Zagoruyko and Komodakis, 2016] with almost 1 17M parameters. The results of our experiments on CIFAR-10 10 are summarised in Tab. Additional experiments on CIFAR- 100 are presented in Sec. C.3. The experimental setup and training schedule et al. [2021]. J. We train our models with batches of 128 elements for 200 epochs, using SGD optimizer and learning rate of 0.1, multiplied by 0.1 after 60, 120, and 160 epochs. The hyperparameters for
taken from Gouk
Distributed under Apache License 2.0

---

Table 2: Performance metrics for different constraints applied WRN-16-10 10 trained CIFAR- Both clipping and division combined with TT layers lead to an increase in metrics (except for ECE), with 256 being the best overall rank. Clipping tends to be a better option in terms of metries but is substantially slower than the division approach. Moreover, clipping the singular values to results in better performance than clipping 1. "Speedup" is speedup of the overhead resulting from singular values control in all the layers in the network. Clipping the whole network w/o decomposition takes 6.2 min while applying division 0.6 sec. "Comp." (compression) is the ratio between the number of parameters of convolutional layers in the original (Rs 16.8M) and decomposed networks.
METHOD
RANK
Acc. f
f
CC f
ECE 4
SPEEDUP t
COMP. 1
95.52
54.22
72.91
2.06
1.0
192
95.21
54.97
73.94
2.84
3.6
BASELINE
256
95.39
55.07
73.73
2.73
2.4
320
95.02
52.32
72.86
2.85
1.8
95.27
53.27
73.44
2,45
1.0
1.0
192
95.25
51.7
73.37
2,49
4.1
3.6
CLIP TO
256
95.12
51.94
72.47
2.43
3.3
2.4
320
95.05
54.44
73.38
2.7
2.3
1.8
55.77
73,27
1.92
1.0
1.0
192
95.45
55.45
73,26
2.55
4.1
3.6
CLIP TO 2
256
95.73
55.83
73.35
2.28
3.3
2.4
320
95.5
54.63
74.34
2.53
2.3
1.8
95.17
53.39
72.94
2.54
1.0
1.0
192
95.71
53.4
74.68
2.4
2.3
3.6
DIVISION
256
95.7
56.28
73.43
2.51
1.5
2.4
320
94.93
52.35
70.89
2.97
1.3
1.8
the experiments with division were taken from Gouk et al. [2021]. The orthogonal loss (11) was used in all experiments with decomposed networks.
Clipping Following Sedghi et al. [2019], we apply the so-called clipping of singular values after computing the singular values of a layer using Theorem 2. particular, we first fix threshold parameter 8, chosen to be or 2 in numerical experiments. Then all singular values greater than ó are replaced with 8. This procedure allows for maintaining maximal singular values at the desired level. After the clipping, we reconstruct the expanded kernel K& of the size n x n X Gin X Cout, which is no longer sparse. Finally, we revert to its original shape by setting K. = (1:k: 1:k, :,:). This operation performed every 100 training iterations; we provide the code for computing singular values in Sec. B.1 of the Appendix. The results suggest that applying TT decomposition not only compresses network by decreasing the number of parameters of convolutional layers, but also speeds up the clipping operation. Network robustness increased when training decomposed layers with clipping, which is confirmed by the results of the AutoAttack module.
Division Even though we can bound the Lipschitz constants of all convolutional layers, the Lipschitz constant of the whole network can be arbitrary. In particular, for WideResNet architecture, the Lipschitz constant would still be unconstrained due to residual connections, fully connected layers, batch normalization, and average pooling layers of the network. Gouk et al. [2021] derived formula for the Lipschitz constants of batch normalization layers and proposed regularizing them too. After each training step, they set the desired values for Lipschitz constants of convolutional, fully connected, and batch normalization layers. To control the Lipschitz constant of convolutional and fully connected layers, they first compute the estimate of the largest singular value of a layer via power iteration. Then they divide the layer by the estimate and multiply it by the desired Lipschitz constant. A similar operation is performed for batch normalization layers, but the Lipschitz constant is computed directly.
In the original paper, this approach proved to be successful for WideResNett166-10 The method itself already increases the robust metrics and accuracy substantially. The pattern remains intact after applying our decomposition to the layers: the robust accuracy of the AutoAttack module improves. Overall, the TT decomposition, applied without any other regularization, performs poorly. However,

---

Acc.
AA
Cifar 1
ECE
93
54
TT+cliging T+
73
52
50
91
40
46
72
24
20
GPAR-110
24
44
20
18
20
228 192 255
280
128
192 256
380
101
128 192 256
300
1u1
256 280
M
Figure 4: Metrics for different ranks on CIFAR- 10 and CIFAR- 100; "full" stands for the uncompressed case, i..., "full" label with TT method is simply the baseline with no TT decomposition and no control singular values, similarly "full" TT+clippin [Sedghi et al., 2019] and "full" TT+division [Gouk al., 2021]. Horizontal dotted lines additionally represent uncompressed baselines.
when used to facilitate the application of a particular method control over the singular values, the decomposition only slightly decreases the accuracy, which might still exceed that of the baseline.
5.3 Experiments with VGG-19
This section presents the results of controlling singular values on the VGG-19 network and two datasets: CIFAR-I0 and CIFAR-100. As previously, we consider the clipping method with the exact computation of singular values [Sedghi et al., 2019] and the application of one iteration of the power method [Gouk et al., 2021]. The clipping parameter is Firstly, we observe that all the considered robust metrics benefit from controlling singular values. This effect more pronounced than for the WRN-16-10 and was also observed in [Gouk al., 2021]. We conjecture that this happens due to the lack of residual connections in VGG-type architectures, which makes them more sensitive to singular values of convolutional layers; residual networks, convolutional layers are only corrections to the signal. In almost all scenarios, we observed that the clipping method performed better than the division with the power method.
0.5. The results are presented in Fig.
Conclusion
We proposed principled sparsity-driven approach controlling the singular values of convolutional layers. We investigated two different families of CNNs and analyzed their performance and robustness under the proposed singular values constraint. The limitations our approach are partly inherited from the encapsulated methods of computing singular values (e.g., the usage of periodic convolutions) and partly stem from the core assumption CNN layer sparsity. Overall, our approach has proven effective for large-scale networks with hundreds of input and output channels, making it stand out compared to intractable methods from the prior art.
As our work is concerned with improving the robustness of neural networks, it positively impacts their reliability. Nevertheless, due to the generality of our approach, it can be used in various application domains, including malicious purposes.
Societal Impact
Acknowledgments
The publication was supported by the grant for research centers in the field of Al provided by the
Analytical Center for the Government of Russian Federation (ACRF) accordance with agreement on the provision of subsidies (identifier of the agreement 000000D730321 0w00007733221P0000000 P5QQ0002) and the agreement with HSE University Ne70-2021-4 00011000199, The calculations were performed in part through the computational resources of HPC facilities at HSE University [Kostenetskly et al., 2021].
10

---

References
Sajid Anwar, Kyuyeon Hwang, and Wonyong Sung. Structured pruning of deep convolutional neural networks. J. Emerg. Technol. Comput. Syst., 13(3), feb 2017. ISSN 1550-4832. doi: 10.1145/3005348.
Peter L Bartlett, Dylan Foster, and Matus J Telgarsky. neural networks. Advances in neural information processing systems, 30, 2017.
margin bounds for
L Susan Blackford, Antoine Petitet, Roldan Pozo. Karin Remington, R Clint Whaley, James Demmel, Jack Dongarra, lain Duff, Sven Hammarling, Greg Henry, et al. An updated set of basic linear algebra subprograms (blas). ACM Transactions on Mathematical Software, (8(2):135-151 2002.
Kumar Chellapilla, Sidd Puri, and Patrice Simard. High performance convolutional neural networks for document processing. In Tenth international workshop on frontiers in handwriting recognition. Suvisoft, 2006.
Moustapha Cisse, Piotr Bojanowski, Edouard Grave, Yann Dauphin, and Nicolas Usunier. Parseval networks: Improving robustness to adversarial examples. International Conference on Machine Learning, pages $54-863. PMLR, 2017.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-fie attacks. In International conference on machine learning, pages 2206-2216. PMLR, 2020.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019.
Timur Garipov, Dmitry Podoprikhin, Alexander Novikov, and Dmitry P. Vetrov. Ultimate tensoriza- tion: compressing convolutional and FC layers alike. CoRR, abs/1611.002/11 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11))139-144, 2020.
lan Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. CoRR, abs/ abs/1442.6652, 2015.
Henry Gouk, Eibe Frank, Bernhard Pfahringer, and Michael J Cree. Regularisation of neural networks by enforcing lipschitz continuity. Machine Learning. 10(())393 316. 2021.
Lars Grasedyck, Daniel Kressner, and Christine Tobler. A literature survey of low-rank tensor approximation techniques. CAMM-Mitteilune
36(1):53-788 2013.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pages 1321-1330. PMLR, 2017.
Julia Gusak, Daria Cherniuk, Alena Shilova, Alexandr Katrutsa, Daniel Bershatsky, Xunyi Zhao, Lionel Eyraud- Eyraud-Dubois, Oleh Shliazhko, Denis Dimitrov, Ivan Oseledets, and Olivier Beaumont. Survey on efficient training of large neural networks. In Proceedings of the Thirty-Firs Inter- national Joint Conference on Artificial Intelligence, IJCAI-22, pages 2494-5501, 7 2022. doi: 10.249663ijcai.20222 2022/769.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corrup- tions and perturbations. Proceedings of the International Conference on Learning Representations, 2019.
Sebastian Holtz, Thorsten Rohwedder, and Reinhold Schneider. On manifolds of tensors of fixed tt-rank. Numerische Mathematik, 120(4):711-71,,011 2012.
A. K. Jain. Fundamentals of digital image processing. Englewood Cliffs, NJ: Prentice Hall, 1989.
Menelaos Kanakis, David Bruggemann, Suman Saha, Stamatios Georgoulis, Anton Obukhov, and Lue Van Gool. Reparameterizing convolutions for incremental multi-task learning without task interference. In Computer Vision ECCV 2020, volume 12365, pages 689-707. Springer, 2020.
11

---

Yong-Deok, Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang, and Dongjun Shin. Com- pression of deep convolutional neural networks for fast and low power mobile applications. In 4th International Conference Learning Representations, ICLR 2016, 05 2016.
Maxim Kodryan, Ekaterina Lobacheva, Maksim Nakhodnov, and Dmitry Vetrov. Training scale- invariant neural networks on the sphere can happen in three regimes. In Advances in Neural Information Processing Systems, 35, 2022.
PS Kostenetskiy, RA Chulkevich, and VI Kozyrev. Hpe resources of the higher school of economics. In Journal of Physics: Conference Series, volume 1740, page 012050, 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009.
Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied handwritten zip code recognition. Neural Computation, 1(4):541-555 1989.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. Snip: Single-shot network pruning based connection sensitivity. In International Conference on Learning Representations, 2019.
Qiyang Li, Saminul Haque, Cem Anil, James Lucas, Roger B Grosse, and Jörn-Henrik Jacobsen. Preventing gradient attenuation in lipschitz constrained convolutional networks. Advances in neural information processing systems, 32, 2019.
Kanglin Liu, Wenming Tang, Fei Zhou, and Guoping Qiu. Spectral regularization for combating mode collapse in gans. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6382-6390, 2019.
Mahdi Pakdaman Nacini, Gregory F Cooper. and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In AAAI, page 2901-2907, 2015.
Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, and Dmitry P Vetrov. Tensorizing neural networks. In Advances in Neural Information Processing Systems, volume 28, Curran Associates, Inc., 2015.
Anton Obukhov, Maxim Rakhuba, Stamatios Georgoulis, Menelaos Kanakis, Dengxin Dai, and Luc Van Gool. T-basis: a compact representation for neural networks. In Proceedings of the 37th International Conference Machine Learning, volume 119, pages 7392-7404. PMLR, 2020.
Anton Obukhov, Maxim Rakhuba, Alexander Liniger, Zhiwu Huang, Stamatios Georgoulis, Dengxin Dai, and Lue Van Gool. Spectral tensor train parameterization of deep learning layers. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130, pages 3547-3555, PMLR, 2021.
Anton Obukhov, Mikhail Usvyatsov, Christos Sakaridis, Konrad Schindler, and Lue Van Gool. Tt-nf: Tensor train neural fields, 2022.
Ivan V Oseledets. Tensor-train decomposition. SIAM Journal on Scientific Computing, 33(5):
2295-2317, 2011.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito,o Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019.
Janis Postels, Mattia Segů, Tao Sun, Luca Daniel Sieber, Lue Van Gool, Fisher Yu, and Federico Tombari. On the practicality of deterministic epistemic uncertainty. In Proceedings of the 39th International Conference Machine Learning, volume 162, pages 17870- 17909. PMLR, 2022.
Amartya Sanyal, Philip H. Torr, and Puneet K. Dokania. Stable rank normalization for improved gen- eralization in neural networks and gans. International Conference on Learning Representations, 2020.
12

---

Hanie Sedghi, Vineet Gupta, and Philip M. Long. The singular values of convolutional layers. In International Conference on Learning Representations. 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
Sahil Singla and Soheil Feizi. Fantastic four: Differentiable and efficient bounds singular values of convolution layers. In International Conference on Learning Representations, 2021a.
Sahil Singla and Soheil Feizi. Skew orthogonal convolutions. In International Conference on Machine Learning, pages 9756-9766. PMLR, 2021b
Konstantin Sozykin, Andrei Chertkov, Roman Schutski, Anh-Huy Phan, Andrzej Cichocki, and Ivan Oseledets. TTOpt: A maximum volume quantized tensor train-based optimization and its application to reinforcement learning. In Advances Neural Information Processing Systems, 35, 2022.
Wenqi Wang. Yifan Sun, Brian Eriksson, Wenlin Wang. and Vaneet Aggarwal. Wide compression: Tensor ring nets. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9329-9338, 2018
Yuichi Yoshida and Takeru Miyato. Spectral norm regularization for improving the generalizability of deep learning, 2017.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Proceedings of the British Machine Vision Conference (BMVC), pages 87.1-87.12 BMVA Press, September 2016.
13


Reviews:
[{"review_id": "RcIV1Gt4jSV", "reviewer": "Reviewer_Cy9m", "summary": "This article proposed a framework based on the Tensor Train (TT) decomposition to decrease the complexity of computing a given convolutional layer's singular values (SVs). The key idea is to decompose the original convolution layers into low-rank factors and calculate the SVs of the factors with smaller sizes instead of the original kernel tensor. The main paper and supplementary materials give detailed proofs of the lemma and theorems. The framework is evaluated on accuracy, robustness, and calibration. The experiments are conducted on LipConv and WideResNet-28-10, using CIFAR-10 as the dataset.", "questions": "1. Section 2. As mentioned in the weakness part, too many topics are mentioned in this section. Though the scope of topics mentioned in this section is enormous, each of them only has short content, which makes the critical point unclear. Especially for the last two (viz. Robustness and Calibration), I think they are metrics to evaluate the framework in the experiments, thus it is not necessary to mention them in the related work section. I suggest making Section 2 more concise and the research gap more obvious.\n2. About the novelty. Including TT, Tucker-2, CP, and Tensor Ring (TR) are also decomposition methods widely utilized to compress the CONV layers. I wonder why TT is employed in this work but not other decomposition methods? \n3. About the choice of TT ranks. How to choose proper ranks is a vital research topic for tensor decomposition, but it is not discussed in this article. I wonder how the ranks are determined in this paper. The selection is based on experimental results? If yes, there should be an ablation study about it. \n4. The experiments are insufficient. I think experiments on two models using CIFAR-10 are not enough. I think the results will be more convincing if more models (e.g., ResNet-18/50) and larger dataset (at least CIFAR-100) are considered.\n5. The results lack analysis. Sections 5.1 & 5.2 are too short, and the results are listed in the Table without further discussion. It will be better if more analysis is provided. Besides, I think the first column in the right part of Table 1 should be “ACC.” instead of “AC.”? The definition of COMP. is also missing, it is not clear how to calculate it and the meaning of the number.\n6. line 288 – 321. I think it is better to move this part to the appendix and use the space to add more experiments, discussion of the results, ablation study of rank, etc.\n", "limitations": "None", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n1. The lemma and theorems are clearly stated, and the proofs are detailed.\n\nWeakness\n1. The originality of the proposed framework lacks significance, since using tensor decomposition to decompose a CONV layer into sequential smaller ones is not novel.\n2. The related work contains too many topics, which instead blurs the key point of this article. It makes the research gap and the motivation of the proposed framework unclear.\n3. This work does not give a clear strategy to find the proper TT ranks, which is of great significance for TT decomposition.\n4. The experiments are not comprehensive and lack discussion and ablation study.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "QKOdb7Yh-hH", "reviewer": "Reviewer_9EwE", "summary": "This paper proposes an efficient method to compute eigenvalues of convolutional layers based on a sparsity-driven approach. The proposed work achieves by proposing a novel framework that compresses the layer representation (section 3.2) based on a tensor decomposition. The authors provide theoretical justification in Lemma 1 and Theorem 1 of their proposed framework. In addition, empirical evidence is provided to demonstrate how effective their tensor decomposition across a wide range of architecture and metrics. \n\nThe contributions are listed as the following:\n1). Proposes a novel framework Tensor-Train (TT) decomposition to compute singular values that reduce computational complexity.\n2). Extend previous work on the exact computation of singular values to strided convolutions\n3). Demonstrated significance of  proposed framework with varying control over the singular values on a few CNN architectures by comparing  their method to the baseline across some important metrics (accuracry, generalization error, and  adversarial robustness).", "questions": "1). Could you please provide some context w/ respect to Figure 1? Such as if you have speed up of x amount, could that lead to requiring less memory in GPU compute? For example, say a researcher only has access to 8 GB GPU could they compress the network by a certain factor to fit onto their GPU. \n\n2). Strided convolutions are a large part of Generative networks in GANs, do you think your work could be built upon to study mode collapse in GAN's based on the singular value composition? As well, did you consider study these types of models in your paper? \n", "limitations": "Limitations are in section 6, assumptions are reasonable. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths-\n1). Works on important problems to the ML/AI community.\n2). Approves upon previous work by reducing the computation complexity and extending work to strided convolution. \n3). Convincing empirical evidence that this framework is effective.\n\nWeakness-\n1). Theoretical claims could be better explained/justified. It's not obvious upon reading this how all the theoretical contributions are related to the work. Theorem 1 and lemma 1 are related to TT framework, and theorem 2 is about explicit singular value formula for strided convolution. To help clear some ambiguity, it might be helpful to put the lemma in appendix and expand up the technical details of theorem 1 and the importance of it for the framework. Then explain how all the theoretical contributions fit into algorithm (1).\n\n2). I believe the paper would benefit from additional experiment, similar to the one in Figure (1), which would include speed up versus some performance metrics (accuracy, ECE). This would be very useful from a practitioner's standpoint\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "NquNO11iH4_", "reviewer": "Reviewer_EXfn", "summary": "The authors of this paper offer a method to compute the singular values (SVs) of convolutional layers efficiently. They propose a method to decompose the matrix $\\mathcal{K}$ representing a convolutional layer to smaller matrices $\\mathcal{K}^{(1)} \\mathcal{K}^{(2)} \\mathcal{K}^{(3)}$ and show that the SVs of $\\mathcal{K}$ can be expressed by the union of the SVs of some smaller matrices related to $ \\mathcal{K}^{(2)}$.", "questions": "Please see the Weaknesses above for the questions.", "limitations": "The limitations are not sufficiently discussed. For example, the assumptions of the lemma and theorems are not specified clearly.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- The decomposition of the matrix for convolutional layers is neat.\n- The experimental results on clean and robust accuracy are improved with the proposed method.\n\n## Weaknesses\n- The assumptions of the lemma and theorems are not specified clearly. Do they apply to all kinds of convolutional layers?\n- It seems that Theorem 2 requires $n \\equiv 0\\ (\\text{mod}\\ s)$, which means the spatial size $n$ has to be divisible by the stride $s$. Does it mean the stride equals the kernel size (i.e., $s = k$)? Many convolutional layers may not satisfy such a requirement.\n- The experiments of computational complexity are too limited. A detailed comparison (in terms of, for example, running time) between the proposed method and prior methods should be added.\n- The correctness of the singular values (SVs) using the proposed method should be validated empirically. For example, the SVs computed by the proposed method can be compared to those from [1]. Note that the method from [1] may only apply to convolutional layers with a stride equaling 1. For other strides, an alternative comparison is [2] by using the inequalities of matrix norms: $r\\|A\\|_\\alpha\\leq\\|A\\|_\\beta\\leq s\\|A\\|_\\alpha$, i.e., by empirically showing that the largest singular value from the proposed method can be bounded by the $\\ell_1$ norm or $\\ell_\\infty$ norm from [2].\n\n[1] Sedghi H, Gupta V, Long PM. The singular values of convolutional layers. arXiv preprint arXiv:1805.10408. 2018 May 26.\n\n[2] Liang Y, Huang D. Large norms of CNN layers do not hurt adversarial robustness. InProceedings of the AAAI Conference on Artificial Intelligence 2021 May 18 (Vol. 35, No. 10, pp. 8565-8573).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "FTzsTun_sD", "reviewer": "Reviewer_eWcT", "summary": "The work proposed a less expansive approach to calculating the singular values of convolutional layers. The proposed approach builds on the assumption that modern neural networks are highly overparametrized in such a way that a sparser surrogate network could achieve similar performance and the method of Tensor-Train (TT) decomposition. The work further empirically shows that a network's robustness could be improved by adding additional orthogonality constraint terms derived from the proposed approach and clipping calculated singular values. ", "questions": "My major concerns are discussed in the Weaknesses section. \n\nAs an additional question, I'm wondering why the constraint discussed in Section 4.2 does not include a term to deal with K2 and why is the constraint useful without controlling K2.\n\nLastly, I would suggest expanding the clipping part discussed in Section 5.2 to a more comprehensive comparison between the proposed method and [1] (perhaps adding some comparisons when [1] is tractable in order to show both the efficiency and accuracy of the proposed method). ", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe overall presentation of the work is clear. The relationships with prior arts are carefully discussed. And the experimental part clearly demonstrates the benefits of applying the proposed method and also discusses many technical details.\n\nWeaknesses:\nThe reviewer found that the title of the work is not well aligned with the actual content. The title indicates that a more efficient approach for calculating singular values of convolutional layers will be proposed, one would expect to directly use the proposed method to calculate singular values of some given network (like the algorithm proposed in [1]). However, the calculation described in this work heavily relies on the decomposition (K = K1 ∘  K2 ∘  K3) and it is not clear to the reviewer how to decompose a given convolutional kernel to such form (please correct me if I'm wrong). Rather, the reviewer believes that the major accomplishment of the work lies in designing networks/constraints to improve the robustness of networks.\nBesides, the experiment section seems a little bit weak to only compare with SOC only (Table 1) and the vanilla setting (Table 2). The effect of the proposed approaches can be better demonstrated If more experiments can be added. \n\n[1] Hanie Sedghi, Vineet Gupta, and Philip M. Long. The singular values of convolutional layers, 2019\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept