LOT: Layer-wise Orthogonal Training on Improving l2 Certified Robustness
Authors: Xiaojun Xu
Linyi Li
Bo Li
Venue: NeurIPS 2022 Accept (2022)
Keywords: Adversarial Robustness
Lipschitz-bounded Models

Abstract: Recent studies show that training deep neural networks (DNNs) with Lipschitz constraints are able to enhance adversarial robustness and other model properties such as stability. In this paper, we propose a layer-wise orthogonal training method (LOT) to effectively train 1-Lipschitz convolution layers via parametrizing an orthogonal matrix with an unconstrained matrix. We then efficiently compute the inverse square root of a convolution kernel by transforming the input domain to the Fourier frequency domain. On the other hand, as existing works show that semi-supervised training helps improve empirical robustness, we aim to bridge the gap and prove that semi-supervised learning also improves the certified robustness of Lipschitz-bounded models. We conduct comprehensive evaluations for LOT under different settings. We show that LOT significantly outperforms baselines regarding deterministic l2 certified robustness, and scales to deeper neural networks. Under the supervised scenario, we improve the state-of-the-art certified robustness for all architectures (e.g. from 59.04% to 63.50% on CIFAR-10 and from 32.57% to 34.59% on CIFAR-100 at radius $\rho=36/255$ for 40-layer networks). With semi-supervised learning over unlabelled data, we are able to improve state-of-the-art certified robustness on CIFAR-10 at $\rho=108/255$ from 36.04% to 42.39%. In addition, LOT consistently outperforms baselines on different model architectures with only 1/3 evaluation time.

LOT: Layer-wise Orthogonal Training on Improving lz Certified Robustness
Xiaojun Xu
Linyi Li
Bo Li
University Illinois jun3, linyi2,
Urbana- Chaapa-C
1bo @1 @illinos.... edu
Abstract
Recent studies show that training deep neural networks (DNNs) with Lipschitz constraints are able to enhance adversarial robustness and other model properties such as stability. In this paper, we propose a layer-wise orthogonal training method (LOT) to effectively train I-Lipschitz convolution layers via parametrizing an orthogonal matrix with an unconstrained matrix. We then efficiently compute the inverse square root of a convolution kernel by transforming the input domain to the Fourier frequency domain. On the other hand, existing works show that semi- supervised training helps improve empirical robustness, we aim to bridge the gap and prove that semi-supervised learning also improves the certified robustness of Lipschitz-houndee models. We conduct comprehensive evaluations for LOT under different settings. We show that LOT significantly outperforms baselines regarding deterministic & certified robustness, and scales to deeper neural networks. Under the supervised scenario, we improve the state-of-the-ar certified robustness for all architectures (e.g. from 59.04% to 63.50 63.50% on CIFAR-10 and from 32.57% 34.59% on CIFAR- at radius p = 36/255 for 40-layer networks). With semi- supervised learning over unlabelled data,
able improve state-of-the-an certified robustness on CIFAR-10 at = 108 108/25, /55 from 36.04% to 42.39%. addition, LOT consistently outperforms baselines different model architectures with only 1/3 evaluation time.
Introduction
Given the wide applications of deep neural networks (DNNs), ensuring their robustness against potential adversarial attacks 77] [1 31 32 is of great importance. There has been a line of research providing defense approaches to improve the empirical robustness of DNNs [88
29,, and
certification methods to certify DNN robustness [14] 35 15
13 33,, Existing certification techniques can categorized as deterministic and probabilistic certifications [14, and in this work we will focus on improving the deterministic & certified robustness by training 1-Lipschitz DNNs.
Although different approaches have been proposed to empirically enforce the Lipschitz constant of the trained model (26). it is still challenging to strictly ensure the I-Lipschitz, which can lead to tight robustness certification. One recent work SOC (23 proposes parametrize the orthogonal weight matrices with the exponential of skew-symmetric matrices (i.e. W exp(V VT)). However, such parametrization will be biased when the matrix norm is constrained and the expressiveness is limited, especially when they rescale V to be small to help with convergence. In this work, we propose a layer-wise orthogonal training approach (LOT) by parameterizing the orthogonal weight matrix with an unconstrained matrix = (VVT) à V. order to calculate the inverse square root for convolution kernels, we will perform Fourier Transformation and calculate the inverse square root of matrices on the frequency domain using Newton's 's iteration. our parametrization, the output is agnostic to the input norm (i.e. scaling V aV does not change the value of W). We show that such
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

parametrization achieves higher model expressiveness and robustness (Section 6.2 and provides more meaningful representation vectors (Section 6.3
In addition, several works have shown that semi-supervised learning will help improve the empirical robustness of models under adversarial training and other settings 4. In this work, we take the first attempt to bridge semi-supervised training with certified robustness based on our I-Lipschitz DNNs. Theoretically, we show that semi-supervised learning can help improve the error bound of Lipschitz-bounded models. We also lower bound the certified radius as a function of the model performance and Lipschitz property. Empirically, we indeed observe that including un-labelled data will help with the certified robustness of 1 I-Lipschitz models, especially at a larger radius (e.g. from 36.04% 42.39% at p 108/255 on CIFAR-10).
We conduct comprehensive experiments to evaluate our approach, and we show that LOT significantly outperforms the state-of-the-an in terms of the deterministic l2 certified robustness. We also conduct different ablation studies to show that (1) (1) LOT can produce more meaningful features for visualization; (2) residual connections help to smoothify the training of LOT model.
Technical contributions, In this work, we aim to train a certifiably robust I-Lipschitz model and also analyze the certified radius of the Lipschitz bounded model under semi-supervised learning.
We propose layer-wise orthogonal training method LOT for convolution layers to train 1-Lipschitz models based on Newton's 's iteration, and thus compute the deterministic certified robustness for the model. We prove the convergence of Newton's 's iteration used in our algorithm.
We derive the certified robustness of lipschitz constrained model under semi- remi-supervised setting, and formally show how semi-supervised learning affects the certified radius.
We evaluate our LOT method under different settings (i.e. supervised and semi-supervised) on different models and datasets. With supervised learning, we show that it significantly outperforms state-of-the ofrthh-art baselines, and on some deep architectures the performance gain over 4%. With supi-supe learning, we further improve certified robust accuracy by over 6% at a large radius.
2 Related Work
Certified Robustness for Lipschitz Constrained Models Several studies have been conducted to explore the Lipschitz-constrained models for certified robustness. 26 first certifies model robustness based on its Lipschitz constant and propose training algorithms to regularize the model Lipschitz. Multiple works (5 19 20
have been proposed to achieve I-Lipschitz during training for linear networks by regularizing or normalizing the spectral norm of the weight matrix. However, when applying these approaches on convolution layers, the spectral norm is bounded by unrolling the convolution into linear operations, which leads to loose Lipschitz bound [27]. Recently, 2 shows that the 1 I-Lipschitz requirement is not enough for a good robust model; rather, the gradient-norm. preserving property is important. Besides these training-time techniques, different approaches have been proposed to calculate a tight Lipschitz bound during evaluation. [6] upper bounds the Lipschitz with semi-definite programming while 111 upper bounds the Lipschitz with polynomial optimization. In this work we aim to effectively train 1-Lipschitz convolution models.
Orthogonal Convolution Neural Networks [16] first proposes directly construct orthogonal convolution operations. Such operations are not only I-Lipschitz, but also gradient norm preserving. which provides a higher model capability and smoother training process (2). BCOP [16] trains orthogonal convolution by iteratively generating 2 x 2 orthogonal kernels from orthogonal matrices. orthogonal convolution with Cayley transformation W =[T- -T- VV-V- V VT)-1 where the convolution inverse is calculated on the Fourier frequency domain. ECO (36] proposes to explicitly normalize all the singular values [22 of convolution operations be 1. So far, the best best-performing orthogonal convolution approach is SOC [23], where they parametrize the orthogonal convolution with W exp(V VT) where the exponential and transpose are defined with the convolution operation. However, one major weakness of SOC is that it will rescale V to be small so that the approximation of exp can converge soon, which will impose a bias on the resulting output space. For example, when V is very small W will be close to I. Such norm norm-dependent property is not desired, and thus we propose a parametrization that is invariant to rescaling. Finally, 24 proposes several techniques for orthogonal CNNs, including a generalized Householder (HH) activation function, a certificate regularizer (CReg) loss, and last layer normalization (LILN). They can be integrated with our training approach to improve model robustness.
25 proposes to parametrize

---

3 Problem Setup
3.1 Lipschitz Constant of Neural Networks and Certified Robustness
Let f: IR - RC denote a neural network for classification, where d is the input dimension and C' is the number of output classes. The model prediction is given by arg max; (z) where / (x)) represents the prediction probability for class c. The Lipschitz constant of the model under p-norm [[(m))-((xall))
Vz1, 22 R.. Unless specified, we will focus on f2-norm and use Lip(f)) to denote (f) in this work. We can observe that the definition of model Lipschitz aligns with robustness property both require the model not to change much w.r.t imput changes. Formally speaking, define MM(x)) = max, f(z); maxj max; f(x);; f(r), to be the prediction gap of f on the input x; then we can guarantee that (Ix will not change its prediction within (x' x r, where
defined as: Lipp(() sup 1 1
r= ==M((e\(VVLipp(()
Therefore, people have proposed to utilize the model Lipschitz to provide certified robustness and investigated training algorithms train model with small Lipschitz constant.
Note that the Lipschitz constant of composed function f = /1 f2 satisfies Lip(/) S Lip(S1) x Lip(f2). Since neural network is usually composed of several layers, can investigate the Lipschitz of each layer to calculate the final upper bound. If we can restrict each layer to be 1 I-Lipschitz, then the overall model will be I-Lipschitz with an arbitrary number of layers.
3.2 Orthogonal Linear and Convolution Operations
Consider a linear operation with equal input and output dimensions = Wz, where æ, E R" and W E _=x=, We say W is orthogonal matrix if WWT WTW I and call y Wr an orthogonal linear operation. The orthogonal operation not only 1-Lipsccitz, but also norm- preserving, ie.. Warll ||a|| for all I If the input and output dimensions do not match, i.e. W E m where n F m. we say W is semi-ore orthooontoooa if either WWT = I or W =I The semi-orthogonal operation is 1-Lipschitz and non-expansive, i.e., |[ S ||2||2-
The orthogonal convolution operation is defined a similar way. Let y = W be an orthogonal operation, where y E Rexwx and W E Rexexkxk We say W is an orthogonal convolution kernel W o WT = WT W = Icone where the transpose here refers to the convolution transpose and Loone denotes the identity convolution kernel. Such orthogonal convolution - I-Lipschitz and norm-presse norm-preserrvng. When the input and output channel numbers are different, the semi-orthogonal convolution kernel W satisfies either W WT Tconv WT o W Tconv and it is 1-Lipschitz and non-expansive.
4 LOT: Layer-wise Orthogonal Training
In this section, we propose our LOT framework to achieve certified robustness ! introduce how LOT layer works to achieve I-Lipschitz. The key idea of our method is to parametrize an orthogonal convolution W with un-constrained convolution kernel W (VVT)- Av. Next, we will propose several techniques to improve the training and evaluation processes our model. Finally, we discuss how semi-supervised learning can help with the certified robustness of our model.
We will first
4.1 1-Lipschitz Neural Networks via LOT
Our key observation is that we can parametrize an orthogonal matrix W E Ruxn with an uncon- strained matrix V E R by W = (VVT)- D addition, this equation also holds true in the case of convolution kernel given any convolution kernel E Rexexkxk where e denotes the channel number and denotes the kernel size, can get orthogonal convolution kernel by W (V - V. where transpose and inverse square root with respect the convolution operations. The orthogonality of W can be verified by W WT = W o WT = Tegnes where is
'The code is available at htttttss://gittei wtt.://.coooooa-ssaaaaae htt.://b.e.o..eteeee.eee///eeeee wrt.:iisssssotttoaoa.a..oo.o Training

---

the identity convolution kernel. This way, we can parametrize an orthogonal convolution layer by training over the mononttrad parameter V.
Formally speaking, we will parametrize an orthogonal convolution layer by Y (V (VOVT)-HOV X, where X, Y E Rexwxw and w 2 k is the input shape. The key obstacle here is how to calculate the inverse square root of a convolution kernel. Inspired by [25] we can leverage the convolution theorem which states that the convolution in the imput domain equals the element-wise multiplication in the Fourier frequency domain. In the case of multi-channel convolution, the convolution corresponds to the matrix multiplication on each pixel location. That is, let FFT : == - C be the 2D Discrete Fourier Transformation and FFT be the inverse Fourier Transformation. We will zero-pad the input to w the original shape is smaller than w. Let X; FF) and V₁,- FF)(V3,) denote the input and kernel on frequency domain, then we have:
FFT(Y):... = (V.r,a,b Vii.a.) V...a..a...
in which multiplication, transpose and inverse square operations are matrix-wise. Therefore, can first calculate (Y) on the frequency domain and perform the inverse Fourier transformation get the final output.
We will use Newton's s iteration to calculate the inverse square root of the positive semi-definite matrix = V...a..V. V* V...a,s in differentiable way [17]. we initialize Yo - A and Zu = and perform the following update rule:
1 Ya+1 Y2(3T 17(33-ZgYe), Ze+1
1 - - ZxYx)
(1)
2
Z& will converge to A when 111-1110 Al < 1. The condition can be satisfied by rescaling the parameter V = aV, noticing that the scaling factor will not change the resulting W. In practice, we execute a finite number of Newton's 's iteration steps and we provide rigorous error bound for this finite iteration scheme Appendix to show that the error will decay exponentially. In addition, we show that although the operation is over complex number domain after the FFT, the resulting parameters will still be real domain, as shown below.
Theorem 4.1. Say J E Cm is unitary so that J'J and V = J for V E mm and V E Cmxmm Let F(V) (VV*)-&V be our transformation. Then F(V) JF(()
Proof. First, notice that VT = =V- = J J Second, we have: (V J'Ji = =(((JVーー))-)3}
so that
J. Thus, we have
- NvV= =(VV-)-AV
Remark. From this theorem it is clear that our returned value JF(V)J J* equals to the transformed version of the original matrix F(V) Rmxm and thus is guaranteed to in the real domain.
Circular Padding vs. Zero Padding When apply the convolution theorem to calculate the convolution Y = o X with Fourier Transformation, the result implicitly uses circular padding. However, in neural networks, zero padding is usually better choice. Therefore, we will first perform zero padding on both sides of input X zero_pape [ooo_pad(X) and calculate the resulting ypod Wa ped with Fourier Transformation. Thus, the implicit circular padding in this process will actually pad the zeros which we padded beforehand. Finally, we truncate the padded part and get our desired output Y = (aancte(YPadd) We empirically observe that this technique helps improve the model robustness shown in Appendix E.3

---

When Input and Output Dimensions Differ In previous discussion, we assume that the convolu- tion kernel V has equal input and output channels. the case W Roue X xkxk where Cout # Cia- we aim to get semi-orthogonal convolution kernel W (i.e. W o WT = conn if Cout < or WT o W Toon if Cout > Cin). As pointed out in 19, calculating W with Newton's s iteration will naturally lead to a semi- remi-orthogonal convolution kernel when the input and output dimensions differ.
Emulating a Larger Stride To emulate the case when the convolution stride is 2, we follow previous works [23] and use the invertible downsampling layer [10), in which the imput dimension Can will be increased by x4 times. Strides larger than 2 can be emulated with similar techniques if needed.
Overall Algorithm Taking the techniques we discussed before, we can get our final LOT convolu- tion layer. The detailed algorithm is shown in Appendix [8] First, we will pad the input prevent the implicit circular padding mechanism and pad the kernel so that they are in the same shape. Next, we perform the Fourier transformation and calculate the output on the frequency domain with Newton's iteration. Finally, we perform inverse Fourier transformation and return the desired output.
Several works have been proposed on orthogonal convolution layers with re-parametrization 25 23 among which the SOC approach via W exp(V VT) has achieved the best performance. Compared with SOC, LOT has the following advantages. First, the parametrization in LOT is norm- corm-independent. Rescaling V V will not change the resulting W. By comparison, V with a smaller norm in SOC will lead to a W closer to the identity transformation. Considering that the norm of V will be regularized during training (e.g. SOC will re-scale V to have small norm; people usually initialize weight to be small and impose 12-regularization during training), the orthogonal weight space in SOC may be biased. Second, we can see that LOT is able to model any orthogonal kernel W by noticing that (WWT) W is W itself; by comparison, SOC cannot parametrize all the orthogonal operations. For example, in the case of orthogonal matrices, the exponential of a skew-symmetric matrix only models the special orthogonal group (i.c. the matrices with +1 determinant). Third, directly handle the case when Ca.. # Cout. while SOC needs to do extra padding so that the channel numbers match. Finally, LOT is more efficient during evaluation, when only need perform the Fourier and inverse Fourier transformation as extra overhead, while SOC needs multiple convolution operations to calculate the exponential. We will further show quantitative comparisons with SOC in Section 6.2
Comparison with SOC
The major limitation of LOT is its large overhead during training, since we need to calculate Newton's 's iteration in each training step which takes more time and memory. In addition, we sometimes observe that Newton's 's iteration is not stable when we perform many steps with 32-bit precision. To overcome this problem, we will pre-calculate Newton's 's iteration with 64-bit precision during evaluation, will introduce in Section 4.2
4.2 Training and Evaluation of LOT
Smoothing the Training Stage In practice, we observe that the LOT layers are highly non-smooth with respect to the parameter V (see Section 6.3 Therefore, the model difficult to converge during the training process especially when the model is deep. To smooth the training, we propose two techniques. First, we will initialize all except bottleneck layers where Cin- Cout with identity parameter V I. The bottleneck layers where Gin = Cout will still be randomly initialized. Second, as pointed out in [12, residual connection helps with model smoothness. Therefore, for the intermediate layers. we will add the 1-Lipschitz residual connection y = Ar - (1 fz) Some work suggests that À here can be trainable 225 while we observe that setting = 0.5 is enough.
Speeding up the Evaluation Stage Notice that after the model is trained, the orthogonal kernel W will no longer change. Therefore, we can pre-calculate Newton's 's iteration and use the result for each evaluation step. Thus, the only runtime overhead compared with a standard convolution layer during evaluation is the Fourier transformation part. In addition, we observe that using 64-bit precision instead of the commonly used 32-bit precision in Newton's 's iteration will help with the numerical stability. Therefore, when we pre-calculate W. we will first transform V into float64. After Newton's iterations, we transform the resulting W back to float32 type for efficiency.
5

---

4.3 Semi-supervised Learning
Existing works have shown that semi- remi-supervised learning helps improve empirical robustness 4 while the impact of semi superviiee learning on certified robustness has not been explored yet. We aim bridge this gap and investigate whether applying semi-supervised learning also benefits the certified robustness of Lipschitz constrained models. As we will discuss in Section [.1] we theoretically show that the certified robustness of Lipschitz-bounded model can be improved by semi-supervised learning. In practice, suppose we have (small) labelled dataset DD = { Ui)} } and a (large) unlabelled dataset ,,, = { we will first train a labeller model Gyr over the labelled dataset DD. Then we use Gpt to assign pseudo-labels to the unlabelled dataset to get D', " Finally, we use the overall dataset D. + D',, to train the student model G as the final model. We
{(zi, (z))
assume that both Gz and G' will use the LOT model architecture.
5 Theoretical Analysis
In this section, we will introduce our theoretical analysis to show that the certified robustness of Linschitz-constrainedd models will benefit from semi-supervised learning. We make similar assumptions with [28] which provides error bound of discrete models in semi-supervised learning, while we will analyze a (Lipschitz-bounded) continuous model and its certified accuracy.
We first introduce data expansion proposed in 28 which we will adopt in our analysis. Let T denote some set of transformations and define B A {x' E T such that ||= - T())] < r to be the set of points within distance r from any of the transformation of x. The neighbourhood of x is defined by N = (z)) n (xx) + 0}. Let P denote the data distribution and P. denote the distribution of data in the i-th class. The expansion property is defined
Definition 5.1 ((a, e)-exnssnoon, as in (28 ). We say that the class-conditin distribution P; satisfies (a, exxpannion if for all V C with PV( < a, the following holds:
P.(N(V)) 2 min min(cP.(V),)
if P. satisfies (a, -)-exannsionn for all classes in P, then we say P satisfies (a, -x-exannsion.
Next, we define the classifier and the loss function the continuous case. For simplicity, we assume a binary classification task here.
Definition 5.2 (Continuous model G and loss functions), We assume a binary classification task,
where G* (z), (z) E {0, 1} are discrete labels, and the trained model is G(z) [0, 1]. We define the loss function Lm! (G, G*) E [0, 1] to be:
T.m (G, G*) = Exex (z)) (x)(1-G(a)) - (1-
and the error of defined as Errm (G) Lm (G,
Remark. Here we provide general setting here, and if assume G(x) (0,1), Lm G*) becomes the 0-1 loss in 28
Then we define the marginal consistency set ST (G; and loss RT (G; in the continuous case. The consistency set includes cases in which the model prediction will not change lot within the neighbourhood, and the consistency loss measures the probability that the model is not consistent over the population.
Definition 5.3 (Marginal consistency set and consistency loss). We define the set ST (G; in which the ratio of marginal prediction change on each input x is no larger than 8 within its neighbourhood:
G(x) G(x²)
G(x) 2 Val E B(x)).
SB
{at
2 5 and
1 - G(x²)
and (B; 8) denotes the marginal consistency loss of the probability that x is not in ST (G;6):
RB 5) P & (G;
Remark. (1) We may use (G) and RT (G) for abbreviation when the choice of å does not lead to confusion; (2) when 6 0, (G) and RB (G) requires that the prediction exactly the same within the neighbourhood, which are reduced to Sy (G) and RG) defined in (28).

---

Finally, we define the certified robustness radius CertR(G) as the average radius over the population in which the model keeps its correct prediction:
CertR(G) (G) BrEx supp S.t.V/l='' - =1l2 < r, =
G -())
Note that when G(r) + G* (z:), the certified radius will O.
5.1 Error Bound and Certified Radius of Lipschitz-bounded Model
Let M(Gpt) = fx: Gpr(z) & (x) denote the set in which the pseudolabel is wrong. We will have similar separation and expansion assumptions on the data distribution as in (28).
Assumption 5.1 (Separation). We assume that is separable with probability 1 - by ground-truth classifier G", i... RB S μ..
be the maximum fraction of incorrectly pseudolabeled examples in any class. We assume that a < 1/3 and P satisfies (a, e)- expansion for ē > 3. We express our bounds in terms of - min /inn1//0.0 Z).
Assumption 5.2 (Expansion). Define & max (MM())))
With these assumptions and let d [0, 2.1, we compare the performance of G and Gpt (i.e. the performance with and without semi-supervised learning) with the following theorem: Theorem 5.1 (Error bound with semi superviee learning). Suppose Assumption 5.1 and 5.2 holds and let G be a minimizer of the following loss function:
G = arg min L(G)= c Lm (G, Gpt)
2c +2+ 2
+ c- RB (G; Err(Gpt)
Then we can upper bound the error of G by:
2c
Errm (G) S c- Emr(ppt)
c-
In addition, we can bound the certified robustness radius w.r.t. the Lipschitz constant below:
Theorem 5.2 (Certified radius with semi-supervised learning). Suppose Assumption [5. holds. Then the certified radius of a model G can be bounded by:
0.5
c+3 e-1
Lm (G, Gpt) 20+2 RT (G:8) )rr(Gp)))
CerlR(G)
c-1
Lip(G)
The proofs of Theorem 5.1 and 5.2 are shown in Appendix
Discussion From Theorem 5.1 we observe that the benefits of semi-supervised learning depend on the error of Gpt and data property. If the data expands and separates well and GpJ accurate, c is large and y is small, which means Errm (G) will be smaller than Err(G Note that is usually small and we assume >> 3, (Gr Err(G,t) will hold true in most cases, indicating that semi-supervised learning helps improve the model performance. From Theorem 5.2 we can observe that given the data distribution and Gpt. the certified radius of the student model CertR (G)) can be power-bounded by 1) how well G learns from Gpt (Lm(G. Gp))): 2) how consistent is (R (G; 6)).
These two factors also correspond to the losses in the actual training process the cross-entropy loss measures how well G learns from Gpt and the Lipschitz constraint measures local consistency for a Lipschitz-bounded mode 2
Evaluation
In this section, we will evaluate our method LOT on different datasets and models compared with the ofate-of-the-art baselines based on the deterministic l certified robustness. We show that on both CIFAR-10 and CIFAR-100, LOT achieves significantly higher certified accuracy under different radii and settings with more efficient evaluation time. In the meantime, we conduct series of ablation studies to analyze the representation power of our LOT, the error control of Newton's 's iterations, and the effects of residual connections. We show that under the semi-supervised setting, the certified accuracy of LOT is further improved and outperforms the baselines.
"This means that, with smaller Lipschitz, the model is more likely to be consistent within neighbourhood. In addition, the CReg loss will also improve local consistency by maximizing the prediction gap.
7

---

Table I: Certified accuracy I-Lipschitz model with CReg loss and HH activation on CIFAR-10/100 in supervised setting. The LLN technique is applied on the CIFAR-100 dataset.
CIFAR-10 10
CIFAR-100
Mean Evaluation Time (sec)
Model
Cony, Type
Vanilla Accuracy
Certified Accuracy at
Vanilla Accuracy
Certified Accuracy
36/255
72/255
108/255
36/255
72/255
108/255
LipConvnet-5
SOC
75.31%
60.37%
45.62%
32.38%
45.82%
32.99%
22.48%
14.79%
2.285
LOT
76.34%
62.07%
47.52%
33.99%
48.38%
34.77%
23.38
15.44%
1.411
LipConvnet-Ii 10
76.21%
62.37%
47.70%
54.15%
34.31%
23.50%
15.66%
3.342
LOT
76.50%
63.12%
48.59 %
35.68%
48.70%
34,82%
23.86 %
15.93%
1,563
LipConvnet- 15
SOC
76.39%
62.96%
48.47%
35.47%
47.61%
34.54%
23.166
15.09%
4.105
LOT
76.86%
63.84%
49.18 %
36.35%
48.99%
34.90%
24.39 %
16.37%
1.627
SOC
76.34%
62.63%
48 69%
36.04%
47 47.84%
34.77%
23.70%
15.8 15.84%
5.142
LipConvnet-20
77.12%
64.30%
49.49%
36.34%
48.81
35.21%
24.37%
16.23%
1.962
LipConvnct-25
SOC
75.21%
61.98%
47.93%
34.92%
46,87%
34.09%
23.41%
15.61%
6.087
LOT
76.83%
64.49%
49.80 %
37.32%
48.93%
35.23%
24.33%
16.59%
2.387
LipConvnet-
SOC
74.23%
60.64%
46.51%
34.08%
46.92%
34.17%
23.21%
15.84%
6.927
1,TT
77.12%
64.36% %
49.98 %
37.30%
49.18%
35.54%
24.24%
16.48%
2.755
LipConvnet-S 35
SOC
74.25%
61.30%
47.60%
35.21%
46.88%
33.64%
23.34%
15.73%
7,870
LOT
76.91%
63.55%
49.05 %
39%
48.25%
34.99%
24.13%
16.25%
3.193
LipConvnct-40
72.59%
59.04%
44.92%
32.87%%
45.03%
32.57%
22.37%
14.76%
8, 8.068
LOT
76.75%
63.50%
49.07% %
36.06%
48.31%
34.59%
23.70%
15.94%
3.595
6.1 Experiment Setup
Baseline- SOC with CReg loss, HH activation and LLN technique In the evaluation, we will mainly compare with SOC [23], as SOC has been shown to outperform other I-Lipschitz models in terms of certified robustness. SOC parameterizes an orthogonal layer with W = exp VT) where V is the convolution kernel and the transpose and exponential are with respect to the convolution operation. Following their setting, we will focus on the CIFAR-10 and CIFAR- 100 datasets and the model architecture varies from LipConvnet-5 to LipConvnet-40.
24 proposes three techniques to improve SOC networks. First, they observe that adding a Certificate Regularization (CReg) loss can help with the certified performance at larger radius: (CReg(r; y) =
-[ (ff max f (xx))) /2). Second, they propose Householder (HH) activation layer, which is generalization of the standard GroupSort activation. Finally, for tasks with large number of classes (e.g. CIFAR- 100), they propose Last Layer Normalization(LLN to only normalize the final output layer instead of orthogonalize. Since these techniques are agnostic to the type of convolution operation, we will further integrate them with LOT for comparison.
Implementation Details We will train and evaluate the LOT network under supervised and un- supervised scenarios. Following previous works [23], we focus on the CIFAR- 10 and CIFAR- 100 datasets and the model architecture varies from LipConvnet-5 to LipConvnet-40. In semi-supervised learning, we use the 500K data introduced in 4 as the unlabelled dataset. To train the LOT network, we will train the model for 200 epochs using momentum SGD optimizer with an initial learning rate 0.1 and decay by 0.1 at the 100-th and 150-th epochs. We use Newton's 's iteration with 10 steps which we observe is enough for convergence (see Appendix (E.4). When CReg loss is applied, we use y = 0.5; when HH activation is applied, we use the version of order 1. We add the residual connection with a fixed = 0.5 for LOT: for SOC, we use their original version, as we observe that residual connections even hurt their performance (see discussions Section We show the certified accuracy at radius E {
255 255 255 ). For the evaluation time comparison, we show the
36 72 108
runtime taken to do a full pass on the testing set evaluated on an NVIDIA RTX A6000 GPU.
6.2 Supervised Scenario
We show the results of supervised learning on CIFAR- 10 and CIFAR-100 in Table [ We can observe that LOT significantly outperforms baselines on both vanilla accuracy and certified accuracy for different datasets. In particular, we observe that the improvement is more significant for deeper models. We owe it to the reason that our LOT layer has a better expressiveness and unbiased parametrization. "Mean evaluation time" column in Table records the evaluation time per instance averaged between CIFAR- 10 and CIFAR-100 models. Since architectures of CIFAR-10 and CIFAR- 100 models differ only in the last linear year, the evaluation time is almost the same. We observe that LOT has a better efficiency during the evaluation stage, since LOT pre-calculates Newton's 's iteration while SOC calculates the exponential in each iteration. A similar conclusion can be observed from results without CReg loss, HH activation and LLN, shown in Table and in Appendix E.5
8

---

Table Performance comparison of I-Lipschitz networks with and without residual connections.
Conv. Type
Residual Connection
Vanilla Accuracy
Certified Accuracy at
Model
36/255
72/255
108/255
76.90%
61.87%
45.79%
31.08%
SOC
76.97%
61.76%
45.46%
30.29%
LipConvnet-20
77. 16% 77.86 %
62.14% 63.54%
45.78%
30.95%
LOT
47.15%
32.12%
Figure 1: Visualizing features the last hidden layer of LipConvnet-22 SOC (left) and LOT (right). Each image corresponds to one randomly chosen neuron from the last hidden layer and optimized maximize the value of the neuron.
6.3 Ablation Studies
In this section, we will perform several ablation studies for LOT. Unless specified, we will use the deep model LipConvnet-20 for evaluation. We the model without CReg loss and HH activation so that we can see the comparison under the standard CNN setting.
Representation analysis Recent works suggest that adversarially robust models will have good regularization-free feature visualization [.. We show the visualization of neurons in the last hidden layer of LipConvNet-20 for both SOC and LOT in Figure More figures are shown in Appendix E. The visualization process is as follows: given a chosen neuron, we will start from random image and take 500 gradient steps maximize the neuron value with step size 1.0 and decay factor 0.1. We can see that LOT indeed generates more meaningful features. We attribute to the good expressiveness power of our model. This indicates that approach indeed leads to better representations.
Error Control Newton's Iterations To see how our Newton's 's iteration approximates the inverse square root, we visualize the maximum singular value (amox) of LOT layers in Appendix E.4] We observe that all the resulting layers have strictly < Lipschitz. Therefore, we can safely conclude that the overall LOT network is I-Lipschitz.
Effect residual connections Empirically, we ob- serve that LOT is non-smooth and therefore need the residual connection to smoothify the model. To verify this phenomenon, approach in [12] and show the loss surface in Fig- [ We can see that LOT without residual connec- tion is highly non-smooth, while after adding resid- ual connections the model becomes much smoother. By comparison, we observe that SOC networks with and without residual connection similar (see Ap- pendix B.2). We hypothesize the reason to be that the exponential parametrization W exp(A) enables residual property implicitly, i.e.,
adopt the visualization
(a) LOT, without residual connection.
(b) LOT, with residual connection.
Figure 2:
with respect to the parameters of LOT with and without residual connections. The figure is plotted by calculating the loss contour two randomly chosen directions of parameters.
The loss landscape [2
1 2
exp A)
X
Ao
X
AoAoX+
where the first two terms are essentially the format of residual connection. We show the results of models with and without residual connections in Table We can observe that our LOT indeed improves with the residual connection, while the performance remains similar or even worse for SOC. Therefore, in the evaluation, use the LOT with residual connection and SOC without it.
Robustness against &ox Empirical Attack To evaluate the robustness of our model against attacks in other fp norms, we evaluate the model against standard f.2 PGD attack [18 with 8/2255 and 50 steps. We show the results on CIFAR-10 with CReg loss and HH activation in Table We can observe that still achieve better empirical robustness the LipConvNet compared with SOC.

---

Table 3: Empirical robust accuracy of I-Lipschitz model with CReg loss and HH activation on CIFAR-10 against &o -PG attack at = 8/ /255..
Model
Cony.
Vanilla accuracy
Robust accuracy
SOC
75.31%
27.18%
LipConvnet-5
LOT
76.34%
27.19%
SOC
76.23%
27.95%
LipConvnet-10
LOT
76.50%
28.71% %
SOC
76.39%
27.84%
LipConvnet-15
LOT
76.86% %
29.10 %
SOC
76.34%
26.17%
LipConvnet-20
LOT
77.12%
29.45%
SOC
75.21% %
28.93%
LipConvnet-25
LOT
76.83%
29.55 %
SOC
74.23%
27.44%
LipConvnet-30
LOT
77.12%
28.92 %
SOC
74.25%
14.09%
LipConvnet-35
LOT
76.91 %
29.32 %
SOC
72.59%
11.68%
LipConvnet-40
LOT
76.75%
28.67 %
Table 4: Performance of 1-Lipschitz networks on the CIFAR-10 dataset with semi-supervised training. Results of different architectures are shown in Table 8 Appendix E.6
Conv. Type
Vanilla Accuracy
Certified Accuracy at p
Model
Setting
36/255
72/255
108/255
Supervised Semi-supervised
76.34%
62.63%
48.69%
36.04%
SOC
70.95%
61.72%
51.78%
42.01
LipConvnet-20
Supervised
77.12 %
64.30%
49.49%
36.34%
LOT
Semi-supervised
71.86%
62.86%
52.24%
42.39%
6.4 Semi-supervised Learning Scenario
We show the results of remiisupprvised learning for one architecture with CReg loss and HH activation in Table The full results are shown in Table S and in Appendix [.6]] We can observe that, with semi-supervised learning, the vanilla accuracy will drop slightly, which will impact the certified accuracy at small radius (p 36/ 36/255). However, the certified accuracy at a large radius will be improved, and the gap is more significant at larger radius (e.g. p 108 (08/255). We improve the previous trae-offthhe-ar certified accuracy by over 6% at = 108 /255. This indeed shows that sepi-super learning can help improve certified robustness. We empirically observe that seni-supervised learning does not help much on CIFAR-100. We owe it to the reason that the vanilla accuracy of the teacher model on CIFAR- 100 is low (less than 50%). Thus, according to Assumption 5.2 and Theorem 5.2 when is low, the certified accuracy would be low.
7 Conclusion
In this work, propose orthogonal convolution layer LOT and build a I-Lipschitz convolution network. We show that LOT network outperforms the previous state-of-the-art certified robustness. We also show that semi-supervised learning can further help with the robustness both theoretically and experimentally.
Acknowledgements
This work partially supported by the NSF grant No. 1910100, NSF CNS No. 2042666,,6, C3 AI, and the Alfred P. Sloan Foundation.
10

---

References
[1] Zeyuan Alu and Yuanzhi Li. Feature purification: How adversarial training performs robust deep learning. In 2021 IEEE 62nd Annual Symposium Foundations of Computer Science (FOCS), pages 977-988. IEEE, 2022.
[2] Cem Anil, James Lucas, and Roger Grosse. Sorting out lipschitz function approximation. In International Conference Machine Learning, pages 291-301. PMLR, 2019.
[3] Dario A Bini, Nicholas J Higham, and Beatrice Meini. Algorithms for the matrix pth root. Numerical Algorithms, 39(4):349-377 078, 2005.
[4] Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John Duchi, and Percy S Liang. Unlabeled data improves adversarial robustness. Advances Neural Information Processing Systems, 32, 2019.
[5] Moustapha Cisse, Piotr Bojanowski, Edouard Grave, Yann Dauphin, and Nicolas Usumier. Parseval networks: Improving robustness to adversarial examples. In International Conference on Machine Learning, pages 854-863. PMLR, 2017.
[6] Mahyar Fazlyab, Alexander Robey, Hamed Hassani, Manfred Morari, and George Pappas. Efficient and accurate estimation of lipschitz constants for deep neural networks, Advances in Neural Information Processing Systems, 32, 2019.
[7] lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. arXiv preprint arXiv: 1eXiv:1412.65777 2014.
[8] Henry Gouk, Eibe Frank, Bernhard Pfahringer, and Michael J Cree. Regularisation of neural networks by enforcing lipschitz continuity. Machine Learning, 110(2):933-46, 2021.
(9] Lei Huang, Li Liu, Fan Zhu, Diwen Wan, Zehuan Yuan, Bo Li, and Ling Shao. Controllable orthogonalization in training dnns. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6429-6438, 2020.
[10] Jöm-Henrik Jacobsen, Arnold Smeulders, and Edouard Oyallon. i-revnet: Deep invertible networks. arXiv preprint arXiv: 07008,, 2018
[11] Fabian Latorre, Paul Rolland, and Volkan Cevher. Lipschitz constant estimation of neural networks via sparse polynomial optimization. arXiv preprint arXiv:2004. 08688, 2020.
[12] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in neural information processing systems, 31, 2018.
[13] Linyi Li, Maurice Weber, Xiaojun Xu, Luka Rimanic, Bhavya Kailkhura, Tao Xie, Ce Zhang, and Bo Li. Tss: Transformation-speeii smoothing for robustness certification. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, 2021.
[14] Linyi Li, Tao Xie, and Bo Li. Sok: Certified robustness for deep neural networks. In 44th IEEE Symposium on Security and Privacy, SP 2023, San Francisco, CA, USA, 22-26 May 2023. IEEE, 2023.
[15] Linyi Li, Jiawei Zhang, Tao Xie, and Bo Li. Double sampling randomized smoothing. In International Conference on Machine Learning. 2022.
[16] Qiyang Li, Saminul Haque, Cem Anil, James Lucas, Roger B Grosse, and Jörn Hen-ik Jacobsen. Preventing gradient attenuation in lipschitz constrained convolutional networks. Advances in neural information processing systems, 32, 2019.
[17] Tsung- Yu Lin and Subhransu Maji. Improved bilinear pooling with cnns. In Gabriel Brostow Kaun Kim, Stefanos Zafeiriou and Krystian Mikolajezyk, editors, Proceedings of the British Machine Vision Conference (BMVC), pages 117.1-117.12. BMVA Press, September 2017.
[18] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 777770606000... 2017.
11

---

[19] Takeru Miyato, Toshiki Kataoka, Masanori Koyama. and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv: 2018.
[20] Haifeng Qian and Mark Wegman. L2-nonexpansive neural networks. arXiv preprint 1eXiv:8802.0790 2018.
[21] Haonan Qiu, Chaowei Xiao, Lei Yang, Xinchen Yan, Honglak Lee, and Bo Li. Semanti- cadv: Generating adversarial examples via attribute-conddtionee image editing. In European Conference Computer Vision, pages 19-37. Springer, 2020.
[22] Hanie Sedghi, Vincet Gupta, and Philip M Long. The singular values of convolutional layers. arXiv preprint 1rXiv:1805.144000 2018
[23] Sahil Singla and Soheil Feizi. Skew orthogonal convolutions. In International Conference on Machine Learning, pages 9756 9766-9766. PMLR, 2021.
[24] Sahil Singla, Surbhi Singla, and Soheil Feizi Improved deterministic 12 robustness on cifar- 10 and c00 In International Conference on Learning Representations, 2021.
[25] Asher Trockman and Zico Kolter. Orthogonalizing convolutional layers with the cayley transform. In International Conference on Learning Representations, 2021.
[26] Yusuke Tsuzuku, Issei Sato, and Masashi Sugiyama. Lipschitz-margin training: Scalable certification of perturbation invariance for deep neural networks. Advances in neural information processing systems, 31, 2018.
[27] Jiayun Wang. Yubei Chen, Rudrasis Chakraborty, and Stella X Yu. Orthogonal convolutional neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11505-11515, 2020.
[28] Colin Wei, Kendrick Shen, Yining Chen, and Tengyu Ma. Theoretical analysis of self-training with deep networks on unlabeled data. In International Conference on Learning Representations, 2020.
[29] Chaowei Xiao, Ruizhi Deng, Bo Li, Taesung Lee, Benjamin Edwards, Jinfeng Yi, Dawn Song, Mingyan Liu, and Ian Molloy. Advit: Adversarial frames identifier based on temporal consistency in videos. In Proceedings of the IEEE/CVP International Conference on Computer Vision, pages 3968-3977, 2019.
[30] Chaowei Xiao, Ruizhi Deng, Bo Li, Fisher Yu, Mingyan Liu, and Dawn Song. Characterizing adversarial examples based on spatial consistency information for semantic segmentation. In Proceedings of the European Conference Computer Vision (ECCV), pages 217-234, 2018.
[31] Chaowei Xiao, Bo Li, Man Zhu, Warren He, Mingyan Liu, and Dawn Song. Generating adversarial examples with adversarial networks. arXiv preprint arXiv: 0001100010211 2018.
[32] Chaowei Xiao, Jun Yan Zhu, Bo Li, Warren He, Mingyan Liu, and Dawn Song. Spatially transformed adversarial examples. In 6th International Conference on Learning Representations, ICLR 2018, 2018
[33] Zhuolin Yang, Linyi Li, Xiaojun Xu, Bhavya Kailkhura, Tao Xie, and Bo Li. On the certi- fied robustness for ensemble models and beyond. In International Conference on Learning Representations, 2022.
[34] Zhuolin Yang, Linyi Li, Xiaojun Xu, Shiliang Zuo, Qian Chen, Pan Zhou, Benjamin I P Rubinstein, Ce Zhang, and Bo Li. Trs: Transferability reduced ensemble via promoting gradient diversity and model smoothness. In Advances in Neural Information Processing Systems, 2021.
[35] Zhuolin Yang, Zhikuan Zhao, Boxin Wang, Jiawei Zhang, Linyi Li, Hengzhi Pei, Bojan Karlaš, Ji Liu, Heng Guo, Ce Zhang, and Bo Li. Improving certified robustness via statistical learning with logical reasoning. NeurlPS, 2022.
[36] Tan Yu, Jun Li, Yunfeng Cai, and Ping Li. Constructing orthogonal convolutions in an explicit manner. In International Conference on Learning Representations, 2021.
12


Reviews:
[{"review_id": "QVSbUL8Su8V", "reviewer": "Reviewer_miXu", "summary": "This paper proposes a training framework named Layer-wise Orthogonal Training (LOT) that aims to train 1-Lipschitz convolution layers effectively and improves the certified robustness of Lipschitz-bounded models. The paper also proves that semi-supervised learning can benefit the robustness of Lipschitz-bounded models. The key idea of LOT is to utilize unconstrained matrices to approximate an orthogonal matrix and to speed up the inverse computation by mapping the matrices to the FFT domain. The experiments are done on CIFAR-10/100, taking LipConvnet with a different number of layers as the backbone, showing LOT outperforms its directly competing framework called SOC. The main contributions are:\n1.\tLOT parametrizes the orthogonal weight matrix W with an unconstrained matrix V, and uses FFT to calculate the inverse square root of matrices faster in the frequency domain.\n2.\tThis paper derives the certified robustness of the Lipschitz constrained model under the semi-supervised setting.\n3.\tLOT improves certified L2 robust accuracy against SOC.\n", "questions": "Further comments:\n1. Sect. 5 on the theory is quite involved in terms of math, while its conclusion/discussion is rather trivial. It sounds a bit like “if it’s good then it’s good”.\n2. The same issue happens for Fig. 1 and its discussion in Sect. 6.3, too. Why are the features from LOT “more meaningful” than those from SOC? Simply because the patterns have higher amplitudes and fluctuations?\n3. Upper half of p.4 shows the FFT-domain operations. I suppose the matrices are holding complex numbers? How does that affect the complexity and stability of the Newton iterations? And how do you guarantee real matrices when mapped back to the spatial domain?\n4. I wonder if your Lipschitz constraint can be imposed on the doubly block Toeplitz convolution operator in the OCNN paper [24] and alike? So that your convolution doesn’t need to be in multiple steps with high complexity and low numerical stability?\n5. One very key issue I have spotted from the paper is the lurking instability in training/inference, as the authors need to deliberately jet up the accuracy to FP64 during training/inference and then trim back to FP32. This is likely due to the inverse operation which can become ill-conditioned. This may also be why only CIFAR10/100 examples are used, without moving on to the larger sizes like 224^2 ImageNet examples? In fact, do you have larger examples beyond 32^2 input sizes?\n6. Following from above, it sounds impossible to quantize your network to INT8, INT4 or even binary as in other CNN compression works?\n7. Last but actually the most important, the experiments are only conducted on LipConvnet which is NOT conventionally used in practice. People in the field know very well that robustness is very much architecture-dependent, where transformers exhibit both high accuracy and robustness. The authors may want to study the leaderboard at http://robust.art/ , to see how real-world networks perform under field tests on ImageNet-class inputs. This in turn renders the LipConvnet characterization of this work purely theoretical yet, frankly, with little practical value as the generalization of the results to the more common modern architectures is doubtful (ConvNeXt, Transformers, etc.). \n\nEditorial:\n1. It is inappropriate to write V=\\alpha V in several places throughout the paper.\n2. line 65. “[23] first certify …”, certifies\n3. line 70, “Recently, [2] show that …”, shows\n", "limitations": "See my questions above.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1. The scale-invariant convolution kernel parametrization can improve robustness further than SOC.\n2. The improvement in the Lipschitz is backed by analytical derivations.\n\nWeaknesses: \n1. The derivation of this paper is specific to l2 attacks only, while attacks are conventionally taking l1, l2 and l_\\infty norms. In fact, the benchmarked SOC paper also shows empirical robustness in other norms.\n2. Only CIFAR10/100 test results are provided, which limits the practicality of the framework.\n3. In Sect. 3.2, the semi-orthogonality in the linear operation context is fine, but then its convolution counterpart sounds hand-waving. Is the latter mathematically equivalent to its linearized counterpart? Can you use a toy example to show how to bridge the former to the latter? If they are not directly corresponding, then a proof that conv(W, W^T)=I_{conv} is needed.\n4. The definition of W in terms of V requires multiple steps of convolution plus inversion in the FFT domain (and thus complex numbers). I wonder, once V is trained and fixed, why can’t W be computed exactly in terms of V, and then the convolution be done in one step using W? Is that due to non-exactness in the orthogonality or what?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5c4WJFv5j8", "reviewer": "Reviewer_YdF6", "summary": "The paper proposes a method to train a 1-Lipschitz neural network to achieve improved certified L2 robustness by ensuring that the weight matrix in each layer is orthogonal. The weight matrix W is parameterized using an unconstrained matrix V as W = (V V^T)^(-1/2) V to ensure orthogonality. This parameterization is invariant to rescaling of the unconstrained matrix V, which the authors show leads to better robustness and expressiveness of the model. The paper also shows how to make convolution operations orthogonal. Lipschitz constraints have been shown to improve the empirical robustness of a model. The aim of this work is to show that such models can achieve better certified robustness as well under the semi-supervised learning setting.", "questions": "1. The matrix V does not really seem to be unrestricted since it needs to satisfy the condition: || I - VV^T|| < 1. Also, it is not clear to me how scaling V can always make it satisfy the condition as stated in line 154, e.g., consider V = zero matrix in a fully connected layer.\n2. How do we initialize V? Do we set it to a particular type of matrix such as an orthogonal matrix?\n3. How do we ensure the condition on V is satisfied after each training step?\n", "limitations": "There does not seem to be any negative societal impact of this work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality: The methods used in the paper such as parameterizing orthogonal weight matrices with unconstrained matrices have been considered before. However, the paper presents a new kind of parameterization that has desirable properties such as scale invariance.\n\nQuality: The paper considers several different convolution operations such as circular padding, zero padding, larger strides, etc and shows ways to orthogonalize them. The paper performs a theoretical analysis to demonstrate the benefits of semi-supervised learning for certified robustness. The method has been evaluated and compared with a previous paper in this area for different model architectures.\n\nClarity: The paper is clearly written and easy to read.\n\nSignificance: The method achieves improved performance over the existing approach of Skew Orthogonal Convolutions (SOC).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5DZstejo80l", "reviewer": "Reviewer_zFGT", "summary": "This work proposed a method that utilizes the convolution theorem and Newton iteration to achieve 1-Lipschitz in convolutional layers of CNN and theoretically justified the effectiveness of semi-supervised training on improving empirical robustness.", "questions": "My major concerns of the paper are stated in the weaknesses section. I would appreciate it and am willing to raise my score if the authors could illustrate more the comparisons between at least [1][2][3] and the proposed method, both theoretically (could be intuitive) and empirically. \n\n[1] Asher Trockman and J Zico Kolter. Orthogonalizing convolutional layers with the Cayley transform. In International Conference on Learning Representations, 2021.\n\n[2] Liu, et al. Convolutional Normalization: Improving Deep Convolutional Network Robustness and Training. In Advances in Neural Information Processing Systems, 2021.\n\n[3] Huang, et al. Controllable orthogonalization in training DNNs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The overall presentation of the work is clear and easy to follow. Although the reviewer did not check all the proofs, the theoretical side of the paper is complete and carefully organized. The paper also talks about the actual implementation (e.g., dealing with padding/stride, etc) and the limitations of the method in great detail.\n\nWeaknesses: \n1. The novelty of the method is somewhat limited. For example, calculating things in the Fourier domain is not new, at least the Cayley transform paper [1] cited in the work and [2] utilize similar idea; also, using Newton iteration to approximate matrix inversion is also explored in previous works [3] of the same domain. \n\n2. The experimental results are not super convincing. The paper mainly conducts experiments to compare with SOC and claims that \"SOC has been shown to outperform other 1-Lipschitz models\". However, the SOC paper mostly only compares with BCOP. I believe a more comprehensive comparison here would make the paper more convincing (at least compare with [1] and [3] which inspired this work?)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept