Exploiting Connections between Lipschitz Structures for Certifiably Robust Deep Equilibrium Models
Authors: Aaron J Havens
Alexandre Araujo
Siddharth Garg
Farshad Khorrami
Bin Hu
Venue: NeurIPS 2023 poster (2023)
Keywords: Deep equilibrium models
Lipschitz networks
certified robustness
TL;DR: We delineate the connections between various Lipschitz parameterizations of DEQ and existing state-of-the-art explicit Lipschitz networks to improve our understanding of achieving better certified robustness for DEQ.

Abstract: Recently, deep equilibrium models (DEQs) have drawn increasing attention from the machine learning community. However, DEQs are much less understood in terms of certified robustness than their explicit network counterparts. In this paper, we advance the understanding of certified robustness of DEQs via exploiting the connections between various Lipschitz network parameterizations for both explicit and implicit models. Importantly, we show that various popular Lipschitz network structures, including convex potential layers (CPL), SDP-based Lipschitz layers (SLL), almost orthogonal layers (AOL), Sandwich layers, and monotone DEQs (MonDEQ) can all be reparameterized as special cases of the Lipschitz-bounded equilibrium networks (LBEN) without changing the prescribed Lipschitz constant in the original network parameterization. A key feature of our reparameterization technique is that it preserves the Lipschitz prescription used in different structures. This opens the possibility of achieving improved certified robustness of DEQs via a combination of network reparameterization, structure-preserving regularization, and LBEN-based fine-tuning. We also support our theoretical understanding with new empirical results, which show that our proposed method improves the certified robust accuracy of DEQs on classification tasks. All codes and experiments are made available at \url{https://github.com/AaronHavens/ExploitingLipschitzDEQ}.

Exploiting Connections between Lipschitz Structures for Certifiably Robust Deep Equilibrium Models
Aaron Havens*
Alexandre Araujo
Siddharth Garg²
Farshad Khorrami
Bin Hu?
CSL & ECE, University Illinois Uramna-Champaign ECE, New York University
Abstract
Recently, deep equilibrium models (DEQs) have drawn increasing attention from the machine learning community. However, DEQs are much less under- stood in terms of certified robustness than their explicit network counterparts. In this paper, we advance the understanding of certified robustness of DEQs via exploiting the connections between various Lipschitz network parameteriza- tions for both explicit and implicit models. Importantly, we show that various popular Lipschitz network structures. including convex potential layers (CPL), SDP-based Lipschitz layers (SLL), almost orthogonal layers (AOL). Sandwich layers, and monotone DEQs (MonDEQ) can all be reparameterized as special equilibrium networks (LBEN) without chang-
cases of the Lipschitz-boundee ing the prescribed Lipschitz constant in the original network parameterization. A key feature of our reparameterization technique is that it preserves the Lip- schitz prescription used different structures. This opens the possibility achieving improved certified robustness of DEQs via combination of network regularization, and LBEN-based fine- tuning. We also support our theoretical understanding with empirical results, which show that our proposed method improves the certified robust accuracy DEQs classification tasks. All codes and experiments are made available at https: gigthu........ com MaronHavvns/Eeae paanns//xxlllieee hanpppooiinnIishhttttlllii
reparameterization, prructure-preservy
Introduction
Recently, deep equilibrium models (DEQs) have drawn increasing attention from the deep learning community [3]. In DEQs, the output is defined as the solution to an input-dependent fixed point equation. Specifically, consider DEQ with input æ and output y. Then is typically determined from x as follows
o < + Un b.). y G= + by:
(1)
where a denotes some nonlinear activation function, e.g. ReLU. DEQs can be viewed as an implicit model that directly solves the fixed point of an infinitely-deey network. Since DEQs have implicit depth, they have the potential to reduce the large memory footprint required to train finite-depth feed-forward networks via explicit back pook-propagation. Noticeably, DEQs have shown great promise in achieving comparable performance on computer vision tasks [4, 33, 6, 28, 23].
Despite great potential, there are many open issues to be explored and understood for DEQs. There has been body of recent works focusing on developing and understanding DEQ structures for addressing well-posedness, training stability, and adversarial robustness [47, 35, 31, 7, 30, 46, 5. 17, 25, 48, 13).
Equal contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

For example, the monotone DEQ (MonDEQ) uses the following parameterization:
2 = α ((11-m)) (1- - A A+ + B - B ) + +z)).)..
Gz + by.
where (A, B,B,U. U, G, b2. by) are the decision variables to be trained, and m is a hyperparameter [47]. In other words, MonDEQ does not treat W as the variable be directly trained and just parameterizes it as W = (1 - m) A A + B - B The advantage of this parameterization is that well- posedness and | -ipschhitssssss (from x to z) are automatically ensured [47, 31J. Another popular // DEQ structure controlling the Lipschitz constant is given by the so-called Lipschitz-bounded equilibrium network (LBEN) parameterization which specifies the weight matrix W in (1) as follows
W I (IG G V "UU'UUUTT vp (2L(2)) + V S +el),
(2)
where Y is a diagonal positive definite matrix, and (V,S) are unconstrained free variables [35]. Using the LBEN parameterization, one can train L-Lipschitz DEQs by solving an unconstrained
optimization problem.
This paper focuses on advancing the understanding of DEQ parameterizations in the context of certified robustness. It is well known that neural networks are susceptible to imperceptible adversarial input perturbations [40]. Hence, there is need to develop certifiably robust models for safety- critical applications. For explicit networks, there has been significant progress on improving certified robustness by using various 1-Lipschitz layer structures such as orthogonal layers [41, 24, 37, 49], almost orthogonal layers (AOL) [34], convex potential layers (CPL) [27]. SDP-based Lipschitz layers (SSL) [2], and Sandwich layers [43]. However, how train DEQs with reasonable certified robustness remains largely open [31]. In this paper, we improve the certified robustness of DEQs via exploiting the connections between various Lipschitz structures for both explicit and implicit models. Importantly, we show that various popular Lipschitz network structures, including convex potential layers (CPL.), SDP-based Lipschitz layers (SL..), almost orthogonal layers (AOL), Sandwich layers, and monotone DEQs (MonDEQ) can all be reparameterized as special cases of the hipschitz-boundde equilibrium networks (LBEN) without changing the prescribed Lipschitz constant in the original network parameterizations. A key feature of our reparameterization technique is that it preserves the Lipschitz prescription used different structures. This feature is particular relevant to training certifiably robust DEQs, and opens the possibility of achieving improved certified robustness of DEQs via a combination of network reparameterization, structure-preserve ABEN-based fine- mueitunnng. We provide empirical results to show that reparameterization technique improves the certified robust accuracy of DEQs on classification benchmarks. Our work sheds new light on the principles for developing 1- 1-Lipschitz DEQ structures and serves meaningful step towards achieving better certified robustness for DEQS.
regularization, and
Additional Related Work. There are also several recent results on certified robustness of DEQs under Lo perturbations [22, 30, 46, 11, 18, 10]. Our work focuses on certified robustness under & perturbations and hence naturally complements these existing works. In this paper, we mainly study the impacts of network parameterizations on certified robustness of DEQs. For explicit models, many other methods for designing certifiably robust models are also available [8, 36, 21, 16, 38, 15). It is an interesting future task to study how to tailor these different methods for improving the certified robust accuracy of DEQs.
2 Background and Preliminaries
Notation. The set of n-dimensional real vectors is denoted by R", For any vector & E R", we denote the associated Euclidean norm as ||||| In addition, we denote the n X n identity matrix and the n zero matrix as In and Og, respectively. The subscripts will be omitted when the dimension is clear from the context. When a matrix P is negative semidefinite (definite), we will use the notation P 3 (-<).. When a matrix P is positive semidefinite (definite), we will use the notation P x (>)00. Given a collection of scalars (a: ) the a, is denoted as diag(a:). For a matrix A, let A and ||A||| denote its transpose and largest singular value, respectively.
diagonal matrix whose (i, i)-th entry is
2.1 Lipschitz Property and Certified Robustness
Suppose we have a prediction model given by an input-output mapping y = f(x). We say that is L-Lipschitz (with respect to the Eg norm) if []f(z;) - f(22)] I L||z1 rall for any (2), 22).

---

The Lipschitz constant can be combined with the prediction margin of f to give certified robust accuracy [42, Proposition 1]. In the past, 1-Lipschitz models have shown great promise in achieving improved certified robustness on classification benchmark problems(29. 41, 24, 37, 27, 34, 2]. In this paper, we focus on designing certifiably robust DEQs via using novel 1-L.ipschitz parameterizations.
2.2 SDPs for Well-posedness and Lipschitz Bounds of DEQs
Given DEQ in the general form (1), the following existing sufficient conditions are useful for testing its well-posednee and Lipschitz properties.
Proposition 2.1 (Theorem in Revay al. [35]). Consider the DEQ model (1) with a being sloped-
restricted on [0, 1]. If there exists a positive definite diagonal matrix I such that the following SDP holds,
- - -TT--
then the DEQ model (1) is well-posed and has finite Lipschitz bound from to y. Proposition 2.2 (Theorem 2 in Revay et al. [35]). Consider the DEQ model (1) with be slope- restricted on [0, 1]. If there exists a positive definite diagonal matrix A such that
1 - AW - T 1 G G --G--AUU AUU A
(4)
L
L
then the DEQ (1) well-posed and I. I--Lipschitz, x to y.
The above propositions can be proved using standard -entrol-theoretic arguments borrowed from the quadratic constraint theory [26, 12]. See [35] for detailed proofs. In general, matrix inequality conditions have been widely adopted for addressing the Lipschitz properties of neural networks [12, 32, 14, 44, 2, 45, 43], and (4) can be viewed as the extension of the LipSDP condition [12] to the DEQ setting. Notice that we use different notations for the decision variables the testing conditions (3) and (4) to emphasize the fact that the solution to (3) may not be solution to (4) even for the same DEQ model. For fixed W. the condition (3) is an SDP that can be efficiently verified by existing solvers. For given (W, U, G). the condition (4) is bilinear in A and hence not an SDP. However, one can easily convexify (4) into an SDP form using the Schur complement lemma. Later, we will develop variant of Proposition 2.2, which can be used to unify the developments of various Lipschitz implicit and explicit models.
2.3 DEQ Parameterizations: MonDEQ and LBEN
Now we briefly review several existing DEQ parameterizations for inducing well-posedness and the Lipschitz property.
The well-posedness issue of DEQs was addressed in [47] using the MonDEQ parame- terization derived from the monotone operator theory. In addition, operator splitting approaches can be used to efficiently compute the unique fixed point of MonDEQ. Recall that MonDEQ adopts the following weight parameterization:
MonDEQ.
(S) where > 0, and (A, B) are free decision variables to be trained. As long as W is parameterized as in (5), the resultant DEQ model (1) is well-posed [47]. Based on examining the unrolled operator splitting iterates, it has been shown [31] that MonDEQ is || Lipschitz from r to z (and hence |||||| a Lipschitz from x to y). If G = 1, then one can enforce MonDEQ to be 1-Lipschitz by adding the constraint |||||
W (
Generalized MonDEQ (G-MonDEE). Revay al. [35] generalizes the MonDEQ parameteriza- tion as follows
= ((A - + ml).
(6)
where is positive definite diagonal matrix to be trained. Obviously, the MonDEQ model is a special case of (6) with o- = I. For the G-MonDEQ parameterization (6), the well-posesness can also be guaranteed. Specifically, one can verify that the SDP condition (3) with W given by (6) holds if we choose r = --1 In this case, the left side of (3) becomes 2AT A 2m/ > O. The Lipschitz constant for the G-MonDEQ parameterization (6) has not been derived before. Later we will present an explicit Lipschitz bound for (6) via constructing analytical solution variant of (4).
3

---

LBEN.
Revay et al. [35] has also proposed the following LBEN parameterization with free decision variables (G, U, V, 5). diagonal matrix variable v/ > and hyperparameter E 0:
1 W = - v 2 ( G+
v - UU - -VV S 2L
cer) -
The LBEN parameterization guarantees the DEQ to be well-posed and L-Lipschitz. Specifically, one can verify that the condition (4) holds with W being defined by (7) and A v y- In this case, the left side of (4) becomes 2 2 The above parameterization allows to train I.-L.ischitz DEQs an unconstrained manner.
I-Lipschitz Explicit Feed-forward Networks: AOL, SLL and Sandwich Layers
For traditional explicit models, there exist several 1 1-Lipschitz parameterizations which can be used for efficient training certifiably robust feed-forward networks. Now we briefly discuss several state-of-the-art I-Lipschitz feed-forward parameterizations.
Orthogonal and Almost Orthogonal Layers. Consider the following standard explicit feed- forward network:
Z0 I, En+1 IW + bn), y WNNN + bn.
where 0, 1,, N Since the compositions of I-Lipschitz functions also I-Lipschitz, one can just parameterize every explicit layer In+1 = o oW + bn) to be I-Lipschitz, and the resultant feed-forward network is L-Lipschitz. ar is 1-Lipschitz' one only needs to ensure ||W,,112 I 1 for all n. An important issue in training such Lipschitz layers for deep explicit networks is gradient vanishing, and this motivates the development of various techniques which introduce gradient norm preservation based on orthogonality. The orthogonality parameterizations just enforce Wn, to satisfy W,T,, W, = 7 [41, 24, 37J. In contrast, the Almost- Orthhronal--aaaa layer (AOL) [34] performs a weight normalization Wn = W,D,,, where Wr, is the free decision variable to be trained, and D, is scaling matrix which is diagonal and defined as²
D2 = diag (E, WW Wols))
(8)
AOL ensures ||W,ll2 < 1. In [34], it has been empirically demonstrated that the trained weight W,, tends to be "almost orthogonal." "
Lipschitz Residual Networks and SLL. Another way to address the gradient vanishing issue in training Lipschitz networks to use residual structure [27, 2]. A general result is given in [2] and shows that the following residual network is guaranteed to be 1-Lipschitz given WT W, = Tn:
Zo = I, In+1 = 2n 2,,T, a (W, In + b,), IN-
For CPL, the choice of Tú is simple, i.e. T, = I|...1l.. To achieve better certified robust accuracy, SLL [2] uses the following more delicate choice of Tn;
T, diag
Σ W
Wn
(9)
n
q
where ( are free positive scalars be trained.
Sandwich Layers. Recently, the sandwich layer has been proposed in [43], and achieves very competitive performances with relatively small models. Based on [43], the following network is guaranteed to be L-Lipschitz:
Zg = VLz, Int1 /2A, 2, 00 (V2 B, + b..).
== V=VLBy + bb
where A,A, BT I, BN I, and 4, is free diagonal matrix whose entries are restricted to be non-negative. One can parameterize [An, BR] to satisfy A, + B, B
using the generalized Cayley transformation [19], and train the above network efficiently.
Almost all the activation functions that are commonly used in practice are L-Lipschitz. If one has E, [W, Wals 0 for then the (i, -h th entry of Dn is just to be 0.

---

3 Network Reparameterizations with Preserved Lipschitz Prescriptions
In this section, we show that various Lipsehitz structures such as MonDEQ, G-MonDEQ, SLL, Sandwich, and AOL can all be reparameterized special cases of LBEN without changing the prescribed Lipschitz constant in the original parameterizations. The feature of preserving the Lipschitz constant prescription between different parameterizations is particularly relevant training certifiably robust DEQs, and we will elaborate on this point later. We emphasize that our result does not indicate that we should abandon all other Lipschitz structures and only use the LBEN parameterization for training certifiably robust DEQs. As a matter of fact, when implementing these different parameterizations for classification tasks, one can potentially incorporate quite different convolution structures and the inductive bias can be very different. The true implication of our theory that one can potentially fine-tune the trained models from MonDEQ. SLL, or other explicit Lipschitz layers by reparameterizing the networks as LBEN and initializing the LBEN training from these reparameterized models. The key to this implication is that the prescribed Lipschitz constant has to be preserved during the reparameteriizttion process. Our theory opens the possibility of achieving improved certified robustness of DEQs via a combination of network reparameterization and L.BEN-based fine-tuning.
3.1 Warm-up: Issues Reparameterizing MonDE. as LBEN and Fix
In this section, reveal deeper connection between MonDEQ and LBEN. As mentioned in [35, Remark 2], any MonDEQ immediately leads to an LBEN for sufficiently large L. However, this trivial reparameteriization of LBEN from MonDEQ does not preserve the prescribed I Lipschitz constant in the original MonDEQ parameterization. Specifically, the original MonDEQ controls its Lipschitz constant by ||0||2 After the trivial reparameterization based on [35, Remark 2), the resultant Lipschtiz constant is no longer ||U||2, In contrast, it can be arbitrarily large. This is problematic for training certifiably robust DEQs, since the increase in the Lipschitz constant will decrease the certified robust accuracy.
m.
Notice that the construction of LBEN relies on Proposition 2.2. The above technical issue hinges upon the fact that Proposition 2.2 cannot be directly applied to prove the Lipschitz constant of MonDEQ. To gain some insight into how to fix this issue, we first discuss how to modify Proposition 2.2 to recover the previous Lipschitz bound of MonDEQ. Specifically, we will use a variant of [35, Theorem 2] (which has been restated as Proposition 2.2). Now we state this new condition.
Theorem 3.1. Suppose the DEQ model (1) is well-posed, and is dlope-restricee on [0, 1). If there exists a positive definite diagonal matrix A such that the following non-strict matrix inequality holds,
1 - AUUTAAA AZO.
1 G
2A - AW T
(10)
then the DEQ (1) is L-Lipschita,rron y.
Proof. We only need to modify the quadratic constraint argument used in [35, Theorem 2]. Originally, a strict matrix inequality is needed to ensure well-posedness. However, well-posedness has been assumed here, and non-strict matrix inequality is sufficient to ensure the L-Lipschitz property. For completeness, a detailed proof is presented in the appendix.
The above variant of the strict matrix inequality (4) gives us a non-strict condition, which can then be used derive Lipschitz properties of MonDEQ. G-MonDEQ, and other explicit models in a unified manner. It turns out that such a non-strict variant is quite crucial for our proposed reparameterization techniques. Recall that originally the LBEN parameterization (7) is derived using Proposition 2.2. Since Theorem 3.1 is the non-strict variant of Proposition 2.2, it is not that surprising that we can use Proposition 2.1 to take care of the well-posedness issue and then combine it with Theorem 3.1 to prove the desired Lipschitz property. We will show that the Lipschitz properties of MonDEQ, G-MonDEQ, and explicit layers such as SLL can also be obtained by providing analytical solutions to (10). More importantly, these new analytical solutions will guide us to find systematic way to reparameterize these structures as LBEN without changing the prescribed Lipschitz constant in the original parameterizations.
5

---

First, we will apply our new theorem to recover the original Lipschitz bound from [31]. The original analysis in [31] relies on examining the operator splitting iterates and bounding the difference between outputs based on their inputs. Alternatively, we can recover this result using the non-strict condition (10). Specifically, we can set = (recalling that we are interested in the Lipschitz constant from x to z), A
T I, = || and W as defined by the MonDEQ parameterization (5).
TōT
Then we can show that the left side of (10) becomes
2m
2
1 1
1
|1/1
1
10/|
ATA
A
L
Li
UUT
1
10
12
A
A
LI
UU
2
1/2 (1 + L
A
A
UUT) = 0.
UU
1
|
Therefore, the condition holds with A = Tilla I and L = ||U1 Notice that MonDEQ is known to be well-posed. Hence Theorem 3.1 can be directly applied to give the conclusion that MonDEQ is WVll -Lipschizz from x 2. If we have G I and want to enforce MonDEQ to 1-Lipschitz, then m we only need to require ||U|||2 < m. There are many ways to parameterize U such that this norm constraint is automatically satisfied. For example, one can enforce V/ymm to be an orthogonal matrix. Similarly, one can also use the scaling trick in AQL to ensure ||U||| S m. See the appendix for more discussions on how to parameterize I-Lipschitz MonDEQ.
1
Next, we discuss how to address the previous issue in reparameterization such that we can reparame- terize 1-Lipschitz MonDEQ as LBEN by exactly setting L = 1 in the LBEN reparameterization. We will leverage the construction of A above, Recall that the free variables in MonDEQ are (A, B, U). The matrices (W, G) are given by
W
B B G=I
Now we confine our scope to I-Lipschitz MonDEQ, i.e. ||U]|2 S m. We will show that given a MonDEQ model (5) with any fixed (A, B,B,UU) U). we can always generate LBEN parameterization in the form of (7) which gives exactly the same input-output mapping behavior from x to y. This means that any trained MonDEQ model can be reparameterized in the form of LBEN with some choice of (M, G, U, V, S).. We will provide explicit formulas for such a choice of (V, U, V,))). We have the following formal statement.
Theorem 3.2. Given a I-Lipschitz MonDEQ model with (A, satisfying ||1||2 m, the following choice of (V, G, G,U,V S, with U being the same one used for MonDEQ gives a valid 1 1-Lipschitz LBEN model which generates the same imput-outper mapping:
B
V = 1U1122,
G=1,
S
e=0 0
11
V =
m
1 2
ATA
A
UUT
20
|
10
2
Proof. The above formula can be easily verified by substituting the expressions for W into both the MonDEQ and LBEN parameterizations and then making simplifications. The only thing that needs be further checked is that the resultant expression for V must be positive semidefinite so we can obtain V by decomposition. Since we know ||U|] S m. we must have
VTV
m
1
1
+
ATA A
UUT
(11)
||0|||
2
|
21
A A
1
1
0
(12)
2
|
Hence the resultant expression for V V is positive semidefinite. This completes the proof.
We can clearly see that in the above result, the Lipschitz constant preserved during the reparameter- ization.

---

3.2 Connections between G-MonDEQ and LBEN
The Lipschitz constant of the G-MonDEQ parameterization (6) has not been explicitly provided by the original work in [35]. Now we will present such an explicit formula via applying (10), Formally, we have the following result.
Theorem 3.3. Consider the G-MonDEQ parameterization (6). It is guaranteed that (6) gives mapping which is
[I Lipschitt a to z.
Proof. We are interested in the Lipschitz constant from z, and hence we choose G I. As ex- plained before, the G-MonDEQ parameterization is known to be well-posed based on Proposition 2.1. Hence apply Theorem 3.1 to upper bound its Lipschitz constant. Specifically,
choose
A T 1--1011 $- and L U'l2 Then the left side of (10) becomes
2
A
-||0-1U1 Φ
--UUUC T Φ-1
L
④
11
(1
A+
TUTTO Φ
UU
0-1)
11112
which is obviously positive semidefinite. Therefore, (10) is feasible for G-MonDEQ with L o "Ull2 This leads to the desired conclusion.
If we choose + I, the above theorem just recovers [31, Theorem 1] as special case. From the above theorem, we can ensure G-MonDEQ to be 1-Lipschitz by enforcing | | S m during training.
Next, we can leverage the above construction of to establish a similar reparameterization result connecting G-MonDEQ and LBEN.
Theorem 3.4. Given a I L-pischiitz G-MonDEQ model with (Ø, A, B, satisfying || S m, the following choice of (V, G, U, V, V,S,)) with U being the same used for G-MonDEQ gives valid 1-Lipschitz LBEN model which generates exactly the same input-ouspul mapping:
= 1=110-101la
G=I,
S=
B. c=0
1-11
VTV
m
1 1) 2
1
AT A
1
1012
21
Proof.
The proof is very similar to the proof of the MonDEQ case and hence omitted.
Again, the prescribed Lipschitz constant is preserved during our reparameterizztiin.
3.3 Connections between SLL and LBEN
Extending the previous analysis, we can further build connections between LBEN and explicit networks such as SLL. The SLL network is parameterized by (W,,, Tr} with WT W, 3 T₁ Again, we can choose (L) G. U. V.S. properly to generate valid 1-Lipschitz LBEN which gives the same input-output mapping. Similar the previous treatment, we will first recover the Lipschitz analysis of SLL to gain some insights on how to construct A for the reparameterization. Notice that feed-forward network always well-posed. Hence we can just rewrite a feed-forward residual network in the form of (1) and then apply Theorem 3.1 to analyze its Lipschitz property. To see this, consider the residual structure Fn+1 = Fn 2W, T,- to (W,,, n on + by,) and xo = x. Denoting
inti o (W,T Fn + bw.) (n 2 1) and Zo Fu. we can express the entire multi-layer residual structure in terms of (Zu, (fu....INN-1 as
n-1
N-1
Int1
WT Tu
Σ
TW T.'4z + bn = Ty
Σ
2W
(13)
km
kmm
for f1....,N N - 1}.
rewrite the SLL parameterization in the DEQ form of (z.y) (ó(W2 + Ux + bw), Gz by) with _
With this description,
T. TT 6(-)
7

---

[id(-)", ..... .() G [I, 2WoT ..... 2W -W U [1,0, .0....000' 0)", and
W being given as the following matrix:
OOO
W
WT
(14)
0
WN-1
-2W
-2WN-1 2-2 0
It is not surprising that the above matrix has lower-triangul triangular structure, since the original SLL network is beed-forward Now it straightforward to verify that (10) feasible (see Appendix C) with L = 1 and A being given by
A = blkdiag (T,2T=). .....,,77-=]) 27
(15)
Now we can reparameterize SLL as LBEN with preserving the I-Lipschitz property as follows. Theorem 3.5. Given any SLL structure parameterized by all n, the resulting input-output relation from to can be exactly recovered with the LBEN parameterization with the following choice of (V, V,S,),
with W, Wn Tn for
V A (as in Eq. (15)), G= [T, 2
-2.N -WWN-
U = T OJ, VV V = (M
(M+M) + MM), = -2M. M, c=0.
2
where M is given M A(I w) +
AUUTA) A) with W being defined by (14).
A detailed proof for the above theorem will be presented in the appendix. We want to comment that one can easily verify that the expression for V is given explicitly by block-diagonal matrix with leading zero block and other remaining blocks defined
(VTV),n. 27 27, W W T
(16)
for (I,...,NN- 11-1 1): Obviously, the resultant matrix for VTV is positive semidefinite. We also emphasize that the above reparameterivation works for both SLL and CPI., as long as one chooses [T,} properly. In addition, it is worth noting that to unconstrained decision variables { which can be useful for finetuning LBEN from SLL.
set v diag (exg(e))(d;)) (d,)) )), leading
Reparameterizing AOL and Sandwich Layers LBEN. Notice that non-residual Lipschitz network structures such as AOL and Sandwich can also be connected to LBEN. For the experimental evaluation, we mainly utilize the SLL initialization of LBEN. We leave the discussions on the explicit connections between AOL, Sandwich, and LBEN to the appendix.
4 Numerical Experiments
In this section, we present some numerical experiments and insights on how to initialize LBEN with other Lipschitz architectures. Such numerical study will demonstrate and support our theory.
4.1 Convolutional LBEN and Implementation Details
For classification tasks, we need embed convolutional layers into the LBEN theoretical framework. Although this is admissible in principal, there are practical aspects that need extra explanations.
When initializing LBEN parameters (V,S) from trained SLL network, we require decomposing 2 {M + MT) constructed from Section 3 (M is given by Theorem 3.5). For convolutional layers, we consider convolution with circular padding which is known to be close under sum, product, and inverse. Hence, all operations required for forming M preserve the convolution structure and so 1 {MM + AM T) is also convolution with circular padding. Furthermore, we can find V by taking the matrix square root of 1 {M + MT) efficiently using the bloek-diagonalization trick of circular convolution (see the appendix for more details).
Initialization of Convolutional LBEN.
'This just means that the first element of the activation is replaced by an identity mapping. If a is slope- restricted on (0, 1]. the resultant e still dost-restricted on j0, 11. and hence Theorem 3.1 can still be applied.

---

Table 1: This table presents results obtained from LBEN models initialized from pre-trained Lipschitz Network (e.g., MonDEQ, SLL). After initialization, the LBEN is then fine-tuned for 40 epochs with very small learning rate. LBEN offers improved éz-certified accuracy over Lipschitz networks counterparts. For MNIST, SLL and LBEN baselines are omitted since LBEN already achieves reasonable performance. We initialized instead from Lip-MonDEQ showcase how our theoretical framework can provide further improvements.
Natural
Certified Accuracy (e)
Datasets
Models
Accuracy
36
72
118
253
250
255
(1) Lip-MonDEQ [31]
0.886
0.886
0.858
0.826
0.634
MNIST
LBEN initialized from Lip-MonDEQ (1)
0.927
0.927
0.910
0.886
0.731
(2) SLL Network [2)
0.654
0.555
0.458
0.363
0.106
CIFARIO
LBEN [35]
0.451
0.361
0.277
0.207
0.043
LBEN initialized from SLL (2)
0.655
0.562
0.472
0.380
0.123
(3) SLLL Network [2]
0.398
0,288
0.207
0.149
0.038
CIFARI00
LBEN [35]
0.292
0.176
0.117
0.078
0.015
LBEN initialized from SLL (3)
0.403
0.291
0.209
0.153
0.041
& Regularization of LBEN Sparsity Structure. Although we endow the weight W with a convolution structure, which will be preserved throughout training. this parameterization uses relatively large kernel. Rather than using, for example, a x 3 kernel applied across the image, our kernel parameterization is the size of the input image of size dCout X dcin X n n, where d corresponds to the number of convolutions). Indeed, the inverse or square root of a convolution with a 3 x 3 kernel applied on an input n x n input is another convolution with a n X n kernel.
(i.e., W is defined as a large kernel
When initializing the LBEN from an existing SLL model, W will be quite sparse due to the original small kernel parameterization. However, this sparsity will not necessarily be preserved after training. This is because the gradients with respect W (defined with kernel 3 x 3) depend on the inverse of W (which has a kernel of size n x n)).
In order to heuristically preserve this sparsity structure and not destroy the inductive bias given by the initial architecture, we impose a small & --eennlyy to non-zero values on the border of the kernel to induce learning small-size kernels. This way, can regularize the structure of W without placing explicit constraints on the LBEN parameterization.
4.2 Experiments: Evaluation of Pretrained Initialization and Discussion
For the experiments, we trained MonDEQ and SLL networks that would serve as initialization for LBEN. To train a I-Lipschitz MonDEQ, we use spectral normalization [29] on the matrix U and set v = 1. For the SL. network (e.g.. (2) in Table 1), we use 4 convolutional layers with circular padding and dense layers. The convolutional and dense layers have and 512 channels/feaurrs, respectively. We use the ReLU nonlinearity which is slope-restricted composition of convolutional and dense LBEN blocks, initialized from the four convolutions and the two dense layers of the pretrained Lipschitz model, respectively. To fine-tune the LBEN after initialization, we use a small learning rate 1e-7 with a erreuularizztio of 0.1 during 40 epochs (for more details, see the publicly available code). The results presented in Table 1.
[0, 1]. Our final LBEN network is
Note that we omit any unconstrained DEQ baselines which may have a higher clean accuracy compared to a constrained model, but very low certified robustness. The resulting Lipschitz constant of the unconstrained model typically much larger than 1 and leads to certified accuracy near zero based on the margin argument. For instance, the work of [31] provides an evaluation of DEQ on CIFARIO for & = 0.01 achieving certified robust accuracy of roughly 10% (estimate is read off graph from [31, Figure 9]). Previously, certified robustness results for the standard perturbation E 36/6//255 255 have been reported for unconstrained DEQ.
LBEN Initialized with Lip.MonDEQ. First, we present small-scale experiment on the MNIST dataset with Mon-MonDEE and LBEN initialized from the trained Lip-MonDEQ. SLL and LBEN baselines are omitted since LBEN already achieves reasonable performance and our aim is to

---

showcase how our theoretical framework can provide further improvements on existing architectures. We observe that the LBEN significantly improves the natural and provable accuracy over the Lip- MonDEQ. This is mainly due to the different Lipschitz constraints used in Lip-MonDEQ and LBEN.
LBEN initialized with SLL. We now present some experiments on CIFARIO and CIFARI00 datasets [20]. First, we trained an SLL network (2) with circular padding with the same hyper- parameters as in [2]. We then compare the natural and certified accuracy between an LBEN trained from scratch (random initialization) and an LBEN initialized from the SLL model. We observe that the LBEN with random initialization offers poor natural and certified accuracy compared to the SLL network for both the CIFARIO and CIFAR 100 datasets. On the other hand, when LBEN initialized from the trained SLL weights, successfully improves upon SL.. by approximately 1 1% in certified accuracy on CIFAR10 10 and marginal improvement on CIFART 100. Despite being marginal improvements over SLL,, these results significantly improve upon the current state-of-the-art (2-certified accuracy for DEQs.
One interpretation of these results that the performance of neural networks is highly-dependent on a good initialization [39, 1]. Additionally, explicit feed-forward I-Lipschitz convolutional networks have important inductive biases that have been crucial for achieving good certified robustness results on image-clasee classification tasks [9]. In contrast, our current understanding on how to incorporate the right inductive biases for LBEN in the context of certified robustness is relative limited. Therefore. fine-tuning LBEN from 1-I-ipschitz layers with good inductive biases (in our case SI..) helps LBEN achieve improved certified robustness via combining the benefits of inductive biases of feed-forward 1 1-Lipschitz networks and the expressive advantage of LBEN over explicit networks.
LBEN initializations with Lipschitz constant other than L = 1. Larger Lipschitz constant parameterizations for LBEN are explored in [35] on CIFARIO (L = 2, 5, 50), which slightly improves clean accuracy. but decreases the empirical robustness when compared to the I-Lipschitz LBEN. The best certified robustness results achieved by our approach on CIFAR 10 and CIFAR100 tasks are indeed achieved by choosing I. = 1. At this moment, the understanding of how to incorporate inductive bias via enforcing convolution structures on 1-Lipschitz layers is relatively matured. Hence choosing L == to make the Lipschitz constant consistent with these structures leads to the best certified robustness result for now. In the future, it is possible that one can improve the certified robustness DEQ for L > 1 by developing new convolutional structures for LBEN.
Remark. The results stated in Table are not on par with the state-of-the thee--------rt art certified robust accuracy obtained by the largest explicit Lipschitz feed-forware models. This is due the following reasons: 1) DEQs are known to be computationally expensive as they require the computation of fixed point for each input x. Therefore, in order keep the computational cost for LBEN models reasonable, we trained small SLL networks. 2) There is technical subtlety of embedding a convolu- tional SLL model into DEQ that requires a much larger convolutional kernel. This also currently prevents us from scaling up to larger vision tasks like TinylmageNet, but may be circumvented in the future. For applications using standard fully-connected layers, the representation memory footprint of equivalent DEQ does not present such an issue. Despite these challenges, to the best of our knowledge, our work provides the best l2-certified accuracy with DEQs.
Conclusion
In this paper, we present a unified algebraic approach for deriving I-Lipschitz DEQ structures via providing analytical solutions to an SDP condition. We show that several Lipschitz structures (e.g.. CPL, SLL, AOL, and MonDEQ) can be reparameterized special cases of LBEN without changing the original Lipschitz constant specifications. We provide explicit formulas for such Lipschitz- preserved reparameterization. Finally, our experiments show that our theory opens the possibility of achieving improved certified robustness of DEQs via a combination of network reparameterization, structure-preserving regularization, and LBEN-basee fine-tuning.
Acknowledgment
A. Havens and B. Hu are generously supported by the NSF award CAREER-2048169. the AFOSR award 2A9550-23-110732. and the IBM/IIDI award 110662-01. A. Araujo. S. Garg, and F. Khorrami
10

---

are supported in part by the Army Research Office under grant number W91 1911NN-21-- and by the New York University Abu Dhabi (NYUAD) Center for Artificial Intelligence and Roboties, funded by Tamkeen under the NYUAD Research Institute Award CG010.
References
[1] A. Agarwala and S. S. Schocnholz. Deep equilibrium networks sensitive to initialization statisties. In International Conference on Machine Learning, pages 136 160. PMLR, 2022.
[2] A. Araujo, A. J. Havens, B. Delattre, A. Allauzen, and B. Hu. A unified algebraic perspective Lipschitz neural networks. International Conference for Learning Representations, 2023.
[3] S. Bai, Z. Kolter, and V. Koltun. Deep equilibrium models. Advances in Neural Information Processing Systems, 32, 2019.
S. Bai, Koltun, and Z. Kolter. Multiscale deep cquilibrium models. Advances in Neural Information Processing Systems, 33:5238-5250, 2020.
[5] S. Bai, Koltun, and J. Kolter. Stabilizing equilibrium models by Jacobian regularization. arXiv preprint 106 14342, 2021.
[6] S. Bai, Geng, Savani, and Kolter. Deep equilibrium optical flow estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 620-630 630, 2022.
[7] T. Chen, J. B. Lasserre, Magron, E. Pauwels. Semialgebraic representation of monotone deep equilibrium models and applications to certification. Advances in Neural Information Processing Syxtems, 177-199997 2021.
[8] J. Cohen, E. Rosenfeld, and Z. Kolter, Certified adversarial robustness via randomized smoothing. In International Conference on Machine Learning, 2019.
191 N. Cohen, O. Sharir, and A. Shashua. On the expressive power of deep learning: A tensor analysis. In Conference on learning theory, pages 698-728. PMLR, 2016.
[10] A. Davydov, S. Jafarpour, M. Abate, F. Bullo, and S, Coogan. Comparative analysis of interval reachability for robust implicit and feedforward neural networks. In 2022 IEEE 61 Conference on Decision and Control (CDC), pages 2073-2078. IEEE, 2022.
[11] L. El Ghaoui, F. Gu, B. Travacca, A. Askari, and A. Tsai, Implicit deep learning. SIAM Journal on Mathematics of Data Science, 3(3)930 958, 2021.
[12] M. Fazlyab, A. Robey, H. Hassani, M. Morari, and G. Pappas. Efficient and accurate estimation of Lipschitz constants for deep neural networks. Advances in Neural Information Processing Systems, 32, 2019.
[13] T. Gao, H. Liu, J. Liu, H. Rajan, and H. Gao. A global convergence theory for deep reLU implicit networks via overpparameterizaaion. In International Conference on Learning Representations, 2022. URL https://
[14] D. Gramlich, P. Pauli, C. W. Scherer, F. Allgöwer, and C. Ebenbauer. Convolutional neural networks as 2-d systems. arXiv preprint arXiv:2303.03042 2023.
[15] K. Hu, Zou, Z. Wang, K. Leino, and M. Fredrikson. Scaling in depth: Unlocking robustness certification on imagenet. arXiv preprint arXiv: /2549, 2023.
[16] Y. Huang, H. Zhang, Y. Shi, J. Z. Kolter, and Anandkumar. Training certifiably robust neural networks with efficient local Lipschitz bounds. Advances in Neural Information Processing Systems, 34, 2021.
[17] S. Jafarpour, A. Davydov, A. Proskurnikov, and F. Bullo. Robust implicit networks via non- non-Euclidee Euclidean contractions. Advances in Neural Information Processing Systems. 34:9857-9888 2021.
[18] S. Jafarpour, M. Abate, A. Davydov, F. Bullo, and S. Coogan. Robustness certificates for implicit neural networks: A mixed monotone contractive approach. In Learning for Dynamics and Control Conference, pages 917-930. PMLR, 2022.
[19] M. Jauch, P. D. Hoff, and D. B. Dunson. Random orthogonal matrices and the Cayley transform. Bernoulli, 1662)1556-11586 2020.
[20] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.

---

[21] K. Leino, Z. Wang. and M. Fredrikson. Globally-robust neural networks. In International Conference on Machine Learning. 2021.
[22] M. Li. Y. Wang, and Z. Lin. Cerdeq: Certifiable deep equilibrium model. In International Conference on Machine Learning, pages 12998 13013. PMLR, 2022.
[23] M. Li, Wang, X. Xie, and Z. Lin. Optimization inspired multi-branch equilibrium modcls. In Inter- national Conference on Learning Representations, 2022. URL abC8iTTXIrk.
[24] Q. Li, S. Haque, C. Anil, J. Lucas, R. B. Grosse, and J.-H. Jacohsen. Preventing gradient attenuation in Lipschitz constrained convolutional networks. In Advances in Neural Information Processing Systems, 2019.
[25] Z. Ling, X. Xie, Q. Wang. Z. Zhang, and Z. Lin. Global convergence of per-parameterized deep equilibrium models. International Conference on Artificial Intelligence and Statistics, pages 767-787. PMLR, 2023.
[26] A. Megretski and Rantzer. System analysis via integral quadratic constraints. IEEE Transactions on Automatic Control, (2266)819-830 1997.
[27] L. Meunier, B. Delattre, A. Araujo, and A. Allauzen, A dynamical system perspective for Lipschitz neural networks. In International Conference on Machine Learning, 2022.
[28] P. Micaelli, A. Vahdat, H. Yin, J. Kautz. and P. Molchanov. Recurrence without recurrence: Stable video landmark detection with deep equilibrium models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22814-22825, 2023.
[29] T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida. Spectral normalization for generative adversarial networks. International Conference on Learning Representations, 2018.
[30] N. Müller, R. Staab, Fischer, and M. Vechev. Effective certification monotone deep equilibrium models. arXiv preprint arXiv:21 2021.
[31] C. Pabbaraju, E. Winston, and J. Z. Kolter. Estimating Lipschitz constants of monotone deep equilibrium models. In International Conference on Learning Representations, 2020.
[32] P. Pauli, A. Koch, J. Berberich, P. Kohler, and F, Allgöwer. Training robust neural networks using Lipschitz bounds. IEEE Control Systems Letters, 6: 6:121, 2021.
[33] A. Pokle, Z. Geng, and J. Z. Kolter. Deep equilibrium approaches to diffusion models. Advances in Neural Information Processing Systems, 35:37975-37990, 2022.
[34] B. Prach and C. H. Lampert. Almost-orthogonal layers for efficient general-purpose Lipschitz networks. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Isruel, October 23-27, 2022, Proceedings, Part XXI, pages 350-365. Springer, 2022.
[35] M. Revay, R. Wang, and R. Manchester. Lipschitz bounded equilibrium networks. arXiv preprint axXi:22000.01332, 2020.
[36] H. Salman, J. Li, I. Razenshteyn, P. Zhang, H. Zhang, S. Bubeck, and G. Yang. Provably robust deep learning via adversarially trained smoothed classifiers. In Advances in Neurul Information Processing Systems, 2019.
[37] S. Singla and S. Feizi. Skew orthogonal convolutions. In International Conference Machine Learning. pages 9756-9766. PMLR, 2021.
[38] S. Singla and S. Feizi. Improved techniques for deterministic 12 robustness. Advances in Neural Information Processing Systems, 2022.
[39] Sutskever, J. Martens, G. Dahl, and G. Hinton. On importance of initialization and momentum in deep learning. In International conference on machine learning, pages 1139-1141 1177. PMLR, 2013.
[40] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014.
[41] A. Trockman et al. Orthogonalizing convolutional layers with the Cayley transform. In International Conference on Learning Representations, 2021.
12

---

[42] Y. Tsuzuku, I. Sato, and M. Sugiyama. Lipschitz maassii training: Scalable certification of perturbation invariance for deep neural networks. In Advances in Neural Information Processing Systems, 2018.
[43] R. Wang and I. R. Manchester. Direct parameterization of Lipschitz-ouunded deep networks. In Interna- tional Conference Machine Learning, 2023.
[44] Z. Wang, G. Prakriya, and S. Jha. A quantitative geometric approach to neural-network smoothness. Advances in Neural Information Processing Systems, 35:34201-34215, 2022.
[45] 7. Wang, S. Jha, et al. Efficient symbolic reasoning for neural-network verification. arXiv preprint arXiv:2303.13588 2023.
[46] C. Wei and J. Z. Kolter. Certified robustness for deep equilibrium models via interval bound propagation.
In International Conference Learning Representations, 2021.
[47] E. Winston and Z. Kolter. Monotone operator equilibrium networks. Advances in neural information processing systems, 33:10718-10728, 2020.
[48] Z. Yang, P. Li, T. Pang, and Y. Liu. Improving adversarial robustness of DEQs with explicit regulations along the neural dynamics. In Proceedings of the 40th International Conference on Machine Learning. 2023.
[49] T. Yu, Li, Y. Cai, and P. Li. Constructing orthogonal convolutions in explicit manner. In International Conference on Learning Representations, 2022.
13

---

A Controlling the Lipschitz Constant in MonDEQ and G-MonDEQ
As discussed Section 3, there are explicit relationships between the MonDEQ and G-MonDEQ parameterizations and the LBEN parameterization. Although nominally the MonDEQ only guarantees well-posedness of the DEQ, we can control the Lipschitz constants through the 0, the imput matrix U and, in the of G-MonDEQ, the additional parameter 4.
was previously shown by [31] that a MonDEQ has a Lipshitz constant wish to parameterize U so that MonDEQ is always I-L-isccitzz, we can choose U = mP where P is an orthogonal matrix. Orthogonal matrices can be parameterized by considering their sub- group of special orthogonal matrices SO(n) through the Cayley transform matrix exponential on skew-symmetric matrices used in [41].
||/|2/m.. we
In the case of G-MonDEQ, a new condition is derived (Theorem 3.3) showing that the G-MonDEQ parameterizaton of (6) is | U1| ///m-Lipscitzz from x 2. this case, we must simultaneously control U and Φ to maintain the Lipschitz bound. A similar approach can be applied to jointly parameterize + 6--0. U. For example, to maintain a Lipschitz bound L, one may practically enforce the constraints 4 2 1/([m) and U U 3 through SLL or Cayley parameterizations [2].
B Additional Proofs
B.1 Proof of Theorem 3.1
Proof. Theorem 3.1 follows almost exactly as in [35, Theorem 2] with an added assumption of well-posedness, i.e. now is assumed that there exists a unique solution = = oW Ux + b.) for any z. We present the details here for completeness.
As formulated in [35], can isolate the nonlinear parts of the DEQ through the following algebraic representation:
v = W2+ Ux b,, ==o(v), Gz + by
(17)
Suppose that given two inputs and x', the two resultant solutions for (17) given by:
Wz + Ux ba: ==o(v), y Gz by
r' Wa' + Un + b,, 2' z'=0(u') Gz' b
We define A, V/ v and A, z. Notice that the nonlinearity is assumed be slope- restricted on [0, 1J. Hence the following inequality must hold for any positive definite diagonal matrix A:
(18)
where the weighted inner product is based on using A as the weight. Now define Ag = x and Ay y/ y. respectively. Obviously, we have A, = WA, + UA, and A, GD.
Now we are ready to show the desired Lipschitz property using the matrix inequality (10). By Schur complement, we have the following equivalent matrix inequality:
2A AW W A G
G
-AU
00
-UA A
LI
Left and right multiplying both sides by [ AT] and [A]
ATJT respectively yields
1 1 A GA, AUA, + LI 2 2 0
AAAA - 24 AWA,
(19)
Using the relations A. WA, UA and Ag GD., we can re-arrange the above inequality as
A - ||ر 2 2A 20,AA, 2(4, A.
(20)
Therefore, must have || < LI |l. and the DEQ model (1) has to be L-Lipschitz.
14

---

B.2 Proof of Theorem 3.5
Proof. Given an SLL residual network, it is clear that define equivalent well-posee DEQ by setting W, U, G as in the theorem statement. If we substitute the parameters (V, U,,,,, into the parameterization in (7), are then left with
(M + -T)= 2
So it remains to show that VTV is positive semidefinite, and then it trivial to see that V can be determined by decomposition.
Notice that the expression for (M + M) is exactly equal to the left side of the matrix inequality condition given by Eq. (10). is simple verify that this sufficient condition (10) is satisfied (see Section C). Therefore we have 1/2(M + T) = O. Due to leading zero block, it necessary that & Because 1 1/2(M + M) positive semidefiiite, we can always compute its decomposition (e.g., via the Cholesky decomposition or matrix square-root). Therefore, W can be recovered by our parameter choice ( G, U, V,S, c), and the proof is complete.
C Feasibility of SLL as LBEN
In Section when discussing the connection between the SLL architecture and LBEN, it is claimed that using the weight structure:
0
W
0
0
W
W.T
2W,
WoTo
U
W
- 2
22 W 2 N-2
2
Tw
0
G [I, 122
2W
satisfies the non-strict LBEN sufficient condition Theorem (3.1) for L and given as follows:
2T
A
27 1
N
We can quickly prove that this true. Plugging W, G. U and A into the inequality (10), all off-diagonal block terms will cancel and we are left with the follow block-diagonal matrix:
2T 2T Wo W
(21)
- - 27
Because it is block-diagonal, it is enough to check positive semidefiniteness of each block. Because each T, defined in the SL.L architecture [2] and satisfies W. W, A Tn. the following conclusion must hold:
27 WTW,F > 2T 2T, n
TTT TT n 0.
Therefore, the SDP sufficient condition (10) is feasible.
15

---

D Further Connections between Existing Lipshitz Architectures and LBEN
D.1 Connections between AOL and LBEN
In similar manner, may also reparameterize AOL as LBEN with L 1. Suppose we have the following multi-layer AOL network structure
Zo = Z
2++1 d(W,Djzytb), (0.....I--1) fo(x) WiDjo b by
where Dj are given by (8). Then the corresponding DEQ may be defined by setting: 2 = [x] .....,,.. IT" (2) = [o(z)) ...."...................... M()) U = [(WoD)) ...... OJT, G [O...., WiDi]
and choosing W as
0
W W1D1
0
Wi-1 0
It can be shown that by choosing A = 1, the AOL network will satisfy the non-strict I-Lipschitz LBEN condition (10). By defining and S in the same way as in Theorem 3.5, we can obtain V (M + MT), which be verified to be positive semidefinite. With this choice of V, S and V I, we have recovered W by the LBEN parameterization (7).
D.2 Connections Between Direct Sandwich Parameterizations and LBEN
Suppose we have the following multi-layer network with the Sandwich layer structure of [43] with prescribed Lipschitz bound of L:
z0 = Var, In+l V2A,, T (onooV222222 Ba + bn), = VLBNEN + bb.
where A, B B I, BN I, and N, is free diagonal matrix whose entries restricted to be non-negative. One can parameterize [An Br] as semi-orthogonal matrix using the generalized Cayley transformation [19], and train the above network efficiently.
can rewrite Sandwich as
yo = Lx, Un+1 = 0222 B, 1 + bn),
y = N 1 IN- +bN.
Then the DEQ may be defined by setting: = [y ..... .NN] o02) = (0(2)) ..... ((NN)
U [[V2L0, Bu) ..... O]T, b- [bj ................................ 1 T G [0...., V2LB, AT 2N by
by. and choosing as
20-1B1. 2o
W
(22)
24 2 BN 2 - 0
Given any Sandwich network, the ouputt relationship can be exactly recovered by LBEN parameterization with the following choice of (v, G, U, V,e):
A diag $2 22 f. 22 N
V --1
G = [0, jo..... 2LB
1 NN-I
U = [[V2L0 Bu) ,...,.' T
VTV V 1 [M M),, S M, ==
1
where M is given by M = A(I WL--W W) 1 zI
(G G AUUUTA).
16

---

As matter of fact, it can be shown that V has the following analytical form:
Ag d
B Φ1
A Φ1
B2 22
V
(23)
AN-I PN-1
B
E
Additional Details of Convolutional LBEN
When it comes to image-classification tasks, one of the most useful architecture biases is the use of convolutional layers which are heavily used in all sate-of-the-art SLL and AOL models, includ- ing our results. Convolutions with circular padding are straightforward to integrate into DEQ by representing them doubly-block circulant weight matrices. particular suppose we have an X ni image input and K E RTx* a convolution kernel and define a doubly-block circulant matrix W = blkcirc Ka..... ......... Kn E R=* where K; cire(K,.). (.:). Then the 2D convolution of the vectorized input x given by the linear operation
Wvec(x) vec(K xx)
(24)
where * is the convolution operation. The concept can be extended to convolutions with multiple input and output channels, where W will be a block-doubly-bloc circulant matrix in Reoeen" Xcn.n
This linear form has several interesting properties linked to the Fourier transform which allows efficient operations. Indeed, for initializing LBEN, must use expensive matrix square-roo operation to compute the LBEN parameter V. More importantly, the operator splitting approach in [47] requires inverse of W for each step of both training and evaluation. Fortunately, there exists an efficient approach for diagonalizing this particular representation of convolutions [41] which has been used previously for convolutional MonDEQs [47]. In particular, given doubly-ble blook circulant weight matrix W E geaat" xc,an? with kernel K R Xein xaxx, we can obtain its block diee-diignnalizttionn as follows from [41, Corollary A.1.1]:
W (Jepss Fn) Segur DST Can (Je, 2 FT')
(25)
where Scnue and So,, are permutations matrices, Fn Un Un. and Un is Fourier basis matrix. Here δ will be block-diagonal matrix with n² blocks of size Cout x Cin and is given by
vav (Ka,a)
Fa vec (Ko.er)
D = ST Dout
Sem.
Fec ()).
vec (Kenst
From here, may compute the inverse square root of W, directly from the kernel K, more efficiently by performing these operations only on the n diagonal blocks. All operations required for training LBEN using the operator splitting approach [47] will leave W within the of these doubly-block circulant matrices, so we can repeatedly apply this diagonalization trick throughout training.
17


Reviews:
[{"review_id": "RcbQ6zJgEw", "reviewer": "Reviewer_PSUC", "summary": "The paper studies the l2-certified robustness of DEQs from the Lipschitz bounded view. They not only proved the advantages of DEQs against other models on certifiable robustness but also show the links between other popular Lipschitz layers like convex potential layers, SDP-based Lipschitz layers, almost orthogonal layers and Lipschtiz bounded equilibrium models. Based on the relation, they can use pre-trained SLL models as initialization to help their DEQs training.", "questions": "Can such fine-tuning technique help other DEQs training for natural tasks like using MDEQ[1] or MOptEqs[2]?\n\n[1] Multiscale deep equilibrium models\n[2] Optimization inspired Multi-Branch Equilibrium Models", "limitations": "Nothing", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Their writing is clear. They show the relationship between other explicit Lipschitz networks and LBENs by a new reparameterizing technique. By using this technique, they use a pre-trained SLL model to help DEQ's training and obtain better certifiable results. Furthermore, I think they may show a good technique to accelerate DEQs training if their parameterization technique can extend to more model training.", "weaknesses": "The results for SLL are much lower than their paper's report. More than 4% lower for natural accuracy and about 10% lower for 72/255 certified accuracy compared with SLL small. \n\nYou may also compare CIFAR-100 and TinyImageNet in your empirical section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful evaluation and comments. Now we provide detailed responses to each of your comments.\n\n##\n**The results for SLL are much lower than their paper's report. More than 4% lower for natural accuracy and about 10% lower for 72/255 certified accuracy compared with SLL small.**\n\nWe agree that this difference in performance requires more explanation. We remark that, due to scalability issues that are inherent to DEQ, we train smaller SLL networks than what is presented for the state-of-the-art in Araujo et al. The SLL network used in our paper consists of $500k$ parameters, where the``SLL Small'' architecture used in Araujo et al has $40m$ parameters, leading to the performance gap. Note that this difference is mainly due to scalability issues of DEQ, however this is still the highest certified accuracy achieved by a DEQ model. We will provide these additional architecture details in the final version.\n\n##\n**You may also compare CIFAR-100 and TinyImageNet in your empirical section.**\n\nDue to scalability issues of training DEQ, applying them to larger datasets like TinyImageNet remains very challenging. We have evaluated LBEN initialized from a small SLL network on CIFAR-100 which achieves the state-of-the-art with respect to certified robustness of any existing DEQ model. Please note there is still a significant performance gap from state-of-the-art due to scalability issues of DEQ. By fine-tuning the LBEN initialized from SLL, we are able to see some improvements on certified robustness.\n|model | clean accuracy | $\\epsilon = 36/255$ | $\\epsilon= 72/255$ | $\\epsilon = 108/255$ | $\\epsilon = 1$ |\n| ---- | ------ | --- | --- | --- | --- |\n| SLL | 0.398 | 0.288 | 0.207 | 0.149 | 0.038 |\n| LBEN | 0.292 | 0.176 | 0.117 | 0.078 | 0.015|\n| LBEN with SLL init |  0.403 | 0.291 | 0.209 | 0.153 | 0.041 |\n\n##\n\n**Can such fine-tuning technique help other DEQs training for natural tasks like using MDEQ[1] or MOptEqs[2]?\n[1] Multiscale deep equilibrium models [2] Optimization inspired Multi-Branch Equilibrium Models**\n\nThank your intriguing question. In order to accommodate MDEQ[1], a modification of the LBEN theory and parameterization would be required to prove the Lipschitz property. This is because MDEQ has a special residual block DEQ structure, where the DEQ assumed in MonDEQ and LBEN are simply single layer DEQ. \n\nFor MOptEqs, it is conceivable to parameterize the multiple parallel branch DEQ structure to guarantee $L$-Lipschitzness of the model and make use of their optimization approach, however this would require new theory since it is not a standard LBEN. This would be very interesting future work.\n"}, {"review_id": "Fnkf78eTOw", "reviewer": "Reviewer_gX3G", "summary": "This paper demonstrates that various widely-used Lipschitz network structures, including convex potential layers (CPL), SDP-based Lipschitz layers (SLL), almost orthogonal layers (AOL), Sandwich layers, and monotone DEQs (MonDEQ), can all be reparameterized as specific cases of the Lipschitz-bounded equilibrium networks (LBEN). This reparameterization does not alter the prescribed Lipschitz constant in the original network parameterization. A notable aspect of our reparameterization technique is that it maintains the Lipschitz prescription utilized in different structures. \n", "questions": "I question whether a 1-Lipschitz is necessary, as a small Lipschitz constant can significantly limit the representation ability. Why is a K-Lipschitz, where K is a limited constant, not acceptable?\n", "limitations": "Overall, while the theoretical proof is elegant, the experimental results do not adequately support the theoretical conclusions.\n", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "\n[Strengths]\n\nThe paper is well-structured and clearly written. \n\nThe authors offer a high-level understanding of CPL, AOL, Sandwich layers, and MonDEQ, which can all be reparameterized as specific cases of LBEN. \n", "weaknesses": "[Weaknesses]\n\nWhile the theoretical results are intriguing, I believe that additional numerical results should be conducted to further validate the theorem. \n\nThe numerical results are quite preliminary and only discuss the MNIST and CIFAR10. I argue that these results do not sufficiently support the paper's theorem.\n\nThe results on CIFAR10 are not convincing. LBEN, when initialized from SLL, does not clearly outperform the SLL network.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thought and thorough evaluation. We respond to your comments as below.\n\n\n**While the theoretical results are intriguing, I believe that additional numerical results should be conducted to further validate the theorem. The numerical results are quite preliminary and only discuss the MNIST and CIFAR10. I argue that these results do not sufficiently support the paper's theorem. The results on CIFAR10 are not convincing. LBEN, when initialized from SLL, does not clearly outperform the SLL network.**\n\nThank you for comment. We have provided additional results on CIFAR100 which are also currently state-of-the-art with respect to certified robustness of DEQ. Please refer to our general comment on the significance of our experimental results to see our explanation on why we believe our numerical results do justify our theoretical claims (our key theoretical claim is that it is crucial to maintain the prescribed Lipschitz constant when reparameterizing other Lipschitz structures as LBEN).\n\nDue to scalability issues of training DEQ, applying them to larger datasets like TinyImageNet remains very challenging. We have evaluated LBEN initialized from a small SLL network on CIFAR-100 which achieves the state-of-the-art with respect to certified robustness of any existing DEQ model. Please note there is still a significant performance gap from state-of-the-art due to scalability issues of DEQ. By fine-tuning the LBEN initialized from SLL, we are able to see some improvements on certified robustness. Despite being marginal, such improvements are enough for giving us the state-of-the-art certified robustness results for DEQs on CIFAR100.\n|model | clean accuracy | $\\epsilon = 36/255$ | $\\epsilon= 72/255$ | $\\epsilon = 108/255$ | $\\epsilon = 1$ |\n| ---- | ------ | --- | --- | --- | --- |\n| SLL | 0.398 | 0.288 | 0.207 | 0.149 | 0.038 |\n| LBEN | 0.292 | 0.176 | 0.117 | 0.078 | 0.015|\n| LBEN with SLL init |  0.403 | 0.291 | 0.209 | 0.153 | 0.041 |\n\n## \n\n**I question whether a 1-Lipschitz is necessary, as a small Lipschitz constant can significantly limit the representation ability. Why is a K-Lipschitz, where K is a limited constant, not acceptable?**\n\nThis is an interesting question and non-obvious aspect of training robust Lipschitz-constrained networks. Larger Lipschitz constants for LBEN are tried in [RWM20] on CIFAR 10 ($L=2,3,5,50$) which slightly improves the clean accuracy, but **decreases the empirical robustness** when compared to the 1-Lipschitz LBEN. We have tried to use different L to improve our certified robustness results, but we did not succeed in achieving better results with larger $L$. The best certified robustness result achieved by our approach on CIFAR10 is indeed based on choosing $L=1$. One intuitive explanation for this is that the understanding of how to incorporate inductive bias via enforcing convolution structures on 1-Lipschitz layers is relatively matured, and hence choosing $L=1$ to make the Lipschitz constant consistent with these structures leads to the best certified robustness result for now. If we use LBEN with different $L$, we will have difficulty in coming up the right features for certifiably robust classification tasks. In contrast, if we use LBEN with $L=1$, we are directly fine-tuning based on the useful features learned by 1-Lipschitz networks. \n\n- [RWM20] Lipschitz-bounded equilibrium networks\n"}, {"review_id": "oY3qvkHEau", "reviewer": "Reviewer_Lz7W", "summary": "The paper addresses the robustness certification of deep equilibrium models (DEQs). It proposes a novel approach that generalizes classical Lipschitz-constrained networks by presenting them as special cases of Lipschitz-bounded equilibrium networks (LBEN). The researchers' contribution is two-fold: first, they provide conditions for a DEQ to be L-Lipschitz and extend these conditions to the reparameterization of DEQs. Second, they establish a connection between SDP-based Lipschitz layers (SLL), almost orthogonal layers (AOL), sandwich layers, and LBEN. This unique approach allows for an improved certified robustness of DEQs, further enhancing the applicability and reliability of these machine learning models.", "questions": "The paper does an excellent job bridging different types of DEQs and 1-Lipschitz neural networks, which I find commendable. However, the experimental part is very limited and doesn't match the ambition of the preceding sections\n\n-For instance, why are there no baselines with unconstrained DEQs?\n\n-Why was a different initialization scheme used for the two experiments? SLL could have been used for both the MNIST and CIFAR datasets.\n\n-The justification for not displaying results for AOL and sandwich layers only focuses on the robustness certification. However, AOL provides more than just Lipschitz guarantees, including quasi orthogonality. The paper would have a broader impact if it considered all different types of 1-Lipschitz parameterizations covered by the LBEN formalization in the experiments.\n", "limitations": "The authors adequately addressed the limitations", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "-The paper interestingly addresses the problem of Lipschitz constant certification for different types of parameterizations. This is a noteworthy contribution to the field.\n\n-The linkage between DEQ and 1-Lipschitz neural network, considering different parameterizations, provides a novel perspective on the analysis of 1-Lipschitz neural networks. The experimental results suggest this could be an efficient approach for initializing DEQ with a Lipschitz certificate.\n\n-The paper is very well written, self-contained, and reader-friendly. The state-of-the-art seems comprehensive and complete.\n", "weaknesses": "-The experimental section of the paper is somewhat limited and doesn't fully cover all the propositions put forth in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and thorough evaluation. We are happy to address your concerns and questions.\n\n**The paper does an excellent job bridging different types of DEQs and 1-Lipschitz neural networks, which I find commendable. However, the experimental part is very limited and doesn't match the ambition of the preceding sections**\n\nThank you for your comment. Please refer to our general comment on the significance of our experimental results to see more details on why we believe our results do justify our theoretical claims.\n\n**-For instance, why are there no baselines with unconstrained DEQs?**\n\nAn unconstrained DEQ will have a higher clean accuracy compared to a constrained model, however in this paper we are concerned with certified robustness. The resulting Lipschitz constant of the unconstrained model is typically much larger than $1$ and leads to certified accuracy near zero based on the margin argument.\n\n[PWK21] does an evaluation on certified robustness on MNIST and CIFAR10 but the certified accuracy obtained is very low. For CIFAR10 with $\\epsilon = 0.01$ (Figure 9 in [PWK21]), the best achieved certified robust accuracy in [PWK21] is roughly 10%. No results for the standard $\\epsilon=36/255$ is given.\n\n**-Why was a different initialization scheme used for the two experiments? SLL could have been used for both the MNIST and CIFAR datasets.**\n\nSorry for the confusion. We will clarify this in our revised draft. There are a few reasons for this. In the original Lip-MonDEQ paper, MNIST is used as a benchmark, where the original SLL work focused on CIFAR benchmarks. Before our work, Lip-MonDEQ already achieved reasonable certified robustness on MNIST and we only include MNIST results to showcase how our theoretical connections between Lip-MonDEQ and LBEN can further improve a Lip-MonDEQ model. The CIFAR setting is where our method really makes a difference, since previously DEQs do not achieve good certified robustness on these tasks. Previously, SLL works well for CIFAR and hence we use SLL as initial conditions when considering CIFAR.\n\n**-The justification for not displaying results for AOL and sandwich layers only focuses on the robustness certification. However, AOL provides more than just Lipschitz guarantees, including quasi orthogonality. The paper would have a broader impact if it considered all different types of 1-Lipschitz parameterizations covered by the LBEN formalization in the experiments.**\n\nThanks for this useful feedback. In the experiments, we have only succeeded with the SLL initializations. Investigating how to achieve good certified robustness results for LBEN initialized from AOL and Sandwich on large datasets is definitely an important future task, and we want to highlight a few challenges here. Although we do show explicitly how LBEN can be initialized from an AOL, and is indeed an equivalent DEQ network at initialization, the quasi-orthogonality property will not be preserved after further training unless additional special structure is assumed about the LBEN parameterization. In addition, AOL typically requires the use of GroupSort activation to address the gradient vanishing issue, and the current LBEN theory does not address GroupSort activations which are not sloped-restricted on [0,1]. In contrast, SLL with ReLU activations automatically address the gradient vanishing issue due to the residual network structure, and can be connected to the current LBEN theory. For sandwich layers, the current convolution forms use stride. How to incorporate consistent convolution structures for LBEN remains an open question. Although our theory does provide a clean connection between LBEN and Sandwich, how to use such a connection for vision tasks requires further study on convolutional LBEN."}, {"review_id": "6qFp8M6wyR", "reviewer": "Reviewer_MZ1P", "summary": "Since the introduction of Deep Equilibrium Models (DEQs), many papers have been written about the certified robustness properties of DEQ models including MonDEQ and GMonDEQ. In addition, many methods have been proposed to study the certified robustness of conventional neural works, such as AOL, SLL, and Sandwich Layers. In this paper, the authors show that all methods aforementioned can be encapsulated under the LBEN framework. The authors conclude this paper by showing experiment results for the MNIST dataset.", "questions": "1. The authors note in their paper that \"one can potentially fine-tune the trained models from MonDEQ, SLL, and AOL by reparameterizing the networks as LBEN and initializing the LBEN training from these reparameterized models\". Why would this be better than simply training an LBEN model? If there is any difference, under which circumstances would one be better than the other?", "limitations": "There are no negative societal impacts of this work", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper is well written and easy to read. I believe the originality and significance of this paper lies in the observation that the LBEN framework can encapsulate methods like MonDEQ and GMonDEQ. In addition, the authors also discuss the reparametrization to fit MonDEQ into the LBEN framework, which the authors correctly identified as non trivial.", "weaknesses": "I believe this paper would benefit from a clear discussion of the application scenarios of the proposed method. For example, when does the proposed method fail to provide satisfactory certified robustness guarantees, and when is it good? In addition, I believe non MNIST/CIFAR experiments should be run in order to demonstrate the effectiveness of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and thorough evaluation. We are happy to address your concerns and questions.\n\n**I believe this paper would benefit from a clear discussion of the application scenarios of the proposed method. For example, when does the proposed method fail to provide satisfactory certified robustness guarantees, and when is it good? In addition, I believe non MNIST/CIFAR experiments should be run in order to demonstrate the effectiveness of the proposed method.**\n\nThis is a very interesting comment, which will add a lot of value to our draft when addressed. The scalability in training LBENs is the main open issue. Our method will provide good certified robustness results when a relatively small 1-Lipschitz structure can be used to provide useful pretrained features for the tasks at hand. \nDue to scalability issues of training DEQ, applying them to larger datasets like TinyImageNet remains very challenging. We tried our method on TinyImageNet, but it failed due to this scalability issue. Then we tried our method (LBEN initialized from a small SLL) on CIFAR-100, and our method does achieve the state-of-the-art with respect to certified robustness of any existing DEQ model on CIFAR100. By fine-tuning the LBEN initialized from SLL, we are able to see some improvements on certified robustness. Despite being marginal, these improvements are sufficient for giving us the state-of-the-art certified robustness results of DEQs on CIFAR100.\n\n|model | clean accuracy | $\\epsilon = 36/255$ | $\\epsilon= 72/255$ | $\\epsilon = 108/255$ | $\\epsilon = 1$ |\n| ---- | ------ | --- | --- | --- | --- |\n| SLL | 0.398 | 0.288 | 0.207 | 0.149 | 0.038 |\n| LBEN | 0.292 | 0.176 | 0.117 | 0.078 | 0.015|\n| LBEN with SLL init |  0.403 | 0.291 | 0.209 | 0.153 | 0.041 |\n\nGenerally, explicit feed-forward convolutional models have very good inductive biases that help them extract useful features and perform well on vision tasks. Although, DEQ are technically more expressive than feed-forward networks, they lack many of these important inductive biases. In our paper we show how one may initialize a DEQ (as LBEN) with the useful features of SLL while maintaining its Lipschitz constant.\n\nAdmittedly, some of the vision benchmarks are not quite state-of-the-art with respect to the best explicit feed-forward SLL models because we are using much smaller initialized models. This is due to 1) scalability issues inherent to training DEQ and 2) a technical subtlety of embedding a convolutional SLL model into a DEQ that requires a much larger convolutional kernel. This also currently prevents us from scaling up to larger vision tasks like TinyImageNet, but may be circumvented in the future. For applications using standard fully-connected layers, the representation memory footprint of an equivalent DEQ is not as much of an issue. If one wants to achieve good certified robustness on very large vision tasks ImageNet for example, explicit models may be more appropriate as of right now. \n\nWe will make sure to address these subtle tradeoffs and applications in the final version of the draft.\n \n## \n\n**The authors note in their paper that \"one can potentially fine-tune the trained models from MonDEQ, SLL, and AOL by reparameterizing the networks as LBEN and initializing the LBEN training from these reparameterized models\". Why would this be better than simply training an LBEN model? If there is any difference, under which circumstances would one be better than the other?**\n\nThe performance of neural networks is highly dependent on good initialization. Additionally, explicit feed-forward 1-Lipschitz convolutional networks have important inductive biases that have been crucial for certifiably robust image-classification tasks.  In contrast, our current understanding on how to incorporate the right inductive biases for LBEN in the context of certified robustness is relative limited. Therefore, fine-tuning LBEN from 1-Lipschitz layers with good inductive biases (such as SLL) can help LBEN achieve good certified robustness via combining the benefits of inductive biases of feed-forward 1-Lipschitz networks and the expressive advantage of LBEN over explicit networks. \n"}]

Author rebuttal:
# General Response

First of all, we would like to thank each reviewer for their constructive feedback. We are glad to see that our theoretical connections on Lipschitz structures were generally well-received and there seems to be many interesting future directions. We would like to clarify some aspects of our experimental results and how they justify the claims we make in our paper.

## General comment about the significance of our experiments
Currently the understanding of the certified robustness of DEQ is very restricted, and hence our work is more exploratory in nature. However, based on the theoretical and empirical contributions, we believe it is fair to claim that our work succeeded in (i) "advancing our understanding of certified robustness of DEQ" and (ii) "improving certified robustness of DEQ on challenging tasks such as CIFAR10."

Based on our theoretical insight, we use SLL to initialize LBEN with $L=1$ (consequently the value of $\sqrt{2}L\epsilon$ is preserved) and improve the SOTA certified robust accuracy of DEQ on CIFAR10 from roughly 10% for $\epsilon=0.01\%$ (this is the result from [PWK21] which uses MonDEQ) to 64.6% for  $\epsilon=0.01\%$ and 56.2% for $\epsilon=36/255$. We were unable to obtain a remotely good certified robustness result training MonDEQ from scratch on CIFAR10. It seems that MonDEQ or G-MonDEQ have difficulty in learning useful features while maintaining the Lipschitz property and good prediction margins starting from scratch. In contrast, residual networks such as SLL are better in extracting useful features from scratch. One interpretation for our empirical results is that we use some useful features learned by SLL and then improve upon it using the expressive power of the DEQ structure. We believe our empirical results are sufficient in supporting our main theoretic idea that maintaining the value of $L$ during reparameterizing models as LBEN is important for certified robustness of DEQ.

Admittedly, some of the results are not quite state-of-the-art with respect to the best explicit feed-forward SLL models because we are using much smaller initialized models. This is due to 1) scalability issues inherent to training DEQ and 2) a technical subtlety of embedding a convolutional SLL model into a DEQ that requires a much larger convolutional kernel. This also currently prevents us from scaling up to larger vision tasks like TinyImageNet, but may be circumvented in the future. For applications using standard fully-connected layers, the representation memory footprint of an equivalent DEQ is not as much of an issue.
 
- [PWK21] Estimating Lipschitz constants of monotone deep equilibrium models

Meta-review: This paper studies the robustness certification of deep equilibrium models. The authors show that various popular Lipschitz network structures can all be reparameterized as special cases of the Lipschitz-bounded equilibrium networks. The results are interesting and clearly presented.
Decision: Accept (poster)