SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. The above formula can be easily verified by substituting the expressions for  $W$  into both the MonDEQ and LBEN parameterizations and then making simplifications. The only thing that needs to be further checked is that the resultant expression for  $V^{\top}V$  must be positive semidefinite so we can obtain  $V$  by decomposition. Since we know  $\| U\| _2\leq m$ , we must have
$$
\begin{array}{l} V ^ {\top} V = \left(\frac {m}{\| U \| _ {2}} - \frac {1}{2}\right) I + \frac {A ^ {\top} A}{\| U \| _ {2}} - \frac {1}{2 \| U \| _ {2} ^ {2}} U U ^ {\top} (11) \\ \succeq \frac {A ^ {\top} A}{\| U \| _ {2}} + \frac {1}{2} \left(I - \frac {1}{\| U \| _ {2} ^ {2}} U U ^ {\top}\right) \succeq 0 (12) \\ \end{array}
$$
Hence the resultant expression for  $V^{\top}V$  is positive semidefinite. This completes the proof.
We can clearly see that in the above result, the Lipschitz constant is preserved during the reparameterization.
3.2 Connections between G-MonDEQ and LBEN
The Lipschitz constant of the G-MonDEQ parameterization (6) has not been explicitly provided by the original work in [35]. Now we will present such an explicit formula via applying (10). Formally, we have the following result.
Theorem 3.3. Consider the  $G$ -MonDEQ parameterization (6). It is guaranteed that (6) gives a mapping which is  $\frac{\|\Phi^{-1}U\|}{m}$ -Lipschitz from  $x$  to  $z$ .
Proof. We are interested in the Lipschitz constant from  $x$  to  $z$ , and hence we choose  $G = I$ . As explained before, the G-MonDEQ parameterization is known to be well-posed based on Proposition 2.1. Hence we can apply Theorem 3.1 to upper bound its Lipschitz constant. Specifically, we can choose  $\Lambda = \frac{1}{\|\Phi^{-1}U\|_2}\Phi^{-1}$ , and  $L = \frac{\|\Phi^{-1}U\|_2}{m}$ . Then the left side of (10) becomes
$$
\begin{array}{l} \frac {1}{L} I + \frac {2}{\| \Phi^ {- 1} U \| _ {2}} A ^ {\top} A - \frac {1}{L \| \Phi^ {- 1} U \| _ {2} ^ {2}} \Phi^ {- 1} U U ^ {\top} \Phi^ {- 1} \\ = \frac {2}{\| \Phi^ {- 1} U \| _ {2}} A ^ {\top} A + \frac {1}{L} \left(I - \frac {1}{\| \Phi^ {- 1} U \| _ {2} ^ {2}} \Phi^ {- 1} U U ^ {\top} \Phi^ {- 1}\right), \\ \end{array}
$$
which is obviously positive semidefinite. Therefore, (10) is feasible for G-MonDEQ with  $L = \frac{\|\Phi^{-1}U\|_2}{m}$ . This leads to the desired conclusion.
If we choose  $\Phi = I$ , the above theorem just recovers [31, Theorem 1] as a special case. From the above theorem, we can ensure G-MonDEQ to be 1-Lipschitz by enforcing  $\| \Phi^{-1}U \|_2 \leq m$  during training.
Next, we can leverage the above construction of  $\Lambda$  to establish a similar reparameterization result connecting G-MonDEQ and LBEN.
Theorem 3.4. Given a 1-Lipschitz  $G$ -MonDEQ model with  $(\Phi, A, B, U)$  satisfying  $\| \Phi^{-1}U \|_2 \leq m$ , the following choice of  $(\Psi, G, U, V, S, \epsilon)$  with  $U$  being the same one used for  $G$ -MonDEQ gives a valid 1-Lipschitz LBEN model which generates exactly the same input-output mapping:
$$
\begin{array}{l} \Psi = \| \Phi^ {- 1} U \| _ {2} \Phi , \quad G = I, \quad S = \frac {1}{\| \Phi^ {- 1} U \|} B, \quad \epsilon = 0 \\ V ^ {\top} V = \left(\frac {m}{\| \Phi^ {- 1} U \| _ {2}} - \frac {1}{2}\right) I + \frac {A ^ {\top} A}{\| \Phi^ {- 1} U \| _ {2}} - \frac {1}{2 \| \Phi^ {- 1} U \| _ {2} ^ {2}} \Phi^ {- 1} U U ^ {\top} \Phi^ {- 1} \succeq 0. \\ \end{array}
$$
Proof. The proof is very similar to the proof of the MonDEQ case and hence omitted.
Again, the prescribed Lipschitz constant is preserved during our reparameterization.
3.3 Connections between SLL and LBEN
Extending the previous analysis, we can further build connections between LBEN and explicit networks such as SLL. The SLL network is parameterized by  $\{W_{n},T_{n}\}$  with  $W_{n}^{\top}W_{n}\preceq T_{n}$ . Again, we can choose  $(\Psi ,G,U,V,S,\epsilon)$  properly to generate a valid 1-Lipschitz LBEN which gives the same input-output mapping. Similar to the previous treatment, we will first recover the Lipschitz analysis of SLL to gain some insights on how to construct  $\Lambda$  for the reparameterization. Notice that the feed-forward network is always well-posed. Hence we can just rewrite a feed-forward residual network in the form of (1) and then apply Theorem 3.1 to analyze its Lipschitz property. To see this, consider the residual structure  $x_{n + 1} = x_{n} - 2W_{n}T_{n}^{-1}\sigma (W_{n}^{\top}x_{n} + b_{n})$  and  $x_0 = x$ . Denoting  $\tilde{x}_{n + 1} = \sigma (W_n^\top x_n + b_n)$  ( $n\geq 1$ ) and  $\tilde{x}_0 = x_0$ , we can express the entire multi-layer residual structure in terms of  $(\tilde{x}_0,\dots ,\tilde{x}_{N - 1})$  as
$$
\tilde {x} _ {n + 1} = \sigma \left(W _ {n} ^ {\top} \left(\tilde {x} _ {0} - \sum_ {k = 0} ^ {n - 1} 2 W _ {k} T _ {k} ^ {- 1} \tilde {x} _ {k + 1}\right) + b _ {n}\right), \quad y = \tilde {x} _ {0} - \sum_ {k = 0} ^ {N - 1} 2 W _ {k} T _ {k} ^ {- 1} \tilde {x} _ {k + 1} \tag {13}
$$
for  $n \in \{1, \ldots, N-1\}$ . With this description, we can rewrite the SLL parameterization in the DEQ form of  $(z, y) = (\phi(Wz + Ux + b_x), Gz + b_y)$  with  $z = [\tilde{x}_0^\top, \ldots, \tilde{x}_l^\top]^\top$ ,  $\phi(\cdot) =$
$[id(\cdot)^{\top}, \sigma(\cdot)^{\top}, \ldots, \sigma(\cdot)^{\top}]^{\top 3}$ ,  $G = [I, -2W_0T_0^{-1}, \ldots, -2W_{N-1}T_{N-1}^{-1}]$ ,  $U = [I, 0, \ldots, 0]^{\top}$ , and  $W$  being given as the following matrix:
$$
W = \left[ \begin{array}{c c c c c} 0 & & & & \\ W _ {0} ^ {\top} & 0 & & & \\ W _ {1} ^ {\top} & - 2 W _ {1} ^ {\top} W _ {0} T _ {0} ^ {- 1} & \ddots & & \\ \vdots & \vdots & \ddots & 0 & \\ W _ {N - 1} ^ {\top} & - 2 W _ {N - 1} ^ {\top} W _ {0} T _ {0} ^ {- 1} & \dots & - 2 W _ {N - 1} ^ {\top} W _ {N - 2} T _ {N - 2} ^ {- 1} & 0 \end{array} \right]. \tag {14}
$$
It is not surprising that the above matrix has a lower-triangular structure, since the original SLL network is feed-forward. Now it is straightforward to verify that (10) is feasible (see Appendix C) with  $L = 1$  and  $\Lambda$  being given by
$$
\Lambda = \operatorname {b l k d i a g} \left(I, 2 T _ {0} ^ {- 1}, \dots , 2 T _ {N - 1} ^ {- 1}\right). \tag {15}
$$
Now we can reparameterize SLL as LBEN with preserving the 1-Lipschitz property as follows.
Theorem 3.5. Given any SLL structure parameterized by  $\{W_{n},T_{n}\}_{n = 0}^{N - 1}$  with  $W_{n}^{\top}W_{n}\preceq T_{n}$  for all  $n$ , the resulting input-output relation from  $x$  to  $y$  can be exactly recovered with the LBEN parameterization with the following choice of  $(\Psi ,G,U,V,S,\epsilon)$ :
$$
\Psi = \Lambda^ {- 1} (a s i n E q. (1 5)), \quad G = [ I, - 2 W _ {0} T _ {0} ^ {- 1}, \dots , - 2 W _ {N - 1} T _ {N - 1} ^ {- 1} ]
$$
$$
U = [ I, 0, \dots , 0 ] ^ {\top}, \quad V ^ {\top} V = \frac {1}{2} (M + M ^ {\top}), \quad S = \frac {1}{2} M, \quad \epsilon = 0,
$$
where  $M$  is given as  $M = \Lambda (I - W) - \frac{1}{2}\left(G^{\top}G + \Lambda UU^{\top}\Lambda\right)$  with  $W$  being defined by (14).
A detailed proof for the above theorem will be presented in the appendix. We want to comment that one can easily verify that the expression for  $V^{\top}V$  is given explicitly by a block-diagonal matrix with a leading zero block and other remaining blocks defined as
$$
\left(V ^ {\top} V\right) _ {n n} = 2 T _ {n} ^ {- 1} - 2 T _ {n} ^ {- 1} W _ {n} ^ {\top} W _ {n} T _ {n} ^ {- 1} \tag {16}
$$
for  $n \in \{1, \dots, N - 1\}$ . Obviously, the resultant matrix for  $V^{\top}V$  is positive semidefinite. We also emphasize that the above reparameterization works for both SLL and CPL, as long as one chooses  $\{T_n\}$  properly. In addition, it is worth noting that one can set  $\Psi = \mathrm{diag}(\exp(d_i))$ , leading to unconstrained decision variables  $\{d_i\}$ , which can be useful for finetuning LBEN from SLL.
Reparameterizing AOL and Sandwich Layers as LBEN. Notice that non-residual Lipschitz network structures such as AOL and Sandwich can also be connected to LBEN. For the experimental evaluation, we mainly utilize the SLL initialization of LBEN. We leave the discussions on the explicit connections between AOL, Sandwich, and LBEN to the appendix.
In this section, we present some numerical experiments and insights on how to initialize LBEN with other Lipschitz architectures. Such numerical study will demonstrate and support our theory.
4.1 Convolutional LBEN and Implementation Details
For classification tasks, we need to embed convolutional layers into the LBEN theoretical framework. Although this is admissible in principal, there are practical aspects that need extra explanations.
Initialization of Convolutional LBEN. When initializing LBEN parameters  $(V, S)$  from a trained SLL network, we require decomposing  $\frac{1}{2}(M + M^{\top})$  as constructed from Section 3 ( $M$  is given by Theorem 3.5). For convolutional layers, we consider convolution with circular padding which is known to be close under sum, product, and inverse. Hence, all operations required for forming  $M$  preserve the convolution structure and so  $\frac{1}{2}(M + M^{\top})$  is also convolution with circular padding. Furthermore, we can find  $V$  by taking the matrix square root of  $\frac{1}{2}(M + M^{\top})$  efficiently using the block-diagonalization trick of circular convolution (see the appendix for more details).
<sup>3</sup>This just means that the first element of the activation is replaced by an identity mapping. If  $\sigma$  is slope-restricted on  $[0,1]$ , the resultant  $\phi$  is still slope-restricted on  $[0,1]$ , and hence Theorem 3.1 can still be applied.
$\ell_1$  Regularization of LBEN Sparsity Structure. Although we endow the weight  $W$  with a convolution structure, which will be preserved throughout training, this parameterization uses a relatively large kernel. Rather than using, for example, a  $3 \times 3$  kernel applied across the image, our kernel parameterization is the size of the input image  $n \times n$  (i.e.,  $W$  is defined as a large kernel of size  $dc_{out} \times dc_{in} \times n \times n$ , where  $d$  corresponds to the number of convolutions). Indeed, the inverse or square root of a convolution with a  $3 \times 3$  kernel applied on an input  $n \times n$  input is another convolution with a  $n \times n$  kernel.
When initializing the LBEN from an existing SLL model,  $W$  will be quite sparse due to the original small kernel parameterization. However, this sparsity will not necessarily be preserved after training. This is because the gradients with respect to  $W$  (defined with kernel  $3 \times 3$ ) depend on the inverse of  $W$  (which has a kernel of size  $n \times n$ ).
In order to heuristically preserve this sparsity structure and not destroy the inductive bias given by the initial architecture, we impose a small  $\ell_1$ -penalty to non-zero values on the border of the kernel to induce learning small-size kernels. This way, we can regularize the structure of  $W$  without placing explicit constraints on the LBEN parameterization.
4.2 Experiments: Evaluation of Pretrained Initialization and Discussion
For the experiments, we trained MonDEQ and SLL networks that would serve as initialization for LBEN. To train a 1-Lipschitz MonDEQ, we use spectral normalization [29] on the matrix  $U$  and set  $m = 1$ . For the SLL network (e.g., (2) in Table 1), we use 4 convolutional layers with circular padding and 2 dense layers. The convolutional and dense layers have 8 and 512 channels/features, respectively. We use the ReLU nonlinearity which is slope-restricted on  $[0,1]$ . Our final LBEN network is a composition of convolutional and dense LBEN blocks, initialized from the four convolutions and the two dense layers of the pretrained Lipschitz model, respectively. To fine-tune the LBEN after initialization, we use a small learning rate of  $1\mathrm{e} - 7$  with a  $\ell_1$ -regularization of 0.1 during 40 epochs (for more details, see the publicly available code). The results are presented in Table 1.
Note that we omit any unconstrained DEQ baselines which may have a higher clean accuracy compared to a constrained model, but very low certified robustness. The resulting Lipschitz constant of the unconstrained model is typically much larger than 1 and leads to certified accuracy near zero based on the margin argument. For instance, the work of [31] provides an evaluation of DEQ on CIFAR10 for  $\varepsilon = 0.01$  achieving certified robust accuracy of roughly  $10\%$  (estimate is read off graph from [31, Figure 9]). Previously, no certified robustness results for the standard perturbation  $\varepsilon = 36 / 255$  have been reported for unconstrained DEQ.
LBEN Initialized with Lip-MonDEQ. First, we present a small-scale experiment on the MNIST dataset with Lip-MonDEQ and LBEN initialized from the trained Lip-MonDEQ. SLL and LBEN baselines are omitted since LBEN already achieves reasonable performance and our aim is to
showcase how our theoretical framework can provide further improvements on existing architectures. We observe that the LBEN significantly improves the natural and provable accuracy over the Lip-MonDEQ. This is mainly due to the different Lipschitz constraints used in Lip-MonDEQ and LBEN.
LBEN initialized with SLL. We now present some experiments on CIFAR10 and CIFAR100 datasets [20]. First, we trained an SLL network (2) with circular padding with the same hyperparameters as in [2]. We then compare the natural and certified accuracy between an LBEN trained from scratch (random initialization) and an LBEN initialized from the SLL model. We observe that the LBEN with random initialization offers poor natural and certified accuracy compared to the SLL network for both the CIFAR10 and CIFAR100 datasets. On the other hand, when LBEN is initialized from the trained SLL weights, it successfully improves upon SLL by approximately  $1\%$  in certified accuracy on CIFAR10 and a marginal improvement on CIFAR100. Despite being marginal improvements over SLL, these results significantly improve upon the current state-of-the-art  $\ell_2$ -certified accuracy for DEQs.
One interpretation of these results is that the performance of neural networks is highly-dependent on a good initialization [39, 1]. Additionally, explicit feed-forward 1-Lipschitz convolutional networks have important inductive biases that have been crucial for achieving good certified robustness results on image-classification tasks [9]. In contrast, our current understanding on how to incorporate the right inductive biases for LBEN in the context of certified robustness is relative limited. Therefore, fine-tuning LBEN from 1-Lipschitz layers with good inductive biases (in our case SLL) helps LBEN achieve improved certified robustness via combining the benefits of inductive biases of feed-forward 1-Lipschitz networks and the expressive advantage of LBEN over explicit networks.
LBEN initializations with Lipschitz constant other than  $L = 1$ . Larger Lipschitz constant parameterizations for LBEN are explored in [35] on CIFAR10 ( $L = 2, 3, 5, 50$ ), which slightly improves clean accuracy, but decreases the empirical robustness when compared to the 1-Lipschitz LBEN. The best certified robustness results achieved by our approach on CIFAR10 and CIFAR100 tasks are indeed achieved by choosing  $L = 1$ . At this moment, the understanding of how to incorporate inductive bias via enforcing convolution structures on 1-Lipschitz layers is relatively matured. Hence choosing  $L = 1$  to make the Lipschitz constant consistent with these structures leads to the best certified robustness result for now. In the future, it is possible that one can improve the certified robustness of DEQ for  $L > 1$  by developing new convolutional structures for LBEN.
Remark. The results stated in Table 1 are not on par with the state-of-the-art certified robust accuracy obtained by the largest explicit Lipschitz feed-forward models. This is due to the following reasons: 1) DEQs are known to be computationally expensive as they require the computation of a fixed point for each input  $x$ . Therefore, in order to keep the computational cost for LBEN models reasonable, we trained small SLL networks. 2) There is a technical subtlety of embedding a convolutional SLL model into DEQ that requires a much larger convolutional kernel. This also currently prevents us from scaling up to larger vision tasks like TinyImageNet, but may be circumvented in the future. For applications using standard fully-connected layers, the representation memory footprint of an equivalent DEQ does not present such an issue. Despite these challenges, to the best of our knowledge, our work provides the best  $\ell_2$ -certified accuracy with DEQs.
In this paper, we present a unified algebraic approach for deriving 1-Lipschitz DEQ structures via providing analytical solutions to an SDP condition. We show that several Lipschitz structures (e.g., CPL, SLL, AOL, and MonDEQ) can be reparameterized as special cases of LBEN without changing the original Lipschitz constant specifications. We provide explicit formulas for such Lipschitz-preserved reparameterization. Finally, our experiments show that our theory opens the possibility of achieving improved robustness of DEQs via a combination of network reparameterization, structure-preserving regularization, and LBEN-based fine-tuning.
A. Havens and B. Hu are generously supported by the NSF award CAREER-2048168, the AFOSR award FA9550-23-1-0732, and the IBM/IIDAI award 110662-01. A. Araujo, S. Garg, and F. Khorrami
are supported in part by the Army Research Office under grant number W911NF-21-1-0155 and by the New York University Abu Dhabi (NYUAD) Center for Artificial Intelligence and Robotics, funded by Tamkeen under the NYUAD Research Institute Award CG010.
[42] Y. Tsuzuki, I. Sato, and M. Sugiyama. Lipschitz-margin training: Scalable certification of perturbation invariance for deep neural networks. In Advances in Neural Information Processing Systems, 2018.
[43] R. Wang and I. R. Manchester. Direct parameterization of Lipschitz-bounded deep networks. In International Conference on Machine Learning, 2023.
[44] Z. Wang, G. Prakriya, and S. Jha. A quantitative geometric approach to neural-network smoothness. Advances in Neural Information Processing Systems, 35:34201-34215, 2022.
[45] Z. Wang, S. Jha, et al. Efficient symbolic reasoning for neural-network verification. arXiv preprint arXiv:2303.13588, 2023.
[46] C. Wei and J. Z. Kolter. Certified robustness for deep equilibrium models via interval bound propagation. In International Conference on Learning Representations, 2021.
[47] E. Winston and J. Z. Kolter. Monotone operator equilibrium networks. Advances in neural information processing systems, 33:10718-10728, 2020.
[48] Z. Yang, P. Li, T. Pang, and Y. Liu. Improving adversarial robustness of DEQs with explicit regulations along the neural dynamics. In Proceedings of the 40th International Conference on Machine Learning, 2023.
[49] T. Yu, J. Li, Y. Cai, and P. Li. Constructing orthogonal convolutions in an explicit manner. In International Conference on Learning Representations, 2022.
A Controlling the Lipschitz Constant in MonDEQ and G-MonDEQ
As discussed in Section 3, there are explicit relationships between the MonDEQ and G-MonDEQ parameterizations and the LBEN parameterization. Although nominally the MonDEQ only guarantees well-posedness of the DEQ, we can control the Lipschitz constants through the hyper-parameter  $m > 0$ , the input matrix  $U$  and, in the case of G-MonDEQ, the additional parameter  $\Phi$ .
It was previously shown by [31] that a MonDEQ has a Lipschitz constant  $L = \| U\| _2 / m$ . If we wish to parameterize  $U$  so that MonDEQ is always 1-Lipschitz, we can choose  $U = mP$  where  $P$  is an orthogonal matrix. Orthogonal matrices can be parameterized by considering their subgroup of special orthogonal matrices  $\mathrm{SO}(\mathfrak{n})$  through the Cayley transform or matrix exponential on skew-symmetric matrices used in [41].
In the case of G-MonDEQ, a new condition is derived (Theorem 3.3) showing that the G-MonDEQ parameterization of (6) is  $||\Phi^{-1}U||_2 / m$ -Lipschitz from  $x$  to  $z$ . In this case, we must simultaneously control  $U$  and  $\Phi$  to maintain the Lipschitz bound. A similar approach can be applied to jointly parameterize  $\Phi^{-1}U$ . For example, to maintain a Lipschitz bound of  $L$ , one may practically enforce the constraints  $\Phi_{ii} \geq 1 / (Lm)$  and  $U^\top U \preceq I$  through SLL or Cayley parameterizations [2].
B Additional Proofs
Proof. Theorem 3.1 follows almost exactly as in [35, Theorem 2] with an added assumption of well-posedness, i.e. now it is assumed that there exists a unique solution  $z = \sigma (Wz + Ux + b_z)$  for any  $x$ . We present the details here for completeness.
As formulated in [35], we can isolate the nonlinear parts of the DEQ through the following algebraic representation:
$$
v = W z + U x + b _ {z}, \quad z = \sigma (v), \quad y = G z + b _ {y} \tag {17}
$$
Suppose that given two inputs  $x$  and  $x'$ , the two resultant solutions for (17) are given by:
$$
v = W z + U x + b _ {z}, \quad z = \sigma (v), \quad y = G z + b _ {y}
$$
$$
v ^ {\prime} = W z ^ {\prime} + U x ^ {\prime} + b _ {z}, \quad z ^ {\prime} = \sigma (v ^ {\prime}), \quad y ^ {\prime} = G z ^ {\prime} + b _ {y}
$$
We define  $\Delta_v = v' - v$  and  $\Delta_z = z' - z$ . Notice that the nonlinearity  $\sigma$  is assumed to be slope-restricted on  $[0,1]$ . Hence the following inequality must hold for any positive definite diagonal matrix  $\Lambda$ :
$$
\left\langle \Delta_ {v} - \Delta_ {z}, \Delta_ {z} \right\rangle_ {\Lambda} \geq 0 \tag {18}
$$
where the weighted inner product is based on using  $\Lambda$  as the weight. Now define  $\Delta_x = x' - x$  and  $\Delta_y = y' - y$ , respectively. Obviously, we have  $\Delta_v = W\Delta_z + U\Delta_x$  and  $\Delta_y = G\Delta_z$ .
Now we are ready to show the desired Lipschitz property using the matrix inequality (10). By Schur complement, we have the following equivalent matrix inequality:
$$
\left[ \begin{array}{c c} 2 \Lambda - \Lambda W - W ^ {\top} \Lambda - \frac {1}{L} G ^ {\top} G & - \Lambda U \\ - U ^ {\top} \Lambda & L I \end{array} \right] \succeq 0
$$
Left and right multiplying on both sides by  $[\Delta_x^\top \Delta_z^\top ]$  and  $[\Delta_x^\top \Delta_z^\top ]^\top$  respectively yields
$$
2 \Delta_ {z} ^ {\top} \Lambda \Delta_ {z} - 2 \Delta_ {z} ^ {\top} \Lambda W \Delta_ {z} - \frac {1}{L} \Delta_ {z} ^ {\top} G ^ {\top} G \Delta_ {z} - 2 \Delta_ {z} ^ {\top} \Lambda U \Delta_ {x} + L \| \Delta_ {x} \| ^ {2} \geq 0 \tag {19}
$$
Using the relations  $\Delta_v = W\Delta_z + U\Delta_x$  and  $\Delta_y = G\Delta_z$ , we can re-arrange the above inequality as
$$
L \left\| \Delta_ {x} \right\| ^ {2} - \frac {1}{L} \left\| \Delta_ {y} \right\| ^ {2} \geq 2 \Delta_ {z} ^ {\top} \Lambda \Delta_ {v} - 2 \Delta_ {z} \Lambda \Delta_ {z} = 2 \langle \Delta_ {v} - \Delta_ {z}, \Delta_ {z} \rangle_ {\Lambda} \geq 0. \tag {20}
$$
Therefore, we must have  $\| \Delta_y\| \leq L\| \Delta_x\|$ , and the DEQ model (1) has to be  $L$ -Lipschitz.
B.2 Proof of Theorem 3.5
Proof. Given an SLL residual network, it is clear that we can define an equivalent well-posed DEQ by setting  $W, U, G$  as in the theorem statement. If we substitute the parameters  $(\Psi, G, U, S, \varepsilon)$  into the parameterization in (7), we are then left with
$$
\frac {1}{2} (M + M ^ {\top}) = V ^ {\top} V
$$
So it remains to show that  $V^{\top}V$  is positive semidefinite, and then it is trivial to see that  $V$  can be determined by decomposition.
Notice that the expression for  $(M + M^{\top})$  is exactly equal to the left side of the matrix inequality condition given by Eq. (10). It is simple to verify that this sufficient condition (10) is satisfied (see Section C). Therefore we have  $1 / 2(M + M^{\top})\succeq 0$ . Due to a leading zero block, it is necessary that  $\varepsilon = 0$ . Because  $1 / 2(M + M^{\top})$  is positive semidefinite, we can always compute its decomposition (e.g., via the Cholesky decomposition or matrix square-root). Therefore,  $W$  can be recovered by our parameter choice  $(\Psi ,G,U,V,S,\epsilon)$ , and the proof is complete.
C Feasibility of SLL as LBEN