SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In Section 3, when discussing the connection between the SLL architecture and LBEN, it is claimed that using the weight structure:
$$
\begin{array}{l} W = \left[ \begin{array}{c c c c c} 0 & & & & \\ W _ {0} ^ {\top} & 0 & & & \\ W _ {1} ^ {\top} & - 2 W _ {1} ^ {\top} W _ {0} T _ {0} ^ {- 1} & \ddots & & \\ \vdots & \vdots & \ddots & 0 & \\ W _ {N - 1} ^ {\top} & - 2 W _ {N - 1} ^ {\top} W _ {0} T _ {0} ^ {- 1} & \ldots & - 2 W _ {N - 1} ^ {\top} W _ {N - 2} T _ {N - 2} ^ {- 1} & 0 \end{array} \right], \quad U = \left[ \begin{array}{c} I \\ 0 \\ \vdots \\ 0 \end{array} \right] \\ G = [ I, - 2 W _ {0} T _ {0} ^ {- 1}, \dots , - 2 W _ {N - 1} T _ {N - 1} ^ {- 1} ] \\ \end{array}
$$
satisfies the non-strict LBEN sufficient condition of Theorem (3.1) for  $L = 1$  and  $\Lambda$  given as follows:
$$
\Lambda = \left[ \begin{array}{c c c c} I & & & \\ & 2 T _ {0} ^ {- 1} & & \\ & & \ddots & \\ & & & 2 T _ {N - 1} ^ {- 1} \end{array} \right].
$$
We can quickly prove that this is true. Plugging in  $W, G, U$  and  $\Lambda$  into the inequality (10), all off-diagonal block terms will cancel and we are left with the follow block-diagonal matrix:
$$
\left[ \begin{array}{c c c c} 0 & & & \\ & 2 T _ {0} ^ {- 1} - 2 T _ {0} ^ {- 1} W _ {0} ^ {\top} W _ {0} T _ {0} ^ {- 1} & & \\ & & \ddots & \\ & & & 2 T _ {N - 1} ^ {- 1} - 2 T _ {N - 1} ^ {- 1} W _ {N - 1} ^ {\top} W _ {N - 1} T _ {N - 1} ^ {- 1} \end{array} \right]. \tag {21}
$$
Because it is block-diagonal, it is enough to check positive semidefiniteness of each block. Because each  $T_{n}$  is defined in the SLL architecture [2] and satisfies  $W_{n}^{\top}W_{n}\preceq T_{n}$ , the following conclusion must hold:
$$
2 T _ {n} ^ {- 1} - 2 T _ {n} ^ {- 1} W _ {n} ^ {\top} W _ {n} T _ {n} ^ {- 1} \succeq 2 T _ {n} ^ {- 1} - 2 T _ {n} ^ {- 1} T _ {n} T _ {n} ^ {- 1} = 0.
$$
Therefore, the SDP sufficient condition (10) is feasible.
D Further Connections between Existing Lipshitz Architectures and LBEN
D.1 Connections between AOL and LBEN
In a similar manner, we may also reparameterize AOL as LBEN with  $L = 1$ . Suppose we have the following multi-layer AOL network structure
$$
x _ {0} = x
$$
$$
x _ {j + 1} = \phi \left(W _ {j} D _ {j} x _ {j} + b _ {j}\right), \quad j \in \{0, \dots , l - 1 \}
$$
$$
f _ {\theta} (x) = W _ {l} D _ {l} x _ {l} + b _ {l}
$$
where  $D_{j}$  are given by (8). Then the corresponding DEQ may be defined by setting:  $z = [x_1^\top, \ldots, x_l^\top]^\top$ ,  $\sigma(z) = [\phi(z_1)^\top, \ldots, \phi(z_l)^\top]^\top$ ,  $U = [(W_0D_0)^\top, \ldots, 0]^\top$ ,  $G = [0, \ldots, W_lD_l]$  and choosing  $W$  as
$$
W = \left[ \begin{array}{c c c c} 0 & & & \\ W _ {1} D _ {1} & 0 & & \\ & \ddots & \ddots & \\ & & W _ {l - 1} D _ {l - 1} & 0 \end{array} \right]
$$
It can be shown that by choosing  $\Lambda = I$ , the AOL network will satisfy the non-strict 1-Lipschitz LBEN condition (10). By defining  $M$  and  $S$  in the same way as in Theorem 3.5, we can obtain  $V^{\top}V = \frac{1}{2}(M + M^{\top})$ , which can be verified to be positive semidefinite. With this choice of  $V, S$  and  $\Psi = I$ , we have recovered  $W$  by the LBEN parameterization (7).
D.2 Connections Between Direct Sandwich Parameterizations and LBEN
Suppose we have the following multi-layer network with the Sandwich layer structure of [43] with a prescribed Lipschitz bound of  $L$ :
where  $A_{n}A_{n}^{\top} + B_{n}B_{n}^{\top} = I$ ,  $B_{N}^{\top}B_{N} = I$ , and  $\Psi_{n}$  is a free diagonal matrix whose entries are restricted to be non-negative. One can parameterize  $[A_{n}\quad B_{n}]^{\top}$  as a semi-orthogonal matrix using the generalized Cayley transformation [19], and train the above network efficiently.
We can rewrite Sandwich as
$$
y _ {0} = \sqrt {L} x, \quad y _ {n + 1} = \sigma (2 \Phi_ {n} ^ {- 1} B _ {n} A _ {n - 1} ^ {\top} \Phi_ {n - 1} y _ {n} + b _ {n}), \quad y = \sqrt {2 L} B _ {N} A _ {N - 1} ^ {\top} \Phi_ {N - 1} y _ {N} + b _ {N}.
$$
Then the DEQ may be defined by setting:  $z = [y_1^\top, \ldots, y_N^\top]^\top$ ,  $\sigma(z) = [\phi(z_1)^\top, \ldots, \phi(z_N)^\top]^\top$ ,  $U = [(\sqrt{2L}\Phi_0^{-1}B_0)^\top, \ldots, 0]^\top$ ,  $b_x = [b_0^\top, \ldots, b_{N-1}^\top]^\top$ .  $G = [0, \ldots, \sqrt{2L}B_NA_{N-1}^\top\Phi_{N-1}]$ ,  $b_y = b_N$ , and choosing  $W$  as
$$
W = \left[ \begin{array}{c c c c} 0 & & & \\ 2 \Phi_ {1} ^ {- 1} B _ {1} A _ {0} ^ {\top} \Phi_ {0} & 0 & & \\ & \ddots & \ddots & \\ & & 2 \Phi_ {N - 1} ^ {- 1} B _ {N - 1} A _ {N - 2} ^ {\top} \Phi_ {N - 2} & 0 \end{array} \right] \tag {22}
$$
Given any Sandwich network, the input-output relationship can be exactly recovered by an LBEN parameterization with the following choice of  $(\Psi, G, U, V, S, \epsilon)$ :
$$
\Lambda = \operatorname {d i a g} \left(\Phi_ {0} ^ {2}, \Phi_ {1} ^ {2}, \dots , \Phi_ {N - 1} ^ {2}\right), \quad \Psi = \Lambda^ {- 1}
$$
$$
G = [ 0, \dots , \sqrt {2 L} B _ {N} A _ {N - 1} ^ {\top} \Phi_ {N - 1} ], U = [ (\sqrt {2 L} \Phi_ {0} ^ {- 1} B _ {0}) ^ {\top}, \dots , 0 ] ^ {\top}
$$
$$
V ^ {\top} V = \frac {1}{2} (M + M ^ {\top}), \quad S = \frac {1}{2} M, \quad \epsilon = 0,
$$
where  $M$  is given by  $M = \Lambda (I - W) - \frac{1}{2L}\left(G^{\top}G + \Lambda UU^{\top}\Lambda\right)$ .
As a matter of fact, it can be shown that  $V$  has the following analytical form:
$$
V = \left[ \begin{array}{c c c c} A _ {0} ^ {\top} \Phi_ {0} & B _ {1} ^ {\top} \Phi_ {1} & & \\ & A _ {1} ^ {\top} \Phi_ {1} & B _ {2} ^ {\top} \Phi_ {2} & \\ & & \ddots & \ddots \\ & & & A _ {N - 1} ^ {\top} \Phi_ {N - 1} \end{array} \quad B _ {N} ^ {\top} \Phi_ {N} \right] \tag {23}
$$
E Additional Details of Convolutional LBEN
When it comes to image-classification tasks, one of the most useful architecture biases is the use of convolutional layers which are heavily used in all state-of-the-art SLL and AOL models, including our results. Convolutions with circular padding are straightforward to integrate into DEQ by representing them as doubly-block circulant weight matrices. In particular suppose we have an  $n \times n$  image input and  $K \in \mathbb{R}^{n \times n}$  a convolution kernel and define a doubly-block circulant matrix  $W = \mathrm{blkcirc}(K_0, \ldots, K_{n-1}) \in \mathbb{R}^{n^2 \times n^2}$ , where  $K_i = \mathrm{circ}(K_{i,:})$ . Then the 2D convolution of the vectorized input  $x$  is given by the linear operation
$$
W \operatorname {v e c} (x) = \operatorname {v e c} (K \star x) \tag {24}
$$
where  $\star$  is the convolution operation. The concept can be extended to convolutions with multiple input and output channels, where  $W$  will be a block-doubly-block circulant matrix in  $\mathbb{R}^{c_{out}n^2\times c_{in}n^2}$ .
This linear form has several interesting properties linked to the Fourier transform which allows efficient operations. Indeed, for initializing LBEN, we must use an expensive matrix square-root operation to compute the LBEN parameter  $V$ . More importantly, the operator splitting approach in [47] requires an inverse of  $W$  for each step of both training and evaluation. Fortunately, there exists an efficient approach for diagonalizing this particular representation of convolutions [41] which has been used previously for convolutional MonDEQs [47]. In particular, given a doubly-block circulant weight matrix  $W \in \mathbb{R}^{c_{out}n^2 \times c_{in}n^2}$  with kernel  $K \in \mathbb{R}^{c_{out} \times c_{in} \times n \times n}$ , we can obtain its block-diagonalization as follows from [41, Corollary A.1.1]:
$$
W = \left(I _ {c _ {o u t}} \otimes F _ {n}\right) S _ {c _ {o u t}} \hat {D} S _ {c _ {i n}} ^ {\top} \left(I _ {c _ {i n}} \otimes F _ {n} ^ {- 1}\right) \tag {25}
$$
where  $S_{c_{out}}$  and  $S_{c_{in}}$  are permutations matrices,  $F_{n} = U_{n} \otimes U_{n}$ , and  $U_{n}$  is a Fourier basis matrix. Here  $\widehat{D}$  will be a block-diagonal matrix with  $n^2$  blocks of size  $c_{out} \times c_{in}$  and is given by
$$
\widehat {D} = S _ {c _ {o u t}} ^ {\top} \left[ \begin{array}{c c c} F _ {n} \operatorname {v e c} \left(K _ {0, 0}\right) & \ldots & F _ {n} \operatorname {v e c} \left(K _ {0, c _ {i n}}\right) \\ \vdots & & \vdots \\ F _ {n} \operatorname {v e c} \left(K _ {c _ {o u t}, 0}\right) & \ldots & F _ {n} \operatorname {v e c} \left(K _ {c _ {o u t}, c _ {i n}}\right) \end{array} \right] S _ {c _ {i n}}
$$
From here, we may compute the inverse or square root of  $W$ , directly from the kernel  $K$ , more efficiently by performing these operations only on the  $n$  diagonal blocks. All operations required for training LBEN using the operator splitting approach [47] will leave  $W$  within the set of these doubly-block circulant matrices, so we can repeatedly apply this diagonalization trick throughout training.