SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 1: Calculating one layer GEQ.
Require: initial state  $\mathbf{z}^{(0)}$  , weight parameter W,  $\mathbf{U}\mathbf{x}\in \mathbb{R}^{chw}$  , channel split  $c_{s}$  patch size  $p$  hidden layer  $\mathbf{W}_h\in \mathbb{R}^{c_sp^2\times 32}$    
Ensure: Get the output  $\mathbf{z}^*$  of  $i$  -th fixed point iteration. def  $g(\mathbf{z}^{(i)};\mathbf{x},\mathbf{U},\mathbf{W},\mathbf{b})$  .. Rearrange  $\mathbf{W}\mathbf{z}^{(i)}\rightarrow \tilde{\mathbf{z}}$ $\mathbf{U}\mathbf{x} + \mathbf{b}\rightarrow \tilde{\mathbf{x}}\in \mathbb{R}^{\frac{chw}{csp^2}\times (c_sp^2)}$ $\tilde{\mathbf{m}} = diag\left(e^{-\gamma \| (\tilde{\mathbf{x}}. - \tilde{\mathbf{z}}.)\mathbf{W}_h\|_2^2}\right)(\tilde{\mathbf{x}} -\tilde{\mathbf{z}})\mathbf{W}_h\mathbf{W}_h^\top$  Rearrange  $\tilde{\mathbf{m}}\rightarrow \mathbf{m}\in \mathbb{R}^{c\times h\times w}$  return  $\mathbf{z}^{(i + 1)} = \sigma \left(\mathbf{W}^\top \mathbf{m}\right)$  with torch.no_grad(): Use anderson algorithm to solve  $\mathbf{z}^{*} = g(\mathbf{z}^{*};\mathbf{x},\mathbf{U},\mathbf{W},\mathbf{b})$  # calculate gradient via phantom gradient: for i in range(5) do  $\mathbf{z}^{*} = 0.2\times \mathbf{z}^{*} + 0.8\times g(\mathbf{z}^{*};\mathbf{x},\mathbf{U},\mathbf{W},\mathbf{b}).$  end for return  $\mathbf{z}^*$
A.2 Proofs for proposition 1
Proposition 1. The outputs of GEQ with Gaussian kernels (Eqn (6)) is the same as Optimized induced Equilibrium Models' output whose output is the optimal solution of the hidden optimization problems:
$$
\min  _ {\mathbf {z}} G (\mathbf {z}; \mathbf {x}) = \mathbf {1} ^ {\top} f (\mathbf {z}) + \frac {1}{2} \| \mathbf {z} \| ^ {2} - \lambda \left\langle \Phi_ {\mathbf {U}} (\mathbf {x} + \mathbf {U} ^ {- 1} \mathbf {b}), \Phi_ {\mathbf {W}} (\mathbf {z}) \right\rangle \tag {17}
$$
where  $\Phi_{\mathbf{W}}(\mathbf{z}) = \left[\mathbf{1},\sqrt{2\gamma}\Phi_{\mathbf{W}}^{(1)}(\mathbf{z}),\dots,\sqrt{\frac{(2\gamma)^i}{i}}\Phi_{\mathbf{W}}^{(i)}(\mathbf{z}),\dots \right]\in \mathbb{R}^{1\times \infty}$  which projects the features to the infinite-dimensional space. And  $\Phi_{\mathbf{W}}^{(i)}:\mathbb{R}^n\to \mathbb{R}^{in^i}$  is the  $k$  -tuple permutation with repetitions formulated as follows:
$$
\Phi_ {\mathbf {W}} ^ {(i)} = \left[ \underbrace {\overbrace {(W x) _ {1} (W x) _ {1} \dots (W x) _ {1}} ^ {i} , \overbrace {(W x) _ {1} (W x) _ {1} \dots (W x) _ {2}} ^ {i} , \dots , \overbrace {(W x) _ {j} (W x) _ {k} \dots (W x) _ {m}} ^ {i} , \dots} _ {i n ^ {i}} \right] \tag {18}
$$
where  $(Wx)_j$  denotes the  $j$ -th element of vector  $\mathbf{W}\mathbf{x}$ . Then the Gaussian kernel can also be regarded as calculating the input features after the weight-tied infinite wide projection  $\Phi_{\mathbf{W}}$  and  $\Phi_{\mathbf{U}}$ .
Proof. We can formulate the OptEqs' hidden optimization problem with Gaussian kernels as below:
$$
\min  _ {\mathbf {z}} G (\mathbf {z}; \mathbf {x}) = \mathbf {1} ^ {\top} f (\mathbf {z}) + \frac {1}{2} \| \mathbf {z} \| ^ {2} - \frac {1}{2 \gamma} e ^ {- \gamma \| \mathbf {U x} + \mathbf {b} - \mathbf {W z} \| _ {2} ^ {2}}, \tag {19}
$$
For  $e^{-\gamma \left\| (\mathbf{U}\mathbf{x} + \mathbf{b} - \mathbf{W}\mathbf{z})\right\| _2^2$  we have
$$
\begin{array}{l} e ^ {- \gamma \| \mathbf {U} \mathbf {x} + \mathbf {b} - \mathbf {W} \mathbf {z} \| _ {2} ^ {2}} = e ^ {- \gamma \| \mathbf {U} \mathbf {x} + \mathbf {b} \| _ {2} ^ {2} - \gamma \| \mathbf {W} \mathbf {z} \| _ {2} ^ {2} + 2 \gamma \langle \mathbf {U} \mathbf {x} + \mathbf {b}, \mathbf {W} \mathbf {z} \rangle}, \\ = e ^ {- \gamma \| \mathbf {U} \mathbf {x} + \mathbf {b} \| _ {2} ^ {2} - \gamma \| \mathbf {W} \mathbf {z} \| _ {2}} e ^ {2 \gamma \langle \mathbf {U} \mathbf {x} + \mathbf {b}, \mathbf {W} \mathbf {z} \rangle}, \tag {20} \\ \end{array}
$$
letting  $\lambda = e^{-\gamma \| \mathbf{U}\mathbf{x} + \mathbf{b}\|_2^2}e^{-\gamma \| \mathbf{W}\mathbf{z}\|_2}$  and we do the Taylor expansion for  $e^{\langle \mathbf{U}\mathbf{x} + \mathbf{b},\mathbf{W}\mathbf{z}\rangle}$ , we have:
$$
e ^ {- \gamma \| \mathbf {U x} + \mathbf {b} - \mathbf {W z} \| _ {2} ^ {2}} = \lambda \sum_ {i = 0} ^ {\infty} \frac {\left(\langle \sqrt {2 \gamma} (\mathbf {U x} + \mathbf {b}) , \sqrt {2 \gamma} (\mathbf {W z}) \rangle\right) ^ {i}}{i !}. \tag {21}
$$
For any  $i$  we have from the permutation theory:
$$
\frac {\left(\langle \sqrt {2 \gamma} (\mathbf {U x} + \mathbf {b}) , \sqrt {2 \gamma} (\mathbf {W z}) \rangle\right) ^ {i}}{i !} = \lambda \left\langle \Phi_ {\mathbf {U}} ^ {(i)} (\mathbf {x} + \mathbf {U} ^ {- 1} \mathbf {b}), \Phi_ {\mathbf {W}} ^ {(i)} (\mathbf {z}) \right\rangle , \tag {22}
$$
where  $\Phi_{\mathbf{W}}^{(i)}(\mathbf{x})$  is the  $i$ -tuple permutation with the repetition for given  $(Wx)_1, (Wx)_2, \ldots, (Wx)_n$ . Each element of  $\Phi_{\mathbf{W}}^{(i)}(\mathbf{x})$  is one possible permutation. Since there are  $n^i$  tuples, then  $\Phi_{\mathbf{W}}^{(i)}$  can project the input features to  $in^i$  space as follows,
$$
\Phi_ {\mathbf {W}} ^ {(i)} = \left[ \underbrace {\overbrace {\left(W x\right) _ {1} \left(W x\right) _ {1} \dots \left(W x\right) _ {1}} ^ {i} , \overbrace {\left(W x\right) _ {1} \left(W x\right) _ {1} \dots \left(W x\right) _ {2}} ^ {i} , \dots , \overbrace {\left(W x\right) _ {j} \left(W x\right) _ {k} \dots \left(W x\right) _ {m}} ^ {i} , \dots} _ {i n ^ {i}} \right]. \tag {23}
$$
Thereby, the hidden optimization problem for our GEQ can be reformulated as,
$$
\min  _ {\mathbf {z}} G (\mathbf {z}; \mathbf {x}) = \mathbf {1} ^ {\top} f (\mathbf {z}) + \frac {1}{2} \| \mathbf {z} \| ^ {2} - \lambda \left\langle \Phi_ {\mathbf {U}} \left(\mathbf {x} + \mathbf {U} ^ {- 1} \mathbf {b}\right), \Phi_ {\mathbf {W}} (\mathbf {z}) \right\rangle \tag {24}
$$
Then our conclusion is proved.
A.3 Proofs for proposition 2
Proposition 2. If input  $\| \mathbf{x}\| _2$  is bounded by  $B$ ,  $\mu \coloneqq \max \left\{\| \mathbf{U}\| _2,\| \mathbf{W}\| _2,\| \mathbf{W}_c\| _2,\| \mathbf{b}\| _2\right\} < 1$ , then we have following results for GEQ and OptEqs with ReLU activations. For  $\forall \delta ,\eta >0$  with probability at least  $1 - \delta$  over the training set of size  $M$ , we have:
$$
\mathcal {L} _ {0} \left(f _ {g e q} ^ {c}\right) \leq \hat {\mathcal {L}} _ {\eta} \left(f _ {g e q} ^ {c}\right) + \sqrt {\frac {1 6 h \ln (2 4 h) \left[ \beta_ {\max } \mu^ {4} B + (2 \mu \beta_ {\max } + 1) (1 - \beta_ {\max } m) \mu B + (1 - \beta_ {\max } m) ^ {2} \right] ^ {2} \mathcal {B} _ {W}}{\eta^ {2} (1 - \beta_ {\max } m) ^ {4} M}} + \frac {\ln (\frac {M \sqrt {M}}{\delta})}{M}, \tag {25}
$$
$$
\mathcal {L} _ {0} (f _ {o p t e q} ^ {c}) \leq \hat {\mathcal {L}} _ {\eta} (f _ {o p t e q} ^ {c}) + \sqrt {\frac {1 6 h \ln (2 4 h) [ \mu^ {3} B + (1 - m) \mu B + (1 - m) ^ {2} ] ^ {2} \mathcal {B} _ {W}}{\eta^ {2} (1 - m) ^ {4} M} + \frac {\ln (\frac {M \sqrt {M}}{\delta})}{M}},
$$
where  $\hat{\mathcal{L}}_{\eta}(f_{geq}^{c})$  denotes the empirical margin loss on the training set, the maximum scaling number is defined by  $\beta_{max}\coloneqq \max_{\mathbf{x}\in \mathcal{D}}e^{-\gamma \| \mathbf{Ux} + \mathbf{b} - \mathbf{Wz}\| _2^2},\mathcal{B}_W\coloneqq \| \mathbf{W}^\top \mathbf{W}\| _F^2 +\| \mathbf{U}\| _F^2 +\| \mathbf{b}\| _2^2 +\| \mathbf{W}_c\| _F^2 +\| \mathbf{b}_c\| _2^2,$  and  $m = \| \mathbf{W}^{\top}\mathbf{W}\| _2$  is less than 1 to ensure the convergence of equilibrium models.
Before the proof our results, we need to introduce a lemma in former work [38] for the perturbation bound for GEQs and reformulated OptEqs as follows.
Lemma 1. Let  $\| \mathbf{W}\| _2\leq m$  and  $\| \overline{\mathbf{W}}\| _2\leq m$ . Then change in the output of the DEqs  $\mathbf{z} = \sigma (\mathbf{W}\mathbf{z} + \mathbf{U}\mathbf{x} + \mathbf{b})$  on perturbation the weights and biases from  $\mathbf{W}$ ,  $\mathbf{U}$ ,  $\mathbf{b}$  to  $\overline{\mathbf{W}}$ ,  $\overline{\mathbf{U}}$ ,  $\overline{\mathbf{b}}$  is bounded as follows:
$$
\begin{array}{l} \left\| f \left(\overline {{\mathbf {W}}} \mathbf {z} + \overline {{\mathbf {U}}} \mathbf {x} + \overline {{\mathbf {b}}}\right) - f (\mathbf {W} \mathbf {z} + \mathbf {U} \mathbf {x} + \mathbf {b}) \right\| _ {2} \leq \\ \frac {\left\| \overline {{\mathbf {W}}} - \mathbf {W} \right\| _ {2} \left\| \mathbf {U} \mathbf {x} + \mathbf {b} \right\| _ {2} + \left\| (\overline {{\mathbf {U}}} - \mathbf {U}) \mathbf {x} \right\| _ {2} + \left\| \overline {{\mathbf {b}}} - \mathbf {b} \right\| _ {2}}{(1 - m) ^ {2}} \tag {26} \\ \end{array}
$$
Like former works [38], we we also introduce another lemma [36] here:
Lemma 2. Let  $f_w$  be any predictor with parameters  $w$ , and let  $P$  denote any distribution on the parameters that are independent of the training data. Then, for any  $\delta, \gamma > 0$ , with probability  $\geq 1 - \delta$  over the training data of size  $M$ , for any  $w$ , and any random perturbation  $u$  such that  $\mathbb{P}\left[\max_x\|f_{w+u}(x)-f_w(x)\|_\infty < \frac{\eta}{4}\right] \geq \frac{1}{2}$ , we have
$$
\mathcal {L} _ {0} \left(f _ {w}\right) \leq \hat {\mathcal {L}} _ {\eta} f _ {w} + 4 \sqrt {\frac {K L (w + u | | P) + \ln \frac {6 M}{\delta}}{M - 1}} \tag {27}
$$
Then we can derive the perturbation bound for the reformulated OptEqs and our GEQ following former settings [38]. First, we also incorporate a fully connected layer at the end as we mentioned in the paper.
$$
f _ {g e q} ^ {c} (\mathbf {x}) = \mathbf {W} _ {c} f _ {g e q} (\mathbf {x}) + \mathbf {b} _ {c}, f _ {\text {o p t e q}} ^ {c} (\mathbf {x}) = \mathbf {W} _ {c} f _ {\text {o p t e q}} (\mathbf {x}) + \mathbf {b} _ {c}. \tag {28}
$$
Since the entries in the perturbations obeying the distribution as  $\mathcal{N}(0,\sigma^2)$ , we have that all the perturbations of weights  $\| \Delta\|$  are bounded by  $\sigma \sqrt{2h\ln(24h)}:\omega$  with probability larger than  $1/2$ .
Since the only difference between OptEqs and monDEQ [51] is the weight parameterization, our reformulate OptEqs is parameterized by  $\mathbf{W}_s = \mathbf{W}^\top \mathbf{W}$  while the monDEQ's weight parameter is parameterized by a series of weights  $\mathbf{W}_{mondeq} = \mathbf{I} + \mathbf{A} + \mathbf{A}^\top +\mathbf{B} + \mathbf{B}^\top$ . Therefore, the perturbation in  $\| \Delta_{\mathbf{W}_s}\|$  is different from former analysis [38]. We have that:
$$
\left\| \Delta_ {\mathbf {W} _ {s}} \right\| _ {2} = \left\| \Delta_ {\mathbf {W}} ^ {\top} \mathbf {W} + \mathbf {W} \Delta_ {\mathbf {W}} ^ {\top} \right\| _ {2} \leq 2 \omega \mu \tag {29}
$$
Then using the above lemma, we have that for all  $x$  with probability at least  $1/2$ .
$$
\begin{array}{l} \| \overline {{f}} _ {o p t e q} ^ {c} (\mathbf {x}) - f _ {o p t e q} ^ {c} (\mathbf {x}) \| _ {2} \leq \| \overline {{\mathbf {W}}} _ {c} \overline {{f}} _ {o p t e q} (\mathbf {x}) + \overline {{\mathbf {b}}} _ {c} - \mathbf {W} _ {c} f _ {o p t e q} ^ {c} (\mathbf {x}) - \mathbf {b} \| _ {2} \\ \leq \frac {2 \mu^ {2} \omega (B + 1)}{(1 - m) ^ {2}} + \frac {2 \mu \omega (B + 1)}{1 - m} + \omega \tag {30} \\ \end{array}
$$
Setting  $\sigma = \frac{\eta(1 - m^2)}{4\sqrt{2h\ln(24h)}(2\mu^3(B + 1) + 2(1 - m)\mu + (1 - m)^2)}$  will make the above perturbation less than  $\frac{\eta}{4}$ . Then we have,
$$
K L \left(\mathbf {W} _ {\cdot} + \Delta_ {\mathbf {W} _ {\cdot}} | P\right) \leq \frac {\mathbf {B} _ {W}}{2 \sigma^ {2}} = \frac {1 6 h \ln (2 4 h) \left(2 \mu^ {3} (B + 1) + 2 (1 - m) \mu (B + 1) + (1 - m) ^ {2}\right) ^ {2}}{\eta^ {2} (1 - m) ^ {4}} \mathcal {B} _ {W} \tag {31}
$$
With the same choice of  $\beta$ 's bound like former work [38],
$$
\frac {\eta (1 - m)}{2 (B + 1)} \leq \beta \leq \frac {\eta (1 - m) \sqrt {M}}{2 (B + 1)}, \tag {32}
$$
we can finally get the upper bound as our OptEqs bound as our proposition shows.
The difference between GEQ and OptEqs is that GEQ's can be viewed as multiplying scalar  $\beta = e^{-\gamma \| \mathbf{U}\mathbf{x} + \mathbf{b} - \mathbf{W}\mathbf{z}\|_2^2}$  with depend on  $\mathbf{x}$  since  $\mathbf{z}$  is also depended on  $\mathbf{x}$ . Setting  $\beta_{\max} = \max_{\mathbf{x} \in \mathcal{D}} \beta(\mathbf{x}) < 1$  and  $\beta_{\min} = \min_{\mathbf{x} \in \mathcal{D}} \beta(\mathbf{x}) > c$ . Assuming  $\beta$  changes a little with respect to the small perturbations on weights, we have:
$$
\begin{array}{l} \mathbf {z} ^ {*} (\mathbf {W}, \mathbf {U}, \mathbf {b}) = \sigma (- \beta \mathbf {W} _ {s} \mathbf {z} ^ {(i)} + \beta \mathbf {W} (\mathbf {U x} + \mathbf {b})) \\ \left\| \mathbf {z} ^ {*} (\overline {{\mathbf {W}}}, \overline {{\mathbf {U}}}, \overline {{\mathbf {b}}}) \right\| _ {2} \leq \frac {\beta_ {\max } \mu \left\| \overline {{\mathbf {W}}} _ {s} - \mathbf {W} _ {s} \right\| _ {2} \left\| \mathbf {U} \mathbf {x} + \mathbf {b} \right\| _ {2}}{(1 - \beta_ {\max } m) ^ {2}} + \frac {\beta_ {\max } \left(\left\| (\overline {{\mathbf {W U}}} - \mathbf {W U}) \mathbf {x} \right\| _ {2} + \left\| \overline {{\mathbf {W b}}} - \mathbf {W b} \right\| _ {2}\right)}{1 - \beta_ {\max } m} \tag {33} \\ \end{array}
$$
With the same setting as above OptEqs, we have:
$$
\| (\overline {{\mathbf {W} \mathbf {U}}} - \mathbf {W} \mathbf {U}) \mathbf {x} \| _ {2} = \| \Delta_ {\mathbf {W}} \overline {{\mathbf {U}}} - \mathbf {W} \Delta_ {\mathbf {U}} \mathbf {x} \| _ {2} \leq 2 \omega \mu \| (\overline {{\mathbf {W} \mathbf {b}}} - \mathbf {W} \mathbf {b}) \mathbf {x} \| _ {2} = \| \Delta_ {\mathbf {W}} \overline {{\mathbf {b}}} - \mathbf {W} \Delta_ {\mathbf {b}} \mathbf {x} \| _ {2} \leq 2 \omega \mu
$$
Then we can obtain that:
$$
\begin{array}{l} \| \overline {{f}} _ {g e q} ^ {c} (\mathbf {x}) - f _ {k e r e q} ^ {c} (\mathbf {x}) \| _ {2} \leq \| \overline {{\mathbf {W}}} _ {c} \overline {{f}} _ {o p t e q} (\mathbf {x}) + \overline {{\mathbf {b}}} _ {c} - \mathbf {W} _ {c} f _ {o p t e q} ^ {c} (\mathbf {x}) - \mathbf {b} \| _ {2} \\ \leq \frac {2 \beta_ {\max} \mu^ {3} \omega (B + 1)}{(1 - \beta_ {\max} m) ^ {2}} + \frac {2 \mu^ {2} \omega \beta_ {\max} (B + 1)}{1 - \beta_ {\max} m} + \frac {\mu \omega (B + 1)}{1 - \beta_ {\max} m} + \omega \\ = \frac {\left(\beta_ {\max} \mu\right) * \left(2 \mu^ {2} \omega (B + 1)\right)}{\left(1 - \beta_ {\max} m\right) ^ {2}} + \frac {\left(2 \mu \beta_ {\max} + 1\right) * \left(\mu \omega (B + 1)\right)}{1 - \beta_ {\max} m} + \omega \tag {35} \\ \end{array}
$$
With the same setting as above, we have:
$$
\begin{array}{l} K L \left(\mathbf {W} _ {\cdot} + \Delta_ {\mathbf {W} _ {\cdot}} | P\right) \leq \frac {\mathbf {B} _ {W}}{2 \sigma^ {2}} \tag {36} \\ = T \mathcal {B} _ {W}, \\ \end{array}
$$
where  $T$  is defined as follows:
$$
T = \frac {1 6 h \ln (2 4 h) \left(2 \left(\beta_ {\max} \mu\right) \mu^ {3} (B + 1) + 2 \left(2 \mu \beta_ {\max} + 1\right) \left(1 - \beta_ {\max} m\right) \mu (B + 1) + \left(1 - \beta_ {\max} m\right) ^ {2}\right) ^ {2}}{\eta^ {2} \left(1 - \beta_ {\max} m\right) ^ {4}} \tag {37}
$$
then we can finally we can get the upper bound as our OptEqs bound as our proposition shows.
A.4 Proofs for proposition 3
Lipschitz constant is the minimal constant for  $f$  and  $\forall \mathbf{x}, \mathbf{y}$  suits the following equation:
$$
\left\| f (\mathbf {x}) - f (\mathbf {y}) \right\| _ {2} \leq L \| \mathbf {x} - \mathbf {y} \| _ {2}. \tag {38}
$$
Thereby, our analysis in Proposition 3 can be viewed as proposing an upper bound for different models. In this section, we are going to prove the Lipschitz upper bounds for our GEQ and OptEqs. First, we restate the proposition as follows:
Proposition 3. If norms for the inputs and outputs are bounded by  $B$ , the spectral norm for the weight parameter  $\mathbf{W}$ ,  $\mathbf{U}$  of equilibrium models with ReLU activation are bounded by  $\mu < 1$  to ensure convergence, then we have the conclusions as below:
$$
\left\| f _ {g e q} \left(\mathbf {x} _ {1}\right) - f _ {g e q} \left(\mathbf {x} _ {2}\right) \right\| _ {2} \leq L _ {g e q} \| \mathbf {x} _ {1} - \mathbf {x} _ {2} \| _ {2} = \frac {\beta_ {\max } \mu^ {2} + \sqrt {\gamma} B \mu^ {3}}{1 - \beta_ {\max } \mu^ {2} - \sqrt {\gamma} B \mu^ {3}} \| \mathbf {x} _ {1} - \mathbf {x} _ {2} \| _ {2}, \tag {39}
$$
$$
\left\| f _ {\text {o p t e q}} \left(\mathbf {x} _ {1}\right) - f _ {\text {o p t e q}} \left(\mathbf {x} _ {2}\right) \right\| _ {2} \leq L _ {\text {o p t e q}} \left\| \mathbf {x} _ {1} - \mathbf {x} _ {2} \right\| _ {2} = \frac {\mu}{1 - \mu^ {2}} \left\| \mathbf {x} _ {1} - \mathbf {x} _ {2} \right\| _ {2}, \tag {40}
$$
Proof. We first prove the inequality for OptEqs:
$$
\begin{array}{l} \left\| f _ {o p t e q} (\mathbf {x} _ {1}) - f _ {o p t e q} (\mathbf {x} _ {2}) \right\| _ {2} = \left\| \mathbf {z} _ {\mathbf {x} _ {1}} - \mathbf {z} _ {\mathbf {x} _ {2}} \right\| _ {2} \leq \left\| \sigma \left(\mathbf {W} ^ {\top} \mathbf {W} \mathbf {z} _ {\mathbf {x} _ {1}} + \mathbf {U} \mathbf {x} _ {1}\right) - \sigma \left(\mathbf {W} ^ {\top} \mathbf {W} \mathbf {z} _ {\mathbf {x} _ {2}} + \mathbf {U} \mathbf {x} _ {2}\right) \right\| _ {2} \\ \leq \left\| \mathbf {W} ^ {\top} \mathbf {W} \left(\mathbf {z} _ {\mathbf {x} _ {1}} - \mathbf {z} _ {\mathbf {x} _ {2}}\right) \right\| _ {2} + \left\| \mathbf {U} \left(\mathbf {x} _ {1} - \mathbf {x} _ {2}\right) \right\| _ {2} \\ \leq \mu^ {2} \| \mathbf {z} _ {\mathbf {x} _ {1}} - \mathbf {z} _ {\mathbf {x} _ {2}} \| _ {2} + \mu \| \mathbf {x} _ {1} - \mathbf {x} _ {2} \| _ {2}, \tag {41} \\ \end{array}
$$
where  $\mathbf{z}_{\mathbf{x}_1}$  denotes the equilibrium states for OptEq with input  $\mathbf{x}_1$ , which means the following equation is satisfied:
$$
\mathbf {z} _ {\mathbf {x} _ {1}} = \sigma \left(\mathbf {W} ^ {\top} \mathbf {W} \mathbf {z} _ {\mathbf {x} _ {1}} + \mathbf {U} \mathbf {x} _ {1}\right). \tag {42}
$$
Reformulating the equations, we can get:
$$
\left\| f _ {\text {o p t e q}} \left(\mathbf {x} _ {1}\right) - f _ {\text {o p t e q}} \left(\mathbf {x} _ {2}\right) \right\| _ {2} \leq \frac {\mu}{1 - \mu^ {2}} \| \mathbf {x} _ {1} - \mathbf {x} _ {2} \| _ {2}. \tag {43}
$$
Then for GEQ, we have:
$$
\begin{array}{l} \left\| \mathbf {z} _ {\mathbf {x} _ {1}} - \mathbf {z} _ {\mathbf {x} _ {2}} \right\| _ {2} \leq \left\| \right. \sigma \left(e ^ {- \gamma \left\| \mathbf {W} \mathbf {z} _ {\mathbf {x} _ {1}} - \mathbf {U} \mathbf {x} _ {1} \right\| _ {2} ^ {2}} \left(- \mathbf {W} ^ {\top} \mathbf {W} \mathbf {z} _ {\mathbf {x} _ {1}} + \mathbf {W} ^ {\top} \mathbf {U} \mathbf {x} _ {1}\right)\right) \\ \left. - \sigma \left(e ^ {- \gamma \left\| \mathbf {W} _ {\mathbf {z} _ {\mathbf {x} _ {2}}} - \mathbf {U} _ {\mathbf {x} _ {2}} \right\| _ {2} ^ {2}} \left(- \mathbf {W} ^ {\top} \mathbf {W}  + \mathbf {W} ^ {\top} \mathbf {U} _ {\mathbf {x} _ {2}}\right)\right) \right\| _ {2} \tag {44} \\ \leq \beta_ {\max} \left\| \mathbf {W} ^ {\top} \mathbf {W} (\mathbf {z _ {x _ {1}}} - \mathbf {z _ {x _ {2}}}) \right\| _ {2} + \left\| \mathbf {W} ^ {\top} \mathbf {U} (\mathbf {x _ {1}} - \mathbf {x _ {2}}) \right\| _ {2} + B \mu^ {2} | \beta_ {\mathbf {x _ {1}}} - \beta_ {\mathbf {x _ {2}}} |, \\ \leq \beta_ {\max } \mu^ {2} \| \mathbf {z} _ {\mathbf {x} _ {1}} - \mathbf {z} _ {\mathbf {x} _ {2}} \| _ {2} + \beta_ {\max } \mu^ {2} \| \mathbf {x} _ {1} - \mathbf {x} _ {2} \| _ {2} + B \mu^ {2} | \beta_ {\mathbf {x} _ {1}} - \beta_ {\mathbf {x} _ {2}} | \\ \end{array}
$$
where  $\mathbf{z}_{\mathbf{x}_1}$  denotes the equilibrium states for GEQ with input  $\mathbf{x}_1$ , and  $\beta_{\mathbf{x}_1}$  is defined as follows:
$$
\beta_ {\mathbf {x} _ {1}} = e ^ {- \gamma \left\| \mathbf {W z} _ {\mathbf {x} _ {1}} - \mathbf {U x} \right\| _ {2} ^ {2}}. \tag {45}
$$
Then with the mean value theorem, we have the following equations:
$$
\begin{array}{l} \left| \beta_ {\mathbf {x} _ {1}} - \beta_ {\mathbf {x} _ {2}} \right| = \left| e ^ {- \gamma \left\| \mathbf {W} \mathbf {z} _ {\mathbf {x} _ {1}} - \mathbf {U} \mathbf {x} _ {2} \right\| _ {2} ^ {2}} - e ^ {- \gamma \left\| \mathbf {W} \mathbf {z} _ {\mathbf {x} _ {1}} - \mathbf {U} \mathbf {x} _ {2} \right\| _ {2} ^ {2}} \right| \\ \leq \sqrt {\gamma} | \| \mathbf {W z _ {x _ {1}}} - \mathbf {U x _ {2}} \| _ {2} - \| \mathbf {W z _ {x _ {x}}} - \mathbf {U x _ {2}} \| _ {2} | \cdot \max _ {c \in \sqrt {\gamma} [ \| \mathbf {W z _ {x _ {1}}} - \mathbf {U x _ {1}} \| _ {2}, \| \mathbf {W z _ {x _ {2}}} - \mathbf {U x _ {2}} \| _ {2} ]} c e ^ {- c ^ {2}} \\ \leq \sqrt {\gamma} | \| \mathbf {W z} _ {\mathbf {x} _ {2}} - \mathbf {U x} _ {2} \| _ {2} - \| \mathbf {W z} _ {\mathbf {x} _ {2}} - \mathbf {U x} _ {2} \| _ {2} | \\ \leq \sqrt {\gamma} \| \mathbf {W} (\mathbf {z} _ {\mathbf {x} _ {1}} - \mathbf {z} _ {\mathbf {x} _ {2}}) \| _ {2} + \| \mathbf {U} (\mathbf {x} _ {1} - \mathbf {x} _ {2}) \| _ {2} \\ \leq \sqrt {\gamma} \mu \left(\left\| \mathbf {z} _ {\mathbf {x} _ {1}} - \mathbf {z} _ {\mathbf {x} _ {2}} \right\| _ {2} + \left\| \mathbf {x} _ {1} - \mathbf {x} _ {2} \right\| _ {2}\right) \tag {46} \\ \end{array}
$$
Then reformulating Eqn (44), we can finally obtain our proposition as below:
$$
\left\| \mathbf {z} _ {\mathbf {x} _ {1}} - \mathbf {z} _ {\mathbf {x} _ {2}} \right\| _ {2} = \left\| f _ {g e q} (\mathbf {x} _ {1}) - f _ {g e q} (\mathbf {x} _ {2}) \right\| _ {2} \leq \frac {\beta_ {\max } \mu^ {2} + \sqrt {\gamma} B \mu^ {3}}{1 - \beta_ {\max } \mu^ {2} - \sqrt {\gamma} B \mu^ {3}} \| \mathbf {x} _ {1} - \mathbf {x} _ {2} \| _ {2} \tag {47}
$$
A.5 Proofs for proposition 4
In this section, we are going to prove the output similarity bounds for our GEQ and OptEqs. First, we restate the proposition as follows:
$$
\kappa_ {g e q} \left(\mathbf {x} _ {1}, \mathbf {x} _ {2}\right) \leq \bar {\kappa} _ {g e q} = \frac {\mu^ {2} D e ^ {- \frac {\gamma}{4} \left(\sigma_ {\min } (\mathbf {U}) ^ {2} \| \mathbf {x} _ {1} - \mathbf {x} _ {2} \| _ {2} ^ {2}\right)} \mathbb {E} \left[ U _ {i} ^ {2} \right] \| \mathbf {x} _ {1} \| \| \mathbf {x} _ {2} \| \left(\sin \theta_ {0} + \left(\pi - \theta_ {0}\right) \cos \theta_ {0}\right)}{2 \pi \left(1 - \beta_ {\max } \mu^ {2}\right) ^ {2}}, \tag {48}
$$
$$
\kappa_ {o p t e q} \left(\mathbf {x} _ {1}, \mathbf {x} _ {2}\right) \leq \overline {{\kappa}} _ {o p t e q} = \frac {\mathbb {E} \left[ U _ {i} ^ {2} \right] \left\| \mathbf {x} _ {1} \right\| \left\| \mathbf {x} _ {2} \right\| \left(\sin \theta_ {0} + (\pi - \theta_ {0}) \cos \theta_ {0}\right)}{2 \pi (1 - \mu^ {2}) ^ {2}}, \tag {49}
$$
where  $\mathbf{x}_1$  and  $\mathbf{x}_2$  are input samples,  $D = e^{\gamma B\|\mathbf{W}\|_2^2}$  and  $\beta_{max} := \max_{\mathbf{x} \in \mathcal{D}} e^{-\gamma \|\mathbf{U}\mathbf{x} - \mathbf{W}\mathbf{z}\|_2^2} < 1$ .  $\theta_0 = \cos^{-1}\left(\frac{\langle\mathbf{x}_1,\mathbf{x}_2\rangle}{\|\mathbf{x}_1\|\|\mathbf{x}_2\|}\right)$  is defined as the angle between the samples.
Proof. Before the proof, we first introduce the following lemma:
Lemma 3. [48] If  $\mathbf{U}$  obeys the spherical Gaussian distributions of variance  $\mathbb{E}[U_i^2]$  and mean 0, then the expectation of the Similarity for the one-layer Neural Network  $\sigma (\mathbf{U}\mathbf{x})$  is:
$$
\kappa_ {N N} \left(\mathbf {x} _ {1}, \mathbf {x} _ {2}\right) = \frac {\mathbb {E} \left[ U _ {i} ^ {2} \right] \left\| \mathbf {x} _ {1} \right\| \left\| \mathbf {x} _ {2} \right\|}{2 \pi} (\sin \theta_ {0} + (\pi - \theta_ {0}) \cos \theta_ {0}) \tag {50}
$$
where  $\theta_0 = \cos^{-1}(\frac{\langle\mathbf{x},\mathbf{y}\rangle}{\|\mathbf{x}\|\|\mathbf{y}\|})$
Letting  $m \coloneqq \| \mathbf{W}_s\| _2 = \| \mathbf{W}^\top \mathbf{W}\| _2 < 1$  and  $\mu \coloneqq \| \mathbf{W}\| _2 < 1$  as our assumptions demonstrate and neglecting the bias  $\mathbf{b}$  for convenience. Then for our reformulated OptEqs, we have:
$$
\begin{array}{l} \mathbf {z} _ {\mathbf {x} _ {1}} ^ {* \top} \mathbf {z} _ {\mathbf {x} _ {2}} \leq \sigma \left(\mathbf {W} _ {s} \mathbf {z} _ {\mathbf {x} _ {1}}\right) ^ {\top} \mathbf {z} _ {\mathbf {x} _ {2}} + \sigma \left(\mathbf {U} \mathbf {x} _ {1}\right) ^ {\top} \mathbf {z} _ {\mathbf {x} _ {2}} \tag {51} \\ \leq \mu^ {2} \sigma \left(\mathbf {z} _ {\mathbf {x} _ {1}}\right) ^ {\top} \mathbf {z} _ {\mathbf {x} _ {2}} + \sigma \left(\mathbf {U} \mathbf {x} _ {1}\right) ^ {\top} \mathbf {z} _ {\mathbf {x} _ {2}} \\ \end{array}
$$