GEQ: Gaussian Kernel Inspired Equilibrium Models
Authors: Mingjie Li
Yisen Wang
Zhouchen Lin
Venue: NeurIPS 2023 poster (2023)
Keywords: equilibirum models
neural networks

Abstract: Despite the connection established by optimization-induced deep equilibrium models (OptEqs) between their output and the underlying hidden optimization problems, the performance of it along with its related works is still not good enough especially when compared to deep networks. One key factor responsible for this performance limitation is the use of linear kernels to extract features in these models. To address this issue, we propose a novel approach by replacing its linear kernel with a new function that can readily capture nonlinear feature dependencies in the input data. Drawing inspiration from classical machine learning algorithms, we introduce Gaussian kernels as the alternative function and then propose our new equilibrium model, which we refer to as GEQ. By leveraging Gaussian kernels, GEQ can effectively extract the nonlinear information embedded within the input features, surpassing the performance of the original OptEqs. Moreover, GEQ can be perceived as a weight-tied neural network with infinite width and depth. GEQ also enjoys better theoretical properties and improved overall performance. Additionally, our GEQ exhibits enhanced stability when confronted with various samples. We further substantiate the effectiveness and stability of GEQ through a series of comprehensive experiments.

GEQ: Gaussian Kernel Inspired Equilibrium Models
Mingjie Li', Yisen Wang Zhouchen Lin 12,33 Lin National Key Lab of General Artificial Intelligence,
School of Intelligence Science and Technology, Peking University 2 Institute for Artificial Intelligence. Peking University Peng Cheng Laboratory
Imjat01 11 @outlook.com, (yisen.wand zlin| J @ @pku.edu.cn
Abstract
Despite the connection established by iitimizaationinduece deep equilibrium models (OptEqs) between their output and the underlying hidden optimization problems, the performance of it along with its related works still not good enough especially when compared to deep networks. One key factor responsible for this performance limitation is the use of linear kernels to extract features these models. To address this issue, we propose novel approach by replacing its linear kernel with a new function that can readily capture nonlinear feature dependencies in the input data. Drawing inspiration from classical machine learning algorithms, we introduce Gaussian kernels as the alternative function and then propose our new equilibrium model, which we refer GEQ can effectively extract the nonlinear information embedded within the input features, surpassing the performance of the original OptEqs. Moreover, GEQ can be perceived as a weight- neural network with infinite width and depth. GEQ also enjoys better theoretical properties and improved overall performance. Additionally, our GEQ exhibits enhanced stability when confronted with various samples. We further substantiate the effectiveness and stability of GEQ through series of comprehensive experiments.
GEQ. By leveraging Gaussian kernels,
Introduction
Deep Neural Networks (DNNs) show impressive performance in many real-world tasks on various data like graphs 43), images [4] 200, sequences [8], and others. However, most neural network structures are constructed by experience or searching on the surrogate datasets [31 57]. Therefore, these architectures cannot be interpretable and such a phenomenon hinders further development. Apart from the current neural network models, traditional machine learning methods like dictionary can design their whole procedure by designing optimization problems with specific regularizers customized from their mathematical modeling and requirements. Thus, these models are easily interpreted. However, the traditional machine learning algorithms' whole procedures do not consider the hidden properties of features and labels. Therefore, they usually perform worse on tasks with more data.
learning [46] 33 subspace clustering (54) and other methods (2 56 55 30
To link two types of models, OptEqs [22 tries to recover the model's 's hidden optimization problem make their model "mathematically explainable". They claim that the output features 2 (we also called the equilibrium state) with respect to input x, which are obtained by solving the fixed-point
"Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

equation in Eqn T is the optimal solution for its hidden optimization problem defined in Equ
z' W "W2 + Ux+ b), y W. +
bes
mjn G(2;x) x) min 1 S(W 12) - (Ux+ b, W IT 2
+
1 W
211
2 1212
where a is the ReLU activation function, U, W, b, W.. b, are learnable parameters. They are trained by optimizing loss functions, such as cross-entropy loss, which are calculated based on the final prediction y derived from 2" as shown in Eqn I W-1 represents an invertible or inverriiee matrix for W, and the function f is positive indicator function, which outputs 1 when Z 2 0 and outputs do in other cases. By choosing this function, the first-order condition for Eqn 2 will contain a non-linear ReLU function, which is the activation function in our GEQ. The equivalence between z" and the optimal solution for the hidden optimization problem 2 enables researchers to not only gain insights into OptEqs' behavior by understanding the underlying hidden optimization problem but also innovate by designing new models based on different problem formulations tailored to specific tasks. For instance, OptEqs introduces module that promotes output sparsity, and Multi-brande OptEqs (MOptEqs) incorporates fusion modules to enhance the diversity among its branches. Despite these advancements and the incorporation of various modules inspired by different vision tasks, the performance of OptEqs and related models still falls short when compared to deep neural networks in image classification. This discrepancy suggests the existence of crucial components that limit the performance of equilibrium models.
To identify such component, we delve into the hidden optimization problem of OptEqs and observe that it can be decomposed into two distinct parts: the regularizer term for the output features and the feature extraction term. While the feature extraction term is crucial as it depends and determines the patterns extracted from the input features, the exploration of the regularizer term has been largely overlooked, with linear kernel functions being the predominant choice for feature extraction. Thereby, we believe that the limitations of previous equilibrium models stem from their feature extraction parts, linear kernels struggle capture complex features effectively. Building upon these insights, we take a step forward by leveraging the widely adopted Gaussian kernel for feature extraction in the hidden optimization problem.
the input
Then by calculating the stationary condition for the above new hidden optimization problem, we propose new type of OptEqs, the Gaussian kernels inspired equilibrium models (GEQ). The model involves a new attentive module induced by its hidden optimization problem and enjoys much better performances classification tasks even compared with deep models. Furthermore, also prove that the new model's outputs are equivalent to the outputs for OptEqs with weight-tied "infinite wide" mappings. Therefore, interesting finding is that model can be regarded "double-inffnite" model because the original OptEqs can be regarded as a weight-tied "infinite deep" model. Apart from the above findings, the utilization of Gaussian kernels also makes proposed model enjoy better generalization abilities. Besides the generalization abilities, we also analyze the stability of our GEQ and find its stability is better on various inputs. We summarize our contributions as follows:
We first reformulate the OptEqs' hidden optimization problem with Gaussian kernels and propose new equilibrium model called GEQ. It contains a new attention module induced by its hidden optimization problem and performs better on real-world datasets.
We find that our GEQ can be regarded as a weight- neural network with both infinite width and depth, and better generalization ability through analysis. Empirical results also confirm the superiority of our GEQ.
We theoretically demonstrate the advantages of the stability of our GEQ compared with former OptEqs on various inputs. We also conduct experiments to validate such advantages.
2 Related Works
2.1 Implicit Models
Most modern deep learning approaches provide explicit computation graphs for forward propagations and we call these models "explici models". Contrary to these models, recent researchers proposed some neural architecture with dynamic computation graphs and we call them "implicit models". A

---

notable example of an implicit model is Neural ODEs 77. its architecture is encoded as a differential system and the implicit ODE solvers they used are equivalent to continuous ResNets that take infinitesimal steps. By representing the entire structure using differential systems, implicit models tap into the black box of traditional neural networks while offering increased flexibility and interpretability. Because of the flexibility and the interpretability of implicit models, the design of implicit models [5 17 draws much attention these days. Many kinds of implicit models have been proposed, including optimization layers ΠΣΑ differentiable physics engines [40] [9], logical structure learning
[0] differential programming [53 and others [25
Among the various implicit models, OptEqs [52 and its multi-branch version MOptEqs [8] stand out as they not only exhibit superior performance compared to other implicit models but also explicitly establish the relationship between their structure and well-defined optimization problem. Therefore, exploring better equilibrium models is a promising direction to achieve more interpretable neural architectures. However, it is worth noting that while OptEqs and its variants have shown promising results, their performance is still not entirely satisfactory, partieularly when compared to the deep explicit models. Other works [5] 42 also show the connection between their architectures and optimization problems, but their performance is also not satisfying. Besides the above general models, there are many works design equilibrium models from the view of optimization to deal with their specific tasks, like implicit graph models 66, 26), certified robust models [27] and image denoising models [5]. However, these models can only work their specific domains.
2.2 Infinite Wide Models and Kernel Methods in Deep Learning
By employing kernel methods estimate the outputs of single-layer networks for various samples, researchers discover that such networks can exhibit characteristics of a Gaussian process (GP) when their parameters randomly initialized with large width limit (35) Building upon this idea, recent researchers have extended these findings to neural networks with multiple layers [24 I0] and other architectures [37 13). These studies primarily focus on weakly-trained models, where the network parameters are randomly initialized and kept fixed throughout the training process except for the last classification layer (2] Despite their "weakly-trained" nature, these models still provide valuable insights applicable to current neural networks. For instance, mean-field theory [4] 16, 19 explains phenomena such as gradient vanishing and exploding during back paoo-propagation, which are relevant not only to single-layer networks but also to other structures like convolutional neural networks (CNNs) and recurrent neural networks (RNNs). Other researchers explore stationary kernels to enhance the interpretability of neural networks by designing different activation functions 34
In addition weakly trained models, recent studies 22 2 introduce the concept of Neural Tangent Kernel (NTK) and its variants. These works have demonstrated that the sample kernel of infinitely wide networks, with appropriate initialization, can converge to a fixed neural tangent kernel when trained using gradient descent with infinitesimal steps (gradient flow). The NTK model is theoretical construct with strict constraints, and its weights are not learned. It is important to note that although our model can also be seen as an infinitely wide model, there are several key differences between our approach and the aforementioned models. Firstly, our model utilizes kernel methods to operate on input features and output features, while the NTK models employ the kernel method on samples. Secondly, our GEQ model can be viewed as employing a "weight-tied infinite wide" projection that is parameterized by learnable parameters, allowing for updates during the training process. This contrasts with NTKs and NTK-DEQ [1] (an equilibrium model constructed with vamilla NTK layers), where the weights are fixed and not learned. Therefore, despite the potential overlap in terminologies used in our paper and NTK-related works, our GEQ model differs significantly.
3 Gaussian kernel Inspired Equilibrium models
3.1 Formulation and Structure of GEQ
Before starting our analysis, we need to reformulate the original formulations of OptEqs' equilibrium equation (1) and hidden optimization problem ( for convenience. We replace W.WT with W...
W : W and replace z with W-1T 2. Then the original OptEqs' optimization problem can be
3

---

reformulated as:
1 min ((z min 1 f(z) + |211 - b, z) W
1
(3)
With the new formulation, we can rewrite the equilibrium equation for OptEqs with input x by calculating Eqn (3's first order stationary condition VG 0 and then reformulate it as follows:
z" (W Wz" + Ux U
where a is the ReLU activation function, U, W, b are learnable parameters trained by optimizing loss functions (like cross-entropy loss). From problem Eqn (3), one can see that the GEQ's outputs try to extract features by minimizing the similarity term with the input feature Ux + b through linear kernel function with some constraints defined in its regulation terms to prevent the trivial outputs. Such an explanation can also extend to other DEQs 3 51 constraints. However, as other traditional machine learning mechanisms show, linear kernel functions cannot perform well when processing complex inputs. We deem that this term will also restrict the performance in equilibrium models. We note that the symmetric constraints won't 't influence the final performance much as many works 32 21 show.
under the symmetric weight
natural consideration arises as to whether we can utilize alternative kernel functions to extract input features for the equilibrium state. However, we find that other equilibrium models employing different kernels with inner products, like the polynomial kernel and sigmoid kerne,, lead a similar structure to OptEqs with appropriate weight re-parameterization and lead to similar empirical results. We provide a detailed discussion of the related models in Appendix [4.6] Thereby, we decide to use the Gaussian kernels, and our new hidden optimization equation is formulated as follows:
min G(z: x) min 1 f(z) + |
1 - Ux+buU
(5)
27
where 7 is the hyperparameter denoting the reciprocal of Gaussian kernels' variance for scaling. Calculating VG = 0 for new G, can get the Gaussian kernel inspired Equilibrium models (GEQ) as the following fixed-point equation:
z
1||Ux+b-WUW 1a:tH-bb-x11: Wx Wz + Ux + b)
Ux+
Compared with linear kernels, Gaussian kernels can easily extract the non-linear relations from the input features and show more stable and powerful performance in SVM and other machine We also find that the formulation of our GEQ is similar adding new attention module to the original equilibrium models. Therefore, our GEQ is supposed to enjoy more representative abilities than the original OptEqs. In the following parts of this section, we will analyze the theoretical advantages of our GEQ against the vanilla OptEqs. And we also empirically evaluate GEQ's performance in the following sections.
learning methods 2I 44
3.2 GEQ equals to the OptEqs with infinite width
Like other Gaussian-related models, our GEQ model can also be regarded as computing similarities by mapping them to an infinitedimmensional space. This allows GEQ to extract input features at the fiinite-dimensione level, enabling the capture of non- non-linear dependencies in the input space. Essentially, GEQ can be seen specialized version of OptEqs operating within the infinite- dimensional space after mapping input features x and output embedding z to this expanded domain.
Proposition 1. The output of our GEQ (Eqn (6)) is the same a special OptEqs' output whose hidden optimization problem defined as follows:
min G(z:x) min 1 f(z)
+
2 | X ( U-' (w(z))
1
where f the positive indicator function and (1 +fa=1 is the Rel.U activation function, b- Walll,, and @w(z) = [1, 2 w (z), /2>>)//TT (z),.... E Rlxx which
(1)
maps the inputs to the infinite- dimeessoad space with / In" - in defined as follows:
T (1) W =
(Wx) J... x)., Wo)o Wo.)... (W.) x)
( x ....WWW x0
(8)

---

where (Wx) 2) denotes the j-th element of vector Wx.
Based on the analysis provided above, it becomes evident that the hidden optimization problem of our GEQ exhibits a similar formulation to a specific OptEqs, whose inputs x and outputs z are mapped to an sidinitediimeniinnlll space using the weight-tied infinite wide mapping 4w and Pu. Given that both GEQ and OptEqs are derived from their respective hidden optimization problems, the equivalence these problems implies the existence of the same equilibrium states for both models. Consequently, our GEQ can be considered an extension of the -ieittteedeph"" OptEqs to the "'ninnteewwitt" domain. Since wider neural networks are generally expected to perform better on classification tasks, we can infer that our GEQ outperforms vanilla equilibrium models like OptEqs. We further support this claim with the theoretical analysis illustrated the subsequent sections.
3.3
Generalization abilities for GEQ
Apart from the above empirical intuition, we are going to prove our GEQ's generalization advan- tages over OptEqs using the generalization bound under the PAC-Bayesian framework [36]. For convenience, we use (x) denotes the equilibrium state z for input x. Then we use the expected margin loss ( at margin 1) of our GEQ on the data distribution DD for classification, which is defined as follows,
Ly = P2 (x.x)>D fjag(X)y S
max S ggeg( (x)(()))) 370
(9)
where fgen (x) W Walgeg(x (x) + b, stands for GEQ's final prediction at input x with learnable parameters We and b and the index j,y here denote the prediction score for certain class. Then we can analyze the generalization bound for our GEQ following the former work's settings (38).
Proposition 2. If input || is bounded by B. μ == max U|l, || 12,, we have following results for GEQ and OptEqs with ReLU activation. For V9, v7 > 0, with probability at least ô over the training set of size M, we have:
W 12, [bbl2) 1, then
16hlu(2Ah) Amaa (2p8max D)(1 MpD (1 m3] Bw
In( My
Coldga) (olFe
M
(10)
16hlu(24h) D + m)BB 1 (1 m)as² Bw Inf MVM]
ColFontee FoonFforcc
x mj'M
M
where Lyggg denotes the empirical margin loss on the training set, the maximum scaling number is [[bll2 ||W++ +
defined by Smax max maxx b/ll Bw and m = [
||W
XED
W|1 is less than to ensure the convergence of equilibrium models.
Remark 1. Bmax < 0.8 and m > 0.9, we can get 1-9maxth the meanwhile, our GEQ's generalization bound is tighter than the original OptEq.
Smaall
I-m and 3p8 I- --mmxct
I-m In
In practical experiments, find that the above conditions for 9max and μt, m satisfied in most cases. And the following experiments also support our above theoretical advantages. However, the above bound is not tight, and how to approximate much tighter bound for equilibrium models still needs exploring.
3.4 GEQ enjoys More Stable Performance
Apart from better performance, Gaussian kernel stands out as one of the most extensively employed kernels in machine learning tasks owing to its stability across various input scenarios. Motivated by this, we aim to investigate whether incorporating Gaussian kernels into our equilibrium models can enhance the model's s stability across diverse inputs. Firstly, we are going estimate output changes with respect to the input perturbations.
Proposition 3. If norms for the imputs and outputs are bounded by B, the spectral norm for the weight parameter W, U of equilibrium models with ReLU activation are bounded by p. 1 to ensure
5

---

convergence, then we have the conclusions as below:
(1 geggg (X2)ll2 S Ingegl(x1 X2ll2 mmaxt 2 + /rB
Smaxt -1 x
x2112,
(11)
B
[Ifopteg (x1) ) - (x2)l2 S Lopteglixx - xell2 = 1 122 |X| X2112:
1
(12)
where X1 and X2 are input samples, Sgen (x.) and Sopteg (x.) denotes the equilibrium states for GEQ and OptEqs given input x., and fmax maXxED --r W2lx <1
Remark 2. If fmax < 0.8, B < 1, and / < 0.2, then Lega < Lopteq-
In practical experiments, we choose different to reach the above condition for 3max and the condition for input B can also be achieved by normalization layers. Although the above bound is not tight, can use it as rough explanation for our GEQ's 's stability, which is demonstrated in the following experiments. How to approximate a much tighter bound for equilibrium models still needs exploring.
Besides having stable outputs under perturbations, a stable model should also show large output differences for different classes to make classification easier. However, the above Lipschitz term can not constrain outputs' similarity when samples are far apart, then we need a new metric for analysis. In line with previous works [8 (4 [0], we assume all weight parameters go to infinite dimensions and analyze the expected output similarity x for model f for inputs x1 and X2 defined below:
x((1, x2) = ((xx))((()) - fuxxx))T(xxx)(((((( fu(xx)p(u)ddu
(13)
with p(u) is the distribution of weight U's vectorization. If is smaller for samples x1 and xę when they belong to different classes, which means they are far away, then the classifier can easily classify these two samples with different labels. The margin for the classification will also be large and easy for the classification of difficult samples. The k's upper bound for GEQ and OptEqs are listed below:
Proposition 4. If norms for the imputs and oulputs are bounded by B, the spectral norm for the weight parameter W of equilibrium models with ReLU activation are bounded by p. < 1 to ensure the convergence, and each row in U obeys the spherical Gaussian distributions (10, E[U?][). Then we have the following conclusions for the expectation of the output similarity for GEQ and OptEqs with respect to input X1 X2 as follows,
De ( (U)² [|x x2 | X1 1/ |/2l||/|xll (sin 0g (m 0b) cos Bu)
Kgeg (x1, x2) S Rgeq"
2T - 2))
(14)
(sin Og + (π 0o) cos 0o)
Koptog [xp (x112)) S Ropteg
(15)
2TT
n²)²
where X1 and X2 are input samples,
BBIWI 3max maxxED e Ux Wall,, J (U)) is the angle between the samples.
U's minimal singular term, and 0g = cos 1 [x)) |xa|||
Remark 3. [[| - x²l12 2 2 2logloo(1//))) /min (U), then Rgeg S Rgeg-
Based the aforementioned analysis, it is evident that our GEQ exhibits smaller output similarity for dissimilar samples. As a result, the predictions made by GEQ are primarily based on the most similar samples, enabling it successfully classify challenging instances. This claim is further supported by the results obtained from our carefully designed experiments.
3.5 Patch Splitting in GEQ
Since different parts of images have different impacts on the image classification, calculating the whole similarity using Gaussian kernels for GEQ is not enough. Inspired by former works [47, we also split the feature map Ux into patches, and then our optimization problem becomes:
N
min G(z; x) min
1 f + 11a11
Σ 27
1
e
(16)
1=1

---

Chamtel Muttiplicitonn
(W sm)
Ax bi - Mx Craa
Uix+ b
Figure I: The sketch map of one layer GEQ's n-th fixed point iteration. x is the input and z (n-1), 1),,) (()) are the output of (n - 1)-th, n-th iteration, and m a middle state.
where x; E RCP is the i-th patch of Ux + while 2, E RSP" is the i-th patch of Wz and WA E Repp"xesd is linear layer to project patches with different size to the constant dimension. e, denotes the channel splitting number, p denotes the patch size, and Chia denotes the hidden dimension of patches after projection. We note that the patch-splitting approach is GEQ's unique feature, as incorporating this technique makes no difference in OptEqs due to its linear kernel. Figure [ provides sketch for GEQ's i-th fixed-point iteration. From the figure, is evident that our GEQ can be viewed as a special OptEqs with additional attention mechanisms to capture the most important regions. Thereby, it can achieve enhanced performance. For a more detailed understanding of the forward procedure in our GEQ, please refer to Appendix A.
4 Empirical Results
4.1 Experiment Settings
In our experiments, we employed parallel GEQs with different input scales like MOptEqs and averaged the output of each branch after average pooling nearest up-sampling to fuse the branches. We use weight normalization to ensure the convergence as MOptEqs and MDEQ, and set y to 0.2/M, where M is the minimum || ž. w among all patches. For the equilibrium calculation, we used the Anderson algorithm in the forward procedure, similar to other implicit models (28]. and applied Phantom gradients [14] for mack-propagation. All models were trained using SGD with step learning rate schedule. We implemented our experiments on the Py platform (39 using RTX-3090. Further details can be found in the Appendix A.6 To compare the performance of our GEQ, we used MOptEqs and MDEQ as benchmark implicit models, which have demonstrated superior performance over OptEqs on image classification tasks. Additionally, we used ResNet-18 and 5e as benchmark explicit models for comparison.
4.2 Results for Image Classification
Firstly, we finish the experiments on CIFAR-10 and CIFAR-100. They are widely used datasets for image classification
small images. In the experiment, we parallel 6 branches GEQ with the input scale is 32, 16, 8, 8, 4, 4 and MOptEqs' architecture setting also the same. The details can be found in the Appendix. As for the comparison, we also conduct experiments of the same training procedure for MDEQ, MOptEqs, and ResNet. The results are listed in Table [ From the results, one can see that our GEQ enjoys clear advantages CIFAR datasets, which demonstrates the powerful generalization ability of other models.
Besides small datasets, we also conducted experiments on large-scale image datasets, as presented in Table 2 The results clearly demonstrate the consistent superiority our GEQ over other models, highlighting its clear advantages. Particularly noteworthy is our GEQ achieves a 2% improvement on ImageNet- against deep model ResNet-50 while consuming approximately half the number of parameters, which emphasizes the effectiveness and efficiency of GEQ on large-scale inputs.
7

---

Model Size
Accuracy
Model Size
Accuracy
ResNet-118
10M
93.5 = 0.2%
ResNet-18
10M
ResNet-50
23M
9.2
ResNet-50
23M
1.19
MDEQ
10M
94.2 +: 0.3%
MDEQ
10M
MOptEqs
8M
94.6 0.2%
MOptEqs
8M
75.6 + 0.2%
GEQ
5M
94.8 + 0.1%
GEQ
5M
76.4 + 0.3%
GEQ
8M
95.6 0.2%
GEQ
8M
78.2 + 0.2%
(a) CIFAR-10.
(b) CIFAR-100,
Table 1: The Empirical results for image classification CIFAR-10 and CIFAR-100.
Model Size
Accuracy
Model Size
Accuracy
ResNet-18
11M
92.3 0.1%
ResNet-11
11M
80.9 + 0.3%
ResNet-50
23M
0.2%
ResNet-50
23M
81.7
MDEQ
10M
91.5 - 0.2%
MDEQ
10M
0,
MOptEqs
10M
92.4 0.2%
MOptEqs
13M
81.5 0.4%
GEQ
6M
92.9 + 0.2%
GEQ
6M
82.2 + 0.2%
GEQ
13M
93.2 + 0.1%
GEQ
13M
83.9 + 0.3%
(a) ImageNette.
(b) ImageNet-100.
Table 2: The Empirical results for image classification on ImageNette and ImageNet- 100.
4.3
Validations on the models' stability
Evaluation on Unseen difficult samples. In order to assess the stability our GEQ model on difficult examples, we conducted experiments using CIFAR-100 super-class classification. CIFAR- 100 consists 20 super classes, each containing five sub-classes We trained our GEQ and MOptEqs models to predict the super-classes using the first four sub-classes from each super-class for training. evaluated the models using both the test set, which includes the first four sub-classes from each super-class (referred to "Known Accuracy"), and separate set of samples from unseen sub-classes (referred to as "Unknown Accuracy"). The classification of the unseen samples is more difficult as they different from the training set. The results of our GEQ and MOptEqs models are presented Table
Known Accuracy
Unknown Accuracy
MOptEqs GEQ
80.1 + 0.3%
77.4 = 0.5%
80.9 + 0.2%
80.1 + 0.6%
Table 3: Empirical rsults on CIFAR-100's super-class classification.
The above table clearly demonstrates that our GEQ model surpasses MOptEqs in achieving superior performance on the challenging task at hand and demonstrates GEQ's stability. Such advantages can be attributed to the fact that GEQ exhibits smaller output similarities compared to OptEqs when input samples are far apart (e.g., samples from different classes). This characteristic can lead to larger margins between different classes, enabling the classifier to be more easily optimized during training. Consequently, our GEQ model excels in accurately classifying difficult unseen samples, further highlighting its stability and superiority
former equilibrium models.
Ablation Studies on corrupted datasets. Apart from difficult samples, we are going to compare the robustness of our GEQ, MOptEqs, and ResNet on the CIFAR-10 corruption dataset, which contains 19 common corruptions including image transformation. image blurring, weather, and digital noises on CIFAR's test datasets. Average results on degrees CIFAR-10 10 corruption datasets list in Figure 2
From the result, one can see that our GEQ based on Gaussian kernels more robust than MOptEqs and ResNet. In particular, our GEQ can show better performance against structured noise and some image transformation. The above results also demonstrate the stability of our GEQ structure.
For example, super class "people" contains five sub classes. "bab"" "girl", "man". "man". "woman"

---

Figure 2: The results for different models under different corruptions.
4.4 Ablation Studies on Saliency Map
The saliency maps generated by GradCAM 45 offer valuable insights into the visual attention of both MOptEqq and GEQ models. These maps highlight the regions of the image that are crucial for the model's predictions. preuentsss the saliency maps obtained for an image from the ImageNet dataset using both models. Upon observation, it becomes evident that GEQ exhibits higher degree of focus the significant regions directly associated with the predicted label "manta". ". contrast, MOptEqs tends to allocate attention to unrelated regions such as the shells. This discrepancy indicates that the attention- module induced by the Gaussian kernel in GEQ enhances the concentration of the model's s attention, resulting in improved performance compared to MOptEqs.
(a) Original Image.
(b) MOptEqs's saliency map. Figure Saliency map for GEQ and MOptEqs on the ImageNet image.
(c) GEQ's saliency map.
4.5 Ablation Studies on Patch splitting
The performance of our GEQ model is influenced by the channel splitting parameter and the patch size. Choosing large values for these parameters causes the kernel to focus mainly on global information while selecting small values makes the kernel concentrate local features. To understand the impact of these choices on model performance, we conducted experiments, the results of which are presented in Figure This figure illustrates the relationship between the channel splitting parameter, patch size, and the model's performance. By analyzing these results, we gain insights into the optimal values for these parameters that yield the best performance for our GEQ model.
The accuracy trend depicted in the figure shows an initial increase followed by a decrease as channel split and patch size increased. Based these empirical results, select patch size of 2 and channel split of for both the CIFAR and ImageNet experiments. These parameter choices are made to optimize the performance of our models on the respective datasets.
4.6 Comparison with other kernel functions
Firstly, we introduce different commonly used kernels, such as polynomial, sigmoid, and Gaussian kernels, to reformulate the hidden optimization problem for equilibrium models. 4 illlstraaess the equilibrium models induced by these kernels. It can observed that equilibrium models with polynomial and sigmoid kernels also incorporate new attentive modules. However, their attentive

---

78
78
77.6
77.6
77.2
77.2
Acc
Acc
76.8
76.8
76.4
76.4
76
76
16 32
64
1
4
8
channel split
patch size
(a) Influence of the channel splitting parameter.
(b) Influence of the patch size parameter.
Figure 4: The influence on the patch size and the channel splitting parameter for our GEQ on CIFAR-100 datasets.
kernels only constrain the input features Ux + b and do not directly affect the activation of the output z". As a result, their performance may be inferior to our GEQ model. To validate these claims, evaluate the performance of different models the CIFAR-100 dataset. Since the fusion module is the primary difference between MOptEqs and OptEqs, we can easily modify the structure of MOptEqs to include different kernel-induced attentive modules using the equations in Table 4 resulting in MOptEqs (Polynomial) and MOptEqs (Sigmoid). The results are presented in Table 5 which clearly demonstrates the superior performance of our GEQ model. For more in-depth analysis of GEQ, we refer readers to the main paper.
Tidhen Optimization
Equilibrium
Lincur
mil., 1 (12) 1|1 (Ux b.z)
Well
W Wz +
Polynomial
ming 1 [()) + (/Ux 2))" WelB
Wz d(Ux b. 23)" (Ux +b)
Sigmoid
mina
1 f(z) 3|121 tanh (Ux b,
W
= Wz' tanh" ((Ux + b. b, (Ux+ b))
min, f(x) Hiali + Cast Wall
Wilutbbb Wa' lew
b)
Table 4: The hidden optimization problems and their related equilibrium models. d > is an integer denoting the polynomial order.
Model Size
Accuracy
MOptEqs
8M
75.6 = 0.2%
MOptEqs (Polynomial)
8M
75.1 4: 0.4% 76.1 +: 0.3%
MOptEqs (Sigmoid)
8M
GEQ
8M
78.2 0.2%
Table 5: Comparison of equilibrium models with different kernel functions CIFAR-100.
Conclusions
In this paper, we introduce novel optinization-induced equilibrium model called GEQ, which utilizes Gaussian kernels in its optimization-indueed framework. Our model incorporates a new attentive module that arises from its novel hidden optimization problem formulation. Notably, GEQ exhibits significantly improved performance in classification tasks, outperforming deep models as well. Morcover, GEQ can be interpreted as weight-tied model with infinite width and depth, highlighting its expressive power. We also provide theoretical analysis demonstrating the superiority of our models in terms of generalization ability and stability compared to previous OptEqs. Empirical results further validate the effectiveness of our proposed approach.
10

---

Acknowledgments
Zhouchen Lin was supported by National Key R&D Program of China (20222D01603022) the major key project of PCL, China (No. PCL2021A12), the NSF China (No. 62276004), and Qualcomm. Yisen Wang was supported by National Natural Science Foundation of China (62006153, 62376010, 92370129), Open Research Projects of Zhejiang Lab (No. 2022RC0AB0S), and Beijing Nova Program (20230484344).
References
[1] Brandon Amos and J Zico Kolter. Optnet: Differentiable optimization as a layer in neural networks. In ICMI., 2017.
[2] Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with infinitely wide neural net. In NeurlPS, 2019.
[3] Shaojie Bai, Zico Kolter, and Vladlen Koltun. Deep equilibrium models. NeurIPS, 2019.
[4] Minmin Chen, Jeffrey Pennington, and Samuel Schoenholz. Dynamical isometry and a mean field theory of rnns: Gating enables signal propagation in recurrent neural networks. In ICML, 2018.
[5] Qi Chen, Yifei Wang. Zhengyang Geng, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Equi- librium image denoising with implicit differentiation. IEEE Transactions on Image Processing. 32:1868-1888 2023.
[6] Qi Chen, Yifei Wang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Optimization-induuce graph implicit nonlinear diffusion. In ICML, 2022.
[7] Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, NeurlPS, 2018.
[8] KR1442 Chowdhary. Natural language processing. Fundamentals of artificial intelligence, pages 603-649, 2020.
[9] Filipe de Avila Belbute- Kevin Smith, Kelsey Allen, Josh Tenenbaum, and J Zico Kolter. End-to-end differentiable physics for learning and control. NeurIPS, 2018.
[10] Alexander G. de G. Matthews, Jiri Hron, Mark Rowland, Richard E. Turner, and Zoubin Ghahramani. Gaussian process behaviour in wide deep neural networks. In ICLR, 2018
[11] Josip Djolonga and Andreas Krause. Differentiable learning of submodular models. NeurlPS, 2017.
[12] Zhili Feng and J Zico Kolter. On the neural tangent kernel of equilibrium models, 2021.
[13] Adrià Mauriga-Alonso, Carl Edward Rasmussen, and Laurence Aitchison. Deep convolutional networks as shallow gaussian processes. In ICLR, 2019.
[14] Zhengyang Geng, Xin-Yu Yu Zhang, Shaojie Bai, Yisen Wang, and Zhouchen Lin. On training implicit models. In NeurlPS, 2021.
[15] Laurent El Ghaoui, Fangda Gu, Bertrand Travacca, and Armin Askari. Implicit deep learning. arXiv preprint /ot::::00001110 15, 2019.
[16] Dar Gilboa, Bo Chang, Minmin Chen, Greg Yang. Samuel S Schoenholz, Ed H Chi, and Jeffrey Pennington. Dynamical isometry and a mean field theory of Istms and grus. arXiv preprint 17Xiv:1901.08987, 2019.
[17] Stephen Gould, Richard Hartley, and Dylan Camphell. Deep declarative networks: A new hope. arXiv preprint 19090 04866, 2019.

---

[18] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. ICML, 2017.
[19] Soufiane Hayou, Arnaud Doucet, and Judith Rousseau. Mean-field behaviour of neural tangent kernel for deep neural networks. arXiv preprint 0rXiv:1905.13654 2019.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[21] Shell Xu Hu, Sergey Zagoruyko, and Nikos Komodakis. Exploring weight symmetry in deep neural networks. Computer Vision and Image Understanding. 187:102786, 2019.
[22] Arthur Jacot, Franck Gabriel, and Clément Hongler. Freeze and chaos for dnns: an NTK view of batch normalization, checkerboard and boundary effects. arXiv preprint arXiv: abs/1907.05515, 2019.
[23] Nikhil Ketkar and Nikhil Ketkar. Stochastic gradient descent. Deep learning with Python: A hands-on introduction, pages 113-132, 2017.
[24] Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. arXiv preprint 2017.
[25] Mingjie Li, Lingshen He, and Zhouchen Lin. Implicit euler skip connections: Enhancing adversarial robustmess via numerical stability. In ICML, 2020.
[26] Mingjie Li, Yifei Wang, Yisen Wang, and Zhouchen Lin. Unbiased stochastic proximal solver for graph neural networks with equilibrium states. In ICLR, 2023.
[27] Mingjie Li, Yisen Wang, and Zhouchen Lin. Cerdeq: Certifiable deep equilibrium model. In ICML, 2022.
[28] Mingjie Li, Yisen Wang, Xingyu Xie, and Zhouchen Lin. Optimization inspired multi-brande equilibrium models. In ICLR, 2021.
[29] Guangean Liu, Shiyu Chang. and Yi Ma. Blind image deblurring using spectral properties of convolution operators. IEEE Transactions on image processing, 23(12):5047-5556 2014.
[30] Guangean Liu and Ping Li. Low-rank matrix completion in the presence of high coherence. IEEE Transactions on Signal Processing, 64(21):5623-5633, 2016.
[31] Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv: /80.::0800000555 2018
[32] Juncheng Liu, Kenji Kawaguchi. Bryan Hooi, Yiwei Wang, and Xiaokui Xiao. Eignn: Efficient infinite-depth graph neural networks. NeurIPS, 2021.
[33] Julien Mairal, Francis Bach, Jean Ponce, and Guillermo Sapiro. Online dictionary learning for sparse coding. In JCML, 2009.
[34] Lassi Meronen, Christabella Irwanto, and Amno Solin. Stationary activations for uncertainty calibration in deep learning. In NewrlPS, 2020.
[35] Radford M Neal. Priors for infinite networks. In Bayesian Learning for Neural Networks, pages 29-53. Springer, 1996.
[36] Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nathan Srebro. A pac- bayesian approach to spectrally-normalizee margin bounds for neural networks. arXiv preprint arXiv: 1707.09564, 2017.
[37] Roman Novak, Lechao Xiao, Yasaman Bahri, Jaehoon Lee, Greg Yang, Jiri Hron, Daniel A. Abolañia, Jeffrey Pennington, and Jascha Sohl-Dickstein. Bayesian deep convolutional networks with many channels are gaussian processes. In ICLR, 2019.
12

---

[38] Chirag Pabbaraju, Ezra Winston, and J Zico Kolter. Estimating lipschitz constants of monotone deep equilibrium models. In ICLR, 2021.
[39] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, perforreffommece deep learning library. In NeurIPS, 2019.
[40] Yi-Ling Qiao, Junbang Liang, Vladlen Koltun, and Ming Lin. Scalable differentiable physics for learning and control. In ICML, 2020.
[41] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In CVPR, 2016.
[42] Max Revay, Ruigang Wang, and Ian R Manchester. Lipschitz bounded equilibrium networks. arXiv preprint urXiv:2010.01732, 2020.
[43]
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE transactions neural networks, 20(1):61-80, 2008.
[44] Bernhard Scholkopf, Kah-Kay Sung, Christopher JC Burges, Federico Girosi, Partha Niyogi, Tomaso Poggio, and Vladimir Vapnik. Comparing support vector machines with gaussian kernels radial basis function classifiers. IEEE transactions Signal Processing, 45(11):2778
2765, 1997.
[45] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient- based localization. In ICCV, 2017.
[46] Ivana Tošié and Pascal Frossard. Dictionary learning. IEEE Signal Processing Magazine, 28(2):27-38, 2011.
[47] Asher Trockman and J Zico Kolter. Patches are all you need? arXiv preprint SXI:2200100979. 2022.
[48] Russell Tsuchida, Fred Roosta, and Marcus Gallagher. Invariance of weight distributions in rectified mlps. In ICML, 2018.
[49] Russell Tsuchida, Suk Yee Yong, Mohammad Ali Armin, Lars Petersson, and Cheng Soon Ong. Declarative nets that are equilibrium models. In ICLR, 2022.
[50] Po-Wei Wang, Priya Donti, Bryan Wilder, and Zico Kolter. Samnet: Bridging deep learning and logical reasoning using differentiable satisfiability solver. In ICML, 2019.
[51] Ezra Winston and J Zico Kolter. Monotone operator equilibrium networks. In NearlPS, 2020.
[52] Xingyu Xie, Qiuhao Wang, Zenan Ling, Xia Li, Guangean Liu, and Zhouchen Lin. Opti- mization induced equilibrium networks: An explicit optimization perspective for understand- ing equilibrium models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 4533)330043361, 2022.
[53] Xingyu Xie, Jianlong Wu, Guangcan Liu, Zhisheng Zhong, and Zhouchen Lin. Differentiable linearized admm. In ICML, 2019.
[54] Chong You, Daniel Robinson, and René Vidal. Scalable sparse subspace clustering by orthogo- nal matching pursuit. In CVPR, 2016.
[55] Hongyang Zhang, Zhouchen Lin, Chao Zhang, and Edward Y Chang. Exact recoverability of robust pca via outlier pursuit with tight recovery bounds. In AAAI, 2015.
[56] Xiao Zhang. Lingxiao Wang, Yaodong Yu, and Quanquan Gu. A primal-dual analysis of global optimality in nonconvex low-rank matrix recovery. In JCMI., 2018.
[57] Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint 1611v:161..01577 2016.
13

---

A Appendix
A.1 Forward Procedure for GEQ
The pseudo-code for our GEQ is listed in Algorithm
Algorithm 1: Calculating one layer GEQ.
Require: initial state (0 weight parameter W, Ux E Rehw channel split cs, patch size p, hidden layer W, E R x32
Ensure: Get the output x of i-th fixed point iteration.
def ;xx U, W.b). b):
Rearrage (z E. Ux+b- - Ux+b- Rop x(c,p")
in = diag e 11 2. 13
(x - 2) W.
Rearrage in m _xxxxx
return (f-1)) = (W m)
with torch.no_graaiie
Use anderson algorithm to solve = 9(""; U, W.
# calculate gradient via phantom gradient:
for i in range(5) do
0.2 x x² 0.8 "x=";x,,>>>>>>>>>>>>>>>>1""" W. b).
end for return zz
A.2 Proofs for proposition 1
Proposition 1. The outputs of GEQ with Gaussian kernels (Eqn (6) is the same as Optimized induced Equilibrium Models' output whose output is the optimal solution of the hidden optimization problems:
min G(z;)((x =1 f(z) + + =|1z|12 X (0 + -)) (.
(17)
where @w (z)
1, 27 (z)
(27]' T (z), E Ix which projects the features to
R= Rin' is the k-tuple permutation with repetitions the infinite- disiteed-mmniionall space. And $ formulated as follows:
(i) W
(W)) (W.)) (Wz) x) (Wx) (W... x)2 (Wx), (W x) (W x)) )msm
(18)
where (Wx), denotes the j-th element of vector Wx. Then the Gaussian kernel can also be regarded as calculating the input features after the weight-tied infinite wide projection 4w and 4u.
Proof. We can formulate the OptEqs' hidden optimization problem with Gaussian kernels as below:
1 Ux
min G(z; =1 f(z) + + |
f(2)
(19)
2
For bb we have
Ux+ Wxll - 1 Wal3
(20)
- Ux+bl| - /
letting ->/||xx+blle Wxllz and do the Taylor expansion for el (Ux+b, Wx) we have:
(Lx27(Ux b), (2z)))))
(21)
i!
14

---

For any we have from the permutation theory:
= x UUTNE 'b), 120 (wll))
(22)
i!
where ( W (x) is the i-tuple permutation with the repetition for given (Wx)1, (Wa)2, (Wx),
Each element of W (x) one possible permutation. Since there are n' tuples, then $ can project the input features to in' space as follows,
2 w
(Wx) (W)) (..r)))............ (Wx)
(Wx)2, ... (Wx), (Wo)R. (Wr)m mm......
(23)
int
Thereby, the hidden optimization problem for our GEQ can be reformulated as,
min G(z: =1 f(z) + [11l X (Tv(x U-- ow (2))
(24)
Then our conclusion is proved.
A.3 Proofs for proposition 2
Proposition 2. If input ||x|| is bounded by B. μ == max (1 [ Well,, [b[l2) 1. then we have following results for GEQ and OptEqs with ReLU activations. For vo, n7 > 0, with probability at least 8 over the training set of size M, we have:
16hlm(24h) Amaup B (2p8 1)(1 mpD (1 m]] Bw
In( My
Coldgee CulFaoe) +
1 Penux m. j' M
M
(25)
16hlu(24h) p m)uB m) Bw
Inf My/3
Colfonted Foppee
x mj'M
M
where Lygggee denotes the empirical margin loss on the training set, the maximum scaling number is | [bbl + || + 1b.l
defined by Amax"
max m e Bw
xED
and m = | Wl2 is less than to ensure the convergence of equilibrium models.
Before the proof results, need introduce lemma in former work 38 for the perturbation bound for GEQs and reformulated OptEqs as follows.
Lemma 1. Let ||W]22 S m and ||W]]2 S m. Then change in the output of the DEqs z a (Wz + Ux + b) on perturbation the weights and biases from W, U, to W, U, b is bounded as follows:
[[ bx f(Wz Ux+
W w
|Xx
-bl2 + (U- - U)x|| + +|b bll2
(26)
(1 mm)²
Like former works 38 we we also introduce another lemma 33
here:
Lemma 2. Let fw be any predictor with parameters and let P denote any distribution the parameters that are independent of the training data. Then, for any i, >> O, with probability
1 8 over the training data of size M, for any w, and any random perturbation such that max, z [] (x) fa (x))oo < Z 3: we have
KL(w + u|(P) + In 6M
Lo(sw) S Log +4V
(27)
M -
Then we can derive the perturbation bound for the reformulated OptEqs and our GEQ following former settings 338 First, we also incorporate fully connected layer at the end as we mentioned in the paper.
Fg (x) Wefgeg (x)
xx)+ + be, opteeg (x) = W Wefopteg (x) + b
(28)
15

---

Since the entries in the perturbations obeying the distribution os W(0, we have that all the perturbations of weights | are bounded by o 2222hl(24h) with probability larger than 1/2.
Since the only difference between OptEqs and monDEQ [51] the weight parameterization, our reformulate OptEqs is parameterized by W. = W W while the monDEQ's weight parameter is parameterized by series of weights W mondeg I + A + A +B+ B", Therefore, the pertubation
in Aww is different from former analysis 38 We have that:
A lll
W W Wdwlla I 2wh
(29)
Then using the above lemma, we have that for all with probability at least 1/2.
[J (x) - Sopter (x)ll2 - W opteq (x) + b. -Wel (x) - bll2
2/ B +1] 2pm (B +1)
(30)
m)²
n11 m(1mm²) Setting 112hln((22))22000041))1 -m
will make the above perturbation less than 2
Then we have,
Bw 202
1B
2(1 +1)00//008000-1-000000-----=---) m)
KL(W. Aw. P)
Bw
172 m)4
(31)
With the same choice of It's bound like former work 38
7 (1-m) m) 2(B + <B3
(1 mMVM
(32)
2(B+1) 1)
we can finally get the upper bound as our OptEqs bound as our proposition shows.
The difference between GEQ and OptEqs is that GEQ's can be viewed as multiplying scaler /
-- Wall with depend x since z also depended on Setting 3max maxxe 3(x)
1 and Bmin minxeD 3(x)= >> c. Assuming 3 changes a little with respect to the small perturbations
on weights, we have:
z (W, U. o0-BW, ( + (WW(UX+X)
8max/ W Wall2/|xx + bll2
PInax (II
WU)x + [Wb Wb]l2)
2 W U,b
Amaxm
(33)
With the same setting above OptEqs, we have:
||
(W)(- U-Aux/l s Wbbb)xx wabbl S 2wn (34)
Then we can obtain that:
[IF geg
fkereg (x)112 < W. of optey (x) + D.b W Wefōpted (x) bbll2
2,3max S (
1) +
2 w/ ( + 1) + mam + 1)
s (1 Buaxe m2
1 Bmaz +
1 Bmax
(8maxtt) * (2 1))
+ (2uBmas + 1) (nw + 1))
+
7.)
Bman
(35)
With the same setting as above, we have:
Bw
KL(W + Aw (P)
20
(36)
=TBw,
where 7 is defined as follows:
T 16/In(24h))(((mmmm))))) 1) 2(243max + 1)(1 Bww.emm)ml(l + (1 8max 22
()
)maxm))
(37)
then finally get the upper bound OptEqs bound as proposition shows.
16

---

A.4 Proofs for proposition 3
Lipschitz constant is the minimal constant for f and Vx, y suits the following equation:
[x - f(y) - x-12-
(38)
Thereby, our analysis in Proposition 3 can be viewed as proposing an upper bound for different models. In this section, we are going to prove the Lipschitz upper bounds for our GEQ and OptEqs. First, we restate the proposition as follows:
Proposition 3. If norms for the inputs and oulputs are bounded by B, the spectral norm for the weight parameter W, U of equilibrium models with ReLU activation are bounded by p to ensure convergence, then we have the conclusions as below:
Smaxt? ? + v7
[[ge((()) - (x2) S Legeg ||x] - X2ll2 1 -mmaxft? B
[x] X2112,
(39)
ll (oopee (opteg(Xx)) 1l2 S Lopteg X2112 1 122 ||x X2212. where x1 and X2 are input samples, Sgen (x.) and Soptog (x. denotes the equilibrium states for GEQ and OptEqs given input x., and 9maz == maXXED e ~>l Wall <1.
(40)
Proof. We first prove the inequality for OptEqs:
[l Sapteg (x2)]l2 2 ||,x,
2x 1|2
$ 1oo (W Wzx, + Ux1) (W Wzx; + Ux2)
$ W W(zx, - 2xz))12 111 + +|(Uxx x2)1l2
Zx1 2xall2 + pl|x1 X2112.
(41)
where Zx, denotes the equilibrium states for OptEq with input X1, which means the following equation is satisfied:
Zx. = o (WT Wzx, + UxI).
(42)
Reformulating the equations, we can get:
[[opttg(X1) Sapteg (x2))l2 5 1 112 |x| x2
(43)
Then for GEQ, have:
00 (e-1l/ Wex, UUxI (Uxill2- (-W Waxi W UXx))
[xx Zx, 112 S
-o -0 W Uxa l12 (-W Wax; + W +WUUx2)) Uxa))11
(44)
S.max TWTW(x 1/2 - | - x2) + BB2 |xx Bxal.
<.maapt ||zx,
2xz ll2 + Amaxle? 2 ||x| x2l12 + BB² 1 - Axyl
where Zx, denotes the equilibrium states for GEQ with input and 8x, is defined as follows:
x = W*,, Ux|
(45)
Then with the mean value theorem, we have the following equations:
[x, - -xxl = Wax, Uxal1
7 wa,, -xx l|
S Wzx,
Ux2l12
WZx.
Uxell21
max
Ce
<Ev Wa,,
Uxi ||-||
Wex,
Uxa|1,]
5 Wix;
Ux2 12 W2x; Ux2l1z1
< W((x)
2x, 2x,)112 + |U(xI - -2))11
S VTP ([|zx|
Zxall2 ||x1 x2ll2)
(46)
Then reformulating Eqn (44), we can finally obtain our proposition as below:
2xg 11 (11))((x1) -Sgog(X2)112 _ Pmaxx/? + BTB
||
||x
X2l12
(47)
Bmaxx v Bu3
17

---

A.5 Proofs for proposition 4
In this section, we are going to prove the output similarity bounds for our GEQ and OptEqs. First, we restate the proposition as follows:
Proposition 4. If norms for the imputs and outputs are bounded by B, the spectral norm for the weight parameter W of equilibrium models with ReLU activation are bounded by μ < 1 ensure the convergence, and each row in U obeys the spherical Gaussian distributions (1, E Then have the following conclusions for the expectation of the output similarity for GEQ and OptEqs with respect to input X1, X2 as follows,
1²² De (0 (U)* x1-1xxxx111 x2 EEU [x] ||||
(sin Oo (TT 0o) cos 0n)
ligeg X1, x2) S Kigg
2 )2
(48)
E[U?] X2
|X|]|] (sin Po (T 0b) cos cos
Kopteq (x1, X2) S Roptey
(49)
2T - 22
where X1 and X2 input samples, D BiI and Smax maXxED Senepeenllle - defined as the angle between the samples.
< 1.
8g cos"
1oss"(( x x
(x)
Proof. Before the proof, we first introduce the following lemma:
Lemma 3. [48] U obeys the spherical Gaussian distributions of variance B[U?] and mean 0, then the expectation of the Similarity for the one-layer Neural Network o (Ux) is:
[N = E X1 ||2|||
(sin 0g + (T - -00) cos 0g)
(50)
2x
where 8g = cos cos- (x,y) |
Letting m == IW I W and μ = W||2 < 1 assumptions demonstrate and neglecting the bias b for convenience. Then for our reformulated OptEqs, we have:
2x1 Ex2 S o(W,lxi T 2x2 + (UUx)) T
Xx2
(51)
S 11²0(ux
)T Txxx= + oUxx)) 1x2
And have
((x))) 2x S SO()) T To (WZzz (Ux:)) (o(xx)) (UXx)
(52)
S ((oo(UX1))
2x, + (Ux,)) (UUX2)
Then
(UxI)) T (UX2))
o(Uxi
Zx2
112 o (Uxi) (UUx)))
(53)
2x1 Tzxa
(1 222
Therefore, we can conclude
EEU X [x2
(sin Oo + (T 0o) cos 0o)
Kopteg X2) 5
(54)
2T((1 222
For GEQ, we -e-x/Uxx-elll and Bmaz xED
Ux Wells
max
2x, 2x2 < Bmax W 2xxx)
2x2 T
Bx W (1x0 T 2x2
(55)
I 2 a (zxi) Zx, + (a,uo((Uxi) Zxx
Like OptEqs, we obtain the following equations for GEQ:
112 1r, Bx o (Ux, ( (Uxx))
2 Zx2
(56)
(1 2))
18

---

Therefore,
1ea Bx, 3x, 1 x1|2||/ (sin 0o + (IT 00) cos 80)
Kgea (21, S
(57)
2n Pm
And we have,
8x Ax, = -x( W Ux: Ux: 1x Ux213)
S -7/2([[W(xx) 2xg -
(58)
S De (a (U)*||> -
where D = 21 the latter two inequality is acquired by Jensen's 's inequality. Then we have,
Kigeg (x1;x2) De H (amin (U)*
EE(? X1 ||a |22|| (sin On (= 0o) 0o)
(59)
2x (1 Smax|(?
A.6 Experiment Settings
A.6.1 Experiments on CIEAR
For our GEQ, we parallel branches with each branch taking the scale of 32, 16, 8, 8, 4 and using the average fusion method for branches' fusion. The output channels for 6 branches or 320 but the mid-channel number(output channel for weight U and W) for the six branches are 64, 128, 128, 128, 256, 256 80, 160, 160, 160,320, 320, 320 with patch size and splitting 8. And the inner MLP inner W output 64 hidden dimension for each patch. We use the SGD (33 optimizer with momentum and step learning rate schedule for all the models. We also RandomAug for all the models for comparison.
all 256
A.6.2 Experiments ImageNette and ImageNet-100
We take the input scale 256 for all models. For our GEQ, we parallel 6 branches with each branch taking the scale of 64, 32, 16, 16, 8, 8 after two downsampling convolutions and using the average fusion method for branches' fusion. The output channels for 6 branches 384 but the mid-channel number(outpar channel for weight U and W) for the six branches are 32, 64, 128, 128, 256, 256 48, 96, 192, 192, 384, 384 with patch size 2 and splitting is 8. And the inner MLP inner W output 128 hidden dimension for each patch. We use the SGD optimizer with momentum and step learning rate schedule for all the models. We also RandomAug for all the models for comparison.
all 256
19


Reviews:
[{"review_id": "E9JxfbtKIM", "reviewer": "Reviewer_8r48", "summary": "In this paper, the authors propose to use Gaussian kernel in OptEqs (optimization-induced deep equilibrium models). Because of the more powerful capability of kernel, it can better capture the dynamic performance than linear models. \n", "questions": "Replacing the linear operator in OptEqs by a nonlinear one is nice. But can the authors explain more why kernel is chosen? why not try e.g., MLP, of which the parameters could be trained.\n\nQuestions for numerical experiments could be found above, especially if the authors want to claim advantages over standard neural networks. \n", "limitations": "not applicable", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "OptEqs is an interesting attempt to describe the training dynamics of DNN and hence making it more powerful (this paper's work) is interesting. (This does not mean that I appreciate the technique contribution of GEQ. To me, this is a natural incremental contribution from OptEqs.) \n\nThe reported performance is quite good, considering that GEQ implies a new model structure. \n", "weaknesses": "Using a Gaussian kernel instead of a linear is simple, which can also naturally lead to a better generalization bound. In other words, the discussion on the generalization bound is interesting but simply saying \"tighter\" is somehow trivial.\n\nIn my point of view, there is essential difference of applying GEQ to ImageNet-100 or the whole ImageNet. \n\nI think that a new point of view of neural networks training itself is already interesting. It is not necessary to defeat standard neural networks. If this is the aim of this paper, the experiments should be enhanced. For example, ImageNet should be considered and \nthe setting (SGD with a step learning rate schedule) is fair but may be not sufficient. The following link gives the links of reported best training strategy: https://paperswithcode.com/sota/ \nCIFAR-10 with ResNet-18 is 95.55; \nCIFAR-100 with ResNet-50-SAM is 85.2\n\nIn one word, I think the current experiments for evaluating GEQ are already good (when ImageNet is included). But it is far from giving the conclusion that GEQ is better than ResNet, etc. \n\n\nMy overall recommendation is currently positive, which is mainly based on numerical experiments. There are still many doubts about the performance. If the answer is not strong, I may lower my score. Also, I feel the theoretical and technique contributions of this paper is weak and hence will not fight for my current score, when other reviewers think this part should be improved. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments. The followings are our responses to your concerns.\n\n# 1. About the contribution of GEQ.\n\nWhile OptEqs has already shown its ability in describing certain DNNs like ResNet, there are many limits on it: \n\n1. Firstly, vanilla OptEqs can only model the behaviors of neural networks as they delve deeply into their architecture. However, its applicability is limited when attempting to analyze neural networks with infinite depth and width simultaneously.\n2. Secondly, vanilla OptEqs is adept at describing fundamental neural networks characterized by linear layers (such as convolutions) with one pointwise non-linear activation module. Yet, it falls short when attempting to analyze network architectures with non-linear attentive modules.\n3. Thirdly, the numerical performances for OptEqs are not satisfying.\n\nTo solve the above problems, we decide to use involve Gaussian kernels in vanilla equilibrium models. The reason is mainly as below:\n\n1. Using the Gaussian kernels can give us more insight into equilibrium models with infinite width since Gaussian kernels are usually related to infinite-dimensional space. Thereby, GEQ can not only be used to analyze deep neural models but also can give some insight into the training dynamics when the model’s depth and width increase both.\n2. To extend equilibrium models for describing networks with some non-linear attentive modules, we decide to involve different non-linear kernels like polynomial, sigmoid, and Gaussian. And the empirical results show that the Gaussian kernel is the best. Thereby, we take a further step in exploring our GEQ’s theoretical and empirical advantages in this paper. As shown in Section 3.5, our GEQ can be viewed as an equilibrium model with attentive modules. Therefore, we may use it to analyze the training dynamics for neural networks with attention.\n3. Our approach offers the potential to enhance both the generalization capability and stability of optimization-induced neural architectures like equilibrium models and provide valuable inspiration for researchers to devise more potent kernels for current models. Prior to our research, no exploration of new equilibrium models starts from our distinctive perspective.\n\n# 2. About numerical experiments.\n\nFirstly, we want to point out that the best result in https://paperswithcode.com/sota/ you report uses the auxiliary data. They won’t get such good performances if they train CIFAR models without auxiliary datasets.\n\nSecondly, we also finish the experiments for ResNet-50 with SAM on CIFAR-10 and CIFAR-100. \n\n## CIFAR-10 with SAM:\n\n| Model | Model Size | Test Acc |  |\n| --- | --- | --- | --- |\n| ResNet-50 | $23$M | $95.5\\pm0.4\\%$ |  |\n| GEQ | $8$M | $\\mathbf{95.9\\pm0.3}\\%$ |  |\n\n## CIFAR-100 with SAM:\n\n| Model | Model Size | Test Acc |\n| --- | --- | --- |\n| ResNet-50 | $23$M | $78.4\\pm0.3\\%$ |\n| GEQ | $8$M | $\\mathbf{78.9\\pm0.2}\\%$ |\n\nOur GEQ can outperform deep ResNets with less than half size even if changing optimizers to SAM.\n\nFurthermore, we also finish the experiments for our GEQ on ImageNet with SGD shown below:\n\n## ImageNet:\n\n| Model | Model Size | Test Acc |\n| --- | --- | --- |\n| ResNet-18 | $13$M | $70.2\\%$ |\n| ResNet-50 | $26$M | $75.1\\%$ |\n| GEQ | $16$M | $\\mathbf{75.9}\\%$ |\n\nFrom the results, one can see that our GEQ can outperform deep ResNets with fewer parameters on larger ImageNet datasets.\n\n## 3. About your concerns on “why kernel is chosen\"？\n\nWe use kernel as we deem that the poor performance for original equilibrium models is caused by the simple term $\\langle \\mathbf{Ux+b}, \\mathbf{z} \\rangle$ in OptEqs original optimization problem. Then inspired by traditional machine learning methods, we naturally choose to use kernel methods like Gaussian kernels. Apart from Gaussian, we also try other common kernels as shown in the Appendix A.1. We also list the formulation and results below.\n\nThe formulation of equilibrium models with commonly used non-linear kernels:\n\n| Kernel | Hidden Optimization Problem | Equilibrium Model |\n| --------- | ---------- | -------------- |\n| Polynomial |  $\\min_{\\mathbf{z}}\\left[\\mathbf{1}^\\top f(\\mathbf{z})+ \\frac{1}{2}\\|\\mathbf{z}\\|_2^2-\\left( \\left\\langle \\mathbf{U}\\mathbf{x}+\\mathbf{b},\\mathbf{z} \\right\\rangle\\right)^d-\\frac{1}{2}\\|\\mathbf{W}\\mathbf{z}\\|_2^2 \\right]$  | $\\mathbf{z}^* =  \\sigma\\left(\\mathbf{W}^\\top\\mathbf{W}\\mathbf{z}^*+ d\\left( \\left\\langle \\mathbf{U}\\mathbf{x}+\\mathbf{b},\\mathbf{z} \\right\\rangle\\right)^{d-1}\\left(\\mathbf{U}\\mathbf{x}+\\mathbf{b}\\right)\\right)$  |\n|Sigmoid | $\\min_{\\mathbf{z}}\\left[\\mathbf{1}^\\top f(\\mathbf{z})+ \\frac{1}{2}\\|\\mathbf{z}\\|_2^2-{\\rm{tanh}}\\left( \\left\\langle \\mathbf{U}\\mathbf{x}+\\mathbf{b},\\mathbf{z} \\right\\rangle\\right) -\\frac{1}{2}\\|\\mathbf{W}\\mathbf{z}\\|_2^2 \\right]$ | $\\mathbf{z}^* =  \\sigma\\left(\\mathbf{W}^\\top\\mathbf{W}\\mathbf{z}^*+ \\left(1 - {\\rm{tanh}}^2\\left( \\left\\langle \\mathbf{U}\\mathbf{x}+\\mathbf{b},\\mathbf{z} \\right\\rangle\\right)\\right)\\left(\\mathbf{U}\\mathbf{x}+\\mathbf{b}\\right)\\right)$ |\n|Gaussian | $\\min_{\\mathbf{z}} \\left[\\mathbf{1}^\\top f(\\mathbf{z}) + \\frac{1}{2}\\|\\mathbf{z}\\|_2^2 - \\frac{1}{2\\gamma}e^{-\\gamma\\|\\mathbf{U}\\mathbf{x}+\\mathbf{b}-\\mathbf{Wz}\\|^2_2}\\right]$ | $\\mathbf{z}^* = \\sigma\\left[e^{-\\gamma\\|\\mathbf{U}\\mathbf{x}+\\mathbf{b}-\\mathbf{Wz}^*\\|^2_2}   \\mathbf{W}^\\top(-\\mathbf{Wz}^* +\\mathbf{U}\\mathbf{x}+\\mathbf{b})\\right]$ |\n\nTheir experiments are shown below, which demonstrate that GEQ is the best.\n\n| Model | Model Size | Accuracy |\n| --- | --- | --- |\n| MOptEqs | $8$M | $75.6\\pm0.2\\%$ |\n| MOptEqs (Polynomial) | $8$M | $75.1\\pm0.4\\%$ |\n| MOptEqs (Sigmoid) | $8$M | $76.1\\pm0.3\\%$ |\n| GEQ | $8$M | $\\mathbf{78.2\\pm0.2\\%}$ |\n\nAbout MLP, if we use an MLP layer in the original $\\langle \\mathbf{Ux+b}, \\mathbf{z} \\rangle$ term to make it become $\\langle \\mathbf{Ux+b}, \\mathbf{W}_{m}\\mathbf{z} \\rangle$.  Since it equals to $\\langle \\mathbf{W}_m^\\top (\\mathbf{Ux+b}), \\mathbf{z} \\rangle$, it will perform almost the same as vanilla OptEqs.\n"}, {"review_id": "6xKFhXviKn", "reviewer": "Reviewer_p1Jk", "summary": "The presented work proposed a new equilibrium model by replacing the linear term in the original formulation into a Gaussian kernel. Analysis and experiments are performed to illustrate the superiority of the proposed model in terms of expressivity, generalization and stability, etc.", "questions": "- I wonder if we can say the expressive power of GEQ is strictly stronger than the original form of OptEqs (in Eq.(3) and Eq.(4)) as the authors proved GEQ can be viewed as OptEqs with some terms projected to infinite-dimensional spaces?\n- Based on my understanding, the Patch Splitting appears to be essential for the proposed model. Since if there is not patch splitting then Eq.(6). would essentially be adding a scalar coefficient to the original update equation, which may not result in significant changes. Am I right? \n\n\n\n", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The stability and infinite-width equavalence analysis appears to be novel. I didn't see similar analysis for other equilibrium models.\n- The exploration of matching the performace of equilibrium models and traditional deep learning models is a meaningful direction. The experiments verified the effectiveness of the proposed model.\n\n-----\nI have decided to raise my rating to this paper to a clear accept because all of my concerns are well address by rebuttal.", "weaknesses": "- In my understanding, the modification made by the presented work is seentially replacing the linear term in OptEqs with an exponential term, which will result in an attention-like module in the forward layer. However, the idea of incorporating an exponential term into the optimization objective and unfolding it into an attention mechanism may not be considered entirely novel. For example [1] also used an exponential term and derived an attention layer from the exponential term, despite their model is not an equilibrium model and they didn't explain it as a Gaussian kernel.\n- According to the Section A.2, the forward process is calculated directly by Eq.(6). However, the exponential term in Eq.(6) looks dangerous, as it is widely known that exponential function often leads to numerical instability in computations. I would expect there to be a normalization term (which leads to a softmax-like term) in Eq.(6). \n- Some expressions are vague, for example it's not clear what does \"feature extraction term\" mean at line 48.\n\nA minor issue: In Proposition 1 there's a term $\\sqrt{{\\boldsymbol{2}} \\gamma} {\\boldsymbol{\\Phi}}_{\\boldsymbol{{W}}}^{({\\boldsymbol{1}})}$, what do the bolded ${\\boldsymbol{2}}, {\\boldsymbol{\\Phi}}$ and superscript $({\\boldsymbol{1}})$ mean? I guess they shouldn't be bolded and it is a typo?\n\n[1] Transformers from an Optimization Perspective", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments. The followings are our responses to your concerns.\n\n# 1. About the exponential term and the difference between our work and [1]\n\nIn fact, the methodology and the motivation of [1] and our work are different:\n\n1. [1]'s optimization problem is for one transformer layer while ours is for the whole equilibrium model.\n2. [1]'s exponential term is applied for self-attention which only calculates the exponential of inputs' self inner product. However, ours is calculated for the difference between the input and our equilibrium model's output. Therefore, our GEQ is more similar to traditional attention modules like in SE-Net[2] and Graph Attention Networks[3] instead of self-attention models.\n3. [1] propose the exponential term because they want to explain softmax in Transformer networks while we use Gaussian kernels to analyze equilibrium models with infinite width.\n\nTherefore, the formulations for our work and [1] are similar. They are two different works.\n\n# 2. About the stability of our GEQ in Equation (6).\n\nIn our work, the exponential term in our GEQ is stable because the weights are constrained by weight normalization to ensure the convergence of Eqn(6):\n\n$\\mathbf{z}^* = \\sigma\\left[e^{-\\gamma\\|\\mathbf{U}\\mathbf{x}+\\mathbf{b}-\\mathbf{Wz}^*\\|^2_2} \\mathbf{W}^\\top(-\\mathbf{Wz}^* +\\mathbf{U}\\mathbf{x}+\\mathbf{b})\\right]$\n\nTherefore, our GEQ's exponential term will be stable if the input is stable, which we use some normalization layers to ensure.\n\n# 3. About the feature extraction term.\n\nAs we can see from OptEqs’s optimization problem:\n\n$\\min_{\\mathbf{z}} G(\\mathbf{z};\\mathbf{x}) = \\min_{\\mathbf{z}}\\left[\\mathbf{1}^\\top \n  f(\\mathbf{z})+ \\frac{1}{2}\\|\\mathbf{z}\\|_2^2-\\left \\langle \\mathbf{U}\\mathbf{x}+\\mathbf{b},\\mathbf{z} \\right\\rangle\n-\\frac{1}{2}\\|\\mathbf{W}\\mathbf{z}\\|_2^2 \\right].$\n\nThe feature extraction term is $\\langle \\mathbf{Ux+b}, \\mathbf{z} \\rangle$, $\\mathbf{x}$ here is input while $\\mathbf{z}$ here is the output. Because when calculating the optimization problem's optimal condition to induce equilibrium models' architecture, it will produce the $\\mathbf{Ux+b}$ term, which can be viewed as extracting useful features. Therefore, we call it the feature extraction term. We'll rewrite such a part to make it more clear in the future version.\n\n# 4. About \"I wonder if we can say the expressive power of GEQ is strictly stronger than the original form of OptEqs\".\n\nAs our empirical results and theoretical analysis show,  GEQ’s performance is better than vanilla OptEqs with the proper choice of $\\gamma$. However, if $\\gamma$ is too large. For example, if $\\gamma=\\infty$, then the optimization problem for GEQ:\n\n$\\min_{\\mathbf{z}} G(\\mathbf{z};\\mathbf{x})=\\min_{\\mathbf{z}} \\left[\\mathbf{1}^\\top f(\\mathbf{z}) + \\frac{1}{2}\\|\\mathbf{z}\\|_2^2 - \\frac{1}{2\\gamma}e^{-\\gamma\\|\\mathbf{U}\\mathbf{x}+\\mathbf{b}-\\mathbf{Wz}\\|^2_2}\\right]$\n\nwill become\n\n$\\min_{\\mathbf{z}} G(\\mathbf{z};\\mathbf{x})=\\min_{\\mathbf{z}} \\left[\\mathbf{1}^\\top f(\\mathbf{z}) + \\frac{1}{2}\\|\\mathbf{z}\\|_2^2\\right]$\n\nthen the final equilibrium state $\\mathbf{z}$ will be $0$, which is a trivial solution. \n\nThereby, we believe the expressive power of our GEQ is better than OptEqs with the proper choice of $\\gamma$. Furthermore, although the choice of $\\gamma$ is important, it won’t be hard to get a proper $\\gamma$ for better performance. As we can see from the empirical section,  we choose one $\\gamma$  for different datasets based on our analysis.\n\n\n\n# 5. About Patch Splitting and its necessity.\n\nPatch splitting is an important term and it is our GEQ's unique feature. Because even if we do patch splitting in OptEqs, its formulation still will be the same as the vanilla OptEqs:\n\n$\\sum_{i=1}^N \\langle (\\mathbf{Ux+b})_i, \\mathbf{z}_i\\rangle = \\langle (\\mathbf{Ux+b}), \\mathbf{z}\\rangle,$\n\nwhere $\\mathbf{z} = [\\mathbf{z}_1,\\mathbf{z}_2,...,\\mathbf{z}_N]$. With the patch-splitting technique, our GEQ can concentrate on different parts based on their similarity like attentive modules.\n\nHowever, we need to clarify your comments on “if there is no patch splitting then Eq.(6). would essentially be adding a scalar coefficient to the original update equation, which may not result in significant changes.” Even without the patch splitting technique, our GEQ is still not the same as trivially scaling OptEqs because the scaling factor is related to $\\mathbf{x}, \\mathbf{z}$ and it is also learnable.\n\n# 6. About “In proposition 1, there’s a term $\\sqrt{2\\gamma}\\Psi_\\mathbf{W}^{(1)}$, what do the bolded 2, $\\Phi$ and superscript (1) mean? I guess they shouldn’t be bolded and it is a typo?”\n\nThey are typos and we’ll correct it in the future version.\n\n[1] Transformers from an Optimization Perspective\n\n[2] Squeeze-and-Excitation Networks\n\n[3] Graph Attention Networks"}, {"review_id": "ataDrFjX0h", "reviewer": "Reviewer_CdMn", "summary": "\nThe authors introduce a deep equilibrium model (6) which ostensibly solves an optimisation problem (5) involving a Gaussian (squared exponential) kernel. The authors describe some possible theoretical properties of the new model, and benchmark it on some problems against some related optimisation-inspired DEQs and resnets.", "questions": "Minor (*not* a reason to reject):\n- Section title for section3 is missing an \"I\".\n- The titles and labels in Figure 2 are too small to read.\n- Why are only Gaussian kernels considered?\n- Many works have pointed out potential issues with using saliency maps such as GradCAM to try and gain insight into neural network predictions. E.g. see Cynthia Rudin's work on explainability versus interpretability. The saliency map visualisation seems to come out of nowhere (not being mentioned anywhere earlier in the paper). Is there a reason why include it in this paper?", "limitations": "I do not see any explicit mention of limitations in this work. That being said, I may have missed some, if they are somewhere in the text.", "rating": 4, "confidence": 3, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "- The idea of deriving DEQ models from optimisation problems is nice, and fits in well with existing literature.", "weaknesses": "Unfortunately, there appear to be many issues with the mathematical results in this paper. I do not believe they are sound. Some of them could potentially be fixable, but I am not certain.\n\n- **Important** The use of the function $f$ is not clear.\n    - Can you elaborate on the definition of $f$? On the second page, you say \"$f$ is a positive indicator function that induces to commonly used ReLU activation\" What does \"induces to\" mean? What exactly is your definition of indicator function? One definition is that given a set $A$, $f_A(z) = 1$ if $z \\in A$ and zero otherwise. This does not seem to reconcile with your definition, though. Perhaps $f$ takes a value of $1$ if $z$ is greater than 0 (elementwise), and zero otherwise? What happens at the value of zero? What is $f$'s (weak) derivative? How is it related to the ReLU?\n    - After equation (3), you mention the first order stationarity condition. Is $f$ here the indicator function? Isn't this function not everywhere differentiable? Even if the derivative were defined, how do you know that $\\nabla G = 0$ implies a minima? \n    - How is a ReLU activation in (4) obtained by differentiation? I could see that the derivative of $relu(a)^2$ with respect to $a$ would be $relu(a)$, but not sure how to obtain it otherwise?\n    - Proposition 1. What is the meaning of the symbol $\\partial f$? How is $(1+\\partial f)^{-1}$ related to the ReLU function?\n- Proposition 1 is not precise. The inner product is not defined, which is quite important because $\\Phi$ lives in some kind of Hilbert space (informally being referred to as $\\mathbb{R}^\\infty$). Can you clarify what $\\infty$-dimensional space means, and what $\\langle \\cdot, \\cdot \\rangle $ means?\n- Proposition 3. Isn't the left hand side of equation (11) zero? This would be a trivial bound, I would have thought.\n- (Minor) Proof of proposition 3. Aren't all the first lines in equation (41) equality, by definition of fixed point?\n- Proposition 4. Can you elaborate on why it is helpful to exhibit a smaller output similarity for dissimilar samples? One can trivially do this for any kernel, by simply rescaling the kernel by a constant factor.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments. The followings are our responses to your concerns.\n\n# 1. About $f$:\n\n## 1) The definition and differentiability of $f$:\n\nThe definition of $f$ is $f(x) = 1$ when  $x \\geq 0$ while $f(x)=\\infty$ when $x<0$.  And $f$ is sub-differentiable and  $\\partial f$ here means the sub-gradients \"[subgradients](https://web.stanford.edu/class/ee364b/lectures/subgradients_notes.pdf)\". In our paper, we set $\\partial f=0$ at $x=0$  for convenience since $0\\in \\partial f$. We'll rewrite them to make it more clear in the future version.\n\n## 2) About the stationary condition:\n\nAlthough $f$ and $G$ are not differentiable everywhere, the stationary condition can also be used with $G$’s sub-gradient, i.e., $0 \\in \\partial G$.  More details in \"[subgradients](https://web.stanford.edu/class/ee364b/lectures/subgradients_notes.pdf)\". We use $0=\\nabla G$  in our paper for convenience since we set $\\partial f=0$ at $x=0$, and our equilibrium also suits the above condition $0 \\in \\partial G$. Therefore, our conclusion is correct although the formulation is a little non-rigorousness. We'll rewrite them to make it more clear in the future version.\n\n## 3) About the derivative of the ReLU function.\n\nWe take the following optimization problem as an example. When you solve the following optimization problem $argmin_{x\\geq0} \\frac{1}{2}||x-y||^2$, you can first reformulate the optimization problem as  $argmin_{x} \\frac{1}{2}||x-y||^2 +f(x)$ where $f$ is defined as above since two optimization problems have the same optimal point.\n\nThen the optimal solution can be achieved by calculating the problems subgradient $0 =  (1+f)(x) -y$ and then we can get $x=(1+\\partial f)^{-1}(y)$ as the optimal solution. And we can also get the original problem's optimal solution as $x=0$ when $y<0$ because $f(x)=\\infty$ when $x<0$ and $x=y$ when $y\\geq 0$. Therefore, $(1+\\partial f)^{-1}(y) = max(y,0)$ which is ReLU and it also calls the proximal operator of $f$.\n\nMore details in \"[Proximal Operators](https://www.math.cuhk.edu.hk/course_builder/1920/math4230/Note10.pdf)\".\n\n# 2. About inner-production, infinity dimension spaces’ definition in Proposition 1.\n\nThe inner product in our paper is$\\langle \\mathbf{x}，\\mathbf{y} \\rangle = \\mathbf{x}^\\top\\mathbf{y}$, which you can find in \"[Inner Product](https://mathworld.wolfram.com/InnerProduct.html)\". And the $\\infty-$dimensional space means the vector $\\mathbf{v}$ in such space has infinite dimension which means $\\mathbf{v}\\in\\mathbb{R}^{\\infty}$.\n\n# 3. About Proposition 3.\n\nAlthough the left lower bound for Eqn(11) is 0, proposition 3 is non-trivial. A tighter Lipschitz upper bound is important because it can make the model more stable. Due to this reason, there are many works (like [1]) working on it. The upper bound can lead researchers to explore the model's most unstable performance in the worst cases. A smaller upper bound means the model is stable and can perform well on many perturbed or corrupted inputs. Thereby, proposition 3's upper bounds try to state the stability of our GEQ is better, which is also confirmed in our experiments (The second experiment in Section 4.3).\n\nAs for the proof in the Appendix, the inequality in Eqn(41) is a typo and I'll correct it in the future version. But it won't influence the final results.\n\n# 4. About Proposition 4.\n\nSmall output similarity for dissimilar samples will make the classification easier. For example, a dataset is easy to classify when all samples belong to the same class cluster together(similar samples should have similar representation) while the distances between cluster centers are large (dissimilar samples’ representation should be different).\n\nTherefore, simply scaling cannot make the classification performance better because it will scale all sample's similarities even if they are similar, which will make samples hard to classify.\n\nApart from that, simply scaling with a scaler may also make OptEqs' outputs trivial. Because  simply scaling by a hyper-parameter $\\eta$  to make the equilibrium function become $\\mathbf{z}^* =  \\sigma\\left(\\eta\\left(\\mathbf{W}^\\top\\mathbf{W}\\mathbf{z}^*+ \\mathbf{U}\\mathbf{x}+\\mathbf{b}\\right)\\right)$ can be considered as reformulating its hidden optimization problem to the following form:\n\n$\\min_{\\mathbf{z}}\\left[\\mathbf{1}^\\top\nf(\\mathbf{z})+ \\frac{1}{2}\\|\\mathbf{z}\\|_2^2-\\eta \\left \\langle \\mathbf{U}\\mathbf{x}+\\mathbf{b},\\mathbf{z} \\right\\rangle\n-\\eta \\frac{1}{2}\\|\\mathbf{W}\\mathbf{z}\\|_2^2 \\right].$\n\nIf $\\eta$  becomes too small, the final output will be more likely to be just optimizing:\n\n$\\min_{\\mathbf{z}}\\left[\\mathbf{1}^\\top\nf(\\mathbf{z})+ \\frac{1}{2}\\|\\mathbf{z}\\|_2^2\\right]$\n\nwhich means the equilibrium model's final output $\\mathbf{z}$ has no relationship with the original input. Therefore, the model’s outputs are useless.\n\n# 5. About \"why only consider Gaussian Kernels\".\n\nWe not only consider Gaussian kernels. We have tried different non-linear kernels like polynomial, sigmoid, and Gaussian in Appendix A.1. However, the empirical results show that the Gaussian kernel is the best. Apart from that, gaussian kernels can also let us analyze the infinite-wide equilibrium model's properties while others cannot. Thereby, we mainly explore GEQ’s theoretical and empirical advantages. \n\nDue to the space limits, I cannot list the results here. However, you can find them in our Appendix or answers for Reviewer 8r48’s problem 3.\n\n# 6. About GradCAM.\n\nHere we use GradCAM to make readers more clear that GEQs with its induced attentive modules (as illustrated in Section 3.5) will make models focus on more semantic-related regions. We choose GradCAM here as it is an easy and widely used way to show. GradCAM is only an auxiliary technique since we have already shown our advantages with theoretical analysis and empirical comparisons.\n\n# 7. About some typos and small captions.\n\nWe'll correct them in the future version.\n\n[1] Estimating Lipschitz constants of monotone deep equilibrium models"}, {"review_id": "lNSP62812m", "reviewer": "Reviewer_sG64", "summary": "This paper makes the observation that conventional DEQ architectures effectively use linear kernels in their feature extraction, and the proposed method in this paper is to use Gaussian Kernels instead of linear kernels. Then, the paper makes the observation that adding Gaussian Kernels to DEQ models produce an architecture that is effectively infinite deep and infinitely wide. The paper then proposes patch splitting, a technique that makes the proposed method enjoy better performance, and shows the effectiveness of the proposed method on CIFAR10, CIFAR100, and saliency map datasets.", "questions": "1. Similar to the weakness section---I am curious whether the proposed model can act as an effective model for large neural nets in general, the same way that the NTK regime approximates large width networks? Can this model be representative of large language models, given its infinite depth and infinite width capacity?\n2. Does the Gaussian Kernel in the proposed model have any natural use cases in practice (that linear kernels, polynomial kernels, etc don't have)?", "limitations": "There are no negative societal impacts of this work", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is clearly written. In terms of contribution, this paper makes the observation that the proposed method is effectively infinitely deep and infinitely wide, which I find to be an interesting thread. Finally, this paper contains convergence bounds and stability analysis, training technique (patch splitting), as well as numerous satisfactory experiment results, and therefore I believe that the proposed method is sufficiently interesting.", "weaknesses": "I believe this paper would benefit more from discussion about the proposed model. For example, now that we know the proposed model is infinitely deep and infinitely wide, what can we say about its connection to NTK? Can this model further some of the NTK analysis? In addition, can this model inspire potential work on Gaussian processes?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments. The followings are our responses to your concerns.\n\n# 1. About the relationship between GEQ and NTK.\n\nBoth GEQ and NTK share a fundamental likeness: they align with the concept of infinite-width models. And we also reference some theoretical results in NTK and Gaussian process models for our analysis. However, our GEQ doesn’t rely on some unrealistic weight distribution. Therefore, the training process for GEQ is much easier, which leads to its superior performance over NTK. Consequently, GEQ's capabilities may have the potential to instigate innovative approaches to NTK model development, making such models more useful and realistic.\n\nAnticipating the future trajectory of GEQ and NTK analyses, we posit that the kernel mechanism intrinsic to GEQ could serve as a wellspring for novel architectural designs within the NTK framework. An intriguing way to explore involves integrating the strengths of both methodologies. This entails harnessing the intra-sample kernel calculation akin to GEQ, which we think is one of the key reasons that GEQ’s performance is better than NTK. Apart from that, the new model should also leverage NTK's global kernel function computed over the entire training dataset. This hybrid approach has the promise of delivering a robust performance akin to GEQ and simultaneously facilitating the acquisition of dataset-specific properties, such as out-of-distribution (OOD) generalization and fairness considerations.\n\nShifting the focus to the advancement of Gaussian process models, GEQ imparts a valuable insight—equilibrium models could constitute a compelling route for expanding Gaussian process models from their non-parametric state to a parameterized one. By embracing this equilibrium framework, these models stand to achieve heightened performance levels and open avenues for significant enhancements.\n\n# 2. About the relationship between GEQ and large neural networks.\n\nRegarding the connection between our GEQ and large neural networks, it is worth noting that our GEQ possesses attributes akin to those of a weight-tied, large deep neural network (as illustrated in [1]) accompanied by attention modules. This resemblance stems from the utilization of exponential terms As such, our GEQ holds promise in providing theoretical insights into the realm of deep and expansive neural networks enriched with attention modules.\n\nTurning to the potential interplay between our GEQ and current large models, such as Large Language Models or Multi-Modal Models, we think our GEQ shows a closer connection with multi-modal models instead of LLM because our GEQ-induced attentive module can be viewed as cross-attention rather than self-attention mechanisms.\n\nFurthermore, our GEQ model itself can scale to large and show better performance than large deep ResNets. For example, on ImageNet:\n\n| Model | Model Size | Test Acc |\n| --- | --- | --- |\n| ResNet-18 | $13$M | $70.2\\%$ |\n| ResNet-50 | $26$M | $75.1\\%$ |\n| GEQ | $16$M | $\\mathbf{75.9}\\%$ |\n\nThe results underscore a notable observation: when our GEQ is scaled up to larger models, it demonstrates superior performance compared to deep ResNets, even while employing fewer parameters, particularly evident when evaluated on larger ImageNet datasets.\n\n# 3. About \"Does the Gaussian Kernel in the proposed model have any natural use cases in practice (that linear kernels, polynomial kernels, etc don't have)?\"\n\nEquilibrium models endowed with linear kernels can be conceptualized as akin to weight-tied ResNets or analogous models, as exemplified in [1]. Consequently, our GEQ, employing a Gaussian kernel, can be likened to weight-tied ResNets, augmented by attention modules like SE-Net [2], Graph Attention Network[3] or etc.\n\n[1] Deep equilibrium Models\n\n[2] Squeeze-and-Excitation Networks\n\n[3] Graph Attention Networks"}]

Author rebuttal:

Meta-review: This paper proposed a new equilibrium model using kernels. The theoretical motivation behind the proposal is quite clear, as the authors demonstrate tighter generalization bounds and smoother layers. The experiments then demonstrate clearly the merits of the approach on several tasks.

The reviewers were generally positive about this work, and the authors adequately answered their concerns. Reviewer CdMn still scores 4, but this is mostly based on minor mathematical misunderstandings that the authors have clarified and not on some concern regarding the method/soundness/evaluation.

 In the final version of the paper, the authors must be clearer about their claims in 3.3 - 3.4: these upper bounds are not tight (this should be explicit!); hence it is a fallacy to claim that "Therefore, our GEQ can show better classification performance on the test set" or "Therefore, we can conclude that our GEQ’s outputs are more stable under perturbations.", this needs to be mitigated as the authors did in discussion with the AC.
Decision: Accept (poster)