Title: GEQ: Gaussian Kernel Inspired Equilibrium Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: equilibirum models

neural networks
Abstract: Despite the connection established by optimization-induced deep equilibrium models (OptEqs) between their output and the underlying hidden optimization problems, the performance of it along with its related works is still not good enough especially when compared to deep networks. One key factor responsible for this performance limitation is the use of linear kernels to extract features in these models. To address this issue, we propose a novel approach by replacing its linear kernel with a new function that can readily capture nonlinear feature dependencies in the input data. Drawing inspiration from classical machine learning algorithms, we introduce Gaussian kernels as the alternative function and then propose our new equilibrium model, which we refer to as GEQ. By leveraging Gaussian kernels, GEQ can effectively extract the nonlinear information embedded within the input features, surpassing the performance of the original OptEqs. Moreover, GEQ can be perceived as a weight-tied neural network with infinite width and depth. GEQ also enjoys better theoretical properties and improved overall performance. Additionally, our GEQ exhibits enhanced stability when confronted with various samples. We further substantiate the effectiveness and stability of GEQ through a series of comprehensive experiments.

GEQ: Gaussian Kernel Inspired Equilibrium Models

Abstract

Despite the connection established by optimization-induced deep equilibrium models (OptEqs) between their output and the underlying hidden optimization problems, the performance of it along with its related works is still not good enough especially when compared to deep networks. One key factor responsible for this performance limitation is the use of linear kernels to extract features in these models. To address this issue, we propose a novel approach by replacing its linear kernel with a new function that can readily capture nonlinear feature dependencies in the input data. Drawing inspiration from classical machine learning algorithms, we introduce Gaussian kernels as the alternative function and then propose our new equilibrium model, which we refer to as GEQ. By leveraging Gaussian kernels, GEQ can effectively extract the nonlinear information embedded within the input features, surpassing the performance of the original OptEqs. Moreover, GEQ can be perceived as a weight-tied neural network with infinite width and depth. GEQ also enjoys better theoretical properties and improved overall performance. Additionally, our GEQ exhibits enhanced stability when confronted with various samples. We further substantiate the effectiveness and stability of GEQ through a series of comprehensive experiments.

1 Introduction

Deep Neural Networks (DNNs) show impressive performance in many real-world tasks on various data like graphs [43], images [41, 20], sequences [8], and others. However, most neural network structures are constructed by experience or searching on the surrogate datasets [31, 57]. Therefore, these architectures cannot be interpretable and such a phenomenon hinders further development. Apart from the current neural network models, traditional machine learning methods like dictionary learning [46, 33], subspace clustering [54] and other methods [29, 56, 55, 30] can design their whole procedure by designing optimization problems with specific regularizers customized from their mathematical modeling and requirements. Thus, these models are easily interpreted. However, the traditional machine learning algorithms' whole procedures do not consider the hidden properties of features and labels. Therefore, they usually perform worse on tasks with more data.

To link two types of models, OptEqs [52] tries to recover the model's hidden optimization problem to make their model 'mathematically explainable'. They claim that the output features ˜ z ∗ (we also called the equilibrium state) with respect to input x , which are obtained by solving the fixed-point

equation in Eqn (1), is the optimal solution for its hidden optimization problem defined in Eqn (2).

where σ is the ReLU activation function, U , ˜ W , b , ˜ W c , b c are learnable parameters. They are trained by optimizing loss functions, such as cross-entropy loss, which are calculated based on the final prediction ˆ y derived from ˜ z ∗ as shown in Eqn (1). ˜ W -1 represents an invertible or pseudo-invertible matrix for ˜ W , and the function f is a positive indicator function, which outputs 1 when x ≥ 0 and outputs ∞ in other cases. By choosing this function, the first-order condition for Eqn 2 will contain a non-linear ReLU function, which is the activation function in our GEQ. The equivalence between z ∗ and the optimal solution for the hidden optimization problem (2) enables researchers to not only gain insights into OptEqs' behavior by understanding the underlying hidden optimization problem but also innovate by designing new models based on different problem formulations tailored to specific tasks. For instance, OptEqs introduces a module that promotes output sparsity, and Multi-branch OptEqs (MOptEqs) incorporates fusion modules to enhance the diversity among its branches. Despite these advancements and the incorporation of various modules inspired by different vision tasks, the performance of OptEqs and related models still falls short when compared to deep neural networks in image classification. This discrepancy suggests the existence of crucial components that limit the performance of equilibrium models.

To identify such a component, we delve into the hidden optimization problem of OptEqs and observe that it can be decomposed into two distinct parts: the regularizer term for the output features and the feature extraction term. While the feature extraction term is crucial as it depends on the input and determines the patterns extracted from the input features, the exploration of the regularizer term has been largely overlooked, with linear kernel functions being the predominant choice for feature extraction. Thereby, we believe that the limitations of previous equilibrium models stem from their feature extraction parts, as linear kernels struggle to capture complex features effectively. Building upon these insights, we take a step forward by leveraging the widely adopted Gaussian kernel for feature extraction in the hidden optimization problem.

Then by calculating the stationary condition for the above new hidden optimization problem, we propose our new type of OptEqs, the Gaussian kernels inspired equilibrium models (GEQ). The model involves a new attentive module induced by its hidden optimization problem and enjoys much better performances on classification tasks even compared with deep models. Furthermore, we also prove that the new model's outputs are equivalent to the outputs for OptEqs with weight-tied 'infinite wide' mappings. Therefore, an interesting finding is that our model can be regarded as a 'double-infinite' model because the original OptEqs can be regarded as a weight-tied 'infinite deep' model. Apart from the above findings, the utilization of Gaussian kernels also makes our proposed model enjoy better generalization abilities. Besides the generalization abilities, we also analyze the stability of our GEQ and find its stability is better on various inputs. We summarize our contributions as follows:

We first reformulate the OptEqs' hidden optimization problem with Gaussian kernels and propose a new equilibrium model called GEQ. It contains a new attention module induced by its hidden optimization problem and performs better on real-world datasets.

We find that our GEQ can be regarded as a weight-tied neural network with both infinite width and depth, and better generalization ability through our analysis. Empirical results also confirm the superiority of our GEQ.

We theoretically demonstrate the advantages of the stability of our GEQ compared with former OptEqs on various inputs. We also conduct experiments to validate such advantages.

2 Related Works

2.1 Implicit Models

Most modern deep learning approaches provide explicit computation graphs for forward propagations and we call these models ' explicit models'. Contrary to these models, recent researchers proposed some neural architecture with dynamic computation graphs and we call them ' implicit models'. A

notable example of an implicit model is Neural ODEs [7], its architecture is encoded as a differential system and the implicit ODE solvers they used are equivalent to continuous ResNets that take infinitesimal steps. By representing the entire structure using differential systems, implicit models tap into the black box of traditional neural networks while offering increased flexibility and interpretability. Because of the flexibility and the interpretability of implicit models, the design of implicit models [15, 17] draws much attention these days. Many kinds of implicit models have been proposed, including optimization layers [11, 1], differentiable physics engines [40, 9], logical structure learning [50], differential programming [53, 43], and others [25, 49].

Among the various implicit models, OptEqs [52] and its multi-branch version MOptEqs [28] stand out as they not only exhibit superior performance compared to other implicit models but also explicitly establish the relationship between their structure and a well-defined optimization problem. Therefore, exploring better equilibrium models is a promising direction to achieve more interpretable neural architectures. However, it is worth noting that while OptEqs and its variants have shown promising results, their performance is still not entirely satisfactory, particularly when compared to the deep explicit models. Other works [51, 42] also show the connection between their architectures and optimization problems, but their performance is also not satisfying. Besides the above general models, there are many works design equilibrium models from the view of optimization to deal with their specific tasks, like implicit graph models [6, 26], certified robust models [27] and image denoising models [5]. However, these models can only work on their specific domains.

2.2 Infinite Wide Models and Kernel Methods in Deep Learning

By employing kernel methods to estimate the outputs of single-layer networks for various samples, researchers discover that such networks can exhibit characteristics of a Gaussian process (GP) when their parameters are randomly initialized with a large width limit [35]. Building upon this idea, recent researchers have extended these findings to neural networks with multiple layers [24, 10] and other architectures [37, 13]. These studies primarily focus on weakly-trained models, where the network parameters are randomly initialized and kept fixed throughout the training process except for the last classification layer [2]. Despite their "weakly-trained" nature, these models still provide valuable insights applicable to current neural networks. For instance, mean-field theory [4, 16, 19] explains phenomena such as gradient vanishing and exploding during back-propagation, which are relevant not only to single-layer networks but also to other structures like convolutional neural networks (CNNs) and recurrent neural networks (RNNs). Other researchers explore stationary kernels to enhance the interpretability of neural networks by designing different activation functions [34].

In addition to weakly trained models, recent studies [22, 2] introduce the concept of Neural Tangent Kernel (NTK) and its variants. These works have demonstrated that the sample kernel of infinitely wide networks, with appropriate initialization, can converge to a fixed neural tangent kernel when trained using gradient descent with infinitesimal steps (gradient flow). The NTK model is a theoretical construct with strict constraints, and its weights are not learned. It is important to note that although our model can also be seen as an infinitely wide model, there are several key differences between our approach and the aforementioned models. Firstly, our model utilizes kernel methods to operate on input features and output features, while the NTK models employ the kernel method on samples. Secondly, our GEQ model can be viewed as employing a "weight-tied infinite wide" projection that is parameterized by learnable parameters, allowing for updates during the training process. This contrasts with NTKs and NTK-DEQ [12](an equilibrium model constructed with vanilla NTK layers), where the weights are fixed and not learned. Therefore, despite the potential overlap in terminologies used in our paper and NTK-related works, our GEQ model differs significantly.

3 Gaussian kernel Inspired Equilibrium models

3.1 Formulation and Structure of GEQ

Before starting our analysis, we need to reformulate the original formulations of OptEqs' equilibrium equation (1) and hidden optimization problem (2) for convenience. We replace ˜ W c ˜ W ⊤ with W c , W := ˜ W ⊤ , and replace z with ˜ W -1 ⊤ ˜ z . Then the original OptEqs' optimization problem can be

reformulated as:

With the new formulation, we can rewrite the equilibrium equation for OptEqs with input x by calculating Eqn (3)'s first order stationary condition ∇ G = 0 and then reformulate it as follows:

where σ is the ReLU activation function, U , W , b are learnable parameters trained by optimizing loss functions (like cross-entropy loss). From problem Eqn (3), one can see that the GEQ's outputs try to extract features by minimizing the similarity term with the input feature Ux + b through a linear kernel function with some constraints defined in its regulation terms to prevent the trivial outputs. Such an explanation can also extend to other DEQs [3, 51] under the symmetric weight constraints. However, as other traditional machine learning mechanisms show, linear kernel functions cannot perform well when processing complex inputs. We deem that this term will also restrict the performance in equilibrium models. We note that the symmetric constraints won't influence the final performance much as many works [32, 21] show.

A natural consideration arises as to whether we can utilize alternative kernel functions to extract input features for the equilibrium state. However, we find that other equilibrium models employing different kernels with inner products, like the polynomial kernel and sigmoid kernel, lead to a similar structure to OptEqs with appropriate weight re-parameterization and lead to similar empirical results. We provide a detailed discussion of the related models in Appendix 4.6. Thereby, we decide to use the Gaussian kernels, and our new hidden optimization equation is formulated as follows:

where γ is the hyperparameter denoting the reciprocal of Gaussian kernels' variance for scaling. Calculating ∇ G = 0 for new G , we can get the Gaussian kernel inspired Equilibrium models (GEQ) as the following fixed-point equation:

Compared with linear kernels, Gaussian kernels can easily extract the non-linear relations from the input features and show more stable and powerful performance in SVM and other machine learning methods [21, 44]. We also find that the formulation of our GEQ is similar to adding a new attention module to the original equilibrium models. Therefore, our GEQ is supposed to enjoy more representative abilities than the original OptEqs. In the following parts of this section, we will analyze the theoretical advantages of our GEQ against the vanilla OptEqs. And we also empirically evaluate GEQ's performance in the following sections.

3.2 GEQ equals to the OptEqs with infinite width

Like other Gaussian-related models, our GEQ model can also be regarded as computing similarities by mapping them to an infinite-dimensional space. This allows GEQ to extract input features at the infinite-dimensional level, enabling the capture of non-linear dependencies in the input space. Essentially, our GEQ can be seen as a specialized version of OptEqs operating within the infinitedimensional space after mapping input features x and output embedding z to this expanded domain.

Proposition 1. The output of our GEQ (Eqn (6)) is the same as a special OptEqs' output whose hidden optimization problem is defined as follows:

where f is the positive indicator function and (1 + ∂f ) -1 is the ReLU activation function, λ = e -γ ∥ Ux + b ∥ 2 2 e -γ ∥ Wz ∥ 2 , and Φ W ( z ) = [ 1 , √ 2 γ Φ ( 1 ) W ( z ) , ..., √ (2 γ ) i /i !Φ ( i ) W ( z ) , ... ] ∈ R 1 ×∞ which maps the inputs to the infinite-dimensional space with Φ ( i ) W : R n → R in i defined as follows:

where ( Wx ) j denotes the j -th element of vector Wx .

Based on the analysis provided above, it becomes evident that the hidden optimization problem of our GEQ exhibits a similar formulation to a specific OptEqs, whose inputs x and outputs z are mapped to an infinite-dimensional space using the weight-tied infinite wide mapping Φ W and Φ U . Given that both GEQ and OptEqs are derived from their respective hidden optimization problems, the equivalence in these problems implies the existence of the same equilibrium states for both models. Consequently, our GEQ can be considered an extension of the "infinite-depth" OptEqs to the "infinite-width" domain. Since wider neural networks are generally expected to perform better on classification tasks, we can infer that our GEQ outperforms vanilla equilibrium models like OptEqs. We further support this claim with the theoretical analysis illustrated in the subsequent sections.

3.3 Generalization abilities for our GEQ

Apart from the above empirical intuition, we are going to prove our GEQ's generalization advantages over OptEqs using the generalization bound under the PAC-Bayesian framework [36]. For convenience, we use f geq ( x ) denotes the equilibrium state z ∗ for input x . Then we use the expected margin loss L η ( f c geq ) at margin η of our GEQ on the data distribution D for classification, which is defined as follows,

̸

where f c geq ( x ) = W c f geq ( x ) + b c stands for GEQ's final prediction at input x with learnable parameters W c and b c , and the index j, y here denote the prediction score for certain class. Then we can analyze the generalization bound for our GEQ following the former work's settings [38].

Proposition 2. If input ∥ x ∥ 2 is bounded by B , µ := max {∥ U ∥ 2 , ∥ W ∥ 2 , ∥ W c ∥ 2 , ∥ b ∥ 2 } < 1 , then we have following results for GEQ and OptEqs with ReLU activation. For ∀ δ, η > 0 , with probability at least 1 -δ over the training set of size M, we have:

where ˆ L η ( f c geq ) denotes the empirical margin loss on the training set, the maximum scaling number is defined by β max := max x ∈D e -γ ∥ Ux + b -Wz ∥ 2 2 , B W := ∥ W ⊤ W ∥ 2 F + ∥ U ∥ 2 F + ∥ b ∥ 2 2 + ∥ W c ∥ 2 F + ∥ b c ∥ 2 2 , and m = ∥ W ⊤ W ∥ 2 is less than 1 to ensure the convergence of equilibrium models.

Remark 1. If β max < 0 . 8 and µ, m > 0 . 9 , we can get β max µ 1 -β max m < 1 1 -m and 2 µβ max +1 1 -β max m ≤ 1 1 -m . In the meanwhile, our GEQ's generalization bound is tighter than the original OptEq.

In practical experiments, we find that the above conditions for β max and µ, m are satisfied in most cases. And the following experiments also support our above theoretical advantages. However, the above bound is not tight, and how to approximate a much tighter bound for equilibrium models still needs exploring.

3.4 GEQ enjoys More Stable Performance

Apart from better performance, Gaussian kernel stands out as one of the most extensively employed kernels in machine learning tasks owing to its stability across various input scenarios. Motivated by this, we aim to investigate whether incorporating Gaussian kernels into our equilibrium models can enhance the model's stability across diverse inputs. Firstly, we are going to estimate output changes with respect to the input perturbations.

Proposition 3. If norms for the inputs and outputs are bounded by B , the spectral norm for the weight parameter W , U of equilibrium models with ReLU activation are bounded by µ < 1 to ensure

convergence, then we have the conclusions as below:

where x 1 and x 2 are input samples, f geq ( x · ) and f opteq ( x · ) denotes the equilibrium states for GEQ and OptEqs given input x · , and β max := max x ∈D e -γ ∥ Ux -Wz ∥ 2 2 < 1 .

In practical experiments, we choose different γ to reach the above condition for β max and the condition for input B can also be achieved by normalization layers. Although the above bound is not tight, we can use it as a rough explanation for our GEQ's stability, which is demonstrated in the following experiments. How to approximate a much tighter bound for equilibrium models still needs exploring.

Besides having stable outputs under perturbations, a stable model should also show large output differences for different classes to make classification easier. However, the above Lipschitz term can not constrain outputs' similarity when samples are far apart, then we need a new metric for analysis. In line with previous works [18, 34, 10], we assume all weight parameters go to infinite dimensions and analyze the expected output similarity κ for a model f for inputs x 1 and x 2 defined below:

with p ( u ) is the distribution of weight U 's vectorization. If κ is smaller for samples x 1 and x 2 when they belong to different classes, which means they are far away, then the classifier can easily classify these two samples with different labels. The margin for the classification will also be large and easy for the classification of difficult samples. The κ 's upper bound for GEQ and OptEqs are listed below:

Proposition 4. If norms for the inputs and outputs are bounded by B , the spectral norm for the weight parameter W of equilibrium models with ReLU activation are bounded by µ < 1 to ensure the convergence, and each row in U obeys the spherical Gaussian distributions N (0 , E [ U 2 i ] I ) . Then we have the following conclusions for the expectation of the output similarity for GEQ and OptEqs with respect to input x 1 , x 2 as follows,

where x 1 and x 2 are input samples, D = e γB ∥ W ∥ 2 2 , β max := max x ∈D e -γ ∥ Ux -Wz ∥ 2 2 , σ min ( U ) is U 's minimal singular term, and θ 0 = cos -1 ( ⟨ x 1 , x 2 ⟩ ∥ x 1 ∥∥ x 2 ∥ ) is the angle between the samples.

Based on the aforementioned analysis, it is evident that our GEQ exhibits a smaller output similarity for dissimilar samples. As a result, the predictions made by GEQ are primarily based on the most similar samples, enabling it to successfully classify challenging instances. This claim is further supported by the results obtained from our carefully designed experiments.

3.5 Patch Splitting in GEQ

Since different parts of images have different impacts on the image classification, calculating the whole similarity using Gaussian kernels for GEQ is not enough. Inspired by former works [47], we also split the feature map Ux into patches, and then our optimization problem becomes:

+

()

-

()

Sum

+-()

C

W

H

-

e

Channel

Multiplication

()

()

-

-

where ˜ x i ∈ R c s p 2 is the i -th patch of Ux + b while ˜ z i ∈ R c s p 2 is the i -th patch of Wz and W h ∈ R c s p 2 × c hid is a linear layer to project patches with different size to the constant dimension. c s denotes the channel splitting number, p denotes the patch size, and c hid denotes the hidden dimension of patches after projection. We note that the patch-splitting approach is a GEQ's unique feature, as incorporating this technique makes no difference in OptEqs due to its linear kernel. Figure 1 provides a sketch for GEQ's i -th fixed-point iteration. From the figure, it is evident that our GEQ can be viewed as a special OptEqs with additional attention mechanisms to capture the most important regions. Thereby, it can achieve enhanced performance. For a more detailed understanding of the forward procedure in our GEQ, please refer to Appendix A.1.

4 Empirical Results

4.1 Experiment Settings

In our experiments, we employed parallel GEQs with different input scales like MOptEqs and averaged the output of each branch after average pooling or nearest up-sampling to fuse the branches. We use weight normalization to ensure the convergence as MOptEqs and MDEQ, and set γ to 0 . 2 /M , where M is the minimum ∥ ˜ x · -˜ z · W h ∥ 2 2 among all patches. For the equilibrium calculation, we used the Anderson algorithm in the forward procedure, similar to other implicit models [28], and applied Phantom gradients [14] for back-propagation. All models were trained using SGD with a step learning rate schedule. We implemented our experiments on the PyTorch platform [39] using RTX-3090. Further details can be found in the Appendix A.6. To compare the performance of our GEQ, we used MOptEqs and MDEQ as benchmark implicit models, which have demonstrated superior performance over OptEqs on image classification tasks. Additionally, we used ResNet-18 and ResNet-50 as benchmark explicit models for comparison.

4.2 Results for Image Classification

Firstly, we finish the experiments on CIFAR10 and CIFAR100 . They are widely used datasets for image classification on small images. In the experiment, we parallel 6 branches GEQ with the input scale is 32 , 16 , 8 , 8 , 4 , 4 and MOptEqs' architecture setting is also the same. The details can be found in the Appendix. As for the comparison, we also conduct experiments of the same training procedure for MDEQ, MOptEqs, and ResNet. The results are listed in Table 1. From the results, one can see that our GEQ enjoys clear advantages on CIFAR datasets, which demonstrates the powerful generalization ability of other models.

Besides small datasets, we also conducted experiments on large-scale image datasets, as presented in Table 2. The results clearly demonstrate the consistent superiority of our GEQ over other models, highlighting its clear advantages. Particularly noteworthy is our GEQ achieves a 2% improvement on ImageNet100 against deep model ResNet50 while consuming approximately half the number of parameters, which emphasizes the effectiveness and efficiency of GEQ on large-scale inputs.

(a) CIFAR10 .

(b) CIFAR100 .

(a) ImageNette.

(b) ImageNet100 .

4.3 Validations on the models' stability

Evaluation on Unseen difficult samples. In order to assess the stability of our GEQ model on difficult examples, we conducted experiments using CIFAR-100 super-class classification. CIFAR100 consists of 20 super classes, each containing five sub-classes 2 . Wetrained our GEQ and MOptEqs models to predict the super-classes using the first four sub-classes from each super-class for training. We evaluated the models using both the test set, which includes the first four sub-classes from each super-class (referred to as "Known Accuracy"), and a separate set of samples from unseen sub-classes (referred to as "Unknown Accuracy"). The classification of the unseen samples is more difficult as they are different from the training set. The results of our GEQ and MOptEqs models are presented in Table 3.

The above table clearly demonstrates that our GEQ model surpasses MOptEqs in achieving superior performance on the challenging task at hand and demonstrates GEQ's stability. Such advantages can be attributed to the fact that GEQ exhibits smaller output similarities compared to OptEqs when input samples are far apart (e.g., samples from different classes). This characteristic can lead to larger margins between different classes, enabling the classifier to be more easily optimized during training. Consequently, our GEQ model excels in accurately classifying difficult unseen samples, further highlighting its stability and superiority over former equilibrium models.

Ablation Studies on corrupted datasets. Apart from difficult samples, we are going to compare the robustness of our GEQ, MOptEqs, and ResNet on the CIFAR10 corruption dataset, which contains 19 common corruptions including image transformation, image blurring, weather, and digital noises on CIFAR's test datasets. Average results on 5 degrees CIFAR10 corruption datasets list in Figure 2.

From the result, one can see that our GEQ based on Gaussian kernels is more robust than MOptEqs and ResNet. In particular, our GEQ can show better performance against structured noise and some image transformation. The above results also demonstrate the stability of our GEQ structure.

contrast

elastic_trans

jpeg_comp

pixelate

brightness

fog

frost

saturate

snow

defocus_blur

gaussian_blur

motion_blur

zoom_blur

gaussian_noise

impulse_noise

shot_noise

spatter

speckle_noise

60

65

70

75

80

85

90

95

MOptEq

G

EQ

ResNet

4.4 Ablation Studies on Saliency Map

The saliency maps generated by GradCAM [45] offer valuable insights into the visual attention of both MOptEqs and GEQ models. These maps highlight the regions of the image that are crucial for the model's predictions. Figure 3 presents the saliency maps obtained for an image from the ImageNet dataset using both models. Upon observation, it becomes evident that GEQ exhibits a higher degree of focus on the significant regions directly associated with the predicted label "manta". In contrast, MOptEqs tends to allocate attention to unrelated regions such as the shells. This discrepancy indicates that the attention-like module induced by the Gaussian kernel in GEQ enhances the concentration of the model's attention, resulting in improved performance compared to MOptEqs.

(a) Original Image.

(b) MOptEqs's saliency map.

(c) GEQ's saliency map.

4.5 Ablation Studies on Patch splitting

The performance of our GEQ model is influenced by the channel splitting parameter and the patch size. Choosing large values for these parameters causes the kernel to focus mainly on global information while selecting small values makes the kernel concentrate on local features. To understand the impact of these choices on model performance, we conducted experiments, the results of which are presented in Figure 4. This figure illustrates the relationship between the channel splitting parameter, patch size, and the model's performance. By analyzing these results, we gain insights into the optimal values for these parameters that yield the best performance for our GEQ model.

The accuracy trend depicted in the figure shows an initial increase followed by a decrease as the channel split and patch size increased. Based on these empirical results, we select a patch size of 2 and a channel split of 8 for both the CIFAR and ImageNet experiments. These parameter choices are made to optimize the performance of our models on the respective datasets.

4.6 Comparison with other kernel functions

Firstly, we introduce different commonly used kernels, such as polynomial, sigmoid, and Gaussian kernels, to reformulate the hidden optimization problem for equilibrium models. Table 4 illustrates the equilibrium models induced by these kernels. It can be observed that equilibrium models with polynomial and sigmoid kernels also incorporate new attentive modules. However, their attentive

Accuracy on different corrupted datasets

4

8

16

32

64

channel split

76

76.4

76.8

77.2

77.6

78

Acc

1

2

4

8

patch size

76

76.4

76.8

77.2

77.6

78

Acc

(a) Influence of the channel splitting parameter.

(b) Influence of the patch size parameter.

kernels only constrain the input features Ux + b and do not directly affect the activation of the output z ∗ . As a result, their performance may be inferior to our GEQ model. To validate these claims, we evaluate the performance of different models on the CIFAR100 dataset. Since the fusion module is the primary difference between MOptEqs and OptEqs, we can easily modify the structure of MOptEqs to include different kernel-induced attentive modules using the equations in Table 4, resulting in MOptEqs (Polynomial) and MOptEqs (Sigmoid). The results are presented in Table 5, which clearly demonstrates the superior performance of our GEQ model. For a more in-depth analysis of GEQ, we refer readers to the main paper.

5 Conclusions

In this paper, we introduce a novel optimization-induced equilibrium model called GEQ, which utilizes Gaussian kernels in its optimization-induced framework. Our model incorporates a new attentive module that arises from its novel hidden optimization problem formulation. Notably, GEQ exhibits significantly improved performance in classification tasks, outperforming deep models as well. Moreover, GEQ can be interpreted as a weight-tied model with infinite width and depth, highlighting its expressive power. We also provide theoretical analysis demonstrating the superiority of our models in terms of generalization ability and stability compared to previous OptEqs. Empirical results further validate the effectiveness of our proposed approach.

Acknowledgments

Zhouchen Lin was supported by National Key R&D Program of China (2022ZD0160302), the major key project of PCL, China (No. PCL2021A12), the NSF China (No. 62276004), and Qualcomm. Yisen Wang was supported by National Natural Science Foundation of China (62006153, 62376010, 92370129), Open Research Projects of Zhejiang Lab (No. 2022RC0AB05), and Beijing Nova Program (20230484344).

References

Brandon Amos and J Zico Kolter. Optnet: Differentiable optimization as a layer in neural networks. In ICML , 2017.

Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. In NeurIPS , 2019.

Shaojie Bai, J Zico Kolter, and Vladlen Koltun. Deep equilibrium models. In NeurIPS , 2019.

Minmin Chen, Jeffrey Pennington, and Samuel Schoenholz. Dynamical isometry and a mean field theory of rnns: Gating enables signal propagation in recurrent neural networks. In ICML , 2018.

Qi Chen, Yifei Wang, Zhengyang Geng, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Equilibrium image denoising with implicit differentiation. IEEE Transactions on Image Processing , 32:1868-1881, 2023.

Qi Chen, Yifei Wang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Optimization-induced graph implicit nonlinear diffusion. In ICML , 2022.

Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, NeurIPS , 2018.

KR1442 Chowdhary. Natural language processing. Fundamentals of artificial intelligence , pages 603-649, 2020.

Filipe de Avila Belbute-Peres, Kevin Smith, Kelsey Allen, Josh Tenenbaum, and J Zico Kolter. End-to-end differentiable physics for learning and control. NeurIPS , 2018.

Alexander G. de G. Matthews, Jiri Hron, Mark Rowland, Richard E. Turner, and Zoubin Ghahramani. Gaussian process behaviour in wide deep neural networks. In ICLR , 2018.

Josip Djolonga and Andreas Krause. Differentiable learning of submodular models. NeurIPS , 2017.

Zhili Feng and J Zico Kolter. On the neural tangent kernel of equilibrium models, 2021.

Adrià Garriga-Alonso, Carl Edward Rasmussen, and Laurence Aitchison. Deep convolutional networks as shallow gaussian processes. In ICLR , 2019.

Zhengyang Geng, Xin-Yu Zhang, Shaojie Bai, Yisen Wang, and Zhouchen Lin. On training implicit models. In NeurIPS , 2021.

Laurent El Ghaoui, Fangda Gu, Bertrand Travacca, and Armin Askari. Implicit deep learning. arXiv preprint arXiv:1908.06315 , 2019.

Dar Gilboa, Bo Chang, Minmin Chen, Greg Yang, Samuel S Schoenholz, Ed H Chi, and Jeffrey Pennington. Dynamical isometry and a mean field theory of lstms and grus. arXiv preprint arXiv:1901.08987 , 2019.

Stephen Gould, Richard Hartley, and Dylan Campbell. Deep declarative networks: A new hope. arXiv preprint arXiv:1909.04866 , 2019.

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In ICML , 2017.

Soufiane Hayou, Arnaud Doucet, and Judith Rousseau. Mean-field behaviour of neural tangent kernel for deep neural networks. arXiv preprint arXiv:1905.13654 , 2019.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , 2016.

Shell Xu Hu, Sergey Zagoruyko, and Nikos Komodakis. Exploring weight symmetry in deep neural networks. Computer Vision and Image Understanding , 187:102786, 2019.

Arthur Jacot, Franck Gabriel, and Clément Hongler. Freeze and chaos for dnns: an NTK view of batch normalization, checkerboard and boundary effects. arXiv preprint arXiv: abs/1907.05715 , 2019.

Nikhil Ketkar and Nikhil Ketkar. Stochastic gradient descent. Deep learning with Python: A hands-on introduction , pages 113-132, 2017.

Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. arXiv preprint arXiv:1711.00165 , 2017.

Mingjie Li, Lingshen He, and Zhouchen Lin. Implicit euler skip connections: Enhancing adversarial robustness via numerical stability. In ICML , 2020.

Mingjie Li, Yifei Wang, Yisen Wang, and Zhouchen Lin. Unbiased stochastic proximal solver for graph neural networks with equilibrium states. In ICLR , 2023.

Mingjie Li, Yisen Wang, and Zhouchen Lin. Cerdeq: Certifiable deep equilibrium model. In ICML , 2022.

Mingjie Li, Yisen Wang, Xingyu Xie, and Zhouchen Lin. Optimization inspired multi-branch equilibrium models. In ICLR , 2021.

Guangcan Liu, Shiyu Chang, and Yi Ma. Blind image deblurring using spectral properties of convolution operators. IEEE Transactions on image processing , 23(12):5047-5056, 2014.

Guangcan Liu and Ping Li. Low-rank matrix completion in the presence of high coherence. IEEE Transactions on Signal Processing , 64(21):5623-5633, 2016.

Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055 , 2018.

Juncheng Liu, Kenji Kawaguchi, Bryan Hooi, Yiwei Wang, and Xiaokui Xiao. Eignn: Efficient infinite-depth graph neural networks. In NeurIPS , 2021.

Julien Mairal, Francis Bach, Jean Ponce, and Guillermo Sapiro. Online dictionary learning for sparse coding. In ICML , 2009.

Lassi Meronen, Christabella Irwanto, and Arno Solin. Stationary activations for uncertainty calibration in deep learning. In NeurIPS , 2020.

Radford M Neal. Priors for infinite networks. In Bayesian Learning for Neural Networks , pages 29-53. Springer, 1996.

Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nathan Srebro. A pacbayesian approach to spectrally-normalized margin bounds for neural networks. arXiv preprint arXiv: 1707.09564 , 2017.

Roman Novak, Lechao Xiao, Yasaman Bahri, Jaehoon Lee, Greg Yang, Jiri Hron, Daniel A. Abolafia, Jeffrey Pennington, and Jascha Sohl-Dickstein. Bayesian deep convolutional networks with many channels are gaussian processes. In ICLR , 2019.

Chirag Pabbaraju, Ezra Winston, and J Zico Kolter. Estimating lipschitz constants of monotone deep equilibrium models. In ICLR , 2021.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS , 2019.

Yi-Ling Qiao, Junbang Liang, Vladlen Koltun, and Ming Lin. Scalable differentiable physics for learning and control. In ICML , 2020.

Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In CVPR , 2016.

Max Revay, Ruigang Wang, and Ian R Manchester. Lipschitz bounded equilibrium networks. arXiv preprint arXiv:2010.01732 , 2020.

Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE transactions on neural networks , 20(1):61-80, 2008.

Bernhard Scholkopf, Kah-Kay Sung, Christopher JC Burges, Federico Girosi, Partha Niyogi, Tomaso Poggio, and Vladimir Vapnik. Comparing support vector machines with gaussian kernels to radial basis function classifiers. IEEE transactions on Signal Processing , 45(11):27582765, 1997.

Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In ICCV , 2017.

Ivana Toši´ c and Pascal Frossard. Dictionary learning. IEEE Signal Processing Magazine , 28(2):27-38, 2011.

Asher Trockman and J Zico Kolter. Patches are all you need? arXiv preprint arXiv:2201.09792 , 2022.

Russell Tsuchida, Fred Roosta, and Marcus Gallagher. Invariance of weight distributions in rectified mlps. In ICML , 2018.

Russell Tsuchida, Suk Yee Yong, Mohammad Ali Armin, Lars Petersson, and Cheng Soon Ong. Declarative nets that are equilibrium models. In ICLR , 2022.

Po-Wei Wang, Priya Donti, Bryan Wilder, and Zico Kolter. Satnet: Bridging deep learning and logical reasoning using a differentiable satisfiability solver. In ICML , 2019.

Ezra Winston and J Zico Kolter. Monotone operator equilibrium networks. In NeurIPS , 2020.

Xingyu Xie, Qiuhao Wang, Zenan Ling, Xia Li, Guangcan Liu, and Zhouchen Lin. Optimization induced equilibrium networks: An explicit optimization perspective for understanding equilibrium models. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(3):3604-3616, 2022.

Xingyu Xie, Jianlong Wu, Guangcan Liu, Zhisheng Zhong, and Zhouchen Lin. Differentiable linearized admm. In ICML , 2019.

Chong You, Daniel Robinson, and René Vidal. Scalable sparse subspace clustering by orthogonal matching pursuit. In CVPR , 2016.

Hongyang Zhang, Zhouchen Lin, Chao Zhang, and Edward Y Chang. Exact recoverability of robust pca via outlier pursuit with tight recovery bounds. In AAAI , 2015.

Xiao Zhang, Lingxiao Wang, Yaodong Yu, and Quanquan Gu. A primal-dual analysis of global optimality in nonconvex low-rank matrix recovery. In ICML , 2018.

Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578 , 2016.

A Appendix

A.1 Forward Procedure for GEQ

The pseudo-code for our GEQ is listed in Algorithm 1.

Algorithm 1: Calculating one layer GEQ.

Require: initial state z (0) , weight parameter W , Ux ∈ R chw , channel split c s , patch size p , hidden layer W h ∈ R c s p 2 × 32

Ensure: Get the output z ∗ of i -th fixed point iteration.

def g ( z ( i ) ; x , U , W , b ) :

Rearrage

˜

m

→

m

return

z

(

i

+1)

=

σ

∈

(

with torch.no_grad():

Use anderson algorithm to solve z ∗ = g ( z ∗ ; x , U , W , b # calculate gradient via phantom gradient:

for i in range(5) do

end for return z ∗

A.2 Proofs for proposition 1

Proposition 1. The outputs of GEQ with Gaussian kernels (Eqn (6)) is the same as Optimized induced Equilibrium Models' output whose output is the optimal solution of the hidden optimization problems:

where Φ W ( z ) = [ 1 , √ 2 γ Φ (1) W ( z ) , ..., √ (2 γ ) i i Φ ( i ) W ( z ) , ... ] ∈ R 1 ×∞ which projects the features to the infinite-dimensional space. And Φ ( i ) W : R n → R in i is the k -tuple permutation with repetitions formulated as follows:

where ( Wx ) j denotes the j -th element of vector Wx . Then the Gaussian kernel can also be regarded as calculating the input features after the weight-tied infinite wide projection Φ W and Φ U .

Proof. We can formulate the OptEqs' hidden optimization problem with Gaussian kernels as below:

For e -γ ∥ ( Ux + b -Wz ) ∥ 2 2 , we have

letting λ = e -γ ∥ Ux + b ∥ 2 2 e -γ ∥ Wz ∥ 2 and we do the Taylor expansion for e ⟨ Ux + b , Wz ⟩ , we have:

R

W

c

⊤

×

h

×

)

m

w

)

For any i we have from the permutation theory:

where Φ ( i ) W ( x ) is the i -tuple permutation with the repetition for given ( Wx ) 1 , ( Wx ) 2 , ..., ( Wx ) n . Each element of Φ ( i ) W ( x ) is one possible permutation. Since there are n i tuples, then Φ ( i ) W can project the input features to in i space as follows,

Thereby, the hidden optimization problem for our GEQ can be reformulated as,

Then our conclusion is proved.

A.3 Proofs for proposition 2

Proposition 2. If input ∥ x ∥ 2 is bounded by B , µ := max {∥ U ∥ 2 , ∥ W ∥ 2 , ∥ W c ∥ 2 , ∥ b ∥ 2 } < 1 , then we have following results for GEQ and OptEqs with ReLU activations. For ∀ δ, η > 0 , with probability at least 1 -δ over the training set of size M, we have:

where ˆ L η ( f c geq ) denotes the empirical margin loss on the training set, the maximum scaling number is defined by β max := max x ∈D e -γ ∥ Ux + b -Wz ∥ 2 2 , B W := ∥ W ⊤ W ∥ 2 F + ∥ U ∥ 2 F + ∥ b ∥ 2 2 + ∥ W c ∥ 2 F + ∥ b c ∥ 2 2 , and m = ∥ W ⊤ W ∥ 2 is less than 1 to ensure the convergence of equilibrium models.

Before the proof our results, we need to introduce a lemma in former work [38] for the perturbation bound for GEQs and reformulated OptEqs as follows.

Lemma 1. Let ∥ W ∥ 2 ≤ m and ∥ W ∥ 2 ≤ m . Then change in the output of the DEqs z = σ ( Wz + Ux + b ) on perturbation the weights and biases from W , U , b to W , U , b is bounded as follows:

Like former works [38], we we also introduce another lemma [36] here:

Lemma 2. Let f w be any predictor with parameters w , and let P denote any distribution on the parameters that are independent of the training data. Then, for any δ, γ > 0 , with probability ≥ 1 -δ over the training data of size M , for any w , and any random perturbation u such that P [ max x ∥ f w + u ( x ) -f w ( x ) ∥ ∞ < η 4 ] ≥ 1 2 , we have

Then we can derive the perturbation bound for the reformulated OptEqs and our GEQ following former settings [38]. First, we also incorporate a fully connected layer at the end as we mentioned in the paper.

Since the entries in the perturbations obeying the distribution os N (0 , σ 2 ) , we have that all the perturbations of weights ∥ ∆ · ∥ are bounded by σ √ 2 h ln(24 h ) : ω with probability larger than 1 / 2 .

Since the only difference between OptEqs and monDEQ [51] is the weight parameterization, our reformulate OptEqs is parameterized by W s = W ⊤ W while the monDEQ's weight parameter is parameterized by a series of weights W mondeq = I + A + A ⊤ + B + B ⊤ . Therefore, the pertubation in ∥ ∆ W s ∥ is different from former analysis [38]. We have that:

Then using the above lemma, we have that for all x with probability at least 1 / 2 .

Setting σ = η (1 -m 2 ) 4 √ 2 h ln(24 h )(2 µ 3 ( B +1)+2(1 -m ) µ +(1 -m ) 2 ) will make the above perturbation less than η 4 . Then we have,

With the same choice of β 's bound like former work [38],

we can finally get the upper bound as our OptEqs bound as our proposition shows.

The difference between GEQ and OptEqs is that GEQ's can be viewed as multiplying scaler β = e -γ ∥ Ux + b -Wz ∥ 2 2 with depend on x since z is also depended on x . Setting β max = max x ∈D β ( x ) < 1 and β min = min x ∈D β ( x ) > c . Assuming β changes a little with respect to the small perturbations on weights, we have:

With the same setting as above OptEqs, we have:

Then we can obtain that:

With the same setting as above, we have:

where T is defined as follows:

then we can finally we can get the upper bound as our OptEqs bound as our proposition shows.

A.4 Proofs for proposition 3

Lipschitz constant is the minimal constant for f and ∀ x , y suits the following equation:

Thereby, our analysis in Proposition 3 can be viewed as proposing an upper bound for different models. In this section, we are going to prove the Lipschitz upper bounds for our GEQ and OptEqs. First, we restate the proposition as follows:

Proposition 3. If norms for the inputs and outputs are bounded by B , the spectral norm for the weight parameter W , U of equilibrium models with ReLU activation are bounded by µ < 1 to ensure convergence, then we have the conclusions as below:

where x 1 and x 2 are input samples, f geq ( x · ) and f opteq ( x · ) denotes the equilibrium states for GEQ and OptEqs given input x · , and β max := max x ∈D e -γ ∥ Ux -Wz ∥ 2 2 < 1 .

Proof. We first prove the inequality for OptEqs:

where z x 1 denotes the equilibrium states for OptEq with input x 1 , which means the following equation is satisfied:

Reformulating the equations, we can get:

Then for GEQ, we have:

where z x 1 denotes the equilibrium states for GEQ with input x 1 , and β x 1 is defined as follows:

Then with the mean value theorem, we have the following equations:

Then reformulating Eqn (44), we can finally obtain our proposition as below:

A.5 Proofs for proposition 4

In this section, we are going to prove the output similarity bounds for our GEQ and OptEqs. First, we restate the proposition as follows:

Proposition 4. If norms for the inputs and outputs are bounded by B , the spectral norm for the weight parameter W of equilibrium models with ReLU activation are bounded by µ < 1 to ensure the convergence, and each row in U obeys the spherical Gaussian distributions N (0 , E [ U 2 i ] I ) . Then we have the following conclusions for the expectation of the output similarity for GEQ and OptEqs with respect to input x 1 , x 2 as follows,

where x 1 and x 2 are input samples, D = e γB ∥ W ∥ 2 2 and β max := max x ∈D e -γ ∥ Ux -Wz ∥ 2 2 < 1 . θ 0 = cos -1 ( ⟨ x 1 , x 2 ⟩ ∥ x 1 ∥∥ x 2 ∥ ) is defined as the angle between the samples.

Proof. Before the proof, we first introduce the following lemma:

Lemma 3. [48] If U obeys the spherical Gaussian distributions of variance E [ U 2 i ] and mean 0 , then the expectation of the Similarity for the one-layer Neural Network σ ( Ux ) is:

where θ 0 = cos -1 ( ⟨ x , y ⟩ ∥ x ∥∥ y ∥ ) .

Letting m := ∥ W s ∥ 2 = ∥ W ⊤ W ∥ 2 < 1 and µ := ∥ W ∥ 2 < 1 as our assumptions demonstrate and neglecting the bias b for convenience. Then for our reformulated OptEqs, we have:

And we have

Then

Therefore, we can conclude

For GEQ, we set β x = e -γ ∥ Ux -Wz ∥ 2 2 and β max := max x ∈D e -γ ∥ Ux -Wz ∥ 2 2 < 1

Like OptEqs, we obtain the following equations for GEQ:

Therefore,

And we have,

where D = e γ ∥ W ∥ 2 2 B , the latter two inequality is acquired by Jensen's inequality. Then we have,

A.6 Experiment Settings

A.6.1 Experiments on CIFAR

For our GEQ, we parallel 6 branches with each branch taking the scale of 32 , 16 , 8 , 8 , 4 , 4 and using the average fusion method for branches' fusion. The output channels for 6 branches are all 256 or 320 but the mid-channel number(output channel for weight U and W ) for the six branches are 64 , 128 , 128 , 128 , 256 , 256 or 80 , 160 , 160 , 160 , 320 , 320 with patch size 2 and c splitting is 8 . And the inner MLP inner W h output 64 hidden dimension for each patch. We use the SGD [23] optimizer with momentum and step learning rate schedule for all the models. We also use RandomAug for all the models for comparison.

A.6.2 Experiments on ImageNette and ImageNet-100

We take the input scale as 256 for all models. For our GEQ, we parallel 6 branches with each branch taking the scale of 64 , 32 , 16 , 16 , 8 , 8 after two downsampling convolutions and using the average fusion method for branches' fusion. The output channels for 6 branches are all 256 or 384 but the mid-channel number(output channel for weight U and W ) for the six branches are 32 , 64 , 128 , 128 , 256 , 256 or 48 , 96 , 192 , 192 , 384 , 384 with patch size 2 and c splitting is 8 . And the inner MLP inner W h output 128 hidden dimension for each patch. We use the SGD optimizer with momentum and step learning rate schedule for all the models. We also use RandomAug for all the models for comparison.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper makes the observation that conventional DEQ architectures effectively use linear kernels in their feature extraction, and the proposed method in this paper is to use Gaussian Kernels instead of linear kernels. Then, the paper makes the observation that adding Gaussian Kernels to DEQ models produce an architecture that is effectively infinite deep and infinitely wide. The paper then proposes patch splitting, a technique that makes the proposed method enjoy better performance, and shows the effectiveness of the proposed method on CIFAR10, CIFAR100, and saliency map datasets.\n\nSTRENGTHS:\nThis paper is clearly written. In terms of contribution, this paper makes the observation that the proposed method is effectively infinitely deep and infinitely wide, which I find to be an interesting thread. Finally, this paper contains convergence bounds and stability analysis, training technique (patch splitting), as well as numerous satisfactory experiment results, and therefore I believe that the proposed method is sufficiently interesting.\n\nWEAKNESSES:\nI believe this paper would benefit more from discussion about the proposed model. For example, now that we know the proposed model is infinitely deep and infinitely wide, what can we say about its connection to NTK? Can this model further some of the NTK analysis? In addition, can this model inspire potential work on Gaussian processes?\n\nQUESTIONS:\n1. Similar to the weakness section---I am curious whether the proposed model can act as an effective model for large neural nets in general, the same way that the NTK regime approximates large width networks? Can this model be representative of large language models, given its infinite depth and infinite width capacity?\n2. Does the Gaussian Kernel in the proposed model have any natural use cases in practice (that linear kernels, polynomial kernels, etc don't have)?\n\nLIMITATIONS:\nThere are no negative societal impacts of this work", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce a deep equilibrium model (6) which ostensibly solves an optimisation problem (5) involving a Gaussian (squared exponential) kernel. The authors describe some possible theoretical properties of the new model, and benchmark it on some problems against some related optimisation-inspired DEQs and resnets.\n\nSTRENGTHS:\n- The idea of deriving DEQ models from optimisation problems is nice, and fits in well with existing literature.\n\nWEAKNESSES:\nUnfortunately, there appear to be many issues with the mathematical results in this paper. I do not believe they are sound. Some of them could potentially be fixable, but I am not certain.\n\n- **Important** The use of the function $f$ is not clear.\n    - Can you elaborate on the definition of $f$? On the second page, you say \"$f$ is a positive indicator function that induces to commonly used ReLU activation\" What does \"induces to\" mean? What exactly is your definition of indicator function? One definition is that given a set $A$, $f_A(z) = 1$ if $z \\in A$ and zero otherwise. This does not seem to reconcile with your definition, though. Perhaps $f$ takes a value of $1$ if $z$ is greater than 0 (elementwise), and zero otherwise? What happens at the value of zero? What is $f$'s (weak) derivative? How is it related to the ReLU?\n    - After equation (3), you mention the first order stationarity condition. Is $f$ here the indicator function? Isn't this function not everywhere differentiable? Even if the derivative were defined, how do you know that $\\nabla G = 0$ implies a minima? \n    - How is a ReLU activation in (4) obtained by differentiation? I could see that the derivative of $relu(a)^2$ with respect to $a$ would be $relu(a)$, but not sure how to obtain it otherwise?\n    - Proposition 1. What is the meaning of the symbol $\\partial f$? How is $(1+\\partial f)^{-1}$ related to the ReLU function?\n- Proposition 1 is not precise. The inner product is not defined, which is quite important because $\\Phi$ lives in some kind of Hilbert space (informally being referred to as $\\mathbb{R}^\\infty$). Can you clarify what $\\infty$-dimensional space means, and what $\\langle \\cdot, \\cdot \\rangle $ means?\n- Proposition 3. Isn't the left hand side of equation (11) zero? This would be a trivial bound, I would have thought.\n- (Minor) Proof of proposition 3. Aren't all the first lines in equation (41) equality, by definition of fixed point?\n- Proposition 4. Can you elaborate on why it is helpful to exhibit a smaller output similarity for dissimilar samples? One can trivially do this for any kernel, by simply rescaling the kernel by a constant factor.\n\nQUESTIONS:\nMinor (*not* a reason to reject):\n- Section title for section3 is missing an \"I\".\n- The titles and labels in Figure 2 are too small to read.\n- Why are only Gaussian kernels considered?\n- Many works have pointed out potential issues with using saliency maps such as GradCAM to try and gain insight into neural network predictions. E.g. see Cynthia Rudin's work on explainability versus interpretability. The saliency map visualisation seems to come out of nowhere (not being mentioned anywhere earlier in the paper). Is there a reason why include it in this paper?\n\nLIMITATIONS:\nI do not see any explicit mention of limitations in this work. That being said, I may have missed some, if they are somewhere in the text.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe presented work proposed a new equilibrium model by replacing the linear term in the original formulation into a Gaussian kernel. Analysis and experiments are performed to illustrate the superiority of the proposed model in terms of expressivity, generalization and stability, etc.\n\nSTRENGTHS:\n- The stability and infinite-width equavalence analysis appears to be novel. I didn't see similar analysis for other equilibrium models.\n- The exploration of matching the performace of equilibrium models and traditional deep learning models is a meaningful direction. The experiments verified the effectiveness of the proposed model.\n\n-----\nI have decided to raise my rating to this paper to a clear accept because all of my concerns are well address by rebuttal.\n\nWEAKNESSES:\n- In my understanding, the modification made by the presented work is seentially replacing the linear term in OptEqs with an exponential term, which will result in an attention-like module in the forward layer. However, the idea of incorporating an exponential term into the optimization objective and unfolding it into an attention mechanism may not be considered entirely novel. For example [1] also used an exponential term and derived an attention layer from the exponential term, despite their model is not an equilibrium model and they didn't explain it as a Gaussian kernel.\n- According to the Section A.2, the forward process is calculated directly by Eq.(6). However, the exponential term in Eq.(6) looks dangerous, as it is widely known that exponential function often leads to numerical instability in computations. I would expect there to be a normalization term (which leads to a softmax-like term) in Eq.(6). \n- Some expressions are vague, for example it's not clear what does \"feature extraction term\" mean at line 48.\n\nA minor issue: In Proposition 1 there's a term $\\sqrt{{\\boldsymbol{2}} \\gamma} {\\boldsymbol{\\Phi}}_{\\boldsymbol{{W}}}^{({\\boldsymbol{1}})}$, what do the bolded ${\\boldsymbol{2}}, {\\boldsymbol{\\Phi}}$ and superscript $({\\boldsymbol{1}})$ mean? I guess they shouldn't be bolded and it is a typo?\n\n[1] Transformers from an Optimization Perspective\n\nQUESTIONS:\n- I wonder if we can say the expressive power of GEQ is strictly stronger than the original form of OptEqs (in Eq.(3) and Eq.(4)) as the authors proved GEQ can be viewed as OptEqs with some terms projected to infinite-dimensional spaces?\n- Based on my understanding, the Patch Splitting appears to be essential for the proposed model. Since if there is not patch splitting then Eq.(6). would essentially be adding a scalar coefficient to the original update equation, which may not result in significant changes. Am I right?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose to use Gaussian kernel in OptEqs (optimization-induced deep equilibrium models). Because of the more powerful capability of kernel, it can better capture the dynamic performance than linear models.\n\nSTRENGTHS:\nOptEqs is an interesting attempt to describe the training dynamics of DNN and hence making it more powerful (this paper's work) is interesting. (This does not mean that I appreciate the technique contribution of GEQ. To me, this is a natural incremental contribution from OptEqs.) \n\nThe reported performance is quite good, considering that GEQ implies a new model structure.\n\nWEAKNESSES:\nUsing a Gaussian kernel instead of a linear is simple, which can also naturally lead to a better generalization bound. In other words, the discussion on the generalization bound is interesting but simply saying \"tighter\" is somehow trivial.\n\nIn my point of view, there is essential difference of applying GEQ to ImageNet-100 or the whole ImageNet. \n\nI think that a new point of view of neural networks training itself is already interesting. It is not necessary to defeat standard neural networks. If this is the aim of this paper, the experiments should be enhanced. For example, ImageNet should be considered and \nthe setting (SGD with a step learning rate schedule) is fair but may be not sufficient. The following link gives the links of reported best training strategy: https://paperswithcode.com/sota/ \nCIFAR-10 with ResNet-18 is 95.55; \nCIFAR-100 with ResNet-50-SAM is 85.2\n\nIn one word, I think the current experiments for evaluating GEQ are already good (when ImageNet is included). But it is far from giving the conclusion that GEQ is better than ResNet, etc. \n\n\nMy overall recommendation is currently positive, which is mainly based on numerical experiments. There are still many doubts about the performance. If the answer is not strong, I may lower my score. Also, I feel the theoretical and technique contributions of this paper is weak and hence will not fight for my current score, when other reviewers think this part should be improved.\n\nQUESTIONS:\nReplacing the linear operator in OptEqs by a nonlinear one is nice. But can the authors explain more why kernel is chosen? why not try e.g., MLP, of which the parameters could be trained.\n\nQuestions for numerical experiments could be found above, especially if the authors want to claim advantages over standard neural networks.\n\nLIMITATIONS:\nnot applicable", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}