SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left\| f _ {\text {o p t e q}} \left(\mathbf {x} _ {1}\right) - f _ {\text {o p t e q}} \left(\mathbf {x} _ {2}\right) \right\| _ {2} \leq L _ {\text {o p t e q}} \left\| \mathbf {x} _ {1} - \mathbf {x} _ {2} \right\| _ {2} = \frac {\mu}{1 - \mu^ {2}} \left\| \mathbf {x} _ {1} - \mathbf {x} _ {2} \right\| _ {2}, \tag {12}
$$
where  $\mathbf{x}_1$  and  $\mathbf{x}_2$  are input samples,  $f_{geq}(\mathbf{x})$  and  $f_{opteq}(\mathbf{x})$  denotes the equilibrium states for GEQ and OptEqs given input  $\mathbf{x}$ , and  $\beta_{max} := \max_{\mathbf{x} \in \mathcal{D}} e^{-\gamma \| \mathbf{Ux} - \mathbf{Wz} \|_2^2} < 1$ .
Remark 2. If  $\beta_{\mathrm{max}} < 0.8$ ,  $B < 1$ , and  $\sqrt{\gamma} < 0.2$ , then  $L_{geq} < L_{opteq}$ .
In practical experiments, we choose different  $\gamma$  to reach the above condition for  $\beta_{\mathrm{max}}$  and the condition for input  $B$  can also be achieved by normalization layers. Although the above bound is not tight, we can use it as a rough explanation for our GEQ's stability, which is demonstrated in the following experiments. How to approximate a much tighter bound for equilibrium models still needs exploring.
Besides having stable outputs under perturbations, a stable model should also show large output differences for different classes to make classification easier. However, the above Lipschitz term can not constrain outputs' similarity when samples are far apart, then we need a new metric for analysis. In line with previous works [18, 34, 10], we assume all weight parameters go to infinite dimensions and analyze the expected output similarity  $\kappa$  for a model  $f$  for inputs  $\mathbf{x}_1$  and  $\mathbf{x}_2$  defined below:
$$
\kappa \left(\mathbf {x} _ {1}, \mathbf {x} _ {2}\right) = \mathbb {E} \left[ f \left(\mathbf {x} _ {1}\right) ^ {\top} f \left(\mathbf {x} _ {2}\right) \right] = \int_ {\mathbb {R}} f _ {\mathbf {u}} \left(\mathbf {x} _ {1}\right) ^ {\top} f _ {\mathbf {u}} \left(\mathbf {x} _ {2}\right) p (\mathbf {u}) d \mathbf {u}, \tag {13}
$$
with  $p(\mathbf{u})$  is the distribution of weight  $\mathbf{U}$ 's vectorization. If  $\kappa$  is smaller for samples  $\mathbf{x}_1$  and  $\mathbf{x}_2$  when they belong to different classes, which means they are far away, then the classifier can easily classify these two samples with different labels. The margin for the classification will also be large and easy for the classification of difficult samples. The  $\kappa$ 's upper bound for GEQ and OptEqs are listed below:
Proposition 4. If norms for the inputs and outputs are bounded by  $B$ , the spectral norm for the weight parameter  $\mathbf{W}$  of equilibrium models with ReLU activation are bounded by  $\mu < 1$  to ensure the convergence, and each row in  $\mathbf{U}$  obeys the spherical Gaussian distributions  $\mathcal{N}(0,\mathbb{E}[U_i^2 ]\mathbf{I})$ . Then we have the following conclusions for the expectation of the output similarity for GEQ and OptEqs with respect to input  $\mathbf{x}_1,\mathbf{x}_2$  as follows,
$$
\kappa_ {g e q} (\mathbf {x} _ {1}, \mathbf {x} _ {2}) \leq \bar {\kappa} _ {g e q} = \frac {\mu^ {2} D e ^ {- \frac {\gamma}{4} \left(\sigma_ {\min } (\mathbf {U}) ^ {2} \| \mathbf {x} _ {1} - \mathbf {x} _ {2} \| _ {2} ^ {2}\right)} \mathbb {E} \left[ U _ {i} ^ {2} \right] \| \mathbf {x} _ {1} \| _ {2} \| \mathbf {x} _ {2} \| _ {2} \left(\sin \theta_ {0} + (\pi - \theta_ {0}) \cos \theta_ {0}\right)}{2 \pi (1 - \beta_ {\max } \mu^ {2}) ^ {2}}, \tag {14}
$$
$$
\kappa_ {o p t e q} \left(\mathbf {x} _ {1}, \mathbf {x} _ {2}\right) \leq \bar {\kappa} _ {o p t e q} = \frac {\mathbb {E} \left[ U _ {i} ^ {2} \right] \left\| \mathbf {x} _ {1} \right\| _ {2} \left\| \mathbf {x} _ {2} \right\| _ {2} \left(\sin \theta_ {0} + (\pi - \theta_ {0}) \cos \theta_ {0}\right)}{2 \pi (1 - \mu^ {2}) ^ {2}}, \tag {15}
$$
where  $\mathbf{x}_1$  and  $\mathbf{x}_2$  are input samples,  $D = e^{\gamma B\|\mathbf{W}\|_2^2}$ ,  $\beta_{max} \coloneqq \max_{\mathbf{x} \in \mathcal{D}} e^{-\gamma \|\mathbf{U}\mathbf{x} - \mathbf{W}\mathbf{z}\|_2^2}$ ,  $\sigma_{\min}(\mathbf{U})$  is  $\mathbf{U}$ 's minimal singular term, and  $\theta_0 = \cos^{-1}(\frac{\langle\mathbf{x}_1,\mathbf{x}_2\rangle}{\|\mathbf{x}_1\|\|\mathbf{x}_2\|})$  is the angle between the samples.
Remark 3. If  $\| \mathbf{x}_1 - \mathbf{x}_2\| _2\geq 2\sqrt{-log(1 / D)} /\sigma_{\mathrm{min}}(\mathbf{U})$  , then  $\overline{\kappa}_{geq}\leq \overline{\kappa}_{geq}$
Based on the aforementioned analysis, it is evident that our GEQ exhibits a smaller output similarity for dissimilar samples. As a result, the predictions made by GEQ are primarily based on the most similar samples, enabling it to successfully classify challenging instances. This claim is further supported by the results obtained from our carefully designed experiments.
3.5 Patch Splitting in GEQ
Since different parts of images have different impacts on the image classification, calculating the whole similarity using Gaussian kernels for GEQ is not enough. Inspired by former works [47], we also split the feature map  $\mathbf{U}\mathbf{x}$  into patches, and then our optimization problem becomes:
$$
\min  _ {\mathbf {z}} G (\mathbf {z}; \mathbf {x}) = \min  _ {\mathbf {z}} \left[ \mathbf {1} ^ {\top} f (\mathbf {z}) + \frac {1}{2} \| \mathbf {z} \| _ {2} ^ {2} - \frac {1}{2 \gamma} \sum_ {i = 1} ^ {N} e ^ {- \gamma \| \left(\tilde {\mathbf {x}} _ {i} - \tilde {\mathbf {z}} _ {i}\right) \mathbf {W} _ {h} \| _ {2} ^ {2}} \right], \tag {16}
$$
Figure 1: The sketch map of one layer GEQ's  $n$ -th fixed point iteration.  $\mathbf{x}$  is the input and  $\mathbf{z}^{(n-1)}$ ,  $\mathbf{z}^{(n)}$  are the output of  $(n-1)$ -th,  $n$ -th iteration, and  $\tilde{\mathbf{m}}$  is a middle state.
where  $\tilde{\mathbf{x}}_i\in \mathbb{R}^{c_sp^2}$  is the  $i$ -th patch of  $\mathbf{U}\mathbf{x} + \mathbf{b}$  while  $\tilde{\mathbf{z}}_i\in \mathbb{R}^{c_sp^2}$  is the  $i$ -th patch of  $\mathbf{W}\mathbf{z}$  and  $\mathbf{W}_h\in \mathbb{R}^{c_sp^2\times c_{hid}}$  is a linear layer to project patches with different size to the constant dimension.  $c_{s}$  denotes the channel splitting number,  $p$  denotes the patch size, and  $c_{hid}$  denotes the hidden dimension of patches after projection. We note that the patch-splitting approach is a GEQ's unique feature, as incorporating this technique makes no difference in OptEqs due to its linear kernel. Figure 1 provides a sketch for GEQ's  $i$ -th fixed-point iteration. From the figure, it is evident that our GEQ can be viewed as a special OptEqs with additional attention mechanisms to capture the most important regions. Thereby, it can achieve enhanced performance. For a more detailed understanding of the forward procedure in our GEQ, please refer to Appendix A.1.
In our experiments, we employed parallel GEQs with different input scales like MOptEqs and averaged the output of each branch after average pooling or nearest up-sampling to fuse the branches. We use weight normalization to ensure the convergence as MOptEqs and MDEQ, and set  $\gamma$  to  $0.2 / M$ , where  $M$  is the minimum  $\| \tilde{\mathbf{x}} - \tilde{\mathbf{z}}. \mathbf{W}_h\|_2^2$  among all patches. For the equilibrium calculation, we used the Anderson algorithm in the forward procedure, similar to other implicit models [28], and applied Phantom gradients [14] for back-propagation. All models were trained using SGD with a step learning rate schedule. We implemented our experiments on the PyTorch platform [39] using RTX-3090. Further details can be found in the Appendix A.6. To compare the performance of our GEQ, we used MOptEqs and MDEQ as benchmark implicit models, which have demonstrated superior performance over OptEqs on image classification tasks. Additionally, we used ResNet-18 and ResNet-50 as benchmark explicit models for comparison.
4.2 Results for Image Classification
Firstly, we finish the experiments on CIFAR-10 and CIFAR-100. They are widely used datasets for image classification on small images. In the experiment, we parallel 6 branches GEQ with the input scale is 32, 16, 8, 8, 4, 4 and MOptEqs' architecture setting is also the same. The details can be found in the Appendix. As for the comparison, we also conduct experiments of the same training procedure for MDEQ, MOptEqs, and ResNet. The results are listed in Table 1. From the results, one can see that our GEQ enjoys clear advantages on CIFAR datasets, which demonstrates the powerful generalization ability of other models.
Besides small datasets, we also conducted experiments on large-scale image datasets, as presented in Table 2. The results clearly demonstrate the consistent superiority of our GEQ over other models, highlighting its clear advantages. Particularly noteworthy is our GEQ achieves a  $2\%$  improvement on ImageNet-100 against deep model ResNet-50 while consuming approximately half the number of parameters, which emphasizes the effectiveness and efficiency of GEQ on large-scale inputs.
Model Size	Accuracy
ResNet-18	10M	93.5 ± 0.2%
ResNet-50	23M	95.2 ± 0.2
MDEQ	10M	94.2 ± 0.3%
MOptEqs	8M	94.6 ± 0.2%
GEQ	5M	94.8 ± 0.1%
GEQ	8M	95.6 ± 0.2%
(a) CIFAR-10.
Model Size	Accuracy
ResNet-18	10M	74.5 ± 0.2%
ResNet-50	23M	77.9 ± 0.1%
MDEQ	10M	74.7 ± 0.3%
MOptEqs	8M	75.6 ± 0.2%
GEQ	5M	76.4 ± 0.3%
GEQ	8M	78.2 ± 0.2%
(b) CIFAR-100.
Table 1: The Empirical results for image classification on CIFAR-10 and CIFAR-100.
Model Size	Accuracy
ResNet-18	11M	92.3 ± 0.1%
ResNet-50	23M	93.0 ± 0.2%
MDEQ	10M	91.5 ± 0.2%
MOptEqs	10M	92.4 ± 0.2%
GEQ	6M	92.9 ± 0.2%
GEQ	13M	93.2 ± 0.1%
(a) ImageNette.
Model Size	Accuracy
ResNet-18	11M	80.9 ± 0.3%
ResNet-50	23M	81.7 ± 0.2%
MDEQ	10M	81.3 ± 0,2%
MOptEqs	13M	81.5 ± 0.4%
GEQ	6M	82.2 ± 0.2%
GEQ	13M	83.9 ± 0.3%
4.3 Validations on the models' stability
Evaluation on Unseen difficult samples. In order to assess the stability of our GEQ model on difficult examples, we conducted experiments using CIFAR-100 super-class classification. CIFAR-100 consists of 20 super classes, each containing five sub-classes $^{2}$ . We trained our GEQ and MOptEqs models to predict the super-classes using the first four sub-classes from each super-class for training. We evaluated the models using both the test set, which includes the first four sub-classes from each super-class (referred to as "Known Accuracy"), and a separate set of samples from unseen sub-classes (referred to as "Unknown Accuracy"). The classification of the unseen samples is more difficult as they are different from the training set. The results of our GEQ and MOptEqs models are presented in Table 3.
(b) ImageNet-100.
Table 2: The Empirical results for image classification on ImageNette and ImageNet-100.
Known Accuracy	Unknown Accuracy
MOptEqs	80.1 ± 0.3%	77.4 ± 0.5%
GEQ	80.9 ± 0.2%	80.1 ± 0.6%
Table 3: Empirical results on CIFAR-100's super-class classification.
The above table clearly demonstrates that our GEQ model surpasses MOptEqs in achieving superior performance on the challenging task at hand and demonstrates GEQ's stability. Such advantages can be attributed to the fact that GEQ exhibits smaller output similarities compared to OptEqs when input samples are far apart (e.g., samples from different classes). This characteristic can lead to larger margins between different classes, enabling the classifier to be more easily optimized during training. Consequently, our GEQ model excels in accurately classifying difficult unseen samples, further highlighting its stability and superiority over former equilibrium models.
Ablation Studies on corrupted datasets. Apart from difficult samples, we are going to compare the robustness of our GEQ, MOptEqs, and ResNet on the CIFAR-10 corruption dataset, which contains 19 common corruptions including image transformation, image blurring, weather, and digital noises on CIFAR's test datasets. Average results on 5 degrees CIFAR-10 corruption datasets list in Figure 2.
From the result, one can see that our GEQ based on Gaussian kernels is more robust than MOptEqs and ResNet. In particular, our GEQ can show better performance against structured noise and some image transformation. The above results also demonstrate the stability of our GEQ structure.
2For example, super class "people" contains five sub classes:"baby", "girl", "man", "man", "woman"
Figure 2: The results for different models under different corruptions.
4.4 Ablation Studies on Saliency Map
The saliency maps generated by GradCAM [45] offer valuable insights into the visual attention of both MOptEqs and GEQ models. These maps highlight the regions of the image that are crucial for the model's predictions. Figure 3 presents the saliency maps obtained for an image from the ImageNet dataset using both models. Upon observation, it becomes evident that GEQ exhibits a higher degree of focus on the significant regions directly associated with the predicted label "manta". In contrast, MOptEqs tends to allocate attention to unrelated regions such as the shells. This discrepancy indicates that the attention-like module induced by the Gaussian kernel in GEQ enhances the concentration of the model's attention, resulting in improved performance compared to MOptEqs.
(a) Original Image.
(b) MOptEqs'saliency map.
(c) GEQ's saliency map.
Figure 3: Saliency map for GEQ and MOptEqs on the ImageNet image.
4.5 Ablation Studies on Patch splitting
The performance of our GEQ model is influenced by the channel splitting parameter and the patch size. Choosing large values for these parameters causes the kernel to focus mainly on global information while selecting small values makes the kernel concentrate on local features. To understand the impact of these choices on model performance, we conducted experiments, the results of which are presented in Figure 4. This figure illustrates the relationship between the channel splitting parameter, patch size, and the model's performance. By analyzing these results, we gain insights into the optimal values for these parameters that yield the best performance for our GEQ model.
The accuracy trend depicted in the figure shows an initial increase followed by a decrease as the channel split and patch size increased. Based on these empirical results, we select a patch size of 2 and a channel split of 8 for both the CIFAR and ImageNet experiments. These parameter choices are made to optimize the performance of our models on the respective datasets.
4.6 Comparison with other kernel functions
Firstly, we introduce different commonly used kernels, such as polynomial, sigmoid, and Gaussian kernels, to reformulate the hidden optimization problem for equilibrium models. Table 4 illustrates the equilibrium models induced by these kernels. It can be observed that equilibrium models with polynomial and sigmoid kernels also incorporate new attentive modules. However, their attentive
(a) Influence of the channel splitting parameter.
(b) Influence of the patch size parameter.
Figure 4: The influence on the patch size and the channel splitting parameter for our GEQ on CIFAR-100 datasets.
kernels only constrain the input features  $\mathbf{U}\mathbf{x} + \mathbf{b}$  and do not directly affect the activation of the output  $\mathbf{z}^*$ . As a result, their performance may be inferior to our GEQ model. To validate these claims, we evaluate the performance of different models on the CIFAR-100 dataset. Since the fusion module is the primary difference between MOptEqs and OptEqs, we can easily modify the structure of MOptEqs to include different kernel-induced attentive modules using the equations in Table 4, resulting in MOptEqs (Polynomial) and MOptEqs (Sigmoid). The results are presented in Table 5, which clearly demonstrates the superior performance of our GEQ model. For a more in-depth analysis of GEQ, we refer readers to the main paper.
Kernel	Hidden Optimization Problem	Equilibrium Model
Linear	minz[1Tf(z)+1/2||z||2-⟨Ux+b,z⟩-1/2||Wz||2]	z* = σ(W^T Wz^* + Ux + b)
Polynomial	minz[1Tf(z)+1/2||z||2-(⟨Ux+b,z⟩^d-1/2||Wz||2]	z* = σ(W^T Wz^* + d⟨Ux+b,z⟩^{d-1}(Ux+b))
Sigmoid	minz[1Tf(z)+1/2||z||2-tanh(⟨Ux+b,z))-1/2||Wz||2]	z* = σ(W^T Wz^* + (1-tanh^2(⟨Ux+b,z)))) (Ux+b))
Gaussian	minz[1Tf(z)+1/2||z||2-1/2e^{-γ||Ux+b-Wz||2}2]	z* = σe^{-γ||Ux+b-Wz||2}2W^T(-Wz^* + Ux + b)
Table 4: The hidden optimization problems and their related equilibrium models.  $d > 1$  is an integer denoting the polynomial order.
Model Size	Accuracy
MOptEqs	8M	75.6 ± 0.2%
MOptEqs (Polynomial)	8M	75.1 ± 0.4%
MOptEqs (Sigmoid)	8M	76.1 ± 0.3%
GEQ	8M	78.2 ± 0.2%
Table 5: Comparison of equilibrium models with different kernel functions on CIFAR-100.
In this paper, we introduce a novel optimization-induced equilibrium model called GEQ, which utilizes Gaussian kernels in its optimization-induced framework. Our model incorporates a new attentive module that arises from its novel hidden optimization problem formulation. Notably, GEQ exhibits significantly improved performance in classification tasks, outperforming deep models as well. Moreover, GEQ can be interpreted as a weight-tied model with infinite width and depth, highlighting its expressive power. We also provide theoretical analysis demonstrating the superiority of our models in terms of generalization ability and stability compared to previous OptEqs. Empirical results further validate the effectiveness of our proposed approach.
Zhouchen Lin was supported by National Key R&D Program of China (2022ZD0160302), the major key project of PCL, China (No. PCL2021A12), the NSF China (No. 62276004), and Qualcomm. Yisen Wang was supported by National Natural Science Foundation of China (62006153, 62376010, 92370129), Open Research Projects of Zhejiang Lab (No. 2022RC0AB05), and Beijing Nova Program (20230484344).
[1] Brandon Amos and J Zico Kolter. Optnet: Differentiable optimization as a layer in neural networks. In ICML, 2017.
[2] Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. In NeurIPS, 2019.
[3] Shaojie Bai, J Zico Kolter, and Vladlen Koltun. Deep equilibrium models. In NeurIPS, 2019.
[4] Minmin Chen, Jeffrey Pennington, and Samuel Schoenholz. Dynamical isometry and a mean field theory of rnns: Gating enables signal propagation in recurrent neural networks. In ICML, 2018.
[5] Qi Chen, Yifei Wang, Zhengyang Geng, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Equilibrium image denoising with implicit differentiation. IEEE Transactions on Image Processing, 32:1868-1881, 2023.
[6] Qi Chen, Yifei Wang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Optimization-induced graph implicit nonlinear diffusion. In ICML, 2022.
[7] Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, NeurIPS, 2018.
[8] KR1442 Chowdhary. Natural language processing. Fundamentals of artificial intelligence, pages 603-649, 2020.
[9] Filipe de Avila Belbute-Peres, Kevin Smith, Kelsey Allen, Josh Tenenbaum, and J Zico Kolter. End-to-end differentiable physics for learning and control. NeurIPS, 2018.
[10] Alexander G. de G. Matthews, Jiri Hron, Mark Rowland, Richard E. Turner, and Zoubin Ghahramani. Gaussian process behaviour in wide deep neural networks. In ICLR, 2018.
[11] Josip Djolonga and Andreas Krause. Differentiable learning of submodular models. NeurIPS, 2017.
[12] Zhili Feng and J Zico Kolter. On the neural tangent kernel of equilibrium models, 2021.
[13] Adrià Garriga-Alonso, Carl Edward Rasmussen, and Laurence Aitchison. Deep convolutional networks as shallow gaussian processes. In ICLR, 2019.
[14] Zhengyang Geng, Xin-Yu Zhang, Shaojie Bai, Yisen Wang, and Zhouchen Lin. On training implicit models. In NeurIPS, 2021.
[15] Laurent El Ghaoui, Fangda Gu, Bertrand Travacca, and Armin Askari. Implicit deep learning. arXiv preprint arXiv:1908.06315, 2019.
[16] Dar Gilboa, Bo Chang, Minmin Chen, Greg Yang, Samuel S Schoenholz, Ed H Chi, and Jeffrey Pennington. Dynamical isometry and a mean field theory of lstms and grus. arXiv preprint arXiv:1901.08987, 2019.
[17] Stephen Gould, Richard Hartley, and Dylan Campbell. Deep declarative networks: A new hope. arXiv preprint arXiv:1909.04866, 2019.
[18] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In ICML, 2017.
[19] Soufiane Hayou, Arnaud Doucet, and Judith Rousseau. Mean-field behaviour of neural tangent kernel for deep neural networks. arXiv preprint arXiv:1905.13654, 2019.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[21] Shell Xu Hu, Sergey Zagoruyko, and Nikos Komodakis. Exploring weight symmetry in deep neural networks. Computer Vision and Image Understanding, 187:102786, 2019.
[22] Arthur Jacot, Franck Gabriel, and Clément Hongler. Freeze and chaos for dnns: an NTK view of batch normalization, checkerboard and boundary effects. arXiv preprint arXiv: abs/1907.05715, 2019.
[23] Nikhil Ketkar and Nikhil Ketkar. Stochastic gradient descent. Deep learning with Python: A hands-on introduction, pages 113-132, 2017.
[24] Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. arXiv preprint arXiv:1711.00165, 2017.
[25] Mingjie Li, Lingshen He, and Zhouchen Lin. Implicit euler skip connections: Enhancing adversarial robustness via numerical stability. In ICML, 2020.
[26] Mingjie Li, Yifei Wang, Yisen Wang, and Zhouchen Lin. Unbiased stochastic proximal solver for graph neural networks with equilibrium states. In ICLR, 2023.
[27] Mingjie Li, Yisen Wang, and Zhouchen Lin. Cerdeq: Certifiable deep equilibrium model. In ICML, 2022.
[28] Mingjie Li, Yisen Wang, Xingyu Xie, and Zhouchen Lin. Optimization inspired multi-branch equilibrium models. In ICLR, 2021.
[29] Guangcan Liu, Shiyu Chang, and Yi Ma. Blind image deblurring using spectral properties of convolution operators. IEEE Transactions on image processing, 23(12):5047-5056, 2014.
[30] Guangcan Liu and Ping Li. Low-rank matrix completion in the presence of high coherence. IEEE Transactions on Signal Processing, 64(21):5623-5633, 2016.
[31] Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055, 2018.
[32] Juncheng Liu, Kenji Kawaguchi, Bryan Hooi, Yiwei Wang, and Xiaokui Xiao. Eignn: Efficient infinite-depth graph neural networks. In NeurIPS, 2021.
[33] Julien Mairal, Francis Bach, Jean Ponce, and Guillermo Sapiro. Online dictionary learning for sparse coding. In ICML, 2009.
[34] Lassi Meronen, Christabella Irwanto, and Arno Solin. Stationary activations for uncertainty calibration in deep learning. In NeurIPS, 2020.
[35] Radford M Neal. Priors for infinite networks. In *Bayesian Learning for Neural Networks*, pages 29-53. Springer, 1996.
[36] Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nathan Srebro. A pac-bayesian approach to spectrally-normalized margin bounds for neural networks. arXiv preprint arXiv: 1707.09564, 2017.
[37] Roman Novak, Lechao Xiao, Yasaman Bahri, Jaehoon Lee, Greg Yang, Jiri Hron, Daniel A. Abolafia, Jeffrey Pennington, and Jascha Sohl-Dickstein. Bayesian deep convolutional networks with many channels are gaussian processes. In ICLR, 2019.
A.1 Forward Procedure for GEQ
The pseudo-code for our GEQ is listed in Algorithm 1.