Optimization inspired Multi-Branch Equilibrium Models
Authors: Mingjie Li
Yisen Wang
Xingyu Xie
Zhouchen Lin
Venue: ICLR 2022 Poster (2022)

Abstract: Works have shown the strong connections between some implicit models and optimization problems. However, explorations on such relationships are limited. Most works pay attention to some common mathematical properties, such as sparsity. In this work, we propose a new type of implicit model inspired by the designing of the systems' hidden objective functions, called the Multi-branch Optimization induced Equilibrium networks~(MOptEqs). The model architecture is designed based on modelling the hidden objective function for the multi-resolution recognition task. Furthermore, we also propose a new strategy inspired by our understandings of the hidden objective function. In this manner, the proposed model can better utilize the hierarchical patterns for recognition tasks and retain the abilities for interpreting the whole structure as trying to obtain the minima of the problem's goal. Comparing with the state-of-the-art models, our MOptEqs not only enjoys better explainability but are also superior to MDEQ with less parameter consumption and better performance on practical tasks. Furthermore, we also implement various experiments to demonstrate the effectiveness of our new methods and explore the applicability of the model's hidden objective function.

Published as a conference paper at ICLR 2022
OPTIMIZATION INSPIRED MULTI-BRANCH EQUILIB- RIUM MODELS
Mingjie Li', Yisen Wang Xingyu Xie', Zhouchen Lin
Key Lab. of Machine Perception (MoE), School of Artificial Intelligence, Peking University Institute for Artificial Intelligence, Peking University
Pazhou Lab, Guangzhou, 510330, China.
ABSTRACT
Works have shown the strong connections between some implicit models and optimization problems. However, explorations on such relationships are lim- ited. Most works pay attention to some common mathematical properties, such as sparsity. In this work, we propose a new type of implicit model inspired by the designing of the systems' hidden objective functions, called the Multi-branch Optimization induced Equilibrium networks (MOptEgs). The model architecture is designed based modelling the hidden objective function for the multi-resolution recognition task. Furthermore, we also pro- pose a new strategy inspired by our understandings of the hidden objective function. In this manner, the proposed model can better utilize the hierar- chical patterns for recognition tasks and retain the abilities for interpreting the whole structure as trying to obtain the minima of the problem's 's goal. Comparing with the state-of-the-art models, our MOptEqs not only enjoys better explainability but are also superior to MDEQ with less parameter consumption and better performance on practical tasks. Furthermore, we also implement various experiments to demonstrate the effectiveness of our new methods and explore the applicability of the model's 's hidden objective function.
INTRODUCTION
Whereas Deep Neural Networks (DNNs) have achieved great success in many real-world lasks such as computer vision and neural language process, the limited interpretability of DNNs greatly hinders their further development.
However, many traditional machine learning methods, matrixx recovery (Zhang et al.
20T 2015; Lin & LT, 2016), subspace clustering (You en all Q1TT)) image deblurring (Lam
2014 and so on, can be interpreted into minimizing the following objective functions:
min f(Z) g(E), s.t. X = AZ + BE. Z.E
where A E Rmxd B mxx, X E mx and f(.) and g(.) are convex functions designed by the modelling of their properties. For this count, the interpretability of such methods is much better than DNNs. Furthermore, these methods can also enjoy state-of-the-art and robust performance on these tasks. We call such interpretability these methods enjoy as tettaticccall interpretability's ie., whether the whole network structure can be summa- rized as a compact mathematical model that can be mathematically analyzed.
Except for these methods, the Optimization Induced Equilibrium Networks (OptEqs) pro- posed by et aL (2021) recover its whole system an optimization problem. The forward propagation for OptEqs tries to solve Eqn (I) to get output y.z" E IR for input x E Rdd.. We call the front part of Eqn (M) as OptEqs' oquilibrium equation, which the central part for equilibrium models. And its forward procedure can be regarded solving the hidden objective functions shown as Equ(O),
"Corresponding author

---

Published as a conference paper at ICLR 2022
z' = =o + Ug(x)= + b), y= Waz,
(1)
min C(2 ((x) min I ((W-T12) (Ug(x)+ 12) + W 1T x1²²
1 12
(2)
W, U. We are learnable weights, g is the convolution layers projecting input x from real- world domain to feature space like other neural architectures, and f is the outputs' penalty.
Although OptEqs' interprctability is satisfying, it still remains some weaknesses worth ex- ploring. First, although OptEqs performss better than some implicit models, these models only consider a single view (or resolution) of the input in their implicit, parts. We call these models single-view implicit models in the following. However, nearly all the state-of-the ahr pattern recognition systems (Lee_ett et all, 20099; Wang et all, 2mn Huang et all 2011 He all, 2016: Burt & Adelson, 1987) benefit from the multi-layer or multi-resolution feature extractors in domains like computer vision and audio processing. Sccondly, limited new modules are proposed in their work, which mainly consider the math propertics of features like sparsity, but whether these properties can benefit image recognition tasks is unexplored.
Besides OptEqs, MDEQ (Baiel 2020) constructs its equilibriumn equation z" = =F(2" xx) and block F with the inspiration of explicit models especially HR.Net Wang et al. (201199)
Then they solve the equilibrium equation by the accelerated algorithm for output z². Al- though it shows the state-of therttrrrrttttee performance with efficient memory cost other implicit models, such strategies will make the model lose "interpretability" since the MDEQ is a black box because its output is solved in implicit way (root-finding method), which makes anal- ysis on its features of intermediate layers almost impossible. Furthermore, its complicated structure also hinders its analysis from mathematical systems. More efficient blocks with good interpretability still needs exploring.
Motivated by the limitations of the above models, we would like to design multi-scale DEQ structure with state-of ootte-of-thh-art the-art performance and preserve mathematical interpretability from the inspiration of OptEqs and multi-scale models. Our contributions are listed below:
We propose multi-branch implicit model, called Multi-branch OptEqs (MOptEqs), which efficiently utilizes different scales inputs with better performance and smaller model size. Furthermore, it still retains its connection to an optimization problem.
We propose some properties modelled as new terms for the model's hidden objective function from our analysis on the relationships between branches and their hicrarchi- cal dependencics. With new problem's 's formulation, we obtain the fusion module in our MOp, called Hierarchical Heritage and Diversity Module (HH&D Module).
Apart from the new module design, we also propose the Perturbation Enhanced strategy (PE) for our MOptEqs from our analysis on the model's hidden objective function. The new strategy cannot only enhance the performance of our MOp(Eqs, but also improve the robustness of our models.
1.1 RELATED WORKS
Implicit Models. Nearly all modern deep learning approaches use explicit models, which provide an explicit computation graph for the forward propagation. In contrast, the compu- tation graphs for implicit models are 'flexible" or can be assumed as having 'infinite" depth. For example, Neural ODEs (Chen et all 2UT Massaroli el all, 2020) encode their neural architectures by a differential system with learnable parameters. Then the implicit ODE solvers they used is equivalent to continuous ResNet taking infinitesimal steps. Further- more, the training process of Neural ODEs can be depicted as finding a differential system of certain type (like heat equations) by updating its learnable parameters which demonstrate that Noural ODE also enjoy interpretability to a cortain extent.
Moreover, DEQ (Baict all 2019 Winston & Kolten, 2021) is another class of implicit models. The central part of DEQ is the designation of the equilibrium equations z" = ; and block F. Its forward procedure is trying solve the equilibrium equations with input to
2

---

Published as a conference paper at ICLR 2022
get the equilibrium state z" output by accelerated algorithms. Since the z" is fixed given FF and x, its inference can be regarded as forwarding an explicit network stacked by the same block F for infinite times. For example, DEQ chooses their FF by one set of Conv+RRLLU while MDEQ constructs an HRNet-like block as theirs. However, no evidence shows that the best block constructions in explicit models can perform the best in the DEQ scheme. The construction of DEQ blocks or equilibrium equations is an open question worth exploring. Furthermore, most DEQ blocks do not have any mathematical insights (like the diffusion process in Neural ODE) and perform totally a black box with limited interpretability.
Since implicit models usually adopt accelerated root-finding algorithms for their forward outputs and backward gradients, they enjoy the advantages of constant and more efficient memory costs compared with DNNs. Due to the above advantages, the design of implicit models draws much attention these days (Ghaoui ellal, 2019; Gouldetaal all, 2019). Apart from the models illustrated above, many kinds of other implicit models are proposed, including differentiable physics engines (Qiao et al. 2021; He Avila Belbute- et all 2018). logical
structure learning (Wang et all 2019)) and implicit neural blocks (I et all, 2121
Model Interpretaiblity. Many researchers nowadays are trying to make their proposed method more interpretable. Although there are many kinds of approaches to achieve this goal, we marginally divide them into two parts: "Empirical" and 'Mathematical" inter- pretability. Many works, such as (Zhang et all 2ntsa Zhang & Zhu,, 218 (Bau et all, 2 attempt to empirically disassemble the black box by characterizing some statistical struc- tural information like the outputs and gradients of neural networks' middle layers. However, these works cannot be directly implemented on DEQ models since implicit ones do not have explicit depths like DNNs, which makes analyzing such models by dissecting each layer's behavior almost impossible. Apart from that, researchers (Diolonga & Krausc, 22017: Amos & Kolten, 2UTE, Xie al, 2ITH: Chan et all 202)) also managed to understand the neural architecture by linking to a mathematical problem. In this way, researchers can analyze the black box by its corresponding mathematical problem. Furthermore, new components can be proposed due to the analysis of these problems. Our model aims to design a new DEQ architecture with proper mathematical interpretability. Compared with former works, our model can perform better on the classification tasks with novel components.
MULTI-BRANCH OPTIMIZED INDUCED EQUILIRIUM MODELS
2.1 THE PROPOSED ARCHITECTURE FOR THE MULTI-BRANCH OPTEQS
Inspired by OptEqq, we first design an objective function for our tasks to solve and then use its first-order stationary conditions as MOptEqs' equilibrium equation. The base of our function is formed by summarizing several objective functions of different scales defined for OptEqs (Details are stated in Appendix I.). However, such a model can not obtain satislying results since these branches are independent. For this count, we need to design some terms describing the dependencies of each branch in the hidden objective function, as shown in the following paragraphs. Then we can obtain the equilibrium equations for our MOptEqs and our MOptEqs' block FF by analyzing the first-order stationary condition of our designed optimization problem.
Hierarchical Heritage Modeling. State-of-the-art explicit models for image tasks are ex- plicitly structured into sequential stages and process different resolutions hicrarchically (Ha of all QUTE, Shelhamer ef all, 2017 Lee ct all, 20ma)) which implics that features should be extracted hierarchically. In other words, the posterior branches should inherit from their prior ones, which we call such property hierarchical heritage. In our work, we are going to formulate such correlation of neighbouring branches in the hidden objective function by adding the following inner- inner-product term into its origin:
(3) where 2, E Rd zj E Rd, Pay E R denotes the Average Downsample (its transpose can be regarded as weighted nearest upsample) Identity matrix suited to the shape of z, and z; (i < j or = 1,i= L). This term estimates the summation for similarity of i-th and j-th branchs channels with the same channel index (corresponding channels).
H(z 2,) 2, PayZ,
3

---

Published as a conference paper at ICLR 2022
When z; and z, are similar, the inner product will be large. Otherwise, the result will become small, which implies that the relationship corresponding channels is weak. In this way, ensure the similarity of corresponding channels of the near branches. Except for the relations between different branches' corresponding channels, the relationships between dis-corresponddng channels explored in the following section.
Diversity Modeling. addition to the hicrarchical relations for corresponding channels, works (Pang et all 2019; Amada el all, 202T) have shown enhancing the diversities between branches can also improve the model's improvements. Therefore, we also consider making
different branches extract various features improve the representation abilities. To achieve such goal, we add a diversity term in the objective function Equ (HI) for <
C
D(zi 23)
Σ kg
vve ((. vec '(Ps52))(*))
(4)
where vec-1 is the inverse vectorization operator converting the vectorized feature z, R4x1 to a matrix in I a xC with C channels and vec (z, E R R² denotes the k- th channel of i-th branch in practice. Since we have already considered the relationships between the corresponding channels in the hicrarchical term, we only tries to estimate the diversitics of dis-corresponding channels of the i-th and j-th branches in this term. As the diversity term goes small, the diversities between the branches become stronger.
The architecture of MOpt..qs and its hidden problem. With the two terms we proposed, we can reformulate the hidden objective for our problem as follows:
min G(zi, . Zxig(x)) apou
min 21....g
1 s(W-1 2:) (U.g(()) b,, ",,,TTT) x)
(5)
(AD (2: 2111) || 1T2l H((x;
g(x) is the input feature for the raw input x, À > is a hyperparameter and z, E Rd:x1 are the final outputs of MOptEqs We can choose I to constrain z; on our demand and will influence model's activation function. If we choose f(x) T(x 2 0} to ensure the outputs to be positive, then the activation function is ReLU, We set D(z ZL+1) D(z z1) and H(ZL, ZL+1) H(z,, 21) when i L to complete the loop. The two terms we added tries to make the corrosponding channels of different branches correlated by maximizing the H term and enhancing the diversitics of different branches by minimizing D term. We note that H and D are not conflict since they are handling different pairs of channels for branches.
As the following proposition shows, we finally get the equilibrium equations (Eqn (E)) for our EQs by calculating the first-order stationary conditions Vz, G= for problem G.
Proposition The proposed brunch structure induced by Equ (G) can be depicted as solving the equilibrium points tt = [z] .... zf. T. E RRE for the following equations:
2 w o (Wl(Z) + Üg(x)+
W
where W
Ú UT. U.IT,b b= (b D
1.
And h(2) [h1(Z], ZL, 22) (z, Z;-1, 1.11,,411 Z(+1) ..., ...,h 1.4..2-11 21-1, (4- and each h is de-
fined as the mapping from Rd x Rd x R to R
M(2 2. 2:41)
P Pat112 + P.T 1.;11)
vec(vec I(P 24 sign - diag(M )
veo(vec (PI Mign[M, diag(M;11s]))
M vec (P (Punt1Z:41) vec -(x)),
M vec
'(x,) vee -(P (PT11.211))
4

---

Published as a conference paper at ICLR 2022
HHD Module
WioeWoo Utats) (ba)
2/
HHD Module
WielWaa Myatx) +b))
Apc
ParcPe 12%
HHD
(a) MOptEqs" structure.
(b) HH&D's structure.
Figure 1: The structure of the MOptEqs and HH&DD Module. Dotted lines in the figure denotes the upsampling, downsampling or identity operators to suit the size of each branch, denotes multiplication operator, @ denotes addition operator, RD operator processes matrix by removing its diagonal, SIGN operator convert each element of a matrix to its sign and the d, CH,W,, with C is the channel number and Hi, W; are the height and width of the feature map.
The equilibrium points for the above formala are also the first-order stationary points for the optimization problem (q
The proof of the proposition is listed in Appendix A In this manner, we obtain the proposed architecture MOpt its forward propagation is equivalent to find the equilibrium points of Equ(E), and such process can also be regarded as solving the stationary points of the optimization problem (E). The whole structure is also shown in Figure M (a) and we also draw figures to illustrate the practical process of h; (HH&D) on its right (Figure M (b)).
We also evaluate the effectiveness of such modelling in Section. 892's experiments.
2.2 THE PROPOSED PERTURBATION ENHANCED STRATEGY FOR MOPrEQS
Researches (Kong et all 2021: Xie all 2020 Tang et all 2021) have discovered that small perturbations can enhance the generalization abilities and varieties of branches for neural architectures. Most efficient works use adversarial perturbations based on gradients (Xi Yuill,, 2020) for such trick. However, such strategy will slow the training process since they need at least one more which consumnn. Unlike these mothods, we can acquire more computation efficient perturbations because our models can be formulated minimizing an objective function we designed. Then we can replace the maximum problem (details in Appendix A.2) for adversarial verturbation with maximizing since we assume MOptEq's performance will go worse if G(z; g(x)) changes a lot at g(x)'s neighborhoods since the forward propagation of MOptEqs minimization G(z; 9x)).
in each training step for perturbations,
architecture's hidden objective function G,
Assumption 1 For well trained OptEqs with its objective function denoted as Giz gxx)) and natural sample x which can be correctly classified. If an input perturbation |001 S con cause one of the following changes:
G( g(x) - 6)
2 Tic,
1| z Lac.
with L1, La > 1, il may lead the OplEqs to the wrong resulls with high probability-
For the second part of the assumption, it's common sense and widely used in the analysis of adversarial robustness (Zhang et al, 202T; Lietal 2020). With the above assumption, we can generate the perturbations by maximizing the objective function Giz: g(x) for our MOptEqs. Since MOptEqs can be regarded parallels several equilibrium models, we decide to inject the perturbations acquired for the prior branch the posterior one, like boosting to enhance the performance.
kind of "implicit" ensembling that
5

---

Published as a conference paper at ICLR 2022
Then the hidden objective function of MOptEqs with G(z:g(x)) is shown follows,
min G(Zl,,..ZL:gl/a Rioin
min EL
I(W, zi) (U:(o(x) 6 - bi, W. 1T
1=1
(AD(z, 2141) 1W-11 2:11² 2211 H1(2,, 2.111)
+
s.t. 6, = argmax 1 f(W. IT 2.) (..((( -6) 6.) b., W. W,' 2.) for i L]. 114
where d0 O. And the problem becomes a bilevel optimization problem. The solution the lower-level problem is 6; e Wi z.). Since the perturbation can be obtained by to the transposed convolution
feeding the output of the activation layer a layer with weight U,, we call it reconstructed perturbation. Compared with the adversarial perturbations based on gradients, our perturbations can be acquired directly by matrix- vector multiplication instead of iteratively built by gradients. Thus, such a process does not require much computation cost. Following the above steps, we obtain harmful perturbed direction for the prior branches and then added them their posterior branches. We note that our perturbation is added to the input feature g(x) instead of the input.
Proposition For a natural sample x and a well trained OptEqs
= (W2' + Ug(xx) -b)
with its hidden objective function denoted as G(z;g(x)), for g'(x) = g(x) and u W; z.), which obeys the constraint ||ō1|c S e, z'n and z" are defined as follows: a argmin G(2; (x)),
argmin G(z; gx)).
If the [W] |U| S and N | w 2 > c, where the element number of g(x),
then
2 - max {((x")g()) 8) - g())) VNellz" 22112]
[ w w-IT - Ne²} -Ne]
The proposition demonsttates if we choose the perturbed direction to be esign W-1T 2) at least one of the changes for the G z² will be around e|| w-ITz"ll and implies each branch may perform bad on the perturbed data according to our assumptions. With the experiments in Section and our assumption is reasonable. Like boosting strategy, we can feed the perturbed data for the prior branch to its posterior branch to enhance the performance; call such method as the Perturbation Enhanced strategy (PE). Following experiments also show that PE can indeed enhance the performance of our MOptEqs.
can conclude that our reconstructed perturbation is useful
2.3 MODEL OPTIMIZATION AND FORWARD CONVERGENCE
Forward Propagation and Implicit Differentiation. Like other equilibrium models, the forward propagation procedure is solving the given equilibrium functions Equ M. For this count, our model also enjoys the constant memory cost advantages as other DEQs. In our work, run root-finding algorithms solve the roots "" izj ...27. of the following problem to reach the equilibrium states, which is the same as MDEQ (shown in Appendix A.))
V2G(zi, Zz; g())
As for the backward propagation, we also adopt the implicit differentiation method widely used in elll, 2020; 2019 Chen all 2018). Instead of tracing the gradients during the forward propagation, the implicit differentiation method directly backpropagates through the equilibrium state using the Jacobian of Te = V2G+ at ž*, For a given loss L = L(Z",y) (where y is the target) and the gradients can be written as
01
Ø1
JT 1 θ
II
a(-)
02
a(-)

---

Published as a conference paper at ICLR 2022
Model
Model Size
Accuracy
Model
Model Size
Accuracy
Neural ODEs
172K
53.7%
ResNet-18
10M
92.9 + 0.2%
Aug. Neural ODFs
172K
60.6%
MonDEQ
1M
89.7%
Single-tire Mo
854K
82.5%
MDEQ
2.53M
92.6 + 0.2%
Deep OptEqs"
199K
87.4%
MDEQ
10M
93.8 1 0.3%
Parallel-OptEas
193K
87.4%
MOptEqs
0.48M
92.9 +: 0.2%
POptEas(sum))
193K
88.4%
MOptEqs
1.9M
94.0 + 0.1%
POp(E4s(conv)
276K
88.9%
MOptEqs
8.1M
94.6%
MOpLEqs (w/o PE)
193K
89.1%
(b) Comparison of the modelss with
MSptEqs
193K
89.5%
multiple-scales. MonDEQ here is the multi-tired monotone DEQ.
(a) Comparison of the models with single-scales
Table 1: Evaluation on CIFAR-10 for different models. Ppraalel-OptEag (POptEqs) mcans our model is built without utilizing HH&D fusion (Stated in Appendix A.J), which formed by paralleling several OptEqs with the mothod stated in the brackets for fusion. "w/o PE* means MOpsEqs is trained without our PE stralegy.
And the calculation of equation:
BC (I To 22- is equivalent to solve root for the following 85
01
mI JT +
As for the root-finding algorithm, we can use Broyden method (Broyden. (Sn)) Anderson Method (Andersom,, [1965; [aa all, 20)11) or other root-finding methods to solve Eq for the equilibrium state and Equ ( for the backward gradient.
Forward Convergence. Like other implicit models, we make some constraints on param- eters in order make the whole MOptEqs (2; (ž here is {z,H-=) be contractive mapping. For the MOptEqs without considering HH&D (Eqn M... the models can easily converge with |Will2 S <<<. 1. But since we use the HH&D module, we need choose proper À to ensure the convergence, around or less than & will be appropriate. We also conduct experiments to verify the convergence in Appendix Aw
EXPERIMENTAL RESULTS
In this section, conduct experiments for the image classificatinn tasks CIFAR- 10, Krizhevsky et all (2009) and ImageNette implemented on the Py- Torch (Paszke ata 2017) platform demonstrate the effectiveness of our model. Details are listed in Appendix A.G.
3.1 COMPARISON OF PRIOR IMPLICIT MODELS
In this part, we decide to verify the superiority of MOpt.qq in two aspects. First, build the single-scale MOptEqs and compare the empirical results with other single-scale implicit models. And then we compare the experimental results with prior implicit models that utilize the multi-scale inputs like MDEQs, the state-of-the-art implicit model with multi-resolution.
Implicit Models with Single View. In this part, we compare our MOpt with other single-view implicit models with comparable model sizes. Like OptEqs who uses three blocks for their experiment, we first construct our small MOptEqs with three branches whose outputs share the same size and the channel number (a) demonstrates that our MOptEqs structure can even enhance the performance for recog- nition tasks under the single-view cases. Compared to the multi-brance model without fusion (POptEqs), our MOptEqs' HH&D efficiently utilize the relationships between different branches and lead the model to better performance. Our superiority also holds compared with former multi branches fusion methods: "sum" (non-parametric module) and "conv" (parametric module). Meanwhile, we can also conclude the effectiveness of our per- turbation enhanced strategy from the table since its performance the best.
32. Results Table

---

Published as a conference paper at ICLR 2022
Model
Model Size
Accuracy
Model
Model Size
Accuracy
MDEQ
2.6M
70.8 1 0.2%
MDEQ
2.5M
90.5 1: 0.2%
11M
72.4 0.2%
10M
91.3 0.3%
MOpsEqs
1.9M
73.4 0.2%
MOptEqs MOptEqs
1.9M
92.1 1 0.2%
MspsEqs
8.1M
74.7%
10M
92.4%
(a) Evaluation on CIFAR-100 for models.
(b) Evaluation on Imagenette for models.
Table 2: Evaluation CIFAR-100 and ImageNette for MDEQ and MOptEqs.
Implicit Models with Multiple Scales. Moreover, we conduct experiments compared with other models which handling multi-scale inputs. Like MDEQ, we construct our MOptEqs with four branches with resolution size equals to 32,16,8,4. Other details can be found in Appendix A.G. From Table (b) and as shown in Table (a), one can see that our MOpt not only outperforms the widely used explicit model ResNet-18 (He et all 2016), but also shows better performance compared with MDEQ with fewer parameters on CIFAR, which is one of the best models. The empirical results for the multi-scale models further verify the superiority of MOptEqs and its strategy.
Apart from experiments on small images, we also conduct experiments on ImageNNette, which a subset of 10 classes from ImageNet. Compared with MDEQ, our MOptEqs consistently perform better shown in Table (b). Nevertheless, one see from the results, the difference between MDEQ and our model becomes much bigger in the CIFAR-100 and Imagenette with the same training hyper-parameters. Such phenomenon demonstrates that the performance of our model is much more stable than MDEQ. Apart from these experiments, we also conducted ablation studies for our models in the following section.
3.2 THE COMPREHENSIVE UNDERSTANDING OF МÖPТEQS
Asange
Sanall Inder
(a) MOpLEqs.
(b) Papt
(c) MOptEqs.
(d) POptEqs.
Figure 2: Visualization of the channels correlations for the MOptEqs and POpt
Visualization of the impact for the HH&D module. In this section, try to estimate the effect of our HH&D module. We finish experiments in this part using branch MOptEqs with 16 channels for each branch to make the visualization more clear.
We tries to verify our HH&D module's effect in two aspects. First, we plot scatter figures for
the values of Σl ΣjEN(O D(a,,,) (denoted as Average D) and EL, ExEN(o H(z..2,) (denoted L(C"-C)
LC
as Average I/) for 100 randomly chosen samples (Nii) denotes neighboring branch of the i-th branch). Since the hidden optimization problem is to maximizing 7. while minimizing D, we find that our MOptEqs (Figure (a)) can induce the output features to reach such goal compared with POptEqs (MOptEqs without HH&D) (Figure ((b)).
Furthermore, we also plot heatmaps of the first two branches' correlation score ( (22)
E R16x16) for a randomly chosen sample shown in Figure (d). One can see that MOptEqs' heatmap (Figure (cc)) more likely to be a diagonal matrix while heatmap Figure (dd) for POptEqs looks random. The heatmap shows that our HH&D can induce the model to perform as our demand, which means corresponding channels for ad- jacent branches due to our architecture design. The visualizations and former results on different datasets verify the effectiveness of our modelling and our HH&D modules' design.
be more correlated, while the dis-corresponding channels are unrelated
8

---

Published as a conference paper at ICLR 2022
60
89
Noptle toaved who Noythi
53
80-
Tees Accuurccyyy
(50
NDytt (146.5)
80
Tecerstucd trtarbltin
PLUE raaia miggernee Lese
30
10a
10
10
10a
Perturbation Size
Iteration
(a)
(b)
(c)
Figure 3: (a) Test accuracy changes with respect to the perturbation size for different perturbed directions. (b) Test Accuracy for small MOptEqs (w. and w/o. PE) under PGD attack with different inner iterations.
(c) Plot of the accuracies for the small MOptEqs without HH&D trained by the extended loss with respect to various
Evaluation of the reconstructed perturbation. We compare perturbation with randomly generated ones in a one-branch well-trained MOptEqs to validate its effectiveness. We first feed a natural sample x to the model and acquire output for and generate our reconstructed perturbation using such output and then feed to the model. Furthermore, add the perturbations generated by Binomial distribution (P(6, = = IP(6; -c) 0.5) and Uniform distribution e[- to input features g(x) for comparison. The result of each distribution averaged for five trials. Figure ((aa) shows our perturbation generated by maximizing G' is much more effective, which validates the effectiveness of our reconstructed
perturbations and the rationality of our assumptions and analysis in Section 2n2.
Robustness of our MOptEqs trained by PE strategy. In addition to improving the generalization abilities for models as shown in S.Il perturbations with proper size may also enhance the robustness of our MOptEqs, shown in FigureB(b) We conduct experiments on small MOptEqs (the same setting as B.T's single-view model). Our perturbation enhanced strategy goes stronger increases. The figure shows that accuracy for the model trained naturally drops quicker than trained by PE strategy. Overall, trained with appropriate reconstructed perturbation can partly improve the robustness of our MOptEqs.
MOptEqs vs. adding Regularizers in training loss. Apart from our HH&D modules, adding D and H. to the training loss is one of the most popular methods for obtaining outputs with certain properties. In order to demonstrate the superiority of our module empirically, conduct experiments for MOptEqs trained by cross-entropy and POptEqs (MOptEqs (w/o. HH&D)) trained by the cross-entropy loss adding 7 Et= (H((,, 2141) XD(z,, 2(+1)) as regularizers (we call it augmented loss). Then we drew its test accuracy for different trained by the augmented loss (A is the same as ours) shown in Figure ((()).
With proper 7, POptEqs trained by additional loss perform better than 7 0. Such phenomenon demonstrates the effectiveness of our HH&D Modelling. Furthermore, the figure demonstrates the superiority of our model since the traditional method is consistently worse than ours. We left some other explorations for our model in the Appendix.
CONCLUSION
We introduce the multi-branch optimization induced equilibrium models (MOptEqs), a new extension of Opt that can utilize multiscale information for recognition tasks and retain its ability to recover to an optimization problem whose solution is equivalent. the equilib- rium states of our model. The model architecture is designed based modelling the hidden objective function for the meltiiresolution recognition task. Furthermore, we also propose new strategy inspired by understandings of the hidden optimization problem. The empirical results show the advantages of our proposed methods. The success of our HH&D module and PE strategy demonstrates the deep link between the optimization problem and neural architecture and may motivate further explorations.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
Yisen Wang is partially supported by the National Natural Science Foundation of China under Grant 62006153, Project 2020BD006 supported by PKU-Baidu Fund, and Open Re- search Projects of Zhejiang Lab (No. 2022RC0AB05).
Zhouchen Lin is supported by the NSF China (No. 61731018), NSFC Tianyuan Fund for Mathematics (No. 12026606), Project 2020BD006 supported by PKU-Baidu Fund, and Qualcomm.
REFERENCES
Takuma Amada, Kazuya Kakizaki, Toshinori Araki, Seng Pei Liew, Joseph Keshet, and Jun Furukawa. Adversarial robustness for face recognition: How introduce ensemble diversity among feature extractors? 2021.
Brandon Amos and J Zico Kolter. Optnet: Differentiable optimization as a layer in neural networks. In International Conference on Machine Learning, pp. 136-145. PMLR, 2017.
Donald G. M. Anderson. Iterative procedures for nonlinear integral equations. Journal of the ACM, 12(4):544 560, 1965. doi: 10.1 10.1145/321296.322 321330.. URL http://ablp. uni-trier de/db/journals/ jacm/
aaaaaaa/aaaaaaaaaaaaaaaaaaaaaaaa jacm htm.m.ttmll... #Andersonn..
Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. Deep equilibrium models. In Hanna M. Wallach, Hugo Larochelle, Alina. Beygelzimer, Florence d' A'Achh--Bu,, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Newral Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 688 699, 2019.
Shaojic Bai, Vladlen Koltun, and J. Zico Kolter. Multiscale deep equilibrium models. In Hugo Larochelle, Marc' Aurclio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020.
Shaojie Bai, Vladlen Koltun, and J. Zico Kolter. Stabilizing equilibrium models by jaco- bian regularization. CoRR, abs/2106.144 14342, 2021. URL
14342
David Bau, Jun-Yan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B. Tenenbaum, William T. Freeman, and Antonio Torralba. GAN dissection: Visualizing and understanding genera- tive adversarial networks. CoRR, abs/1811.10597, 2018. URL http://arxiv: wwwprivi.v..r/veeo abs/
181. 10597
Charles G Broyden. A class of methods for solving nonlinear simultaneous equations. Mulh- emotics of computation, 19(92):577-553) 1965.
Peter J Burt and Edward H Adelson. The laplacian pyramid as compact image code. In Readings in computer vision, pp. 671-679. Elsevier, 1987.
Kwan Ho Ryan Chan, Yaodong Yu, Chong You, Haozhi Qi, John Wright, and Yi Ma. Deep networks from the principle of rate reduction. arXiv preprint Niv:2010.147000 2020.
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grau- man, Nicolò Cesa-Bianche and Roman Garnelt (eds.), Advances in Neural Informalion Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 6572-6583, 2018
Filipe de Avila Belbute-Peres Kevin Smith, Kelsey Allen, Josh Tenenbaum, and J Zico Kolter. End-to-end differentiable physics for learning and control. Advances in neural information processing systems, 31:7178 -1889, 2018.
10

---

Published as a conference paper at ICLR 2022
Josip Djolonga and Andreas Krause. Differentiable learning of submodular models. Advances in Neural Information Processing Systems, 30:1013-1023 2017.
Zhengyang Geng, Meng-Hao Guo, Hongxu Chen, Xia Li, Ke Wei, and Zhouchen Lin. Is attention better than matrix decomposition? In International Conference on Learning Representations, 2021.
Laurent El Ghaoui, Fangda Gu, Bertrand Travacca, and Armin Askari. Implicit deep learn- ing. CoRR, abs/ 11119998803.33. 2019.
Stephen Gould, Richard Hartley, and Dylan Campbell. Deep declarative networks: A new hope. CoRR, 0bss/190900450 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pottern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016. doi: 10.1109/CCV 2VP..200....0
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In 2017 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 2261-2269. IEEE Computer Society, 2017. doi: 10. 10.1100/CVPR.2012222.
Kezhi Kong, Guohao Li, Mucong Ding, Zuxuan Wu, Chen Zhu, Bernard Ghanem, Gavin Taylor, and Tom Goldstein. Flag: Adversarial data augmentation for graph neural net- works. arXiv preprint ar Xrxiv:2010.0900 0..999, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Honglak Lee, Roger Grosse, Rajesh Ranganath, and Andrew Y Ng. Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Tn Proceedings of the 26th annual international conference on machêne learning, pp. 609-616, 2009a.
Honglak Lee, Roger B. Grosse, Rajesh Ranganath, and Andrew Y. Ng. Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In An- drea Pohoreckyj Danyluk, Léon Bottou, and Michael L.. Tittman (eds.), Proceedings of the 26th Annual International Conference
Machine Learndng, 1CML 2009, Montreal, Que- bec, Canada, June 1 14-18, 2009, volume 382 of ACM International Conference Proceeding Series, pp. 609-616. ACM, 2009b. doi: 10.1145 10.1145/1553374.1553 11//115533331353555
Jia Li, Cong Fang, and Zhouchen Lin. Lifted proximal operator machines. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 4181 4188, 2019.
Mingjie Li, Lingshen He, and Zhouchen Lin. Implicit euler skip connections: Enhancing adversarial robustness via numerical stability. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 5874-5883. PMLR, 2020.
Guangean Liu and Ping Li. Low-rank matrix completion in the presence of high coherence. IEEE Transactions on Signal Processing, (4(21):5623 5633, 2016.
Guangean Liu, Shiyu Chang, and Yi Ma. Blind image deblurring using spectral properties of convolution operators. IEEE Transaction image processing, 23 12):5047-5056, 2014.
Stefano Massaroli, Michael Poli, Jinkyoo Park, Atsushi Yamashita, and Hajime Asama. Dissecting neural odes. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria- Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Process- ing Systems 33: Annuual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020.
Michael C Mozer. A focused back propagatnnn algorithm for temporal pattern recognition. Complex systems, 3(4):349 381, 1989.
11

---

Published as a conference paper at ICLR 2022
Tianyu Pang, Kun Xu, Chao Du, Ning Chen, and Jun Zhu. Improving adversarial robustness via promoting ensemble diversity. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 4970-4979. PMLR, 2019.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differ- entiation in pytorch. In NIPS- W, 2017.
Yi-Ling Qiao, Junbang Liang, Vladlen Koltun, and Ming Lin. Scalable differentiable physies for learning and control. In International Conference on Machine Learning, pp. 7847-7856 PMLR, 2020.
Evan Shelhamer, Jonathan Long, and Trevor Darrell. Fully convolutional networks for semantic segmentation. IEEE Trans. Pattern Anal. Mach. Intell., 39(4):640 6510, 2017.
doi:
Shiyu Tang, Ruíhao Gong, Yan Wang, Aishan Liu, Jiakai Wang, Xinyun Chen, Fengwei Yu, Xianglong Liu, Dawn Song, Alan Yuille, et al. Robustart: Benchmarking robustness on architecture design and training techniques. arXiv preprint ar KKf::209.052211 2021.
Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, Wenyu Liu, and Bin Xiao. Deep high- resolution representation learning for visual recognition. CoRR, 0as/1908.079199 2019a.
Po-Wei Wang, Priya Donti, Bryan Wilder, and Zico Kolter. Satnet: Bridging deep learning and logical reasoning using a differentiable satisfiability solver. In International Confer- ence Machine Learning, pp. 6545 6554. PMLR, 2019b.
Ezra Winston and J. Zico Kolter. Monotone operator cquilibrium networks. In Hugo Larochelle, Marc' Aurclio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tion Lin (cds.), Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NewrIPS 2020, December 6- 6-12, 2020, virtual, 2020.
Cihang Xie and Alan L. Yuille. Intriguing properties of adversarial training at scale. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Elhiopia, April 26-30, 2020. OpenReview.ne,, 2020.
Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan Yuille, and Quoe V Le. Adver- sarial examples improve image recognition. In Proceedings of the IEEE/CV Conference on Computer Vision and Pallern Recogrilioo,, pp. 819 828, 2020.
Xingyu Xie, Jianlong Wu, Guangean Liu, Zhisheng Zhong, and Zhouchen Lin. Differen- tiable linearized admm. In International Conference on Machine Learning, pp. 1911- PMLR, 2019.
Xingyu Xie, Jianlong Wu, Guangean Liu, Zhisheng Zhong, and Zhouchen Lin. Optimization induced deep equilibrium networks. arXiv preprint arXiv:2105. 13228, 2021.
Chong You, Daniel Robinson, and René Vidal. Scalable sparse subspace clustering by orthogonal matching pursuit. In Proceedings of the IEEE conference on compuler vision and pallern recognilion, pp. 3918 3927, 2016.
Bohang Zhang, Tianle Cai, Zhou Lu, Di He, and Liwei Wang. Towards certifying I infinity) robustness using neural networks with infinity-dist neurons. CoRR, abs/2102.05363,
2021.
Hongyang Zhang, Zhouchen Lin, Chao Zhang, and Edward Y Chang. Exact recoverability of robust pea via outlier pursuit with tight recovery bounds. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
12

---

Published as a conference paper at ICLR 2022
Quanshi Zhang and Song-Chun Zhu. Visual interpretability for deep learning: survey. arXiv preprint arXiv:1802 100.0440, 2018.
Quanshi Zhang, Ying Nian Wu, and Song-Chur Zhu. Interpretable convolutional neu- ral networks. In 2018 IEEE Conference on Computer Vision and Pattern Recogni- tion, CVPR 2018, Sall Lake City, UT, USA, June 18-22, 2018, pp. 8827 8836. Com- puter Vision Foundation / IEEE Computer Society, 2018a. doi: 10.1109/CVPR. 2018.00920. CRL kttp://openaccees: thecvf. wtw./ccooccooo com/content, cvpr ctwp_cveeesseeeeee 2018//ttl//hhaeee Interpretable Convolutional Neural CVPR 2018 paper .html.
Xiao Zhang, Lingxiao Wang, Yaodong Yu, and Quanquan Gu. primal-dual analysis of global optimality in nonconvex low-rank matrix recovery. In International conference machine learning, pp. 5862-5871, PMLR, 2018b.
13

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 PARALLELING MULTI-BRANE OPTEQS
Experiments show that even simple one-layer OptEqs module enjoys impressive generaliza- tion abilities. we view a simple one-layer OptEqq module as a powerful feature generator, we can use an 'ensemble' scheme on the DEQs by paralleling several DEQ modules. Then we get the multi-branch implicit structure (formulated as Equ M.), which is capable of utilizing samples of tmulti-reoluuton but also retains the capability of recovering to an optimization problem.
zj W]o Uig(x) +
(10)
z WoWW + ULg(x) bz),
where z denotes the oquilibrium outputs of i-th branch, we call this structure Parallel- Opt (qSptEqs). Like (Xie_et_a) 202T) stated for OptEqs, the POptEqs also be depicted as solving the following problem (Equ II] W, invertible and o(-) activation function,
min G(z ZLig(x)) = min ZL
f[W," 1T 2;) - (U.g(x) + b;, W.
21
==1
(11)
1
+ 2
| 1T
2
12
2
where gxx is the imput feature with raw imput z, E 12dx1 x1 and f added for the activation function. For example
the positive orthant f(x) = If= 2 0} we use ReLU activation, since (I + Of) is the ReLU activation function, other activation functions can be found Li et all (2019). The calculation of the above structure (Equ M.) is equivalent to solving the stationary points Vx, 0.
A.2 ADVERSARIAL PERTURBATIONS.
Adversarial perturbations ions aim to cheat well-trained neural networks with small or unnolice- able changes on natural images. The adversarial perturbations can be obtained by solving the following maximum problem for input Xnat and label y:
max C(fNN(X,,) 8); SEC
where C is the feasible set of 8 in case the change is too large and usually chosen to be an l-infinite ball. Due to the neural network is a non-convex function for 6, obtaining the perturbation is not easy. The most popular method is the PGD-k method which solves the above problem by implementing the projected gradient method by lo times. Although the performance of such method is satisfactory in most cases, it makes the whole training procedure + 1 times slower than the origin. In our work, we replace the above objective function with our model's hidden objective function. Since the hidden optimization prob- lem is convex with respect to the perturbation, we can get the perturbation much easier. Moreover, such perturbation can also enhance the generalization abilities, shown in the experiments.
A.3 PROOFS FOR PROPOSITION
In this section, we are going to prove that the equilibrium equation Eqn also the first order stationary points for the optimization problem
14

---

Published as a conference paper at ICLR 2022
Proof The equilibrium equations Eqr M in the proposition con be seporated into different branches as follows:
21 W. a (W2h1 (21, + U1g(x) b1),
W, a (W,h; (Z,, 2, + + b;),
(12)
21 W a (W The (ZL, 21-1, + ULg(x) + bL),
where h4 RR Rdd+ Rdd-1 Rd can be defined as:
1
h; h;(2;, (1,4; 2z-11
(P 1a++17++1 P P. 1-1
vec (vec -1 ((P (Pw4112+1)ssi sign M diag
(13)
A
vec(vec- -1 (P.-1:Z-- sign M diag(M,
2
where
M = vcc ( (Pis-12z41) Tvoc
vvc (z,).
Firsl, we need to prove that
h(z,, Zi-1, Bi+1) = V2,
Σ
(DD(2,, - H(z, 2i+1)]
(14)
i=1
The derivatives for the right term can be divided as the derivatives for H. and D:
H[Zz, 22 Σ z P 1k 1k++1
Vx. (z. 121+ + 2T_, P. 11))))
Pus+12i41 + + 1-12---11
which the first term of hi.
Then we are going colculate the derivatives for D:
1
D(zk, 2 2 = Vz,
vec ((2z)(1)T vec vec-1(PPaatp ((Pk44126+1) (k2)
k= ki= k2
Ivec (2; vec (P 1264 (tea)
x;
vvc (roc-((z--1))) vcc vec-1(B vPP-1(Pi-
=1 K2
vec Vvcc-1 (x))
|vec (2,) vec ((P5+1144)))
k1=1 katki
vve--1(z--)) (Zz (Ki) vec - (P--1z ]]
vvec
11 kg
with the following equation holds for the sampling matrir Pi-1 and sampling is done for each channel independently:
vec vec- (2;-1) (k:)T vec
(P:-1122))k))) (
vec VPPT 1111,) 11 ) TT vec (2,)
15

---

Published as a conference paper at ICLR 2022
The equation can be formulated as:
V 2,
Dizk, zk+1) vec Vvec-1(z:)
Σ Σ vec (2:) vec "Pee"(PPrn (P2s412441)(
= kogthy
vec (P;-1;;--1) (K4)T vec vec (z;;
Since
Vvec-1(z.)
vec |vec-(z;)) (2;
vec vec (oll)Tvee--ee (Pes411.+1) (Kz))
k1 11 k2
Vvec 1(x))())
Σ
vec (2,) vec ( vec--
(Pott Vvec 1(x,)00:
#1= ka
sign(vec vec (Pss1(((r1))))))))))))) " (Pst412244)
sign(vec' ( vec (Ps t( 11 (hz))
ka CC
vec (P iit 2i Jsig vvc - '( (Pr++12:41) vvc diag(vec (114122+1) -vec vec (
= (P (PP++1 sigg [M diag (Mi++1))
In the same manner, we can get,
V vec-1 (x))
vec (P (PTT (K:) vec (z,)
= ( (4--
diag(M;-a))
Taking all the terms together, we've proved the correctness of Equ L4 Then can proved the relationships between Eqn IB and the first-order stationary points for Equ W. Take z; for an example:
0 V n
z) + 1 /TWW 2,) (U.)(x) + b,, IT
(AD(z,, 2i41) + W-11 2 H(zz Z 1))
2=1
0 (I+ - b.) -1(zi,,, 2i 2i+1)
Then we can get the equilibrium equation:
2, W. (W (WW,,lz,, 21,, + Uzg(x) + bb).
the proof is the same for all i. In this manner, we proved the relations between the structure and the optimization, problems. The proof for the proposition is complete.
A.4 PROOFS FOR PROPOSITION
Proof 2 From the preliminaries above, the difference belween G(z**g((x) 8) and G(x)) with 8 ( is as follows: ((x 8) G(z":g(x)) Since x² is the minimizer of G al g(x), the above function can be converted as follows:
g(())) G(z): (csign(U w IT 2"), U w W-TT"" 2T
G(z":g(x) 8) G(z"; :x()) > ( (e ToeenuuT-iie WT U 1T
= U w IT x (csign(U W 1T 2²), U NTWW- 1T -T - -z'))
16

---

Published as a conference paper at ICLR 2022
From the structure of OptEqs, can get:
GSIGN(U W W U W IT W- (2 - =2")
u W-ll 1T (z" 211]111
u W-1T (z" - 2
S (o(Wz' Ug(x) + +b) o( U(g(x) 8) + b)) ll2
SVNello(W2 + +Ug(x) + +b) - oW2" U(g(x) - 6) + bbll2
SVNcl Ug(x) +b Wz - U(((x) - bbll2
WVNelIW(z* -z")+ + +U0l2
WVNe -2") /Ne U8
SVNcllz" z 112 N22
The inequality is aequired because of the Lipshilzness of common activalion function (ReLU U and Leacky ReLUU) and the bounds weight malrices. With the above resulls, we can get:
G(z': 6) - -G(zz: C z VNellz" =2'12-N0 Ne²
Then we can obtain the proposition:
max ((zz" - G(zz)g(())) -
1 -2 fellUT w-IT
- 2"111 Ne²}
The proofs are similar when considering the HH&D modules.
A.5 FORWARD PROCESS OF OUR MOPTEQS' IMPLICIT PART.
The forward propagation is illustrated as follows:
Algorithm Forward Process of our MOptEqs implicit part.
Require: Input x, initial points {z, perturbation size c, 6g 0, maximum perturbation size
Ensure: Equilibrium points(2:]{
Calculate the imput feature g(x).
The calculation of V2G(2;g(x)) = the same as finding the equilibrium points
To V+++2=2
Use Broyden (Broyden (+66 or Anderson Method (Anderson, (M or others to find
roots of the following equation like MDEQ:
To(2; =
return ž
{z [m
And the calculation of To is shown follows:
17

---

Published as a conference paper at ICLR 2022
Algorithm 2: The Calculation of To V&G at (x))
Require: Input feature g(x), initial points {2i perturbation size c, dg 0, maximum perturbation size e. Ensure: Output Tg VzC(Z;g(x)) +
for i E R do h1, k2 t neighbor of in L-loop
Calculate Zpi by h; (Shown in Figure D (b)):
Zpi s- (2; 2 21
if Use PE Method then
yr - o (W.z ppi + U.(g(x) - 81-1) + bi)
d; + cS.GN(U?
else
y: - oo + Ugg(x) + b,)
end if
2: < W. yi
end for
return (z M
A.6 EXPERIMENTS DETAILS OP MÖPTEQs FOR THE EXPERIMENTS CLASSIFICATION.
A.6.1 CIFAR-10
For classification tasks of single-view models, we set the channel number of each branch to 32, À = 0.01 and perturbation size e = 0.1 for the small MOptEqs with three branches for the single-view comparison as OptEqs. The batch size is set to be 128 for all the experiments.
As for the multi-scale models' comparison, we use four branches with each branch take imputs with resolutions equals 32, 16, 8, 4 like MDEQ, the output channel number for each branch 256 for MOptEqs with 1.9M learnable parameters. At the same time, we set the channel number to be 128 for MOpt with 0.48M learnable parameters. And set = 0.001 and perturbation size c 0.1 for the experiments.
We use the widely used SGD algorithm for training procedure. We set weight decay to be 0.001 and the initial learning rate start from 0.1 and decay by 0.1 at 100, 150, 175-th epoch with 200 epochs in total like others. We use the standard data augmentation for all the experiments.
Our implementation for small MDEQ only changes the channel number of MDFQ's 10M model to obtain the comparable size with our model, without changing the training settings and its pyree-parameters.
A.6.2 CIFAR-100
Except for CIFAR-10. 10, we also finish the experiments on CIFAR-100 classification to fur- ther verify our model's effectiveness. The model we used is the same as the models used in the multi-scale comparison, only changing the output from 10 to 100 for classification. MDEQ with 11M parameter is the same model as (Bai of all 2020) proposed for CIFAR-10 experiment with only changing the output from 10 to 100 for CIFAR-100. Moreover, the hyper-parameter setting is also the same as (Bai all 2021) proposed in the paper.
We the widely used SGD algorithm for training procedure. We set weight decay to be 0.001 and the initial learning rate start from 0.1 and decay by 0.1 at 100, 150, 175-th epoch with 200 epochs in total like others. We use the standard data augmentation for all the experiments.
Phttps//xww..estorooto... kriz/cifar. html
18

---

Published as a conference paper at ICLR 2022
10
10
Relative Error
10
Relative
10
10
A=0.005
A=0.01
A=0.1
10
10
10
30
40
50
10
20
30
40
50
Iteration Numbers
Iteration Numbers
(a) Plot of small MOptEqs' convergence with respect to different A.
(b) Plot of multi-scale
MOptEqs (A 0.001) convergence.
Figure 4: The convergent iterations' of our forward root-finding procedure used for our MOptEqs (A = 0.001) forward propagation. Relative error is defined |
17
A.6.3 IMAGENETTE
Besides the classification for images with small size, we also conduct the experiments on the Imagenette which is a subset of 10 classes from ImageNet with 9469 training photos and its test set consists of 3925 images. Furthermore, we use the full-size version of Imagenette further verify the superiority of our models on large-scale images. The model architecture for our MQp and MDFQ (for comparison) is the same the ones used in the multi- scale models' comparison for CIFAR-IC 10 except adding two downsampling layers in the head of models to downsample the input size from 256 to 64. As for the syper-parametent only change the weight decay to 5e -5, batch size to be 32 and extend the whole training epochs to 200 with 100-th, 150-th, 175-th for the lcarning rate docay for all the models. All the aper-pprrseetess for both MOptEqs and MDEQs are hardly changed during all the datasets.
A.7 CONVERGENCE VALIDATION FOR MÖPTËQS.
As shown in Figure (a), the Anderson method can quickly find the equilibrium points for our small MOptEq used for single-view comparison. From the figure, one can see that MOpLEqs with proper À (around smaller than I T around 0.03 in this circunstance) can ensure the whole mappings be contractive. But too large À will make the model hard to converge or even fail.
Apart from plotting the convergent iterations' of our forward root-finding procedure for the small MOptEqs (shown in Figure (a)), we also draw figures to show the convergence of the forward propagation for our multi-scale MOptEqs (A = 0.001) used in the imagenette and CIFAR experiments Figure g (b) shown. We use the pro-trained model Imagenette whose prediction accuracy is reported the above sections. One can see that our multi-scale MOptEqs also quickly converge the equilibrium points we expected.
A.8 ABLATION STUDIES ON THE IMPACT OF D AND H. PART
In this section, we analyzed the influence of our D and H term for single-branch MOptEs model trained without PE method shown Table. 3 We set X 0.02 for the experiment. From the table, one see that the contribution of the H term is slightly higher than the diversity term. But considering both two terms can significantly improve the results comparing with MOptEqs without considering these two terms.
3 "https:/ htass://gittub.ooo www./oossaaaaeoc hot.a///....ssaa///auttttee /imagenettee/ 'htrps:/ www. image-net.org/
19

---

Published as a conference paper at ICLR 2022
Model
Natural Accuracy
Esp
89.1%
D
88.6%
H
88.4%
D,
87.4%
Table 3: Evaluation on MOptEqs trained with and without considering D and part.
A.9 EXPERIMENTS ON THE IMAGENET
Except CIFAR and Imagenette, we also conducted experiments on ImageNet with 13M parameters shown as below:
Model
Model Size
Natural Accuracy
ResNet
13M
70.3%
HR-Net
13M
72.3%
MOptEqs
13M
72.5%
Table 4: Evaluation on MOpLEqs on ImageNet.
From the results, one that our model achieve the satisfying performance ImageNet compared with the statc-of-the-art modcls.
A.10 PE METHOD AND ROBUSTNESS
From the above experiments, one bustness of our MOptEqs. In this section, we conduct experiments to see whether the PE methods can further enhance the robustness of MOptEqs when use adversarial training methods on these models. We finish comparisons of PGD-3 training for the single-view MOptEq with or without PE, and the result is shown as follows: From the table, we can
that the PE method may also improve the
Model
Natural Accuracy PGD-20 Accuracy(c 255
PGD-3 w/o PE PGD-3 w. PE
78.60%
37.18%
78.40%
38.05%
Table 5: Evaluation MOptEqs trained by PGD-3 with and without PE method.
conclude that our PE method can improve the robustness of our MOptEqs model trained by the adversarial training method. Such ad advantages also demonsttate the superiority of our models' mathematical interpretability since the PE method acquired based on the model's hidden optimization problem.
A.11 OTHER TRAINING METHODS: FIXED POINT METHOD AND ONE-STEP GRADIENTS
A.11.1
METHODS
Since our model is not so complicated, we can use the fixed point method for iteration for the forward propagation. While for the back propagation, we can use BPTT (Mozed, (ا88]) and (Geng el all 202T) proposed method, which only backward the final forward iterations using the chain rule:
ac ac aT a(-) 0T a(-)
where E are the learnable parameters for the implicit models, T denotes the output for the I-th fixed-point iteration. Since the back-propagation of such strategy is similar
20

---

Published as a conference paper at ICLR 2022
to backwards only single layer of DNN module without the calculation of root-finding algorithm for the Jacobian in the implicit differentiation, the computational cost much less than the implicit method. However, the gradient approximated by such method is not accurate and may cause the performance drop. However, as we illustrated below, such method can be regarded trade-off if the computational resources of training not enough.
A.11.2 EMPIRICAL RESULTS AND COMPUTATIONAL EFFICIENCY ONE-STPE GRADIENT
We take the ImageNette dataset an example, the training memory cost, forward time, and test accuracy are shown as follows (each batch contains 32 images and is finished on GTX 1080Ti for 200 epochs):
Model
Memory/Batel Infer Time/ Total Time
Accuracy
MDEQ
8GB
1.06s
9h
91.42%
MOptEq(one step)
3GB
0.26s
3h
91.21%
Table 6: The comparison of the compulational cost of MDEQ and our MOptEqs (trained
by one-slep gradient).
The results shows that the one-step gradient methods can save a lot computation resources while with slightly drop the performance.
21


Reviews:
[{"review_id": "g18hgKrWxp9", "reviewer": "Reviewer_8Er5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n* Even though incorporating different input resolutions to improve the performance of the model is not novel, I believe that the design of the architecture to incorporate multiple scales is still an interesting problem.\n* I checked the proofs and I believe that the work is technically correct. \n* Empirically, the proposed method achieves impressive results on CIFAR-10, CIFAR-100, and ImagImagenette and it is convincing that the proposed method outperforms MDEQ on these small-scale experiments.\n\nConcerns:\n* I have one concern about the additional computational/memory overhead of “unrolling”. I believe that the model size (number of parameters) is relatively small for CIFAR-10 experiments. However, for larger models (and more complex datasets such as ImageNet), do you think your method would scale to these settings?\n* Following up on the previous concerns, I recommend the authors clearly state the additional overhead (wall-clock time/additional memory) in the results section to describe the potential limitations of the approach.\n* Another concern I have is that, since the proposed method is trained by unrolling, I wonder if it is fair to compare against MDEQ? I believe that the direct comparison is unfair. For example, the proposed method does not have useful properties of equilibrium models (e.g. constant memory). However, we are solely comparing the performance between MDEQ and the proposed approach. \n* In the introduction, the authors mention that OptEqs have “mathematical interpretability”. Does this necessary correspond to, is it easier to interpret what the network is doing? If not, what is the core benefit of “mathematical interpretability”?", "summary_of_the_paper": "The paper introduces a multi-scale network (called Multi-Branch OptEqs or MOptEqs) that extends on recent results of [1]. Specifically, the authors propose to use several DEQ models to incorporate samples of multi-resolutions. The authors further propose a modified training strategy (called perturbation enhanced training) to improve the performance of the model. Empirically, the proposed model shows better performances compared to the previous DEQ models on CIFAR-10, CIFAR-100, and Imagenette datasets.\n\n[1] Xie, Xingyu, et al. \"Optimization Induced Equilibrium Networks.\" arXiv preprint arXiv:2105.13228 (2021).", "summary_of_the_review": "I believe the idea of the paper is interesting and the authors show the impressive results on CIFAR experiments. However, there are some concerns regarding scalability (since the proposed method performs unrolling), additional overhead, and experimental fairness. Most importantly, the proposed approach is trained with unrolling to a fixed depth and does not have a property of equilibrium models. Hence, I give a score of 5.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "aM8UGeRdVZS", "reviewer": "Reviewer_2WhH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrengths: \n1. Performance: The proposed method obtains decent improvement in performance and reduction in model size as compared to prior work on this topic.\n2. Explainibility and ablation studies: The detailed ablation study shows the effect of design  choices such as Hierarchical Heritage and Diversity modules, perturbation size and regularizers.  \n\n\nWeaknesses/Questions\n1. Writing: The paper is not well-written and organized. The key ideas are not properly organized and explained. For someone who is not familiar with the implicit models literature, I had to spend  time carefully re-reading the paper to make sure that I understood all the details.\n2. Somewhat limited novelty?: The key ideas proposed in the paper (multi-scale, Hierarchical Heritage, Diversity) are quite common in the explicit models (i.e., standard neural networks). It is not clear to me if adapting them directly to implicit models is quite a significant advance.\n3. How would the approach scale and compare to state-of-the-art architectures (or even a larger ResNet) trained on ImageNet? \n", "summary_of_the_paper": "This paper proposes a new implicit model for deep learning called Multi-branch Optimization induced Equilibrium networks (MOptEqs), which models the hidden objective function for making efficiently use of different scales inputs. The fusion model uses two ideas:  Hierarchical Heritage, which ensures the similarity of corresponding channels of the near branches and (2)Diversity Module, which optimizes diversity in non-corresponding channels of neighboring braches. Taken together, these design choices lead to better performance and smaller model size on image recognition tasks on CIFAR-10 and CIFAR-100 datasets, as compared to  prior work on Equilibrium networks that use single/multiple branch architectures.\n", "summary_of_the_review": "The paper proposes some interesting ideas for using multi-scale inputs and feature hierarchy and diversity for training equilibrium models. The results on CIFAR-10 look promising, but I have some concerns about the writing and the novelty of the work. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8mc2pkDKWYb", "reviewer": "Reviewer_DkRL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The proposed method is straight forward and easy to follow, the newly proposed terms in the objective function have good motivations;\n2. Model sizes in multiple-scale experiments are much smaller than related works, which illustrates the effectiveness and efficiency of the proposed method;\n\nWeaknesses, questions & comments:\n1. The performance improvements seem to be trivial, although it is impressive to see the model size comparison in multiple-scale experiment. It would be great if the author could provide comparisons of different methods with same model size (results of MDEQ with smaller model size and results of MOptEqs with larger model size). It would be interesting to see whether the proposed method also has good scalability;\n2. More experiments on higher dimensional datasets are suggested;\n3. Could the author provide some ablation studies investigating the contribution of each component (inner-product term and diversity term)?\n4. Are the compared methods in tables trained with any kind of perturbation or not? It would clearer if all the methods are compared in two different settings: with and without perturbation;", "summary_of_the_paper": "The paper propose a new method which can efficiently utilize inputs with different scales, the propose method obtains better quantitative results in experiments.", "summary_of_the_review": "This paper propose a new method, which can handle different input scales. The  proposed method is well-presented with good motivation, obtains better results with smaller model sizes in experiments. There are several questions remaining, I'm looking forward to the responses and discussions. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2kt3rZQrBDs", "reviewer": "Reviewer_61kJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Though authors include the concepts of interpretability in the motivation of this work. I cannot see the connections between interpretability to this work. Did authors define the interpretability as explanation of behaviors of CNN, such as LIME explains DNN for computer vision using the attribution to features? Could you please give an example to demonstrate how the proposal interpret the DNN.", "summary_of_the_paper": "I don't quite understand this work. It seems authors proposed a new deep equilibrium model through incorporating a layer of optimization into the network. More specifically, authors may have extended the work MDEQ from two aspects: (1) inclusion of layers of deep neural network as optimizer and (2) introducing multi-branch architecture design. Experiments based on CIFAR demonstrate the advantage of the proposed solution in their settings. Theortical analysis with two propositions justify the properties of proposals.", "summary_of_the_review": "I failed to collect evidence to reject this paper, so I believe it might be a solide work. I hope authors could address the interpretability issues here. I hope AC and SPC could hear from other reviewers.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a multi-scale network that uses DEQ models to incorporate samples at multiple resolutions. The authors also propose a training strategy to improve the performance of the model. The authors investigate the interest of the approach through ablation and explainability, weighing the value of hierarchical heritage, diversity modules, perturbation size, and regularization penalties. 

The reviewers appreciated that the authors tackled the problem of incorporating multiple scales and the “impressive results” on CIFAR-10, CIFAR-100. The reviewers also expressed concerns regarding the computational assessment, in particular the additional computational/memory overhead of unrolling and what the authors mean by “explainability” in their experimental evaluation. The reviewers also made suggestions to organize the paper better. 

The authors submitted responses to the reviewers' comments. After reading the response, updating the reviews, and discussion, the reviewers who took part in the discussion considered that they are “satisfied by the response” and the “major concerns have been addressed”.  The feedback provided was already fruitful and the final version should be already improved. The ablative analysis and comparison to baseline is careful and thorough.

Accept. Poster.
Decision: Accept (Poster)