Year: 2021

Abstract:
Works have shown the strong connections between some implicit models and optimization problems. However, explorations on such relationships are limited. Most works pay attention to some common mathematical properties, such as sparsity. In this work, we propose a new type of implicit model inspired by the designing of the systems’ hidden objective functions, called the Multi-branch Optimization induced Equilibrium networks (MOptEqs). The model architecture is designed based on modelling the hidden objective function for the multi-resolution recognition task. Furthermore, we also propose a new training strategy inspired by our understandings of the hidden objective function. In this manner, the proposed model can better utilize the hierarchical patterns for recognition tasks and retain the abilities for interpreting the whole structure as trying to obtain the minima of the problem’s goal. Comparing with the state-of-the-art models, our MOptEqs not only enjoys better explainability but are also superior to MDEQ with less parameter consumption and better performance on practical tasks. Furthermore, we also implement various experiments to demonstrate the effectiveness of our new methods and explore the applicability of the model’s hidden objective function.

heading: 1 Introduction
text: Whereas Deep Neural Networks (DNNs) have achieved great success in many real-world tasks such as computer vision and neural language process, the limited interpretability of DNNs greatly hinders their further development.
However, many traditional machine learning methods, e.g.,matrix recovery (Zhang et al., 2018b; 2015; Liu & Li, 2016), subspace clustering (You et al., 2016), image deblurring (Liu et al., 2014) and so on, can be interpreted into minimizing the following objective functions:
min Z,E f(Z) + g(E), s.t. X = AZ + BE.
where A ∈ Rm×d1 , B ∈ Rm×d2 , X ∈ Rm×n, and f(·) and g(·) are convex functions designed by the modelling of their properties. For this count, the interpretability of such methods is much better than DNNs. Furthermore, these methods can also enjoy state-of-the-art and robust performance on these tasks. We call such interpretability these methods enjoy as "mathematical interpretability", i.e., whether the whole network structure can be summarized as a compact mathematical model that can be mathematically analyzed.
Except for these methods, the Optimization Induced Equilibrium Networks (OptEqs) proposed by Xie et al. (2021) recover its whole system for recognition to an optimization problem. The forward propagation for OptEqs tries to solve Eqn.1 to get output y, z∗ ∈ Rd for input x ∈ Rdin . We call the front part of Eqn.1 as OptEqs’ equilibrium equation, which is the central part for equilibrium models. And its forward procedure can be regarded as solving the hidden objective functions shown as Eqn.2,
z∗ = W>σ(Wz∗ +Ug(x) + b), y = Wcz ∗, (1)
min z G(z; g(x)) = min z
[ 1> f(W−1>z)− 〈 Ug(x) + b,W−1>z 〉 + 1
2 ‖W−1>z‖2 − 1 2 ‖z‖2
] . (2)
W,U,Wc are learnable weights, g is the convolution layers projecting input x from real-world domain to feature space like other neural architectures, and f is the penalty for the outputs.
Although the interpretability for OptEqs is good, it still remains some weaknesses worth exploring. First, although OptEqs performs better than some implicit models, these models only consider a single view (or resolution) of the input in their implicit parts. We call these models single-view implicit models in the following. However, nearly all the state-of-the-art pattern recognition systems (Lee et al., 2009b; Wang et al., 2019a; Huang et al., 2017; He et al., 2016; Burt & Adelson, 1987) benefit from the multi-layer or multi-resolution feature extractors in domains like computer vision and audio processing. Secondly, new modules induced by the objective function they proposed are limited, which mainly consider the math properties of the output features like sparsity, but whether these properties can benefit image recognition tasks is unexplored.
Besides OptEqs, MDEQ (Bai et al., 2020) constructs its equilibrium equation z∗ = F(z∗;x) and block F with the inspiration of explicit models especially HRNet Wang et al. (2019a). Then they solve the equilibrium equation by the accelerated algorithm for output z∗. Although it shows the state-of-the-art performance with efficient memory cost as other implicit models, such strategies will make the model lose "interpretability" since the MDEQ is a black box because its output is solved in an implicit way (root-finding method), which makes analysis on its features of intermediate layers almost impossible. Furthermore, its complicated structure also hinders its analysis from mathematical systems. More efficient blocks with good interpretability still needs exploring.
Motivated by the limitations of the above models, we would like to design a multi-scale DEQ structure with state-of-the-art performance and preserve mathematical interpretability from the inspiration of OptEqs and multi-scale models. The contributions for our work are listed below:
• We propose a multi-branch implicit model, called Multi-branch OptEqs (MOptEqs), which efficiently utilizes different scales inputs with better performance and smaller model size. Furthermore, it still retains its connection to an optimization problem.
• We propose some properties modelled as new terms for the model’s hidden objective function from our analysis on the relationships between branches and their hierarchical dependencies. With new problem’s formulation, we obtain the fusion module in our MOptEqs, called Hierarchical Heritage and Diversity Module (HH&D Module).
• Apart from the new module design, we also propose the Perturbation Enhanced training strategy (PE) for our MOptEqs from our analysis on the model’s hidden objective function. The new strategy cannot only enhance the performance of our MOptEqs, but also improve the robustness of our models.

heading: 1.1 Related Works
text: Implicit Models. Nearly all modern deep learning approaches use explicit models, which provide an explicit computation graph for the forward propagation. In contrast, the computation graphs for implicit models are "flexible" or can be assumed as having "infinite" depth. For example, Neural ODEs (Chen et al., 2018; Massaroli et al., 2020) encode their neural architectures by a differential system with learnable parameters. Then the implicit ODE solvers they used is equivalent to a continuous ResNet taking infinitesimal steps. Furthermore, the training process of Neural ODEs can be depicted as finding a differential system of a certain type (like heat equations) by updating its learnable parameters which demonstrate that Neural ODE also enjoy interpretability to a certain extent.
Moreover, DEQ (Bai et al., 2019; Winston & Kolter, 2020) is another class of implicit models. The central part of DEQ is the designation of the equilibrium equations z∗ = F(z∗;x) and block F . Its forward procedure is trying to solve the equilibrium equations with input x to get the equilibrium state z∗ as output by accelerated algorithms. Since the z∗ is fixed given F and x, its inference can be regarded as forwarding an explicit network stacked by the same block F for infinite times. For example, DEQ chooses their F by one set of Conv+ReLU
while MDEQ constructs an HRNet-like block as theirs. However, no evidence shows that the best block constructions in explicit models can perform the best in the DEQ scheme. The construction of DEQ blocks or equilibrium equations is an open question worth exploring. Furthermore, most DEQ blocks do not have any mathematical insights (like the diffusion process in Neural ODE) and perform totally a black box with limited interpretability.
Since implicit models usually adopt accelerated root-finding algorithms for their forward outputs and backward gradients, they enjoy the advantages of constant and more efficient memory costs compared with DNNs. Due to the above advantages, the design of implicit models draws much attention these days (Ghaoui et al., 2019; Gould et al., 2019). Apart from the models illustrated above, many kinds of other implicit models have been proposed, including differentiable physics engines (Qiao et al., 2020; de Avila Belbute-Peres et al., 2018), logical structure learning (Wang et al., 2019b) and implicit neural blocks (Li et al., 2020).
Model Interpretaiblity. Many researchers nowadays are trying to make their proposed method more interpretable. Although there are many kinds of approaches to achieve this goal, we marginally divide them into two parts: "Empirical" and "Mathematical" interpretability. Many works, such as (Zhang et al., 2018a; Zhang & Zhu, 2018; Bau et al., 2018), attempt to empirically disassemble the black box by characterizing some statistical or structural information like the outputs and gradients of neural networks’ middle layers. However, these works cannot be directly implemented on DEQ models since implicit ones do not have explicit depths like DNNs, which makes analyzing such models by dissecting each layer’s behavior almost impossible. Apart from that, researchers (Djolonga & Krause, 2017; Amos & Kolter, 2017; Xie et al., 2019; Chan et al., 2020) also managed to understand the neural architecture by linking it to a mathematical problem. In this way, researchers can analyze the black box by its corresponding mathematical problem. Furthermore, new components can be proposed due to the analysis of these problems. Our model aims to design a new DEQ architecture with proper mathematical interpretability. Compared with the former works, our model can deal with the classification tasks with better performances with novel components.

heading: 2 Multi-Branch Optimized Induced Equilirium Models

heading: 2.1 The proposed architecture for the Multi-Branch OptEqs
text: Inspired by OptEqs, we first design an objective function for our tasks to solve and then use its first-order stationary conditions as MOptEqs’ equilibrium equation. The base of our function is formed by summarizing several objective functions of different scales defined for OptEqs (Details are stated in Appendix.A.1). However, such a model can not obtain satisfying results since these branches are independent. For this count, we need to design some terms describing the dependencies of each branch in the hidden objective function, as shown in the following paragraphs. Then we can obtain the equilibrium equations for our MOptEqs and our MOptEqs’ block F by analyzing the first-order stationary condition of our designed optimization problem.
Hierarchical Heritage Modeling. State-of-the-art explicit models for image tasks are explicitly structured into sequential stages and process different resolutions hierarchically (He et al., 2016; Shelhamer et al., 2017; Lee et al., 2009a), which implies that features should be extracted hierarchically. In other words, the posterior branches should inherit from their prior ones, which we call such property hierarchical heritage. In our work, we are going to formulate such correlation of neighbouring branches in the hidden objective function by adding the following inner-product term into its origin:
H(zi, zj) = z>i Pi:jzj (3) where zi ∈ Rdi , zj ∈ Rdj ,Pi:j ∈ Rsi×sj denotes the Average Downsample (its transpose can be regarded as a weighted nearest upsample) or Identity matrix suited to the shape of zi and zj (i < j or j = 1, i = L). This term only estimate the summation for the similarity of i-th and j-th branch’s channels with the same channel index, which we call them corresponding channels, as shown in the above equation.
When zi and zj are similar, the inner product will be large. Otherwise, the result will become small, which implies that the relationship of corresponding channels is weak. In this
way, we can ensure the similarity of corresponding channels of the near branches. Except for the relations between different branches’ corresponding channels, the relationships between dis-corresponding channels are explored in the following section.
Diversity Modeling. In addition to the hierarchical relations for corresponding channels, works (Pang et al., 2019; Amada et al., 2021) have shown enhancing the diversities between branches can also improve the model’s improvements. For this account, we also consider making different branches extract various features to improve the representation abilities of different branches. To achieve such a goal, we add a diversity term in the objective function Eqn.(4) if we assume that i < j,
D(zi, zj) = C∑
k1=1 C∑ k2=1 k2 6=k1 |vec−1(zi)(k1)>vec−1(Pi:jzj)(k2)|, (4)
where vec−1 is the inverse vectorization operator convert the vectorized feature zi ∈ Rdi×1 to a matrix in R di C ×C with C channels and vec−1(zi)(k) ∈ R di C ×1 denotes the k-th channel of i-th branch in practice. Since we have already considered the relationships between the corresponding channels in the hierarchical term, we only tries to estimate the diversities of dis-corresponding channels of the i-th and j-th branches in this term. As the diversity term goes small, the diversities between the branches become stronger.
The architecture of MOptEqs and its hidden problem. With the two terms we proposed, we can reformulate the hidden objective for our problem by replacing the L2 norm with our new modelling as follows:
min z1,...,zL G(z1, ..., zL; g(x)) = min z1,...,zL L∑ i=1 [ 1>f(W−1>i zi)− 〈 Uig(x) + bi,W −1> i z 〉 + 1
2 (λD(zi, zi+1) + ‖W−1>zi‖2 −H(zi, zi+1))
] .
(5)
g(x) is the input feature for the raw input x, λ > 0 is a hyperparameter and zi ∈ Rdi×1 are the final outputs of MOptEqs. We can choose f to constrain zi on our demand and will influence model’s activation function. If we choose f(x) = I{x ≥ 0} to ensure the outputs to be positive, then the activation function is ReLU. We set D(zL, zL+1) = D(zL, z1) and H(zL, zL+1) = H(zL, z1) when i = L to complete the loop. The two terms we added tries to make the corresponding channels of different branches correlated by maximizing the H term and enhancing the diversities of different branches by minimizing D term. We note that H and D are not conflict since they are handling different pairs of channels for branches. As the following proposition shows, we finally get the equilibrium equations (Eqn.(6)) for our MOptEqs by calculating the first-order stationary conditions ∇ziG = 0 for problem G.
Proposition 1 The proposed multi-branch structure induced by Eqn(5) can be depicted as solving the equilibrium points z̃∗ := [z>∗1 , ..., z>∗L ] > ∈ R ∑L i=1 di for the following equations:
z̃ = W̃>σ(W̃h(z̃) + Ũg(x) + b̃)
where W̃ = W1 . . . WL  ,Ũ = [U>1 , ...,U>L ]>, b̃ = [b>1 , ...,b>L ]>. (6)
And h(z̃) = [h1(z1, zL, z2)>, .., hi(zi, zi−1, zi+1)>, .., hL(zL, zL−1, z1)>]> and each hi is defined as the mapping from Rdi × Rdi+1 × Rdi−1 to Rdi ,
h(zi, zi−1, zi+1) = 1
2 (Pi:i+1zi+1 +P
> i−1:izi−1)
− λ 2 vec(vec−1(Pi:i+1zi+1)sign[Mi:i+1 − diag(Mi:i+1)])
− λ 2 vec(vec−1(P>i−1:izi−1)sign[M > i−1:i − diag(Mi−1:i)]),
Mi:i+1 =vec−1(Pi:i+1zi+1)>vec−1(zi),
Mi−1:i =vec−1(zi)>vec−1(P>i−1:izi−1).
The equilibrium points for the above formula are also the first-order stationary points for the optimization problem Eqn(5).
The proof of the proposition is listed in Appendix.A.3. In this manner, we obtain the proposed architecture MOptEqs, its forward propagation is equivalent to find the equilibrium points of Eqn(6), and such process can also be regarded as solving the stationary points of the optimization problem (5). The whole structure is also shown in Figure.1 (a) and we also draw figures to illustrate the practical process of hi (HH&D) on its right (Figure.1 (b)). We also evaluate the effectiveness of such modelling in Section. 3.2’s experiments.

heading: 2.2 The proposed Training Strategy for MOptEqs
text: Researches (Kong et al., 2020; Xie et al., 2020; Tang et al., 2021) have discovered that small perturbations can enhance the generalization abilities and varieties of branches for neural architectures. Most efficient works use adversarial perturbations based on gradients (Xie & Yuille, 2020) for such a trick. However, such a strategy will slow the training process since they need at least one more back-propagation in each training step for perturbations, which is time-consuming. Unlike these methods, we can acquire more computation efficient perturbations because our models can be formulated as minimizing an objective function we designed. Then we can replace the maximum problem (details in Appendix.A.2) for adversarial perturbation with maximizing our architecture’s hidden objective function G, since we assume MOptEq’s performance will go worse if G(z; g(x)) changes a lot at g(x)’s neighborhoods since the forward propagation of MOptEqs is minimization G(z; g(x)).
Assumption 1 For a well trained OptEqs with its objective function denoted as G(z; g(x)) and a natural sample x which can be correctly classified. If an input perturbation ‖δ‖∞ ≤ can cause one of the following changes:
|G(z′∗; g(x)− δ)−G(z∗; g(x))| ≥ L1 , ‖z′∗ − z∗‖2 ≥ L2 .
with L1, L2 > 1, it may lead the OptEqs to the wrong results with high probability.
For the second part of the assumption, it’s common sense and widely used in the analysis of adversarial robustness (Zhang et al., 2021; Li et al., 2020). With the above assumption, we can generate the perturbations by maximizing the objective function G(z; g(x)) for our MOptEqs. Since our MOptEqs can be regarded as a kind of "implicit" ensembling that parallels several equilibrium models, we decide to inject the perturbations acquired for the prior branch to the posterior one, like boosting to enhance the performance.
Then the hidden objective function of MOptEqs with G(z; g(x)) is shown as follows,
min z1,...,zL G(z1, ...,zL; g(x)) = min z1,...,zL L∑ i=1 [ 1>f(W−1>i zi)− 〈 Ui(g(x)− δi−1) + bi,W−1>i z 〉 + 1
2
( λD(zi, zi+1) + ‖W−1>zi‖2 −H(zi, zi+1) )] ,
s.t. δi = argmax ‖δi‖∞≤
1>f(W−1>i zi)− 〈 Ui(g(x)− δi) + bi,W−1>i zi 〉 for i ∈ [1,L].
where δ0 = 0. And the problem becomes a bilevel optimization problem. The solution to the lower-level problem is δi = sign(U>i W −1> i zi). Since the perturbation can be obtained by feeding the output of the activation layer σ(Wz + Ug(x) + b) to the transposed convolution layer with weight Ui, we call it reconstructed perturbation. Compared with the adversarial perturbations based on gradients, our perturbations can be acquired directly by matrix-vector multiplication instead of iteratively built by gradients. For this account, such a process does not require much computation cost. Following the above steps, we obtain a harmful perturbed direction for the prior branches and then added them to their posterior branches. We note that our perturbation is added to the input feature g(x) instead of the raw input.
Proposition 2 For a natural sample x and a well trained OptEqs
z∗ = W>σ(Wz∗ + Ug(x) + b)
with its hidden objective function denoted as G(z; g(x)), for g′(x) = g(x) − δ and δ = sign(U>i W −1> i zi), which obeys the constraint ‖δ‖∞ ≤ . z′∗ and z∗ are defined as follows:
z′∗ = argmin z G(z; g′(x)),
z∗ = argmin z G(z; g(x)).

heading: If the ‖W‖2, ‖U‖2 ≤ 1 and 1N ‖U
text: >W−1>z∗‖1 , where N is the element number of g(x), then
max { G(z′∗; g(x)− δ)−G(z∗; g(x)), √ N ‖z′∗ − z∗‖2 } ≥ 1
2
{ ‖U>W−1>z∗‖1 −N 2 } .
The proposition demonstrates if we choose the perturbed direction to be sign(U>W−1>z), at least one of the changes for the G or z∗ will be around ‖U>W−1>z∗‖1 and implies each branch may perform bad on the perturbed data according to our assumptions. With the experiments in Section.3.2, we can conclude that our reconstructed perturbation is useful and our assumption is reasonable. Like boosting strategy, we can feed the perturbed data for the prior branch to its posterior branch to enhance the performance; we call such method as the Perturbation Enhanced training strategy (PE). Following experiments also show that PE can indeed enhance the performance of our MOptEqs.

heading: 2.3 Model Optimization and Forward Convergence
text: Forward Propagation and Implicit Differentiation. Like other equilibrium models, the forward propagation procedure is solving the given equilibrium functions Eqn. 7. For this count, our model also enjoys the constant memory cost advantages as other DEQs. In our work, run root-finding algorithms to solve the roots z̃∗ = [z∗1, ...z∗L] of the following problem to reach the equilibrium states, which is the same as MDEQ (shown in Appendix.A.5):
∇z̃G(z1, ..., zL; g(x)) = 0. (7)
As for the backward propagation, we also adopt the implicit differentiation method widely used in (Bai et al., 2020; 2019; Chen et al., 2018). Instead of tracing the gradients during the forward propagation, the implicit differentiation method directly backpropagates through the equilibrium state using the Jacobian of Tθ = ∇z̃G+ z̃ at z̃∗. For a given loss l = L(z̃∗,y) (where y is the target) and the gradients can be written as
∂l ∂(·) = ∂l ∂z̃∗ (I− JTθ |z̃ ∗)−1 ∂Tθ ∂(·) . (8)
And the calculation of ∂l∂z̃∗ (I − JTθ |z̃ ∗)−1 is equivalent to solve root m for the following equation:
m(I− JTθ |z̃∗) + ∂l
∂z̃∗ = 0. (9)
As for the root-finding algorithm, we can use Broyden method (Broyden, 1965), Anderson Method (Anderson, 1965; Bai et al., 2021) or other root-finding methods to solve Eqn(7) for the equilibrium state and Eqn(9) for the backward gradient.
Forward Convergence. Like other implicit models, we need to make some constraints on parameters in order to make the whole MOptEqs TΘ(z̃;x) (z̃ here is {zi}Li=1) be a contractive mapping. For the MOptEqs without considering HH&D (Eqn.10), the models can easily converge with ‖Wi‖2 ≤ ζ < 1. But since we use the HH&D module, we need to choose proper λ to ensure the convergence, around or less than 1C will be appropriate, which does not need ζ to be too small. We also conduct experiments to verify the convergence in Fig.4 in Appendix. A.7.

heading: 3 Experimental Results
text: In this section, we conduct experiments for the image classification tasks on CIFAR-10,CIFAR100 Krizhevsky et al. (2009) and ImageNette implemented on the PyTorch (Paszke et al., 2017) platform to demonstrate the effectiveness of our model. Details are listed in Appendix..A.6.

heading: 3.1 Comparison of Prior Implicit Models
text: In this part, we decide to verify the superiority of MOptEqs in two aspects. First, we build the single-scale MOptEqs and compare the empirical results with other single-scale implicit models. And then we compare the experimental results with prior implicit models that utilize the multi-scale inputs like MDEQs, the state-of-the-art implicit model with multi-resolution.
Implicit Models with Single View. In this part, we compare our MOptEqs with other single-view implicit models with comparable model sizes. Like OptEqs who uses three blocks for their experiment, we first construct our small MOptEqs with three branches whose outputs share the same size and the channel number C = 32. Results in Table.1 (a) demonstrates that our MOptEqs structure can even enhance the performance for recognition tasks under the single-view cases. Compared to the multi-branch model without fusion (POptEqs), our MOptEqs’ HH&D can efficiently utilize the relationships between different branches and lead the model to better performance. Our superiority also holds compared with former multi branches fusion methods: "sum" (non-parametric module) and "conv" (parametric module). Meanwhile, we can also conclude the effectiveness of our perturbation enhanced training strategy from the table since its performance is the best.
Implicit Models with Multiple Scales. Moreover, we conduct experiments compared with other models which handling multi-scale inputs. Like MDEQ, we construct our MOptEqs with four branches with resolution size equals to 32,16,8,4. Other details can be found in Appendix.A.6. From Table.1 (b) and as shown in Table.2 (a), one can see that our MOptEqs not only outperforms the widely used explicit model ResNet-18 (He et al., 2016), but also shows better performance compared with MDEQ with fewer parameters on CIFAR, which is one of the best models. The empirical results for the multi-scale models further verify the superiority of MOptEqs and its training strategy.
Apart from experiments on small images, we also conduct experiments on ImageNette, which is a subset of 10 classes from ImageNet. Compared with MDEQ, our MOptEqs consistently perform better shown in Table.2 (b). Nevertheless, as one can see from the results, the difference between MDEQ and our model becomes much bigger in the CIFAR-100 and Imagenette with the same training hyper-parameters. Such a phenomenon demonstrates that the performance of our model is much more stable than MDEQ. Apart from these experiments, we also conducted ablation studies for our models in the following section.

heading: 3.2 The comprehensive understanding of MOptEqs
text: Visualization of the impact for the HH&D module. In this section, we try to estimate the effect of our HH&D module. We finish experiments in this part using three-branch MOptEqs with 16 channels for each branch to make the visualization more clear.
We tries to verify our HH&D module’s effect in two aspects. First, we plot scatter figures for the values of ∑L i=1 ∑ j∈N(i)D(zi,zj) L(C2−C) (denoted as Average D) and ∑L i=1 ∑ j∈N(i)H(zi,zj) LC (denoted as Average H ) for 100 randomly chosen samples (N (i) denotes neighboring branch of the i-th branch). Since the hidden optimization problem is to maximizing H while minimizing D, we find that our MOptEqs (Figure.2(a)) can induce the output features to reach such goal compared with POptEqs (MOptEqs without HH&D) (Figure.2(b)).
Furthermore, we also plot heatmaps of the first two branches’ correlation score (|vec−1(z2)vec−1(z1)>| ∈ R16×16) for a randomly chosen sample shown in Figure.2(c)(d). One can see that MOptEqs’ heatmap (Fig.2(c)) is more likely to be a diagonal matrix while heatmap Fig.2(d) for POptEqs looks random. The heatmap shows that our HH&D can induce the model to perform as our demand, which means corresponding channels for adjacent branches can be more correlated, while the dis-corresponding channels are unrelated due to our architecture design. The visualizations and former results on different datasets verify the effectiveness of our modelling and our HH&D modules’ design.
Evaluation of the reconstructed perturbation. We compare our perturbation with randomly generated ones in a one-branch well-trained MOptEqs to validate its effectiveness. We first feed a natural sample x to the model and acquire output z for x and generate our reconstructed perturbation using such output and then feed it to the model. Furthermore, we add the perturbations generated by Binomial distribution (P(δi = ) = P(δi = − ) = 0.5) and Uniform distribution U [− , ] to input features g(x) for comparison. The result of each distribution is averaged for five trials. Figure.3(a) shows our perturbation generated by maximizing G is much more effective, which validates the effectiveness of our reconstructed perturbations and the rationality of our assumptions and analysis in Section.2.2.
Robustness of our MOptEqs trained by PE strategy. In addition to improving the generalization abilities for models as shown in Sec.3.1, perturbations with proper size may also enhance the robustness of our MOptEqs, shown in Figure.3(b). We conduct experiments on small MOptEqs (the same setting as Sec.3.1’s single-view model). Our perturbation enhanced training strategy goes stronger as increases. The figure shows that accuracy for the model trained naturally drops quicker than trained by PE strategy. Overall, trained with appropriate reconstructed perturbation can partly improve the robustness of our MOptEqs.
MOptEqs vs. adding Regularizers in training loss. Apart from our HH&D modules, adding D and H to the training loss is one of the most popular methods for obtaining outputs with certain properties. In order to demonstrate the superiority of our module empirically, we conduct experiments for our MOptEqs trained by cross-entropy and POptEqs (MOptEqs (w/o. HH&D)) trained by the cross-entropy loss adding γ ∑L i=1(H(zi, zi+1)− λD(zi, zi+1)) as regularizers (we call it augmented loss). Then we drew its test accuracy for different γ trained by the augmented loss (λ is the same as ours) shown in Figure.3(c).
With proper γ, POptEqs trained by additional loss can perform better than γ = 0. Such phenomenon demonstrates the effectiveness of our HH&D Modelling. Furthermore, the figure demonstrates the superiority of our model since the traditional method is consistently worse than ours. We left some other explorations for our model in the Appendix due to the space limit.

heading: 4 Conclusion
text: We introduce the multi-branch optimization induced equilibrium models (MOptEqs), a new extension of OptEqs that can utilize multiscale information for recognition tasks and retain its ability to recover to an optimization problem whose solution is equivalent to the equilibrium states of our model. The model architecture is designed based on modelling the hidden objective function for the multi-resolution recognition task. Furthermore, we also propose a new training strategy inspired by our understandings of the hidden optimization problem. The empirical results show the advantages of our proposed methods. The success of our HH&D module and PE training strategy demonstrates the deep link between the optimization problem and neural architecture and may motivate further explorations.

heading: A Appendix

heading: A.1 Paralleling Multi-Branch OptEqs
text: Experiments show that even a simple one-layer OptEqs module enjoys impressive generalization abilities. If we view a simple one-layer OptEqs module as a powerful feature generator, we can use an "ensemble" scheme on the DEQs by paralleling several DEQ modules. Then we get the multi-branch implicit structure (formulated as Eqn.10), which is capable of utilizing samples of multi-resolution but also retains the capability of recovering to an optimization problem.
z∗1 = W > 1 σ(W1z ∗ 1 + U1g(x) + b1),
......
z∗L = W > Lσ(WLz ∗ L + ULg(x) + bL),
(10)
where z∗i denotes the equilibrium outputs of i-th branch, we call this structure ParallelOptEqs (POptEqs). Like (Xie et al., 2021) stated for OptEqs, the POptEqs can also be depicted as solving the following problem (Eqn.11) if Wi is invertible and σ(·) is activation function,
min z1,...,zL G(z1, ..., zL; g(x)) = min z1,...,zL L∑ i=1 [ 1>f(W−1>i zi)− 〈 Uig(x) + bi,W −1> i z 〉 + 1
2 ‖W−1>i zi‖ 2 − 1 2 ‖zi‖2
] ,
(11)
where g(x) is the input feature with raw input x, zi ∈ Rdi×1 and f added for the activation function. For example f is the positive orthant f(x) = I{x ≥ 0} if we use ReLU activation, since (I + ∂f)−1 is the ReLU activation function, other activation functions can be found in Li et al. (2019).The calculation of the above structure (Eqn.10) is equivalent to solving the stationary points ∇ziG = 0.

heading: A.2 Adversarial Perturbations.
text: Adversarial perturbations aim to cheat well-trained neural networks with small or unnoticeable changes on natural images. The adversarial perturbations can be obtained by solving the following maximum problem for input xnat and label y:
max δ∈C
L(fNN (xnat + δ); y),
where C is the feasible set of δ in case the change is too large and usually chosen to be an l-infinite ball. Due to the neural network is a non-convex function for δ, obtaining the perturbation is not easy. The most popular method is the PGD-k method which solves the above problem by implementing the projected gradient method by k times. Although the performance of such a method is satisfactory in most cases, it makes the whole training procedure k + 1 times slower than the origin. In our work, we replace the above objective function with our model’s hidden objective function. Since the hidden optimization problem is convex with respect to the perturbation, we can get the perturbation much easier. Moreover, such perturbation can also enhance the generalization abilities, as shown in the experiments.

heading: A.3 Proofs for Proposition.1
text: In this section, we are going to prove that the equilibrium equation Eqn.1 is also the first order stationary points for the optimization problem 5.
Proof 1 The equilibrium equations Eqn.1 in the proposition can be separated into different branches as follows:
z1 = W > 1 σ (W1h1 (z1, zL, z2) + U1g(x) + b1) ,
......
zi = W > i σ (Wihi (zi, zi−1, zi+1) + Uig(x) + bi) ,
......
zL = W > Lσ (WLhL (zL, zL−1, z1) + ULg(x) + bL) ,
(12)
where hi : Rdi × Rdi+1 × Rdi−1 → Rdi can be defined as:
hi(zi, zi−1, zi+1) = 1
2 (Pi:i+1zi+1 + P
> i−1:izi−1)
− λ 2 vec
( vec−1(Pi:i+1zi+1)sign [Mi:i+1 − diag(Mi:i+1)] ) − λ
2 vec(vec−1
( P>i−1:izi−1)sign [ M>i−1:i − diag(Mi−1:i) ]) ,
(13)
where Mi:i+1 = vec−1(Pi:i+1zi+1)>vec−1(zi).
First, we need to prove that
h(zi, zi−1, zi+1) = − 1
2 ∇zi L∑ i=1 [λD(zi, zi+1)−H(zi, zi+1)] (14)
The derivatives for the right term can be divided as the derivatives for H and D:
∇zi L∑ k=1 H(zk, zk+1) = ∇zi L∑ k=1 z>k Pk:k+1zk+1
= ∇zi(z>i Pi:i+1zi+1 + z>i−1Pi−1:izi) = Pi:i+1zi+1 + P > i−1:izi−1
which is the first term of hi. Then we are going to calculate the derivatives for D:
∇zi L∑ k=1 D(zk, zk+1) = ∇zi L∑ k=1 c∑ k1=1 c∑ k2 6=k1 |vec−1(zk)(k1)>vec−1(Pk:k+1zk+1)(k2)|
= ∇zi  c∑ k1=1 c∑ k2 6=k1 |vec−1(zi)(k1)>vec−1(Pi:i+1zi+1)(k2)|
+ c∑ k1=1 c∑ k2 6=k1 |vec−1(zi−1)(k1)>vec−1(Pi−1:izi)(k2)|  = vec
∇vec−1(zi)  c∑ k1=1 c∑ k2 6=k1 |vec−1(zi)(k1)>vec−1(Pi:i+1zi+1)(k2)|
+ c∑ k1=1 c∑ k2 6=k1 |vec−1(zi−1)(k1)>vec−1(Pi−1:izi)(k2)|  with the following equation holds for the sampling matrix Pi−1:i and sampling is done for each channel independently:
vec−1(zi−1)(k1)>vec−1(Pi−1:izi)(k2) = vec−1(P>i−1:izi−1) (k1)>vec−1(zi)(k2)
The equation can be formulated as:
∇zi 1
2 L∑ k=1 D(zk, zk+1) = vec ∇vec−1(zi)  c∑ k1=1 c∑ k2 6=k1 |vec−1(zi)(k1)>vec−1(Pi:i+1zi+1)(k2)|
+ c∑ k1=1 c∑ k2 6=k1 |vec−1(P>i−1:izi−1)(k1)>vec−1(zi)(k2)|  Since
∇vec−1(zi) c∑
k1=1 c∑ k2 6=k1 |vec−1(zi)(k1)>vec−1(Pi:i+1zi+1)(k2)|
= ∇vec−1(zi)(1)  c∑ k1=1 c∑ k2 6=k1 |vec−1(zi)(k1)>vec−1(Pi:i+1zi+1)(k2)|  , ...,∇vec−1(zi)(C)(...) 
=  L∑ k2 6=1 sign(vec−1(zi)(1)>vec−1(Pi:i+1zi+1)(k2))vec−1(Pi:i+1zi+1)(k2),
...,
L∑ k2 6=C sign(vec−1(zi)(C)>vec−1(Pi:i+1zi+1)(k2))vec−1(Pi:i+1zi+1)(k2)  =vec−1(Pi:i+1zi+1)sign[vec−1(Pi:i+1zi+1)>vec−1(zi)− diag(vec−1(Pi:i+1zi+1)>vec−1(zi))] =vec−1(Pi:i+1zi+1)sign[Mi:i+1 − diag(Mi:i+1)],
In the same manner, we can get,
∇vec−1(zi) c∑
k1=1 c∑ k2 6=k1 |vec−1(P>i−1:izi−1)(k1)>vec−1(zi)(k2)|
=vec−1(P>i−1:izi−1)sign[M > i−1:i − diag(Mi−1:i)]
Taking all the terms together, we’ve proved the correctness of Eqn.14. Then we can proved the relationships between Eqn.12 and the first-order stationary points for Eqn.5. Take zi for an example: 0 = ∇zi L∑ i=1 [ 1>f(W−1>i zi)− 〈 Uig(x) + bi,W −1> i z 〉 + 1 2 ( λD(zi, zi+1) + ‖W−1>zi‖2 −H(zi, zi+1)
)] 0 = W−1i (I + ∂f)(W −1> i zi)−W −1 i (Uig(x) + bi)− h(zi, zi−1, zi+1)
Then we can get the equilibrium equation:
zi = W > i σ(Wihi(zi, zi−1, zi+1) + Uig(x) + bi),
the proof is the same for all i. In this manner, we proved the relations between the structure and the optimization problems. The proof for the proposition is complete.

heading: A.4 Proofs for Proposition.2
text: Proof 2 From the preliminaries above, the difference between G(z′∗; g(x)−δ) and G(z∗; g(x)) with δ = SIGN(U>W−1>z∗) is as follows:
G(z′∗; g(x)− δ)−G(z∗; g(x)) = G(z′∗; g(x))−G(z∗; g(x)) + 〈 sign(U>W−1>z∗),U>W−1>z′∗ 〉 Since z∗ is the minimizer of G at g(x), the above function can be converted as follows:
G(z′∗; g(x)− δ)−G(z∗; g(x)) ≥ 〈 sign(U>W−1>z∗),U>W−1>z′∗ 〉 = ‖U>W−1>z∗‖1 − 〈 sign(U>W−1>z∗),U>W−1>(z∗ − z′∗) 〉
From the structure of OptEqs, we can get:
SIGN(U>W−1>z∗)>U>W−1>(z∗ − z′∗) ≤ ‖U>W−1>(z∗ − z′∗)‖1 ≤ √ N ‖U>W−1>(z∗ − z′∗)‖2
≤ √ N ‖U> ( σ(Wz∗ + Ug(x) + b)− σ(Wz′∗ + U(g(x)− δ) + b) ) ‖2 ≤ √ N ‖σ(Wz∗ + Ug(x) + b)− σ(Wz′∗ + U(g(x)− δ) + b)‖2 ≤ √ N ‖Wz∗ + Ug(x) + b−Wz′∗ −U(g(x)− δ)− b‖2 ≤ √ N ‖W(z∗ − z′∗) + Uδ‖2 ≤ √ N ‖W(z∗ − z′∗)‖2 + √ N ‖Uδ‖2 ≤ √ N ‖z′∗ − z∗‖2 +N 2
The inequality is acquired because of the Lipshitzness of common activation function (ReLU and Leacky ReLU) and the bounds on weight matrices. With the above results, we can get:
G(z′∗; g(x)− δ)−G(z∗; g(x)) ≥ ‖U>W−1>z∗‖1 − √ N ‖z′∗ − z∗‖2 −N 2
Then we can obtain the proposition:
max { G(z′∗; g(x)− δ)−G(z∗; g(x)), √ N ‖z′∗ − z∗‖2 } ≥ 1
2
{ ‖U>W−1>z∗‖1 −N 2 }
The proofs are similar when considering the HH&D modules.
A.5 Forward Process of our MOptEqs’ implicit part.
The forward propagation is illustrated as follows:
Algorithm 1: Forward Process of our MOptEqs’ implicit part. Require: Input x, initial points {zi}Li=1, perturbation size , δ0 = 0, maximum perturbation size . Ensure: Equilibrium points{zi}Li=1 Calculate the input feature g(x). The calculation of ∇z̃G(z̃; g(x)) = 0 is the same as finding the equilibrium points
Tθ = ∇z̃G+ z̃ = z̃
. Use Broyden (Broyden, 1965) or Anderson Method (Anderson, 1965) or others to find roots of the following equation like MDEQ:
Tθ(z̃; g(x)) = z̃
return z̃ = {z∗i }Li=1
And the calculation of Tθ is shown as follows:
Algorithm 2: The Calculation of Tθ = ∇z̃G+ z̃ at (z̃, g(x)). Require: Input feature g(x), initial points {zi}Li=1, perturbation size , δ0 = 0, maximum perturbation size . Ensure: Output Tθ = ∇z̃G(z̃; g(x)) + z̃ for i ∈ RANGE(1, L) do k1, k2 ← neighbor of i in L-loop Calculate zpi by hi (Shown in Fig.1 (b)): zpi ← hi(zi, zi−1, zi+1)
if Use PE Method then yi ← σ(Wizpi + Ui(g(x)− δi−1) + bi) δi ← SIGN(U>i yi) else yi ← σ(Wizpi + Uig(x) + bi) end if zi ←W>i yi
end for return {z∗i }Li=1
A.6 Experiments details of MOptEqs for the experiments Classification.
A.6.1 CIFAR-10
For classification tasks of single-view models, we set the channel number of each branch to 32, λ = 0.01 and perturbation size = 0.1 for the small MOptEqs with three branches for the single-view comparison as OptEqs. The batch size is set to be 128 for all the experiments.
As for the multi-scale models’ comparison, we use four branches with each branch take inputs with resolutions equals 32, 16, 8, 4 like MDEQ, the output channel number for each branch is 256 for MOptEqs with 1.9M learnable parameters. At the same time, we set the channel number to be 128 for MOptEqs with 0.48M learnable parameters. And we set λ = 0.001 and perturbation size = 0.1 for the experiments.
We use the widely used SGD algorithm for training procedure. We set weight decay to be 0.001 and the initial learning rate start from 0.1 and decay by 0.1 at 100, 150, 175-th epoch with 200 epochs in total like others. We use the standard data augmentation for all the experiments.
Our implementation for small MDEQ only changes the channel number of MDEQ’s 10M model to obtain the comparable size with our model, without changing the training settings and its hyper-parameters.
A.6.2 CIFAR-100
Except for CIFAR-10, we also finish the experiments on CIFAR-100 classification2 to further verify our model’s effectiveness. The model we used is the same as the models used in the multi-scale comparison, only changing the output from 10 to 100 for classification. MDEQ with 11M parameter is the same model as (Bai et al., 2020) proposed for CIFAR-10 experiment with only changing the output from 10 to 100 for CIFAR-100. Moreover, the hyper-parameter setting is also the same as (Bai et al., 2020) proposed in the paper.
We use the widely used SGD algorithm for training procedure. We set weight decay to be 0.001 and the initial learning rate start from 0.1 and decay by 0.1 at 100, 150, 175-th epoch with 200 epochs in total like others. We use the standard data augmentation for all the experiments.
2https://www.cs.toronto.edu/ kriz/cifar.html
‖T j‖2 .
A.6.3 ImageNette
Besides the classification for images with small size, we also conduct the experiments on the Imagenette3, which is a subset of 10 classes from ImageNet4 with 9469 training photos and its test set consists of 3925 images. Furthermore, we use the full-size version of Imagenette5 further to verify the superiority of our models on large-scale images. The model architecture for our MOptEqs and MDEQ (for comparison) is the same as the ones used in the multi-scale models’ comparison for CIFAR-10 except adding two downsampling layers in the head of models to downsample the input size from 256 to 64. As for the hyper-parameters, we only change the weight decay to be 5e − 5, batch size to be 32 and extend the whole training epochs to 200 with 100-th, 150-th, 175-th for the learning rate decay for all the models. All the hyper-parameters for both MOptEqs and MDEQs are hardly changed during all the datasets.

heading: A.7 Convergence validation for our MOptEqs.
text: As shown in Figure.4(a), the Anderson method can quickly find the equilibrium points for our small MOptEq used for single-view comparison. From the figure, one can see that MOptEqs with proper λ (around or smaller than 1C , around 0.03 in this circumstance) can ensure the whole mappings be contractive. But too large λ will make the model hard to converge or even fail.
Apart from plotting the convergent iterations’ of our forward root-finding procedure for the small MOptEqs (shown in Figure.4 (a)), we also draw figures to show the convergence of the forward propagation for our multi-scale MOptEqs (λ = 0.001) used in the imagenette and CIFAR experiments as Figure.4 (b) shown. We use the pre-trained model on Imagenette whose prediction accuracy is reported in the above sections. One can see that our multi-scale MOptEqs can also quickly converge to the equilibrium points as we expected.
A.8 Ablation Studies on the impact of D and H part
In this section, we analyzed the influence of our D and H term for a single-branch MOptEq model trained without PE method shown in Table. 3. We set λ = 0.02 for the experiment. From the table, one can see that the contribution of the H term is slightly higher than the diversity term. But considering both two terms can significantly improve the results comparing with MOptEqs without considering these two terms.
3https://github.com/fastai/imagenette/ 4https://www.image-net.org/ 5https://s3.amazonaws.com/fast-ai-imageclas/imagenette2.tgz

heading: A.9 Experiments on the ImageNet
text: Except CIFAR and Imagenette, we also conducted experiments on ImageNet with 13M parameters shown as below:
From the results, one can see that our model can achieve the satisfying performance on ImageNet compared with the state-of-the-art models.

heading: A.10 PE method and robustness
text: From the above experiments, one can see that the PE method may also improve the robustness of our MOptEqs. In this section, we conduct experiments to see whether the PE methods can further enhance the robustness of MOptEqs when we use adversarial training methods on these models. We finish comparisons of PGD-3 training for the single-view MOptEq with or without PE, and the result is shown as follows: From the table, we can conclude
that our PE method can improve the robustness of our MOptEqs model trained by the adversarial training method. Such advantages also demonstrate the superiority of our models’ mathematical interpretability since the PE method is acquired based on the model’s hidden optimization problem.
A.11 Other Training methods: Fixed Point method and One-step gradients

heading: A.11.1 Methods
text: Since our model is not so complicated, we can use the fixed point method for L iteration for the forward propagation. While for the back propagation, we can use BPTT (Mozer, 1989) and (Geng et al., 2021) proposed method, which only backward the final forward iterations using the chain rule:
∂L ∂(·) = ∂L ∂T LΘ ∂T LΘ ∂(·)
where θ ∈ Θ are the learnable parameters for the implicit models, T LΘ denotes the output z for the L-th fixed-point iteration. Since the back-propagation of such a strategy is similar
to backwards only a single layer of DNN module without the calculation of root-finding algorithm for the Jacobian in the implicit differentiation, the computational cost is much less than the implicit method. However, the gradient approximated by such a method is not accurate and may cause the performance drop. However, as we illustrated below, such a method can be regarded as a trade-off if the computational resources of training are not enough.

heading: A.11.2 Empirical Results and Computational Efficiency of One-stpe Gradient
text: We take the ImageNette dataset as an example, the training memory cost, forward time, and test accuracy are shown as follows (each batch contains 32 images and is finished on 2 x GTX 1080Ti for 200 epochs):
The results shows that the one-step gradient methods can save a lot computation resources while with slightly drop on the performance.

Review summaries:
This paper proposes a new implicit model for deep learning called Multi-branch Optimization induced Equilibrium networks (MOptEqs), which models the hidden objective function for making efficiently use of different scales inputs. The fusion model uses two ideas:  Hierarchical Heritage, which ensures the similarity of corresponding channels of the near branches and (2)Diversity Module, which optimizes diversity in non-corresponding channels of neighboring braches. Taken together, these design choices lead to better performance and smaller model size on image recognition tasks on CIFAR-10 and CIFAR-100 datasets, as compared to  prior work on Equilibrium networks that use single/multiple branch architectures. 

The paper introduces a multi-scale network (called Multi-Branch OptEqs or MOptEqs) that extends on recent results of [1]. Specifically, the authors propose to use several DEQ models to incorporate samples of multi-resolutions. The authors further propose a modified training strategy (called perturbation enhanced training) to improve the performance of the model. Empirically, the proposed model shows better performances compared to the previous DEQ models on CIFAR-10, CIFAR-100, and Imagenette datasets.  [1] Xie, Xingyu, et al. "Optimization Induced Equilibrium Networks." arXiv preprint arXiv:2105.13228 (2021).

I don't quite understand this work. It seems authors proposed a new deep equilibrium model through incorporating a layer of optimization into the network. More specifically, authors may have extended the work MDEQ from two aspects: (1) inclusion of layers of deep neural network as optimizer and (2) introducing multi-branch architecture design. Experiments based on CIFAR demonstrate the advantage of the proposed solution in their settings. Theortical analysis with two propositions justify the properties of proposals.