Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new implicit model for deep learning called Multi-branch Optimization induced Equilibrium networks (MOptEqs), which models the hidden objective function for making efficiently use of different scales inputs. The fusion model uses two ideas:  Hierarchical Heritage, which ensures the similarity of corresponding channels of the near branches and (2)Diversity Module, which optimizes diversity in non-corresponding channels of neighboring braches. Taken together, these design choices lead to better performance and smaller model size on image recognition tasks on CIFAR-10 and CIFAR-100 datasets, as compared to  prior work on Equilibrium networks that use single/multiple branch architectures.

main_review: 
Strengths: 
1. Performance: The proposed method obtains decent improvement in performance and reduction in model size as compared to prior work on this topic.
2. Explainibility and ablation studies: The detailed ablation study shows the effect of design  choices such as Hierarchical Heritage and Diversity modules, perturbation size and regularizers.  


Weaknesses/Questions
1. Writing: The paper is not well-written and organized. The key ideas are not properly organized and explained. For someone who is not familiar with the implicit models literature, I had to spend  time carefully re-reading the paper to make sure that I understood all the details.
2. Somewhat limited novelty?: The key ideas proposed in the paper (multi-scale, Hierarchical Heritage, Diversity) are quite common in the explicit models (i.e., standard neural networks). It is not clear to me if adapting them directly to implicit models is quite a significant advance.
3. How would the approach scale and compare to state-of-the-art architectures (or even a larger ResNet) trained on ImageNet? 

summary_of_the_review: The paper proposes some interesting ideas for using multi-scale inputs and feature hierarchy and diversity for training equilibrium models. The results on CIFAR-10 look promising, but I have some concerns about the writing and the novelty of the work.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a multi-scale network (called Multi-Branch OptEqs or MOptEqs) that extends on recent results of [1]. Specifically, the authors propose to use several DEQ models to incorporate samples of multi-resolutions. The authors further propose a modified training strategy (called perturbation enhanced training) to improve the performance of the model. Empirically, the proposed model shows better performances compared to the previous DEQ models on CIFAR-10, CIFAR-100, and Imagenette datasets.

[1] Xie, Xingyu, et al. "Optimization Induced Equilibrium Networks." arXiv preprint arXiv:2105.13228 (2021).
main_review: Pros:
* Even though incorporating different input resolutions to improve the performance of the model is not novel, I believe that the design of the architecture to incorporate multiple scales is still an interesting problem.
* I checked the proofs and I believe that the work is technically correct. 
* Empirically, the proposed method achieves impressive results on CIFAR-10, CIFAR-100, and ImagImagenette and it is convincing that the proposed method outperforms MDEQ on these small-scale experiments.

Concerns:
* I have one concern about the additional computational/memory overhead of “unrolling”. I believe that the model size (number of parameters) is relatively small for CIFAR-10 experiments. However, for larger models (and more complex datasets such as ImageNet), do you think your method would scale to these settings?
* Following up on the previous concerns, I recommend the authors clearly state the additional overhead (wall-clock time/additional memory) in the results section to describe the potential limitations of the approach.
* Another concern I have is that, since the proposed method is trained by unrolling, I wonder if it is fair to compare against MDEQ? I believe that the direct comparison is unfair. For example, the proposed method does not have useful properties of equilibrium models (e.g. constant memory). However, we are solely comparing the performance between MDEQ and the proposed approach. 
* In the introduction, the authors mention that OptEqs have “mathematical interpretability”. Does this necessary correspond to, is it easier to interpret what the network is doing? If not, what is the core benefit of “mathematical interpretability”?
summary_of_the_review: I believe the idea of the paper is interesting and the authors show the impressive results on CIFAR experiments. However, there are some concerns regarding scalability (since the proposed method performs unrolling), additional overhead, and experimental fairness. Most importantly, the proposed approach is trained with unrolling to a fixed depth and does not have a property of equilibrium models. Hence, I give a score of 5.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper propose a new method which can efficiently utilize inputs with different scales, the propose method obtains better quantitative results in experiments.
main_review: Strengths:
1. The proposed method is straight forward and easy to follow, the newly proposed terms in the objective function have good motivations;
2. Model sizes in multiple-scale experiments are much smaller than related works, which illustrates the effectiveness and efficiency of the proposed method;

Weaknesses, questions & comments:
1. The performance improvements seem to be trivial, although it is impressive to see the model size comparison in multiple-scale experiment. It would be great if the author could provide comparisons of different methods with same model size (results of MDEQ with smaller model size and results of MOptEqs with larger model size). It would be interesting to see whether the proposed method also has good scalability;
2. More experiments on higher dimensional datasets are suggested;
3. Could the author provide some ablation studies investigating the contribution of each component (inner-product term and diversity term)?
4. Are the compared methods in tables trained with any kind of perturbation or not? It would clearer if all the methods are compared in two different settings: with and without perturbation;
summary_of_the_review: This paper propose a new method, which can handle different input scales. The  proposed method is well-presented with good motivation, obtains better results with smaller model sizes in experiments. There are several questions remaining, I'm looking forward to the responses and discussions.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: I don't quite understand this work. It seems authors proposed a new deep equilibrium model through incorporating a layer of optimization into the network. More specifically, authors may have extended the work MDEQ from two aspects: (1) inclusion of layers of deep neural network as optimizer and (2) introducing multi-branch architecture design. Experiments based on CIFAR demonstrate the advantage of the proposed solution in their settings. Theortical analysis with two propositions justify the properties of proposals.
main_review: Though authors include the concepts of interpretability in the motivation of this work. I cannot see the connections between interpretability to this work. Did authors define the interpretability as explanation of behaviors of CNN, such as LIME explains DNN for computer vision using the attribution to features? Could you please give an example to demonstrate how the proposal interpret the DNN.
summary_of_the_review: I failed to collect evidence to reject this paper, so I believe it might be a solide work. I hope authors could address the interpretability issues here. I hope AC and SPC could hear from other reviewers.
review_title: null

Meta-review: The paper proposes a multi-scale network that uses DEQ models to incorporate samples at multiple resolutions. The authors also propose a training strategy to improve the performance of the model. The authors investigate the interest of the approach through ablation and explainability, weighing the value of hierarchical heritage, diversity modules, perturbation size, and regularization penalties. 

The reviewers appreciated that the authors tackled the problem of incorporating multiple scales and the “impressive results” on CIFAR-10, CIFAR-100. The reviewers also expressed concerns regarding the computational assessment, in particular the additional computational/memory overhead of unrolling and what the authors mean by “explainability” in their experimental evaluation. The reviewers also made suggestions to organize the paper better. 

The authors submitted responses to the reviewers' comments. After reading the response, updating the reviews, and discussion, the reviewers who took part in the discussion considered that they are “satisfied by the response” and the “major concerns have been addressed”.  The feedback provided was already fruitful and the final version should be already improved. The ablative analysis and comparison to baseline is careful and thorough.

Accept. Poster.
Decision: Accept (Poster)