Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose a method to obtain multiple local optimal solutions around the existing one. The algorithm consists of moving away from the local minima along some random direction, but biasing the direction with a true gradient of the loss.
The solutions are then combined into a single result by taking them as an ensemble. 

I quite like the idea behind the the algorithm. Combining several nearby solutions might help the overall performance of the method. However, I see several important problems with the presented method. 

First, it is not clear how sensitive the algorithm is to the the learning rates \rho_1 and \rho_2. I believe that they would very much depend on the problem at hand and even the loss surface around the local function. 

Second, I had trouble understanding the theoretical ideas behind. The paper is a bit hard to read and would benefit a lot from proofreading. For example, SEP is never formally defined. It says at the bottom of p.4 that "SEPs (i.e. nearby LOSs)". So what is the difference between SEP and LOS? It is not clear why the variable updates are written as a state dynamical system instead of regular iterations. Was that necessary in any way?  It is not well described how exactly TRUST-TECH works and finds LOS. How exaclty high-quality local optima defined and how is it different from other local minima?
review_title: Algorithm to explore the space around local minima

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a new method for applying the TRUST-TECH method to the ensemble of deep neural networks (DNNs). When applying TRUST-TECH to a deep neural network, it is difficult to determine the direction and exit point. This paper introduces Dynamic Searching Paths (DSP) to solve these problems. The proposed method can apply TRUST-TECH method to DNNs using Stochastic Gradient Descent (SGD) with minor memory overhead. 

[+] Propose a method of applying TRUST-TECH to find local optimal solutions (LOS) of DNNs\
[+] Introduce DSP for exploration in high-dimensional parameter space\
[+] High ensemble performance through DSP-TT

[-] Claims not sufficiently proven or supported\
The paper repeatedly claims that the solution found by the proposed method is high quality and diversity. However, it is hard to find a theoretical and experimental basis for high-quality solutions. Indeed, the definition of a high-quality solution is also unclear. The following is the part that mentioned the quality of the local solution on page 1.
"A popular metric for the quality of a local solution is to measure its generalization capability, which shows the gap between the training and testing performances."
According to this explanation, high quality should have a small gap between training and testing performances, but none of the paper's experiments can support this.

Also, the relationship between diversity and performance is not convincing. The paper uses parameter distance and output correlation as measures for diversity. Tables 2 and 3 show that the relationship between these metrics and performance is very weak. In many cases, models with high output correlation have better performance, or models with close distances sometimes have higher performance. It can be seen from two aspects: these metrics are not suitable for measuring diversity, or the relationship between diversity and performance is weak.

[-] Unconvincing baseline results\
According to the FGE paper, ResNet-164 has errors of 20.2 and 4.54 with 150 epochs budget in CIFAR-100 and CIFAR-10, respectively, and 18.67 and 4.21 in 300 epochs. This is an almost similar result to DSP-TT with 200 epochs budget. However, in this paper, the FGE method uses 200 epochs and has errors of 20.75 and 4.67 in CIFAR 100 and CIFAR-10, respectively. Since it uses more budget, the performance should be higher, but it seems that FGE is not implemented properly. The explanation about this result is missing, making it difficult to trust the paper's experimental results.

[-] Minor comments\
The image quality in Figure 1 is poor, so it is difficult to identify text even if it is enlarged.

The proposed method is somewhat novel in the aspect of suggesting TRUST-TECH for an ensemble of DNNs. However, it is difficult to give a high score because the experimental results do not support 
review_title: Method for applying to TRUST-TECH neural networks ensemble

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper  proposes an intersting Dynamic Search Path TRUST-TECH training method for deep neural nets. In contrast to other global solvers, the proposed method efficiently explores the parameter space in a systematic way. Specifically, a Dynamic Searching Path method is proposed to make the original TRUST-TECH applicable to deep neural networks. Then,  the batch evaluation formula is used to increase the algorithm efficiency. Furthermore, the DSP-TT Ensembles is construed to  improve the model performance. I think the idea is interesting. The experimental results validate the effectiveness of the proposed method. The writing is good.  
review_title: Interesting idea; good performance

final_score: rating: 6: Marginally above acceptance threshold
review_content: ##########################################################################

Summary:
 
The paper describes a technique based on the modified generalized gradient descent for finding multiple high-quality local optima of deep neural networks. The search method does not require re-initialization of the model parameters and can be carried out in a single training session. Identified local optima are then used to build model ensembles which appear to outperform several other ensembling approaches.


##########################################################################

Reasons for score: 

 
Overall, I vote for accepting. While I was not entirely satisfied with how the paper is written and how the approach is introduced and explained, I find the results to be quite interesting. While I do not know the literature sufficiently well, I think this method is original and well-founded.

 
##########################################################################

Pros: 

1. While being simple and intuitive, the proposed method appears to succesfully and efficiently identify multiple high-quality local optima of a model.

2. Possibly even more interestingly, ensembles containing corresponding models appear to outperform other alternative approaches. While inference with ensembles of models can be quite costly (growing with the number of models in the ensemble) and a similar or better accuracy could potentially be achieved with larger simple models sharing the same computational cost, this result is nevertheless very promising.

3. The experimental methodology appears to be sound and some illustrative examples (like those shown in Figure 3) are interesting and insightful.

 
##########################################################################

Cons: 

1. Certain parts of the publication are not entirely well written and some sentences are a bit confusing. Also, the text contains quite a few misprints. Some more serious mistakes can be found, for example, in Table 2 (DenseNet results) and central equations (5) and (6), which seem to use a wrong sign for the gradient (current sign seems to correspond to gradient ascent and not descent thus maximizing the loss and not minimizing it). Also, as a very minor comment, I believe that, strictly speaking, the gradient (with respect to $\Delta$) of the loss in these equations should be computed at $\phi_{\omega_0}(t_{i-1})$ because otherwise the right-hand side of these equations would be dependent on $\Delta(t_{i})$.

2. In my opinion, the discussion in Section 3 could be clarified and simplified. Furthermore, I believe that the method could be explained and analyzed a bit better. For example, it would appear that the proposed difference equation (6) can be written as a gradient descent on the modified loss function with the added quadratic term $\sim (\rho_1/\rho_2) \Delta^2$. If correct, I find this simple perspective much more natural and insightful. This quadratic component can essentially flip the Hessian in the vicinity of the starting local minimum thus causing the trajectory to be repelled from it. This simple view also appears to have implications for what kind of local minima of the original loss (their Hessians) could finally attract such training trajectories, potential shifts due to finite $\rho_1$, and the role that the decay of $\rho_1/\rho_2$ could play in the process of convergence.

3. The related work section contains just a few ensemble papers and none after 2018. It would appear that this section could be expanded and include some more recent papers at least for reference.

##########################################################################

Questions during rebuttal period: 

Please address and clarify the cons above. (I will update my score based on the authors reply.)

Unfortunately, I am not an expert in this field, but two papers I came across doing a very quick search appear to be somewhat relevant: "Local minima found in the subparameter space can be effective for ensembles of deep convolutional neural networks" and "MEAL: Multi-Model Ensemble via Adversarial Learning". I am not sure these particular papers need to be included in the prior work section, but I do think that this publication would benefit from a more in-depth literature overview.

##########################################################################

Post-rebuttal.

Thanks for a detailed response that clarified some of my questions. I think the overall quality of the paper increased and I am happy to see additional information (like additional literature and an ablation study in Section 5.4) and a somewhat improved explanation of the core idea. However, I am still hesitant to give this paper a higher rating, in part because I find Section 5.4 to be poorly written and somewhat confusing (it lacks any sort of conclusion or insight and I cannot read Figure 4 at all, the text is too small) and partly because the paper does not clearly put its results in the context of the recent model ensemble progress (and thus makes me doubt the impact of this result on the field; although it does seem to be promising compared to the mentioned baselines).
review_title: Conceptually simple, but seemingly effective way of finding diverse local optima and building model ensembles

Meta-review: This work proposes a method to discover neighboring local optima around an existing one. Reviewers all found the idea interesting but argued that the paper needed more work. In particular, some of the claims are too informal or not sufficiently supported and the reviewers found the key section were difficult to follow. The paper should be resubmitted after improving the presentation of the results.
Decision: Reject