Constructing Multiple High-Quality Deep Neural Networks: A TRUST-TECH Based Approach
Authors: Zhiyong Hao
Hsiao-Dong Chiang
Bin Wang
Venue:  (2021)
Keywords: Nonlinear Dynamical Systems
Global Optimization
Deep Neural Networks
Ensemble.

Abstract: The success of deep neural networks relied heavily on efficient stochastic gradient descent-like training methods. However, these methods are sensitive to initialization and hyper-parameters. 
In this paper, a systematical method for finding multiple high-quality local optimal deep neural networks from a single training session, using the TRUST-TECH (TRansformation Under Stability-reTaining Equilibria Characterization) method, is introduced. 
To realize effective TRUST-TECH searches to train deep neural networks on large datasets, a dynamic search paths (DSP) method is proposed to provide an improved search guidance in TRUST-TECH method. 
The proposed DSP-TT method is implemented such that the computation graph remains constant during the search process, with only minor GPU memory overhead and requires just one training session to obtain multiple local optimal solutions (LOS). To take advantage of these LOSs, we also propose an improved ensemble method. Experiments on image classification datasets show that our method improves the testing performance by a substantial margin. Specifically, our fully-trained DSP-TT ResNet ensmeble improves the SGD baseline by 20\% (CIFAR10) and 15\%(CIFAR100). Furthermore, our method shows several advantages over other ensembling methods. 

Under review as a conference paper at ICLR 2021
CONSTRUCTING MULTIPLE HIGH-QUALITY DEEP NEURAL NETWORKS: A TRUST-TECH-BASE AP- PROACH
Anonymous authors Paper under double-blind review
ABSTRACT
The success of deep neural networks relied heavily efficient stochastic gradient descent-like training methods. However, these methods are sensitive to initializa- tion and syper-parameters. In this paper, a systematical method for finding mul- tiple high-quality local optimal deep neural networks from a single training ses- sion, using the TRUST-TECH (TRansformation Under Stability- raainin Equi- libria Characterization) method, is introduced. To realize effective TRUST-TECH searches to train deep neural networks large datasets, dynamic search paths (DSP) method is proposed to provide an improved search guidance in TRUST- TECH method. The proposed DSP-TT method is implemented such that the com- putation graph remains constant during the search process, with only minor GPU memory overhead and requires just one training session obtain multiple local optimal solutions (LOS). To take advantage of these LOSs, we also propose an improved ensemble method. Experiments on image classification datasets show that our method improves the testing performance by substantial margin. Specif- ically, our stainet DSP-TT ResNet ensmeble improves the SGD baseline by 15% (CIFAR10) and (CI(CIAR Furthermore, our method shows several advantages over other ensembling methods.
INTRODUCTION
Due to the high redundancy on parameters of deep neural networks (DNN), the number of local optima huge and can grow exponentially with the dimensionality of the parameter space (Auer et al. (1996); Choromanska et al. (2015); Dauphin et al. (2014b)). It still remains challenging task to locate high-quality optimal solutions in the parameter space, where the model performs satisfying both training and testing data. A popular metric for the quality of a local solution is to measure its generalization capability, which is commonly defined as the gap between the training and testing performances (LeCun et al. (2015)). For deep neural networks with high expressivity, the training error is near zero, so that it suffices to use the test error to represent the generalization gap. Generally, local solvers do not have the global vision of the parameter space, so there is no guarantee that starting from a random initialization can locate a high-quality local optimal solution. On the other hand, one can apply a non-local solver in the parameter space to find multiple optimal solutions and select the high-quality ones, Furthermore, one can improve the DNN performance by ensembling these high-quality solutions with high diversity.
TRUST-TECH plays an important role in achieving the above goal. In general, it computes high- quality optimal solutions for general nonlinear optimization problems, and the theoretical founda- tions can be bound in (Chiang & Chu (199)); Lee & Chiang (2004)). It helps local solvers escape from one local optimal solution (LOS) and search for other LOSs. has been successfully ap- plied in guiding the Expectation Maximization method to achieve higher performance (Reddy et al.) (2008)), training ANNs (Chiang & Reddy (2007): Wang & Chiang (201T)), estimating finite mix- ture models (Reddy et al. (2008)), and solving optimal power flow problems (Chiang et al. (2009); Zhang & Chiang (2020). Additionally, does not interfere with existing local or global solvers, but cooperates with them. TRUST-TECH efficiently searches the neighboring subspace of the promis- ing candidates for new LOSs in a tier-by-tier manner. Eventually, a set of high-quality LOSs can be found. The idea of TRUST-TECH method is the following: for given loss surface of op-

---

Under review as a conference paper at ICLR 2021
timization problem, each LOS has its own stability region. If one start from one local optimum, and track the loss values along a given direction, we will find an exit point where loss start to de- crease steadily, which means another stability region corresponding to nearby LOS is found. By following a trajectory in the stability region, another LOS is computed.
We propose an optima exploring algorithm designed for DNNs that is able to find high-quality local optima in a systematic way, and thereby form optimal and robust ensembles. Normally for deep neural network, exit points can hardly be found by original TRUST-TECH due to the huge dimensionality. So, in this work we introduce the Dynamic Searching Paths (DSP) method instead of fixed directions. We set the search directions to trainable parameters. After an exploration step forward along the current direction, we calibrate the direction using the current gradient. By doing so, the method can benefit from not only the mature Stochastic Gradient Descent (SGD) training paradigm with powerful GPU acceleration capability, but also exit points can be easily found.
The overall DSP-TT method consists of four stages. First, we train the network using local solvers to get tier-0 local optimal solution. Second, our proposed Dynamic Search Path TRUST-TECH (DSP-TT) method is called to find nearby solutions in a tier-by-tier manner. Third, a selection process is performed so that candidates with high quality are chosen. Finally, ensembles are built with necessary fine-tunings on selected member networks. To the best of our knowledge, this paper is the first one to search for multiple solutions on deep neural networks in systematical way.
Our major contributions and highlights are summarized as follows:
We propose the Dynamic Searching Path (DSP) method that enables exploration on high- dimensional parameter space efficiently.
We show that combining TRUST-TECH method with DSP (DSP-TT) is effective in finding multiple optimal solutions on deep neural networks systematically.
We design and implement the algorithm efficiently that it obtains multiple local solutions within one training session with minor GPU memory overhead.
We develop the DSP-TT Ensembles of solutions found by DSP-TT with high quality and diversity for further improving the DNN performance.
2 RELATED WORK
The synergy between massive numbers of parameters and nonlinear activations in deep neural net- works leads to the existence of multiple LOSs trained on specific dataset. Experiments show that different initializations lead to different solutions with various qualities (Dauphín et al. (2014a)). Even with the same initialization, the network can converge to different solutions depending on the loss function and the solver (Im et al. (2016)) Many regularization techniques are therefore pro- posed to foree the network to converge
better solution, some of which are proven to be useful and popular (Kingma& & Ba (015); ; Srivastava et al. (2014); loffe & $2egedy((0015)) However, is still mysterious how these regularized solutions compared to the global optimum.
There are researchers that focus on characterizing different local optima and investigating the inter- nal relations among them. It is claimed in (Hochreiter & Schmidhuber (1997): Keskar et al. (2016) that sharp minima prevent deep neural networks from generalizing well on the testing dataset. Later, Dinh et al. (2017) argued that the definition of flatness in (Keskar al. (2016) is problematic and came up with an example where solutions with different geometries can have similar test time per- formances. Li et al. (2018) designed new visualization method that rebuilt the correspondence between the sharpness of the minimizer and the generalization capability. On the other hand, some researchers apply meta-heuristic a algorithms to obtain a better local minimizer (Gudise & Venayag- amoorthy (2003): Zhang et al. (2007); [uang (2004); Leung et al. (2003). However, these methods were either designed for obsolete toy models or on explicit benchmark objective functions where there are analytical forms for global optimum, and therefore the effectiveness of these algorithms on deep architectures and large datasets seems unconvincing. Moreover, the advantage of the global searching ability seems to be crippled when it comes to deep neural networks, and the minimizers they found are still local. Recently, Garipov et al. (2018) reveal the relation among local optima by building pathways, called Mode Connectivities, as simple as polygonal chains or Bezier curves that
2

---

Under review as a conference paper at ICLR 2021
connect any two local optima. Draxler et al. (2018) also found similar results at the same time, al- though they used the Nudged Elastic Band (Jonsson et al. (1998) method from quantum chemistry.
To address the issue of converging to suboptimal solutions, a great deal of research efforts were directed to ensembles. Xie et al. (2013) proposed horizontal and vertical ensembles that combine the output of networks different training epochs. [aine& & Aila (2016) used group of models
with different regularization and augmentation conditions to create variety. MMghimi et al. (2016)
borrowed the concept of boosting create a strong ensemble of CNNs. [Imaillv et al. (0188)
found that averaging weights from different iterations leads to flatter solutions than from SGD and helps in generalization. Huang et al. (2017a) proposed method that obtains ensembles by col- lecting several local minima along single training process using a cyclical learning rate schedule. Zhang et al. (2020) used similar approach, but with the sampling capability that fully exploits each mode. (aaripvv et al. (2018) developed the Fast Geometric Ensembling based on Mode Connectivity. Although the methods in these papers obtain multiple networks within one training session, these ensembles are still largely dependent on initialization. While these ensemble methods performs bet- ter than a single network, naive randomly initialized ensemble is still the best choice when training they explore different
budget is unconstrained. |Fort al. (020) explained this phenomenon modes in function space compared to weight averaging. [Shen et al.] (2019) improved the ensemble inference efficiency via teacher-student paradigm distilling the knowledge of an ensemble into one single network. Yanng et al.] (2020) built ensembles by randomly initialize on a subparameter space, aiming alleviate the exponentially growing number of local minima on deep networks. Wang & Chiang (201T) used the TRUST-TECH (Chiang & Chu (1966): Lee & Chiang (2004); Chiang & Alberto (2015) method to perform a systematic search for diversified minimizers to obtain their ensembles. They implemented TRUST-TECH for training and constructing high-qualit ensembles of artificial neural networks and showed that their method consistently outperforms other training methods. We generalize this method and tailor it for deep architectures and to work efficiently with popular local solvers in deep learning.
TRUST-TECH METHOD FOR MULTIPLE OPTIMAL SOLUTIONS
3.1 TRUST-TECH METHODOLOGY
Another category of methods has been developed in recent years for systematically computing a set of local optimal so- lutions in deterministic manner. This family of methods is termed TRUST-TECH methodology, standing for Transforma- tion Under Saability-reTainiine Equilibria Characterization. It is based on the following transformations:
Tier-2 a
00
the transformation of a local optimal solution (LOS) of a nonlinear optimization problem into a stable equilib- rium point (SEP, Chiang & Chu (1966) of a continuous nonlinear dynamical system.
Her-1
Tier
Tier-1
TiersP
(ii) the transformation of the search space of nonlinear opti- mization problems into the union of the closure of stabil- ity regions of SEPs.
60,
Tier-1
Tier-2
Hence, the optimization problem (i.e. the problem of finding LOSs) is transformed into the problem of finding SEPs, and therefore we use the terms LOS and SEP interchangeably in the following discussion. It will become clear that the stability regions of SEPs play an important role in finding these local optimal solutions. We note that, given LOS, its correspond- ing first-tier LOSs are defined as those optimal solutions whose corresponding stability boundaries have a non-empty intersec- tion with the stability boundary of the LOS (Chiang & Chu (1996); Lee & Chiang (2004)). The definition of the stability boundary and its characterization can be found in Chiang & Fekih-Ahmed (199)). Similarly, its second-tier LOOs are defined as those optimal solutions whose
Given LOS (i.c. wa tier-zero LOS), the corresponding tier- LOSs are W1.1. W1.2, W1.s. Similarly, its tier-2 LOSs are W2,1, W2,2, w2,3. Note that the corresponding stability boundaries
Figure 1:
tier-2 have non-empty intersection with the stability bound- aries of its tier-1 LOSs.
3

---

Chiang & Under review as a conference paper at ICLR 2021
corresponding stability boundaries have non-empty intersection with the stability boundary of first-tier LOSs (Chiang & Chu (199)); Lee & Chiang (2004)). See fig. | for an illustration.
We consider general nonlinear unconstrained optimization problem defined as follows:
min c(x)
(1)
where D C R" + R is assumed to be continuously differentiable and D the set of feasible points (or search space). A point x² E D is called local minimum c(x*) I c(x) for all D
with | x"1
for > 0.
To systematically search for multiple LOSs, a generalized negative gradient system based on the objective eq. is constructed and is described by
dr di -rradR c(x) -R(x)- Ve(x) f(x(t))
(2)
where the state vector ztt) of this dynamic system belongs to the Euclidean space R and the function f IR" - R satisfies the sufficient condition for the existence and the uniqueness of the solutions. R(x) positive definite symmetric matrix (also known as the Riemannian metric) that generalizes various training algorithms. For example, if R(x) = (identity), it is naive gradient descent algorithm. R(x)) = I (x) J(x) (I is the Jucobian matrix), then it is the Gauss-Newton method. If R(x)= J(x) ((x) + pl, it becomes the gevenberg-Marquurrll (LM) algorithm.
The Theorem of the Equilibrium Points and Local Optima (Lee Chiang (2004)) shows one nice property of the gradient system (2). which is the critical point of the optimization problem ( is a (asymptotically) SEP of the dynamic system (2). i.c. 5 is a SEP of (2) if and only if 2 is an isolated local minimum for TT. Hence, the task of finding the LOSs of (1) can be achieved by
finding the corresponding SEPs of (2). In short, TRUST-TECH a dynamical method designed to systematically compute multiple LOSs with the following features:
is a systematic and deterministic method to escape from a LOS towards another LOS,
(ii) finds multiple LOSs in a tier-by-tier manner (see fig. D and
(iii) has solid theoretical foundation (Chiang & Chu 1996): Lee & Chiang 2004
Alberto 2015); Zhang & Chiang (0200).
Another distinguishing feature of TRUST-TECH is its ability to guide a local method and/or meta- heuristic method for effective computation set of LOSs or even the global optimal solution.
3.2 SYSTEMATIC SEARCH ON DEEP NEURAL NETS
The center idea is to find multiple LOSs in tier-by-tier manner. On small scale problems, applying fixed searching directions is proven to be effective in practice (Chiang et al. (2009); Reddy et al. (2008); Chiang & Reddy (2007); ) Waang & Chiang (201T)). In these applications, either random directions or eigen-vectors of the objective Hessian evaluated at each SEP were used. But in training deep neural networks, finding proper direction is challenging. For a deep neural network, when searching along a random and fixed direction, the loss value will grow indefinitely. Another issue is that the computational cost of the original TRUST-TECH is high. Specifically, it assumes cheap evaluation of the objective function each search step. However, in supervised learning of large dataset, only the empirical loss is accessible instead of the ground-truth objective function. This means evaluating the loss function for the entire training set, which is almost impossible since is limited by computational restrictions.
Our method follows the paradigm of the TRUST-TECH.
To tackle both challenges, we propose the Dynamic Search Path (DSP) method that enables explo- ration on deep neural networks' parameter space. Furthermore, we apply the DSP method serve as the search paths for TRUST-TECH (DSP-TT), Details are discussed in section 3.2.1 An example of one-tier DSP-TT method is shown Algorithm I
3.2.1 1 OBTAINING DYNAMIC SEARCHING PATHS FOR TRUST-TECH
In this section, go through the details how to construct dynamic searching paths during TRUST-TECH computation and how it helps converge to nearby LOSs, Construction of search- ing path is inspired by the mode connectivity proposed in Garipov al. (2018), in which the authors
4

---

Under review as a conference paper at ICLR 2021
Algorithm One-Tier DSP-TT Search
1: procedure "1SeARcH(mmdde dataset, maziter, batchsize)
2: Initialize paths, candidates = {model parameters]
D intialize search paths and solution set.
3: for k + 1 maziter do
batch + c
batchsize)
Pl: P2 f- Schedu le e(iter, maniter, exittfooudd)
update learning rates b Randomly select one path D. forward search step check for exit kth path
6:
Aş + Sel Select (paths)
7:
Update(model, P1)
emit. found +- Checkexit Ak, batch) if ewil ceiilffuree found then
10:
We,k + LocalSoLvez(modier Ae, dataset) condidates = candidates (wa.k
b converge to tier-1 optimum b update solution set
11:
12:
else
13:
Calibrate (model, P2y Aky batch)
P calibration step
14:
return condidates
found there exists low-loss "tunnels" between different LOSs. But Mode Connectivity is used to find a high-accuracy pathway between two local solutions by optimizing the expectation form distribution on the path. Our focus is finding the proper searching directions towards nearby optima when starting from one LOS. They also claimed that path @g cannot be explicitly learned when given one starting point. However, we find that such a construction is possible. Specifically, by redesigning the objective and combining the exploration capability of TRUST-TECH and the exploitation capability of SGD, another local optimum can be found starting from one LOS. More generally, by using such an optimization-bait based math-finding technique, one can find multiple tier-one SEPs (i.e. nearby LOSs) simultaneously.
uni-
Region of
Region 60
Exit Point
%
sraldey of 0b
Begion of (B
(a)
(b)
Figure 2: (a) Phase portrait of two-dimensional objective surface, wp and wi are two SEPs; (b) demon- stration of the DSP methad on the same objective. Black arrows represent the DSP path, blue vectors represent forward search steps, and red vectors represent calibration steps.
To do this, we first train the neural network to obtain a LOS wp E RIneel, Then we define a trainable search direction vector d, E R mot] (randomly initialized do), that during TRUST-TECH search from Wg. instantancous parameter vector at step i represented as (wb + d,).
At each step i, DSP updates the direction d, as:
d, (1) d + pz(1) f(wo + d.)
The first term describes P1 (i) d: the original TRUST-TECH search with no direction calibration, where (i) E (0, Pmaz] is the step size schedule for exploration phase whose value increases from O to Pmaz w.r.t. step i. The second term is the DSP calibration term, where p2(14) is the step size schedule for the calibration, descenty represents a general local descent solvers, such as Gradient Descent, Newton's 's Method, etc. f(.) is the dynamics defined in Equation (2). where various local solvers can be applied here. The stopping criteria of Pi (t;) is determined dynamically by either an exit point is found or Pmazz is reaches.
5

---

Under review as a conference paper at ICLR 2021
The above steps repeats until (wy + d,) converges another LOS, which call it tier-1 solution associated with wo. An intuitive demonstration of this process is shown in Figure 2b
Our proposed scheme is scalable to performing multiple search directions starting from one LOS. To do this, we initialize multiple directions, and at each step, each search direction is updated via eq. (3) It is also worth noting that during training, the computation graph size is the same as the original network, since the algorithm only picks one direction to be included in the computation graph. Thus, minor memory overhead is introduced in practice. As for the computational efficiency, our proposed method evaluates objectives on mini-batches instead of the entire dataset, and determines the stopping criteria by an exponential moving average of past batch evaluations. To further stabilize the stochastic behavior caused by mini-batch evaluations, buffer variables are used to determine the state transition between up (loss values are climbing current stability region) and down (reaches nearby stability region and the loss decreases steadily). These resolve the efficiency issue of the original TRUST-TECH on large scale supervised learning problems.
DSP-TT ENSEMBLES OF DEEP NEURAL NETWORKS
When training budget is less constrained, high-quality of each tier-I solution is emphasized as having better test accuracy than the tier-0 network. On the other hand, for building ensembles with a limited budget, high-quality is emphasized more on the diversity among the collection of local optimal neural networks found to better serve the ensemble, in stead of on single network.
With the proposed DSP-TT, set of optimal network parameters with high accuracy can be found systematically given enough training budget, and with limited budget, the high diversity among tier- O and tier-1 solutions still remedies the weaker performance on tier-1 networks when serving the ensemble. Individual qualities are guaranteed because the starting point of any search is already LOS from mature SGD-based solvers with high quality, which is also shown from the experiments, especially in Table 4 As for diversity, SEPs (i.e. optimal parameter values, or LOS)
separated by at least two stability regions because each SEP has its own stability region. It is necessary to initialize parameters in different stability regions in order to find multiple optimal solutions. The proposed TRUST-TECH based method is systematic in characterizing stability regions while other heuristic-based algorithms is also high due to the mutual exclusiveness of stability regions.
not. And therefore, the diversity among SEPs found by our method
The high-quality LOSs with high diversity further motivate us to build ensembles to make a more robust and accurate model than each single member. First, a list of candidates with high quality and diversity are selected. After that, a fine-tuning process is executed if necessary to help any under- fitted candidates toward greater convergence. Since the searching process already integrates the gradient information, the fine-tuning in our algorithm requires little effort. fact, as shown in the experiments, fine-tuning does not show benefit for the ensembling performance, so this procedure ignored by default. Finally, we build the final ensembles by either averaging (regression) or voting for (classification) the outputs. Sophisticated ensembling methods can be applied here, however is out of the scope of this paper.
5 EXPERIMENTS
Exit point verification is run using MLPs on UCI-wine and MNIST datasets. Further experiments are run using VGG-16 16 (Sioonyan & Zisserman (2014), DenseNet- 100-BC (Huang et al. (2017))) and ResNet- 164 (He et al. (2016) on CIFAR datasets. The program is developed on PyTorch framework. Each configuration is run multiple times and the average performance are shown.
Hyperparameters Training budget: DenseNet has 300 epochs of training budget, and ResNet/VGG has 200 epochs. Batch size: 128 for VGG and ResNet, and 64 for DenseNet. DSP-IT parameters: P1 increases 0.001 per iteration, P2 is 0.1x of the initial tier-0 learning rate. Fine-tuning phase requires 10 epochs per solution. All others: DenseNet follows Huang al. (2017b), VGG and ResNet follows (Garipov et al. (2018).
For DTP-TT ensembles, exit points are usually found in negligible time (e.g. around Imin CIFAR compared to a full training which takes hours). So 50 epochs are given to one tier of DSP-TT search with all exit points, while the rest of the budget are given to tier-0 training.

---

Under review as a conference paper at ICLR 2021
020
021
-11
Sund Seeioe
brand
-12
022
034-12
-04
6.3
93
(a)
(b)
(c)
Figure 3: (a) Loss progress on the training and testing set of MNIST during a DSP-TT(full gradient) search. (b) Error rate progress on the training and testing set of CIFARI00 during a DSP-TT(batched) search. (c) Exit Point Verification: Points along the search path near the exit point (top left with "X" marker) are sampled and then integrated until convergence. The points before (red) and after (blue) exit converge to different LOSs.
5.1 EXIT POINT VERIFICATION
Exit points play important role in TRUST-TECH method in finding multiple local optimal lutions. Figures 3a and 3b shows full gradient and batch version of a loss change with respect to the DSP-TT search iterations along one search path. The loss value first goes up, escaping from the tier-0 solution. At a certain point, the loss reaches local maximum and then goes down, suggesting that the search path hits the stability boundary and enters a nearby stability region.
To further verify that an exit point lies on the stability boundary, we do the following visualization: Several points along the search path near the exit point sampled. Then forward integration (gradient descent with small step size) is executed starting from each sample. Trajectories are plotted by projecting the parameter space onto two random orthogonal directions. Due high computation cost, this process is only simulated using a 1-layer MLP with 5 neurons (61 parameters) trained on UCI-wine dataset. Each integration process is executed for 50,000 steps with step size of 0.01. As shown in fig. 3c| The points before (red) and after (blue) exit converge two different points on the 2D projection space. We also observe the cosine between initial and updated search directions remains close to 1.0 throughout the search process, suggesting that gradients only calibrate extreme dimensions of the initial direction, but does not interfere with the remaining majority of dimensions.
5.2 TIER-BY-TIER SEARCH
The proposed DSP-TT computes: tier- one (from tier-zero LOS) and 5 tier- two (from the best tier-one LOS) LOSs. Among these, perform the following ensembles: Tier- (5 tier-one LOSs); Tier- I-tune (5 tier-one LOSs, each with fine- tuning); Tier-0-1 (1 tier-zero and 5 tier-one LOSs); Tier-0-1-2 (1 tier-zero. 5 tier-one and 5 tier-two LOSs). We use SGD as the local solver and DenseNet the architec- ture. As shown in table [ all DSP-TT- enhanced ensembles outperform the base- line model. Although Tier-0---2 performs mostly best among all, is sufficient to use Tier-0-1 in practice for efficiency, and therefore we use Tier-O-I in all the following experiments. From table I we also find that although fine-tuning individuals can improve its performance. This shows that the diversity introduced by our algorithm dominates the fine-tuning improvements by individuals. So in later experiments, all fine-tunings are neglected.
CIFARIO
CIFAR
err (%)
loss
err (%)
loss
Baseline
5.92*
24.15*
Tier-1
4.65
0.1622
22.29
0.9014
Tier-I-tum
4.64
0.1670
22.29
0.9206
Tier-0-1
4.34
0.1433
20.85
0.8323
Ticr-0-1-2
4.36
0.1396
20.56
0.8090
Table Error rate and cross-entropy loss the test of CIFARIO and CIFAR100 datascts with different tier settings. (Baseline: the DenseNet- 100 100-BC trained by SGD. ": num- bers from Huang al. (2017))
performance, it does not help much the ensembles
7

---

Under review as a conference paper at ICLR 2021
Test Error (%)
Output Correlation
(rchitecturo(Budget)
Method(#models)
C10
C100
C10
C100
Individual()
6.75
27.4'
S
Huang al.
2017a))
6.57"
26.4'
/
VGG16(200)
FGE(22)
Gampov et al.
2018
6.48"
25.70"
DSP-TT(6)
6,56
25.64
0.976
0.957
Individual(1)
4.72
21.50
SSE(5)
4.67
20.68
0.917
0.815
ResNet-164(200)
PGE(6)
4.67
20.75
0.944
0.902
DSP-TT(6)
4.53
20.20
0.898 /
0.773
Individual(I)
5.92
24.15
SSE(6)
4,48
21.08
0.946
0.901
DenseNet- DenseNet-100(300)
FGE(6)
4.82
22.82
0.992
0.991
DSP-TT(6)
4.34
20.85
0.903
0.832
Performance comparison among ensemble methods various architectures CIFAR datasets. (Out- put Correlation: mean Pearson Correlation Coefficients among members. ": numbers from Garipov et al.) (2018). * numbers from Huang et al. (b). Source codes for FGE http://ittub..ommmiaaaapoooo mode-connectivity) and SSE (htts://github.cc hwt..//aaaaaaaggnaaaoooooiiinnn tively, and present the best results we could achieve.)
Table 2:
are from the authors, respec-
Test Error (%)
Parameter Distance
Output Correlation
#modols#models; Budget)
C10
C1O0
CTO
CTO0
CTO
C100
Individuai (12000
4.72
21.50
mndividual-Ens(6. (200)
4.35
19.79
42.19
71.64
0,912
0.756
6SE(6;200) (Huang et al. 201 7
4.67
20.68
29.70
44.18
0.917
0.815
FGE(6:200
(Gariov 2018)
4.67
20.75
12.70
19.06
0.944
0.902
DSP-TT-0- DSP-TT-O-1(62200)
4.53
20.20
39.41
69.09
0.898
0.773
DSP-TT-OO-1ffull-rrin 1200)
3.99
18.67
35.82
82.11
0.929
0.810
Table 3: Detailed comparison with ResNet- 164 trained on the CIFAR datasets. (Parameter Distance: Euclidean
distance of parameters.)
5.3 COMPARISON WITH OTHER ENSEMBLE ALGORITTMM
In this section, we compare our method with other popular ensemble methods (Huang et al. 2017a); Garipov et (2018) deep learning. Results are shown in tables Manle 3
Besides accuracy, member diversity is another major quality for ensembles. Ide- ally, we want all members perform relatively well, while each member learns some knowledge that differs from that of others. We measure the output corre- lation (Huang In table And a more detailed analysis in table ( shows that both parameter distance and output correlation by DSP-TT Ensembles are better than SSE and FGE, and are similar level to those of Individual En-
al. (2017a)) and the parameter distance (Garipov et al.) (2018). the correlation by DSP-TT outperforms other ensemble methods.
Test Error (%)
CIFAR-10
CIFAR-100 100
Tier-O
5.00
21.98
sembles (multiple networks trained from scratch). Moreover, our fully trained DSP-TT Ensembles out- performs Individual Ensembles, and improves the individual baseline by 15% (CIFAR10) and 13% (CIFARIOM) 100). table 4 shows that fully trained tier- networks performs least as good as the tier-0 network. This suggests that training from an exit point found by DSP-TT method better than from random initialization. It is notable that in multiple cases, FGE members are more correlated, indicat- ing that these members are not multiple LOSs, but perturbations near one LOS, From this perspective, FGE be regarded fine-tuning around one local optimal point.
Tier- 1(#1)
4.86
21.58
Tier-
4.68
21.30
((#3)
4.78
21.72
Tier-1
4.89
21.98
Tier-1
4.73
21.72
Table 4: error the DSP-TT-O-1-füu 1-fT----ffllltrall Ensemble from table tier-1 networks performs at least as good as the tier-O network.
error rate of individual networks of
8

---

Under review as a conference paper at ICLR 2021
From the hardware side, DSP-TT search process introduces minor overhead to the GPU memory usage. Specifically, baseline training of ResNet- 164 takes 3819Mb GPU memory, which increases to 3921Mb during DSP-TT search. This justifies our previous claim that TRUST-TECH does not increase the size of the computation graph with only a little additional overhead.
5.4 ABLATION TEST ON DSP-TT HYPERPARAMETERS
The key hyperparameters for DSP-TT are PI (pace of search step) and p2 (step size of calibration step) defined in Section 3.2. In this part we test the sensitivity of the two. We perform tests on a grid of deL de P2) pairs, and record (1) the number of iterations to finish DSP-TT search for exit points, (2) average P1 of each search path when an exit point is reached, and (3) average distance between the search origin (tier-0 solution) and each exit point. As shown in Figure A DSP-TT is insensitive to P2. And figs. [4b and 4e (1) #1 and the distance between tier-0 and exit points are highly correlated, and (2) The surface becomes flat after the increment speed of P1 passes 5e 4, suggesting that other stability regions are reached.
(b)
(c)
Figure 4: Sensitivity test #1 p2. X-axis: increment rate of p1; Y-axis: pa; Z-axis: (a) Running iterations when all exit points reached. (b) Average #1 when each exit point is reached. (c) Average distance from the search origin when each exit point is reached.
CONCLUSION AND FUTURE WORK
In this paper, we propose a novel Dynamic Search Path TRUST-TECH training method for deep neural nets. Unlike other global solvers, our proposed method efficiently explores the parameter space in systematic way. To make the original TRUST-TECH applicable to deep neural networks, we first develop the Dynamic Searching Path (DSP) method. Second, we adopt the batch evaluation formula to increase the algorithm efficiency. Additionally, to further improve the model perfor- mance, we build the DSP-TT Ensembles. Test cases show that our proposed training method helps individual models obtain better performance, general purposed, so that can be applied to various architecture with various local solver.
when tier-1 search is applied. Our method is
Moreover, it is observed from Table percentage improvements in rate is not as significant as that in loss. This suggests that the cross-entropy loss may be the bottleneck for further improve- ments in performance for classification tasks. Thus, designing proper loss function that can be more sensitive to classification accuracy would be a valuable topic in the future.
REFERENCES
Peter Auer, Mark Herbster, and Manfred K Warmuth. Exponentially many local minima for single neurons. In Advances in Neural Information Processing Systems 8, pp. 316-322. 1996.
Hsiao-Dong Chiang and Luis F. C. Alberto. Stability Regions of Nonlinear Dynamical Systems: Theory, Estimation, and Applications. Cambridge University Press, Cambridge, U.K., 2015.
Hsiao- Hsiao-Dong Chiang and Chia-Chi Chu. A systematic search method for obtaining multiple local op- timal solutions of non- conelinearr programming problems. IEEE Transactions on Circuits and Systems I: Fandamental Theory and Applications, 43(2):99-009, 1996.

---

Under review as a conference paper at ICLR 2021
Hsiao- Isiao-Dong Chiang and L. Femie-Ahmed. Quasi-stability regions of nonlinear dynamical systems: Theory. IEEE Transactions on Circuits and Systems 1: Fundamental Theory and Applications, 43:627-635, 1996.
Hsiao-Dong Chiang and Chandan K. Reddy. Trust-tech based neural network training. In Interna- tional Joint Conference on Neural Networks, pp. 90-95, 2007.
Hsiao-Dong Chiang. Bin Wang, and Quan- Quan-Yuan Jiang. Applications of trust-tech methodology in optimal power flow of power systems. In Optimization in the Energy Industry, pp. 297-318, 2009.
Anna Choromanska, Mikael Henaff, Michael Mathieu, Gérard Ben Arous, and Yann LeCun. The loss surfaces of multilayer networks. In Artificial Intelligence and Statistics, 2015.
Yann N. Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non-convex opti- mization. In Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2, pp. 2933-29229 2941, 2014a.
Yann N Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in nigh-dimensional non-convex op- timization. In Advances in Neural Information Processing Systems, 2014b.
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pp. 1019-1028. PMLR, 06-11 Aug 2017.
Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In Proceedings of the 35th International Conference on Machine Learning (PMLR), pp. 1309-1318, 2018.
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape per- spective. arXiv:1912.0275: 2020.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, and Andrew Gordon Wilson Loss surfaces, mode connectivity, and fast ensembling of dnns. In 32nd Conference on Neural Information Processing Systems (NeurIPS), 2018.
V.G. Gudise and G.K. Venayagamoorthy. Comparison of particle swarm optimization and back- propagation as training algorithms for neural networks. In Proceedings of the 2003 IEEE Swarm Intelligence Symposium. SIS 03, 2003.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural Computation, 9:1-42, 1997.
Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E. Hopcroft, and Kilian Q Weinberger. Snapshot ensembles: Train I, get M for free. In 5th International Conference on Learning Rep- resentations (ICLR), 2017a.
Gao Huang, Zhuang Liu, Laurens der Maaten, and Kilian Q. Weinberger. Densely nected convolutional networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017b.
Daniel Jiwoong Im, Michael Tao, and Kristin Branson. An empirical analysis of deep network loss surfaces. arXiv:1612.04010, 2016.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In The 32th International Conference on Machine Learning (ICML), 2015.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry P. Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. Conference on Uncertainty in Artificial Intelligence (UAI), 2018
10

---

Under review as a conference paper at ICLR 2021
Hannes Jonsson, Greg Mills, and Karsten W. Jacobsen. Classical and Quantum Dynamics in Con- densed Phase Simulations, chapter Nudged Elastic Band Method for Finding Minimum Energy Paths of Transitions. 1998.
Chia-Feng Juang. A hybrid of genetic algorithm and particle swarm optimization for recurrent network design. IEEE Transactions on Systems, Man, and Cybernetics, 34:997-1006, 2004.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. NKKYv1660,048366 2016.
Diederik Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization. International Conference Learning Representations (ICLR), 2015.
Samuli Laine and Timo Aila.
Temporal ensembling for semi-supervised learning.
arXiv:1610.02242,, 2016.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553)436-4444 2015. doi: 10. 153../
Jacwook Lee and Hsiao-Dong Chiang. A dynamical vujectory-based methodology for systemati- cally computing multiple optimal solutions of general nonlinear programming problems. IEEE Transactions Automatic Control, (9(6))888-89) 2004.
Frank H. F. Leung, H. K. Lam, S. H. Ling, and Peter K. S. Tam. Tuming of the structure and parameters of a neural network using an improved genetic algorithm. IEEE Transactions on Neural Networks, 14:79-88, 2003.
Hao Li, Zheng Xu, Gavin Taylor, and Tom Goldstein. Visualizing the loss landscape of neural nets. In NeurlPS, 2018. URL http://///pppeeeeeeeeeet///ooaee
Mohammad Moghimi, Mohammad Saberian, Jian Yang, Li-Jia Li, Numo Vasconcelos, and Serge Belongie. Boosted convolutional neural networks. In British Machine Vision Conference (BMVC), York, UK, 2016.
Chandan K. Reddy, Hsiao- Doooo Chiang, and Bala Rajaratnam. Trust-tech-based expectation maxi- mization for learning finite mixture models. IEEE Transactions Pattern Analysis and Machine Intelligence, 30:1 146-1157, 2008. 2008
Zhiqiang Shen, Zhankui He, and Xiangyang Xue. Meal: mudti-model ensemble via adversarial learning. In AAAI, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv 1409.1550 09 2014.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way prevent neural networks from overfitting. Journal of Machine Learning Research, 19999999988 2014.
Bin Wang and Hsiao-Dong Chiang. Elite: Ensemble of optimal input-pruned neural networks using trust-tech. IEEE Transactions Neural Networks,
201
Jingjing Xie, Bing Xu, and Chuang Zhang. Horizontal and vertical ensemble with deep representa- tion for classification. arXiv: 1306.2799, 2013.
Yongquan Yang, Haijum Lv, Ning Chen, Yang Wu, Jiayi Zheng, and Zhongxi Zheng. Local minima found in the subparameter space be effective for ensembles of deep convolutional neural networks. Pattern Recognition, 109(2021), August 2020.
Jing-Ru Zhang, Jun Zhang, Tat-Ming Lok, and Michael R. Lyu. A hybrid particle swarm o propaggtono algorithm for feedforward neural network training. Applied Math- ematics and Computation,
2007.
Ruqi Zhang, Chunyuan Li, Jianyi Zhang, Changyou Chen, and Andrew Gordon Wilson. Cyclical stochastic gradient meme for bayesian deep learning. In ICLR, 2020.

---

Under review as a conference paper at ICLR 2021
Yong-Feng, Zhang and Hsiao- Chiang. Enhanced elite-load: A novel empsoatt methodology constructing short-term load forecasting model for industrial applications. IEEE Transactions on Industrial Informatics, 16:2325-2333 2020.
12


Reviews:
[{"review_id": "qIdnaGNyr1", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new method for applying the TRUST-TECH method to the ensemble of deep neural networks (DNNs). When applying TRUST-TECH to a deep neural network, it is difficult to determine the direction and exit point. This paper introduces Dynamic Searching Paths (DSP) to solve these problems. The proposed method can apply TRUST-TECH method to DNNs using Stochastic Gradient Descent (SGD) with minor memory overhead. \n\n[+] Propose a method of applying TRUST-TECH to find local optimal solutions (LOS) of DNNs\\\n[+] Introduce DSP for exploration in high-dimensional parameter space\\\n[+] High ensemble performance through DSP-TT\n\n[-] Claims not sufficiently proven or supported\\\nThe paper repeatedly claims that the solution found by the proposed method is high quality and diversity. However, it is hard to find a theoretical and experimental basis for high-quality solutions. Indeed, the definition of a high-quality solution is also unclear. The following is the part that mentioned the quality of the local solution on page 1.\n\"A popular metric for the quality of a local solution is to measure its generalization capability, which shows the gap between the training and testing performances.\"\nAccording to this explanation, high quality should have a small gap between training and testing performances, but none of the paper's experiments can support this.\n\nAlso, the relationship between diversity and performance is not convincing. The paper uses parameter distance and output correlation as measures for diversity. Tables 2 and 3 show that the relationship between these metrics and performance is very weak. In many cases, models with high output correlation have better performance, or models with close distances sometimes have higher performance. It can be seen from two aspects: these metrics are not suitable for measuring diversity, or the relationship between diversity and performance is weak.\n\n[-] Unconvincing baseline results\\\nAccording to the FGE paper, ResNet-164 has errors of 20.2 and 4.54 with 150 epochs budget in CIFAR-100 and CIFAR-10, respectively, and 18.67 and 4.21 in 300 epochs. This is an almost similar result to DSP-TT with 200 epochs budget. However, in this paper, the FGE method uses 200 epochs and has errors of 20.75 and 4.67 in CIFAR 100 and CIFAR-10, respectively. Since it uses more budget, the performance should be higher, but it seems that FGE is not implemented properly. The explanation about this result is missing, making it difficult to trust the paper's experimental results.\n\n[-] Minor comments\\\nThe image quality in Figure 1 is poor, so it is difficult to identify text even if it is enlarged.\n\nThe proposed method is somewhat novel in the aspect of suggesting TRUST-TECH for an ensemble of DNNs. However, it is difficult to give a high score because the experimental results do not support ", "extra_scores": {}, "extra_text": {}}, {"review_id": "isuc3f_nW4k", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a method to obtain multiple local optimal solutions around the existing one. The algorithm consists of moving away from the local minima along some random direction, but biasing the direction with a true gradient of the loss.\nThe solutions are then combined into a single result by taking them as an ensemble. \n\nI quite like the idea behind the the algorithm. Combining several nearby solutions might help the overall performance of the method. However, I see several important problems with the presented method. \n\nFirst, it is not clear how sensitive the algorithm is to the the learning rates \\rho_1 and \\rho_2. I believe that they would very much depend on the problem at hand and even the loss surface around the local function. \n\nSecond, I had trouble understanding the theoretical ideas behind. The paper is a bit hard to read and would benefit a lot from proofreading. For example, SEP is never formally defined. It says at the bottom of p.4 that \"SEPs (i.e. nearby LOSs)\". So what is the difference between SEP and LOS? It is not clear why the variable updates are written as a state dynamical system instead of regular iterations. Was that necessary in any way?  It is not well described how exactly TRUST-TECH works and finds LOS. How exaclty high-quality local optima defined and how is it different from other local minima?", "extra_scores": {}, "extra_text": {}}, {"review_id": "eeBWynzxuto", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper  proposes an intersting Dynamic Search Path TRUST-TECH training method for deep neural nets. In contrast to other global solvers, the proposed method efficiently explores the parameter space in a systematic way. Specifically, a Dynamic Searching Path method is proposed to make the original TRUST-TECH applicable to deep neural networks. Then,  the batch evaluation formula is used to increase the algorithm efficiency. Furthermore, the DSP-TT Ensembles is construed to  improve the model performance. I think the idea is interesting. The experimental results validate the effectiveness of the proposed method. The writing is good.  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "4pkewgbl6up", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n \nThe paper describes a technique based on the modified generalized gradient descent for finding multiple high-quality local optima of deep neural networks. The search method does not require re-initialization of the model parameters and can be carried out in a single training session. Identified local optima are then used to build model ensembles which appear to outperform several other ensembling approaches.\n\n\n##########################################################################\n\nReasons for score: \n\n \nOverall, I vote for accepting. While I was not entirely satisfied with how the paper is written and how the approach is introduced and explained, I find the results to be quite interesting. While I do not know the literature sufficiently well, I think this method is original and well-founded.\n\n \n##########################################################################\n\nPros: \n\n1. While being simple and intuitive, the proposed method appears to succesfully and efficiently identify multiple high-quality local optima of a model.\n\n2. Possibly even more interestingly, ensembles containing corresponding models appear to outperform other alternative approaches. While inference with ensembles of models can be quite costly (growing with the number of models in the ensemble) and a similar or better accuracy could potentially be achieved with larger simple models sharing the same computational cost, this result is nevertheless very promising.\n\n3. The experimental methodology appears to be sound and some illustrative examples (like those shown in Figure 3) are interesting and insightful.\n\n \n##########################################################################\n\nCons: \n\n1. Certain parts of the publication are not entirely well written and some sentences are a bit confusing. Also, the text contains quite a few misprints. Some more serious mistakes can be found, for example, in Table 2 (DenseNet results) and central equations (5) and (6), which seem to use a wrong sign for the gradient (current sign seems to correspond to gradient ascent and not descent thus maximizing the loss and not minimizing it). Also, as a very minor comment, I believe that, strictly speaking, the gradient (with respect to $\\Delta$) of the loss in these equations should be computed at $\\phi_{\\omega_0}(t_{i-1})$ because otherwise the right-hand side of these equations would be dependent on $\\Delta(t_{i})$.\n\n2. In my opinion, the discussion in Section 3 could be clarified and simplified. Furthermore, I believe that the method could be explained and analyzed a bit better. For example, it would appear that the proposed difference equation (6) can be written as a gradient descent on the modified loss function with the added quadratic term $\\sim (\\rho_1/\\rho_2) \\Delta^2$. If correct, I find this simple perspective much more natural and insightful. This quadratic component can essentially flip the Hessian in the vicinity of the starting local minimum thus causing the trajectory to be repelled from it. This simple view also appears to have implications for what kind of local minima of the original loss (their Hessians) could finally attract such training trajectories, potential shifts due to finite $\\rho_1$, and the role that the decay of $\\rho_1/\\rho_2$ could play in the process of convergence.\n\n3. The related work section contains just a few ensemble papers and none after 2018. It would appear that this section could be expanded and include some more recent papers at least for reference.\n\n##########################################################################\n\nQuestions during rebuttal period: \n\nPlease address and clarify the cons above. (I will update my score based on the authors reply.)\n\nUnfortunately, I am not an expert in this field, but two papers I came across doing a very quick search appear to be somewhat relevant: \"Local minima found in the subparameter space can be effective for ensembles of deep convolutional neural networks\" and \"MEAL: Multi-Model Ensemble via Adversarial Learning\". I am not sure these particular papers need to be included in the prior work section, but I do think that this publication would benefit from a more in-depth literature overview.\n\n##########################################################################\n\nPost-rebuttal.\n\nThanks for a detailed response that clarified some of my questions. I think the overall quality of the paper increased and I am happy to see additional information (like additional literature and an ablation study in Section 5.4) and a somewhat improved explanation of the core idea. However, I am still hesitant to give this paper a higher rating, in part because I find Section 5.4 to be poorly written and somewhat confusing (it lacks any sort of conclusion or insight and I cannot read Figure 4 at all, the text is too small) and partly because the paper does not clearly put its results in the context of the recent model ensemble progress (and thus makes me doubt the impact of this result on the field; although it does seem to be promising compared to the mentioned baselines).", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes a method to discover neighboring local optima around an existing one. Reviewers all found the idea interesting but argued that the paper needed more work. In particular, some of the claims are too informal or not sufficiently supported and the reviewers found the key section were difficult to follow. The paper should be resubmitted after improving the presentation of the results.
Decision: Reject