Constructing Multiple High-Quality Deep Neural Networks: A TRUST-TECH Based Approach

1 INTRODUCTION . Due to the high redundancy on parameters of deep neural networks ( DNN ) , the number of local optima is huge and can grow exponentially with the dimensionality of the parameter space ( Auer et al . ( 1996 ) ; Choromanska et al . ( 2015 ) ; Dauphin et al . ( 2014b ) ) . It still remains a challenging task to locate high-quality optimal solutions in the parameter space , where the model performs satisfying on both training and testing data . A popular metric for the quality of a local solution is to measure its generalization capability , which is commonly defined as the gap between the training and testing performances ( LeCun et al . ( 2015 ) ) . For deep neural networks with high expressivity , the training error is near zero , so that it suffices to use the test error to represent the generalization gap . Generally , local solvers do not have the global vision of the parameter space , so there is no guarantee that starting from a random initialization can locate a high-quality local optimal solution . On the other hand , one can apply a non-local solver in the parameter space to find multiple optimal solutions and select the high-quality ones . Furthermore , one can improve the DNN performance by ensembling these high-quality solutions with high diversity . TRUST-TECH plays an important role in achieving the above goal . In general , it computes highquality optimal solutions for general nonlinear optimization problems , and the theoretical foundations can be bound in ( Chiang & Chu ( 1996 ) ; Lee & Chiang ( 2004 ) ) . It helps local solvers escape from one local optimal solution ( LOS ) and search for other LOSs . It has been successfully applied in guiding the Expectation Maximization method to achieve higher performance ( Reddy et al . ( 2008 ) ) , training ANNs ( Chiang & Reddy ( 2007 ) ; Wang & Chiang ( 2011 ) ) , estimating finite mixture models ( Reddy et al . ( 2008 ) ) , and solving optimal power flow problems ( Chiang et al . ( 2009 ) ; Zhang & Chiang ( 2020 ) ) . Additionally , it does not interfere with existing local or global solvers , but cooperates with them . TRUST-TECH efficiently searches the neighboring subspace of the promising candidates for new LOSs in a tier-by-tier manner . Eventually , a set of high-quality LOSs can be found . The idea of TRUST-TECH method is the following : for a given loss surface of an op- timization problem , each LOS has its own stability region . If one start from one local optimum , and track the loss values along a given direction , we will find an exit point where loss start to decrease steadily , which means another stability region corresponding to a nearby LOS is found . By following a trajectory in the stability region , another LOS is computed . We propose an optima exploring algorithm designed for DNNs that is able to find high-quality local optima in a systematic way , and thereby form optimal and robust ensembles . Normally for a deep neural network , exit points can hardly be found by original TRUST-TECH due to the huge dimensionality . So , in this work we introduce the Dynamic Searching Paths ( DSP ) method instead of fixed directions . We set the search directions to be trainable parameters . After an exploration step forward along the current direction , we calibrate the direction using the current gradient . By doing so , the method can benefit from not only the mature Stochastic Gradient Descent ( SGD ) training paradigm with powerful GPU acceleration capability , but also exit points can be easily found . The overall DSP-TT method consists of four stages . First , we train the network using local solvers to get a tier-0 local optimal solution . Second , our proposed Dynamic Search Path TRUST-TECH ( DSP-TT ) method is called to find nearby solutions in a tier-by-tier manner . Third , a selection process is performed so that candidates with high quality are chosen . Finally , ensembles are built with necessary fine-tunings on selected member networks . To the best of our knowledge , this paper is the first one to search for multiple solutions on deep neural networks in a systematical way . Our major contributions and highlights are summarized as follows : • We propose the Dynamic Searching Path ( DSP ) method that enables exploration on highdimensional parameter space efficiently . • We show that combining TRUST-TECH method with DSP ( DSP-TT ) is effective in finding multiple optimal solutions on deep neural networks systematically . • We design and implement the algorithm efficiently that it obtains multiple local solutions within one training session with minor GPU memory overhead . • We develop the DSP-TT Ensembles of solutions found by DSP-TT with high quality and diversity for further improving the DNN performance . 2 RELATED WORK . The synergy between massive numbers of parameters and nonlinear activations in deep neural networks leads to the existence of multiple LOSs trained on a specific dataset . Experiments show that different initializations lead to different solutions with various qualities ( Dauphin et al . ( 2014a ) ) . Even with the same initialization , the network can converge to different solutions depending on the loss function and the solver ( Im et al . ( 2016 ) ) . Many regularization techniques are therefore proposed to force the network to converge to a better solution , some of which are proven to be useful and popular ( Kingma & Ba ( 2015 ) ; Srivastava et al . ( 2014 ) ; Ioffe & Szegedy ( 2015 ) ) . However , it is still mysterious how these regularized solutions are compared to the global optimum . There are researchers that focus on characterizing different local optima and investigating the internal relations among them . It is claimed in ( Hochreiter & Schmidhuber ( 1997 ) ; Keskar et al . ( 2016 ) ) that sharp minima prevent deep neural networks from generalizing well on the testing dataset . Later , Dinh et al . ( 2017 ) argued that the definition of flatness in ( Keskar et al . ( 2016 ) ) is problematic and came up with an example where solutions with different geometries can have similar test time performances . Li et al . ( 2018 ) designed a new visualization method that rebuilt the correspondence between the sharpness of the minimizer and the generalization capability . On the other hand , some researchers apply meta-heuristic algorithms to obtain a better local minimizer ( Gudise & Venayagamoorthy ( 2003 ) ; Zhang et al . ( 2007 ) ; Juang ( 2004 ) ; Leung et al . ( 2003 ) ) . However , these methods were either designed for obsolete toy models or on explicit benchmark objective functions where there are analytical forms for global optimum , and therefore the effectiveness of these algorithms on deep architectures and large datasets seems unconvincing . Moreover , the advantage of the global searching ability seems to be crippled when it comes to deep neural networks , and the minimizers they found are still local . Recently , Garipov et al . ( 2018 ) reveal the relation among local optima by building pathways , called Mode Connectivities , as simple as polygonal chains or Bezier curves that connect any two local optima . Draxler et al . ( 2018 ) also found similar results at the same time , although they used the Nudged Elastic Band ( Jonsson et al . ( 1998 ) ) method from quantum chemistry . To address the issue of converging to suboptimal solutions , a great deal of research efforts were directed to ensembles . Xie et al . ( 2013 ) proposed horizontal and vertical ensembles that combine the output of networks at different training epochs . Laine & Aila ( 2016 ) used a group of models with different regularization and augmentation conditions to create variety . Moghimi et al . ( 2016 ) borrowed the concept of boosting to create a strong ensemble of CNNs . Izmailov et al . ( 2018 ) found that averaging weights from different iterations leads to flatter solutions than from SGD and helps in generalization . Huang et al . ( 2017a ) proposed a method that obtains ensembles by collecting several local minima along a single training process using a cyclical learning rate schedule . Zhang et al . ( 2020 ) used similar approach , but with the sampling capability that fully exploits each mode . Garipov et al . ( 2018 ) developed the Fast Geometric Ensembling based on Mode Connectivity . Although the methods in these papers obtain multiple networks within one training session , these ensembles are still largely dependent on initialization . While these ensemble methods performs better than a single network , naive randomly initialized ensemble is still the best choice when training budget is unconstrained . Fort et al . ( 2020 ) explained this phenomenon as they explore different modes in function space compared to weight averaging . Shen et al . ( 2019 ) improved the ensemble inference efficiency via a teacher-student paradigm distilling the knowledge of an ensemble into one single network . Yang et al . ( 2020 ) built ensembles by randomly initialize on a subparameter space , aiming to alleviate the exponentially growing number of local minima on deep networks . Wang & Chiang ( 2011 ) used the TRUST-TECH ( Chiang & Chu ( 1996 ) ; Lee & Chiang ( 2004 ) ; Chiang & Alberto ( 2015 ) ) method to perform a systematic search for diversified minimizers to obtain their ensembles . They implemented TRUST-TECH for training and constructing high-quality ensembles of artificial neural networks and showed that their method consistently outperforms other training methods . We generalize this method and tailor it for deep architectures and to work efficiently with popular local solvers in deep learning . 3 TRUST-TECH METHOD FOR MULTIPLE OPTIMAL SOLUTIONS 3.1 TRUST-TECH METHODOLOGY Another category of methods has been developed in recent years for systematically computing a set of local optimal solutions in a deterministic manner . This family of methods is termed TRUST-TECH methodology , standing for Transformation Under Stability-reTaining Equilibria Characterization . It is based on the following transformations : ( i ) the transformation of a local optimal solution ( LOS ) of a nonlinear optimization problem into a stable equilibrium point ( SEP , Chiang & Chu ( 1996 ) ) of a continuous nonlinear dynamical system . ( ii ) the transformation of the search space of nonlinear optimization problems into the union of the closure of stability regions of SEPs . Hence , the optimization problem ( i.e . the problem of finding LOSs ) is transformed into the problem of finding SEPs , and therefore we use the terms LOS and SEP interchangeably in the following discussion . It will become clear that the stability regions of SEPs play an important role in finding these local optimal solutions . We note that , given a LOS , its corresponding first-tier LOSs are defined as those optimal solutions whose corresponding stability boundaries have a non-empty intersection with the stability boundary of the LOS ( Chiang & Chu ( 1996 ) ; Lee & Chiang ( 2004 ) ) . The definition of the stability boundary and its characterization can be found in Chiang & Fekih-Ahmed ( 1996 ) . Similarly , its second-tier LOSs are defined as those optimal solutions whose corresponding stability boundaries have a non-empty intersection with the stability boundary of first-tier LOSs ( Chiang & Chu ( 1996 ) ; Lee & Chiang ( 2004 ) ) . See fig . 1 for an illustration . We consider a general nonlinear unconstrained optimization problem defined as follows : min x c ( x ) ( 1 ) where c : D ⊂ Rn → R is assumed to be continuously differentiable and D the set of feasible points ( or search space ) . A point x∗ ∈ D is called a local minimum if c ( x∗ ) ≤ c ( x ) for all x ∈ D with ‖x− x∗‖ < σ for σ > 0 . To systematically search for multiple LOSs , a generalized negative gradient system based on the objective eq . ( 1 ) is constructed and is described by dx dt = −gradR c ( x ) = −R ( x ) −1 · ∇c ( x ) = f ( x ( t ) ) ( 2 ) where the state vector x ( t ) of this dynamic system belongs to the Euclidean space Rn and the function f : Rn → Rn satisfies the sufficient condition for the existence and the uniqueness of the solutions . R ( x ) is a positive definite symmetric matrix ( also known as the Riemannian metric ) that generalizes various training algorithms . For example , if R ( x ) = I ( identity ) , it is a naive gradient descent algorithm . If R ( x ) = J ( x ) > J ( x ) ( J is the Jacobian matrix ) , then it is the Gauss-Newton method . If R ( x ) = J ( x ) > J ( x ) + µI , it becomes the Levenberg-Marquardt ( LM ) algorithm . The Theorem of the Equilibrium Points and Local Optima ( Lee & Chiang ( 2004 ) ) shows one nice property of the gradient system ( 2 ) , which is the critical point of the optimization problem ( 1 ) is a ( asymptotically ) SEP of the dynamic system ( 2 ) . i.e . x̄ is a SEP of ( 2 ) if and only if x̄ is an isolated local minimum for ( 1 ) . Hence , the task of finding the LOSs of ( 1 ) can be achieved by finding the corresponding SEPs of ( 2 ) . In short , TRUST-TECH is a dynamical method designed to systematically compute multiple LOSs with the following features : ( i ) it is a systematic and deterministic method to escape from a LOS towards another LOS , ( ii ) it finds multiple LOSs in a tier-by-tier manner ( see fig . 1 ) , and ( iii ) has a solid theoretical foundation ( Chiang & Chu ( 1996 ) ; Lee & Chiang ( 2004 ) ; Chiang & Alberto ( 2015 ) ; Zhang & Chiang ( 2020 ) ) . Another distinguishing feature of TRUST-TECH is its ability to guide a local method and/or a metaheuristic method for effective computation of a set of LOSs or even the global optimal solution .

Summary: This paper proposes a new method for applying the TRUST-TECH method to the ensemble of deep neural networks (DNNs). When applying TRUST-TECH to a deep neural network, it is difficult to determine the direction and exit point. This paper introduces Dynamic Searching Paths (DSP) to solve these problems. The proposed method can apply TRUST-TECH method to DNNs using Stochastic Gradient Descent (SGD) with minor memory overhead.