HyperGAN:  Exploring the Manifold of Neural Networks

We introduce HyperGAN , a generative network that learns to generate all the weight parameters of deep neural networks . HyperGAN first transforms low dimensional noise into a latent space , which can be sampled from to obtain diverse , performant sets of parameters for a target architecture . We utilize an architecture that bears resemblance to generative adversarial networks , but we evaluate the likelihood of samples with a classification loss . This is equivalent to minimizing the KL-divergence between the generated network parameter distribution and an unknown true parameter distribution . We apply HyperGAN to classification , showing that HyperGAN can learn to generate parameters which solve the MNIST and CIFAR-10 datasets with competitive performance to fully supervised learning , while learning a rich distribution of effective parameters . We also show that HyperGAN can also provide better uncertainty than standard ensembles . This is evaluated by the ability of HyperGAN-generated ensembles to detect out of distribution data as well as adversarial examples . We see that in addition to being highly accurate on inlier data , HyperGAN can provide reasonable uncertainty estimates . 1 INTRODUCTION . Since the inception of deep neural networks , it has been found that it is possible to train from different random initializations and obtain networks that , albeit having quite different parameters , achieve quite similar accuracy ( Freeman & Bruna , 2016 ) . It has further been found that ensembles of deep networks that are trained in such a way have significant performance advantages over single models ( Maclin & Opitz , 2011 ) , similar to the classical bagging approach in statistics . Ensemble models also have other benefits , such as being robust to outliers and being able to provide variance or uncertainty estimates over their inputs ( Lakshminarayanan et al. , 2017 ) . In Bayesian deep learning , there is a significant interest in having a probabilistic interpretation of network parameters and modeling a distribution over them . Earlier approaches mostly utilize dropout as a Bayesian approximation , by randomly setting different parameters to zero and thus integrating over many possible networks . ( Gal & Ghahramani , 2016 ) showed that networks with dropout following each layer are equivalent to a deep Gaussian process ( Damianou & Lawrence , 2013 ) marginalized over its covariance functions . They proposed MCdropout as a simple way to estimate model uncertainty . These approximations are not well aligned with current training patterns of neural networks . Applying dropout to every layer results in over-regularization and underfitting of the target function . Moreover , dropout does not integrate over the full variation of possible models , only those which may be reached from one ( random ) initialization . As another interesting direction , hypernetworks ( Ha et al. , 2016 ) are neural networks which output parameters for a target neural network . The hypernetwork and the target network together form a single model which is trained jointly . The original hypernetwork produced the target weights as a deterministic function of its own weights , but Bayesian Hypernetworks ( BHNs ) ( Krueger et al. , 2017 ) , and Multiplicative Normalizing Flows ( MNF ) ( Louizos & Welling , 2016 ) generate model parameters by sampling a Gaussian prior . However , these approaches use normalizing flows to transform a simple prior into a sample of the more complicated posterior , which are composed of only bijective , invertible functions . This limits their scalability and the variety of learnable functions . In this paper we explore an approach which focuses on generating all the parameters of a neural network , without assuming any fixed noise models on parameters . To keep our method scalable , we avoid utilizing invertible functions as in normalizing flow approaches , and instead utilize the ideas from generative adversarial networks ( GANs ) . We especially observe recent Wasserstein Autoencoder ( Tolstikhin et al. , 2017 ) approaches . These approaches have demonstrated an impressive capability to model complicated , multimodal distributions . In our approach , a random noise vector is first encoded to a number of different random vectors , then each random vector is used to generate all parameters within one layer of a deep network . The generator is then trained with conventional maximum likelihood ( classification/regression ) on the parameters it generates , and an adversarial regularization keeps it from collapsing onto only one mode . In this way , it is possible to generate much larger networks than the dimensionality of the latent code , making our approach capable of generating all the parameters of a deep network with a single GPU . As an example , in our experiments on CIFAR-10 we start from a 256-dimensional latent vector and generate all 50 , 000+ parameters in one pass , consuming only 4GB GPU memory . This shows that deep networks may indeed span a low-dimensional manifold , and could spur further thoughts and research . 1.1 SUMMARY OF CONTRIBUTIONS . We propose HyperGAN , a novel approach for generating all the parameters for a target network architecture using a modified GAN , and we do so starting from a small Gaussian noise vector which scales well with the size of the output . Our approach is different from strict Bayesian approaches since we do not attempt to model the entire posterior , i.e. , from an observed network we can not compute its probability of it being generated . Moreover , we use deterministic generators , our only source of stochasticity is the initial prior sample . After our GAN is trained , one can directly generate many diverse , well-trained deep models without needing to further train or fine-tune them . The diversity of the models we can generate is beyond just adding dropout or scaling factors , which is shown by the superior performance of ensembles of the generated networks . We believe HyperGAN is widely applicable to a variety of tasks . One area where populations of diverse networks show promise is in uncertainty estimation and anomaly detection . We show through a variety of experiments that populations of networks sampled from HyperGAN are able to approximate the data distribution such that it can detect out of distribution samples . We show that we can provide a reasonable measure of uncertainty by calculating the entropy within the predictive distribution of sampled networks . Our method is straightforward , as well as easy to train and sample from . We hope that we can inspire future work in estimation of the manifold of neural networks . 2 RELATED WORK . Generating parameters for neural networks has been framed in contexts other than the Bayesian approaches described above . The hypernetwork framework ( Ha et al. , 2016 ) has come to describe models where one network directly supervises the weight updates of another network . ( Pawlowski et al. , 2017 ) uses hypernetworks to generate layer-wise weights for a target architecture given some auxiliary noise as input . The auxiliary noise is noted to be independent between layers , while we impose structure in our inputs to create diversity in our generated samples . Computer vision methods often use data driven approaches as seen in methods such as Spatial Transformer networks ( Jaderberg et al. , 2015 ) , or Dynamic Filter networks ( Jia et al. , 2016 ) . In these methods the filter parameters of the main network are conditioned on the input data , receiving contextual scale and shift updates from an auxiliary network . Our method , however , generates parameters for the whole network . Furthermore our predicted parameters are highly nonlinear functions of the input , instead of simple affine transformations based on the input examples . Recently , ( Lakshminarayanan et al. , 2017 ) proposed Deep Ensembles , where adversarial training was applied to standard ensembles to smooth the predictive variance . However , adversarial training is an expensive training process . Adversarial examples are generated for each batch of data seen . We seek a method to learn a distribution over parameters which does not require adversarial training . Meta learning approaches use different kinds of weights in order to increase the generalization ability of neural networks . The first proposed method is fast weights ( Hinton & Plaut , 1987 ) which uses an auxiliary ( slow ) network to produce weight changes in the target ( fast ) network , acting as a short term memory store . Meta Networks ( Munkhdalai & Yu , 2017 ) build on this approach by using an external neural memory store in addition to multiple sets of fast and slow weights . ( Ba et al. , 2016 ) augment recurrent networks with fast weights as a more biologically plausible memory system than Neural Turing Machines . Unfortunately , the generation of each predicting network requires querying the base ( slow ) learner many times . Many of the methods presented there , along with hyperparameter learning ( Lorraine & Duvenaud , 2018 ) , and the original hypernetwork , propose learning target weights which are deterministic functions of the training data . Our method instead captures a distribution over parameters , and provides a cheap way to directly sample full networks from a learned distribution . 3 HYPERGAN . Taking note from the original hypernetwork framework for generating neural networks ( Ha et al. , 2016 ) , we coin our approach HyperGAN . Standard GAN training dictates that we train a generator G to model the target distribution , aided by a discriminator network D. This entails acquiring a large dataset of trained neural network parameters , and providing samples to D to judge the output of the generator . However , a large collection of neural networks would be extremely costly to build . Instead , our data distribution is built online by evaluating the performance of our generated parameters at each step . We begin with assuming that a neural network F ( x ; θ ) with input x and parameters θ consisting of a given architecture withN layers , and a training setM with inputs and targets ( X , Y ) = { xi , yi } Mi=1 . The standard training regime consists of computing a loss function L ( F ( x ; θ ) , y ) and updating the parameters θ with backpropagation untilL is minimized . This works if we only want a point estimate of θ . However , if we want to generate more than one non-trivial network , some diversity is needed . Our approach to creating diversity is similar to a regular GAN architecture in that we start by drawing a random sample s ∼ S = N ( 0 , Ij ) , where 0 is an all-zero vector and Ij is a j × j identity matrix , and generate parameters using a generator from s. Figure 1 shows the HyperGAN architecture . Distinct from the standard GAN , we use parallel , untied generators to form the parameters of each layer . In order for that to work , we observe that parameters between network layers are not independent , but are strongly correlated . The parameters of each layer depend on the output and the parameters of the preceding layers . Therefore , the generated parameters must be correlated to produce well-performing neural networks . We propose to add a Mixer Q which maps s ∼ S to the mixed latent space Z ∈ RNd , d < j . An Nd-dimensional Q ( s ) is then partitioned into N layer embeddings [ q1 , . . . , qN ] , each being a d-dimensional vector . Finally , we use N parallel generators G = { G1 ( q1 ) . . . GN ( qn ) } to generate the parameters θG for all layers in F . After generation , we can evaluate the new modelF ( x , θG ) on the training set . We define an objective which minimizes the error of generated parameters with respect to a task loss L : inf G , Q Es∼SE ( x , y ) ∼ ( X , Y ) [ L ( F ( x ; G ( Q ( s ) ) ) , y ) ] ( 1 ) At each training step we generate a different network G ( Q ( s ) ) from a random s ∼ S , and then evaluate the loss function on a mini-batch from the training set . The resulting loss is backpropagated through the generators until θG minimizes the target loss L. The formulation in ( 1 ) presents a problem : the codes sampled from Q ( s ) will certainly collapse to the maximum likelihood ( ML ) estimate ( when L is a log-likelihood ) . This means that the generators will learn a very narrow approximation of Θ , and indeed we see this happen in tables 3 and 4 . To assure that the parameters are well distributed , we added an adversarial constraint on the mixed latent spaceD ( Q ( s ) ) so that it should not deviate too much from a Gaussian prior P . This constraint makes it closer to the generated parameters and ensures that Q ( s ) itself does not collapse to always outputting the same latent code . With this we arrive at the HyperGAN objective : inf G , Q Es∼SE ( x , y ) ∼ ( X , Y ) [ L ( F ( x ; G ( Q ( s ) ) ) , y ) ] − βD ( Q ( s ) , P ) ( 2 ) Where β is a hyperparameter , andD is the regularization term which penalizes the distance between the prior and the distribution of latent codes . In practice D could be any distance function between two distributions . We choose to parameterize D as a discriminator network D that output probabilities , and use the adversarial loss ( Goodfellow et al. , 2014 ) to approximate D ( Q ( s ) , P ) . Note that while P and S are both multivariate Gaussians , they are distinct distributions as seen in figure 1 . D : = − N∑ i=1 ( logD ( pi ) + log ( 1−D ( qi ) ) ) ( 3 ) Note that we find it difficult to learn a discriminator in the output ( parameter ) space because the dimensionality is high and there is no structure in those parameters to be utilized as in images ( where CNNs can be trained ) . Our experiments show that regularizing in the latent space works well . We hypothesize this is because of overparametrization in θ . The latent space initializes with random projections , which have a restricted isometry property according to the Johnson-Lindenstrauss lemma , hence if the mixed latent factor q is Gaussian , a random projection from it preserves the distance and diversity from q to θ . If θ is indeed severely overparametrized , and it is possible to generate diverse parameters by maximizing the likelihood , then the generators would not collapse to a single θ , since that would require breaking the restricted isometry from the initialization . This framework is general and can be adapted to a variety of tasks and losses . In this work , we show that HyperGAN can operate in both classification and regression settings . For multi-class classification , the generators and mixer are trained with the cross entropy loss function : LH = M−1 M∑ i=1 yi log ( F ( xi ; θ ) ) where θ = { G1 ( q1 ) , . . . , Gn ( qn ) } ( 4 ) For regression tasks we replace the cross entropy term with the mean squared error function : Lmse = M−1 M∑ i=1 ( yi −F ( xi ; θ ) ) 2 where θ = { G1 ( q1 ) , . . . , Gn ( qn ) } ( 5 )

Summary: This works propose a new approach to learn to sample (or generate) the parameters of a deep neural networks to solve a task. They propose a new architecture inspired by hyper networks and adversarial auto-encoders, where the parameters of the networks are generated from a low dimensional latent space. By using an ensemble of networks sampled with their approach they're able to get state of the art results on uncertainty estimation.