HyperGAN:  Exploring the Manifold of Neural Networks

We introduce HyperGAN , a generative network that learns to generate all the weight parameters of deep neural networks . HyperGAN first transforms low dimensional noise into a latent space , which can be sampled from to obtain diverse , performant sets of parameters for a target architecture . We utilize an architecture that bears resemblance to generative adversarial networks , but we evaluate the likelihood of samples with a classification loss . This is equivalent to minimizing the KL-divergence between the generated network parameter distribution and an unknown true parameter distribution . We apply HyperGAN to classification , showing that HyperGAN can learn to generate parameters which solve the MNIST and CIFAR-10 datasets with competitive performance to fully supervised learning , while learning a rich distribution of effective parameters . We also show that HyperGAN can also provide better uncertainty than standard ensembles . This is evaluated by the ability of HyperGAN-generated ensembles to detect out of distribution data as well as adversarial examples . We see that in addition to being highly accurate on inlier data , HyperGAN can provide reasonable uncertainty estimates . 1 INTRODUCTION . Since the inception of deep neural networks , it has been found that it is possible to train from different random initializations and obtain networks that , albeit having quite different parameters , achieve quite similar accuracy ( Freeman & Bruna , 2016 ) . It has further been found that ensembles of deep networks that are trained in such a way have significant performance advantages over single models ( Maclin & Opitz , 2011 ) , similar to the classical bagging approach in statistics . Ensemble models also have other benefits , such as being robust to outliers and being able to provide variance or uncertainty estimates over their inputs ( Lakshminarayanan et al. , 2017 ) . In Bayesian deep learning , there is a significant interest in having a probabilistic interpretation of network parameters and modeling a distribution over them . Earlier approaches mostly utilize dropout as a Bayesian approximation , by randomly setting different parameters to zero and thus integrating over many possible networks . ( Gal & Ghahramani , 2016 ) showed that networks with dropout following each layer are equivalent to a deep Gaussian process ( Damianou & Lawrence , 2013 ) marginalized over its covariance functions . They proposed MCdropout as a simple way to estimate model uncertainty . These approximations are not well aligned with current training patterns of neural networks . Applying dropout to every layer results in over-regularization and underfitting of the target function . Moreover , dropout does not integrate over the full variation of possible models , only those which may be reached from one ( random ) initialization . As another interesting direction , hypernetworks ( Ha et al. , 2016 ) are neural networks which output parameters for a target neural network . The hypernetwork and the target network together form a single model which is trained jointly . The original hypernetwork produced the target weights as a deterministic function of its own weights , but Bayesian Hypernetworks ( BHNs ) ( Krueger et al. , 2017 ) , and Multiplicative Normalizing Flows ( MNF ) ( Louizos & Welling , 2016 ) generate model parameters by sampling a Gaussian prior . However , these approaches use normalizing flows to transform a simple prior into a sample of the more complicated posterior , which are composed of only bijective , invertible functions . This limits their scalability and the variety of learnable functions . In this paper we explore an approach which focuses on generating all the parameters of a neural network , without assuming any fixed noise models on parameters . To keep our method scalable , we avoid utilizing invertible functions as in normalizing flow approaches , and instead utilize the ideas from generative adversarial networks ( GANs ) . We especially observe recent Wasserstein Autoencoder ( Tolstikhin et al. , 2017 ) approaches . These approaches have demonstrated an impressive capability to model complicated , multimodal distributions . In our approach , a random noise vector is first encoded to a number of different random vectors , then each random vector is used to generate all parameters within one layer of a deep network . The generator is then trained with conventional maximum likelihood ( classification/regression ) on the parameters it generates , and an adversarial regularization keeps it from collapsing onto only one mode . In this way , it is possible to generate much larger networks than the dimensionality of the latent code , making our approach capable of generating all the parameters of a deep network with a single GPU . As an example , in our experiments on CIFAR-10 we start from a 256-dimensional latent vector and generate all 50 , 000+ parameters in one pass , consuming only 4GB GPU memory . This shows that deep networks may indeed span a low-dimensional manifold , and could spur further thoughts and research . 1.1 SUMMARY OF CONTRIBUTIONS . We propose HyperGAN , a novel approach for generating all the parameters for a target network architecture using a modified GAN , and we do so starting from a small Gaussian noise vector which scales well with the size of the output . Our approach is different from strict Bayesian approaches since we do not attempt to model the entire posterior , i.e. , from an observed network we can not compute its probability of it being generated . Moreover , we use deterministic generators , our only source of stochasticity is the initial prior sample . After our GAN is trained , one can directly generate many diverse , well-trained deep models without needing to further train or fine-tune them . The diversity of the models we can generate is beyond just adding dropout or scaling factors , which is shown by the superior performance of ensembles of the generated networks . We believe HyperGAN is widely applicable to a variety of tasks . One area where populations of diverse networks show promise is in uncertainty estimation and anomaly detection . We show through a variety of experiments that populations of networks sampled from HyperGAN are able to approximate the data distribution such that it can detect out of distribution samples . We show that we can provide a reasonable measure of uncertainty by calculating the entropy within the predictive distribution of sampled networks . Our method is straightforward , as well as easy to train and sample from . We hope that we can inspire future work in estimation of the manifold of neural networks . 2 RELATED WORK . Generating parameters for neural networks has been framed in contexts other than the Bayesian approaches described above . The hypernetwork framework ( Ha et al. , 2016 ) has come to describe models where one network directly supervises the weight updates of another network . ( Pawlowski et al. , 2017 ) uses hypernetworks to generate layer-wise weights for a target architecture given some auxiliary noise as input . The auxiliary noise is noted to be independent between layers , while we impose structure in our inputs to create diversity in our generated samples . Computer vision methods often use data driven approaches as seen in methods such as Spatial Transformer networks ( Jaderberg et al. , 2015 ) , or Dynamic Filter networks ( Jia et al. , 2016 ) . In these methods the filter parameters of the main network are conditioned on the input data , receiving contextual scale and shift updates from an auxiliary network . Our method , however , generates parameters for the whole network . Furthermore our predicted parameters are highly nonlinear functions of the input , instead of simple affine transformations based on the input examples . Recently , ( Lakshminarayanan et al. , 2017 ) proposed Deep Ensembles , where adversarial training was applied to standard ensembles to smooth the predictive variance . However , adversarial training is an expensive training process . Adversarial examples are generated for each batch of data seen . We seek a method to learn a distribution over parameters which does not require adversarial training . Meta learning approaches use different kinds of weights in order to increase the generalization ability of neural networks . The first proposed method is fast weights ( Hinton & Plaut , 1987 ) which uses an auxiliary ( slow ) network to produce weight changes in the target ( fast ) network , acting as a short term memory store . Meta Networks ( Munkhdalai & Yu , 2017 ) build on this approach by using an external neural memory store in addition to multiple sets of fast and slow weights . ( Ba et al. , 2016 ) augment recurrent networks with fast weights as a more biologically plausible memory system than Neural Turing Machines . Unfortunately , the generation of each predicting network requires querying the base ( slow ) learner many times . Many of the methods presented there , along with hyperparameter learning ( Lorraine & Duvenaud , 2018 ) , and the original hypernetwork , propose learning target weights which are deterministic functions of the training data . Our method instead captures a distribution over parameters , and provides a cheap way to directly sample full networks from a learned distribution . 3 HYPERGAN . Taking note from the original hypernetwork framework for generating neural networks ( Ha et al. , 2016 ) , we coin our approach HyperGAN . Standard GAN training dictates that we train a generator G to model the target distribution , aided by a discriminator network D. This entails acquiring a large dataset of trained neural network parameters , and providing samples to D to judge the output of the generator . However , a large collection of neural networks would be extremely costly to build . Instead , our data distribution is built online by evaluating the performance of our generated parameters at each step . We begin with assuming that a neural network F ( x ; θ ) with input x and parameters θ consisting of a given architecture withN layers , and a training setM with inputs and targets ( X , Y ) = { xi , yi } Mi=1 . The standard training regime consists of computing a loss function L ( F ( x ; θ ) , y ) and updating the parameters θ with backpropagation untilL is minimized . This works if we only want a point estimate of θ . However , if we want to generate more than one non-trivial network , some diversity is needed . Our approach to creating diversity is similar to a regular GAN architecture in that we start by drawing a random sample s ∼ S = N ( 0 , Ij ) , where 0 is an all-zero vector and Ij is a j × j identity matrix , and generate parameters using a generator from s. Figure 1 shows the HyperGAN architecture . Distinct from the standard GAN , we use parallel , untied generators to form the parameters of each layer . In order for that to work , we observe that parameters between network layers are not independent , but are strongly correlated . The parameters of each layer depend on the output and the parameters of the preceding layers . Therefore , the generated parameters must be correlated to produce well-performing neural networks . We propose to add a Mixer Q which maps s ∼ S to the mixed latent space Z ∈ RNd , d < j . An Nd-dimensional Q ( s ) is then partitioned into N layer embeddings [ q1 , . . . , qN ] , each being a d-dimensional vector . Finally , we use N parallel generators G = { G1 ( q1 ) . . . GN ( qn ) } to generate the parameters θG for all layers in F . After generation , we can evaluate the new modelF ( x , θG ) on the training set . We define an objective which minimizes the error of generated parameters with respect to a task loss L : inf G , Q Es∼SE ( x , y ) ∼ ( X , Y ) [ L ( F ( x ; G ( Q ( s ) ) ) , y ) ] ( 1 ) At each training step we generate a different network G ( Q ( s ) ) from a random s ∼ S , and then evaluate the loss function on a mini-batch from the training set . The resulting loss is backpropagated through the generators until θG minimizes the target loss L. The formulation in ( 1 ) presents a problem : the codes sampled from Q ( s ) will certainly collapse to the maximum likelihood ( ML ) estimate ( when L is a log-likelihood ) . This means that the generators will learn a very narrow approximation of Θ , and indeed we see this happen in tables 3 and 4 . To assure that the parameters are well distributed , we added an adversarial constraint on the mixed latent spaceD ( Q ( s ) ) so that it should not deviate too much from a Gaussian prior P . This constraint makes it closer to the generated parameters and ensures that Q ( s ) itself does not collapse to always outputting the same latent code . With this we arrive at the HyperGAN objective : inf G , Q Es∼SE ( x , y ) ∼ ( X , Y ) [ L ( F ( x ; G ( Q ( s ) ) ) , y ) ] − βD ( Q ( s ) , P ) ( 2 ) Where β is a hyperparameter , andD is the regularization term which penalizes the distance between the prior and the distribution of latent codes . In practice D could be any distance function between two distributions . We choose to parameterize D as a discriminator network D that output probabilities , and use the adversarial loss ( Goodfellow et al. , 2014 ) to approximate D ( Q ( s ) , P ) . Note that while P and S are both multivariate Gaussians , they are distinct distributions as seen in figure 1 . D : = − N∑ i=1 ( logD ( pi ) + log ( 1−D ( qi ) ) ) ( 3 ) Note that we find it difficult to learn a discriminator in the output ( parameter ) space because the dimensionality is high and there is no structure in those parameters to be utilized as in images ( where CNNs can be trained ) . Our experiments show that regularizing in the latent space works well . We hypothesize this is because of overparametrization in θ . The latent space initializes with random projections , which have a restricted isometry property according to the Johnson-Lindenstrauss lemma , hence if the mixed latent factor q is Gaussian , a random projection from it preserves the distance and diversity from q to θ . If θ is indeed severely overparametrized , and it is possible to generate diverse parameters by maximizing the likelihood , then the generators would not collapse to a single θ , since that would require breaking the restricted isometry from the initialization . This framework is general and can be adapted to a variety of tasks and losses . In this work , we show that HyperGAN can operate in both classification and regression settings . For multi-class classification , the generators and mixer are trained with the cross entropy loss function : LH = M−1 M∑ i=1 yi log ( F ( xi ; θ ) ) where θ = { G1 ( q1 ) , . . . , Gn ( qn ) } ( 4 ) For regression tasks we replace the cross entropy term with the mean squared error function : Lmse = M−1 M∑ i=1 ( yi −F ( xi ; θ ) ) 2 where θ = { G1 ( q1 ) , . . . , Gn ( qn ) } ( 5 )

Summary: This paper proposes a technique for learning a distribution over parameters of a neural network such that samples from the distribution correspond to performant networks. The approach effectively encourages sampled parameters to have low loss on the training set, and also uses an adversarial loss to encourage the distribution of parameters to be Gaussian distributed. This approach can improve performance slightly by using ensembling and can be useful for uncertainty estimates for out-of-distribution examples. The approach is tested on a few simple problems and is shown to work well.