Meta Internal Learning

1 Introduction . In the field of internal learning , one wishes to learn the internal statistics of a signal in order to perform various downstream tasks . In this work , we focus on Single image GANs [ 31 , 32 , 13 , 8 ] , which present extremely impressive results in modeling the distribution of images that are similar to the input image , and in applying this distribution to a variety of applications . However , given that there is no shortage of unlabeled images , one may ask whether a better approach would be to model multiple images , and only then condition the model on a single input image . Doing so , one could ( i ) benefit from knowledge and feature sharing between the different images , ( ii ) better define the boundaries between the distribution obtained from the input image and those of other images , ( iii ) possibly avoid the costly training phase for a novel image , and instead employ feedforward inference , and ( iv ) mix different single-image models to create novel types of images . ∗Corresponding author - bensadoun @ mail.tau.ac.il 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . From the algorithmic standpoint , this multi-image capability can be attempted using various forms of conditioning . For example , one can add a one-hot vector as an input , or , more generally , a vector signature , and train multiple images using the same single image method . One can also add a complete layer of a conditioning signal to the RGB input . Alternatively , one can employ StyleGANlike conditioning and modify the normalization of the layers [ 16 ] . More generally , observing that this scenario is a meta-learning problem , one can employ methods such as MAML [ 5 ] for learning a central network and its variants per image . After performing many such attempts over a long period of time , we were unable to achieve a desirable level of performance with any of these methods . Instead , we advocate for a meta-learning solution that is based on hypernetworks [ 9 ] . Hypernetworks consist of two main components : a primary network g that performs the actual computation , and the hypernetwork f that is used for conditioning . The parameters ( weights ) of g are not learned conventionally . Instead , they are given as the output of f for the conditioned input signal . Following a single-image GAN setting with a hierarchical structure , we have two hypernetworks fg and fd , which produce the weights of the multiple generators and discriminators for input image I dynamically . Our method allows for training on multiple images at once , obtaining similar results for various applications to those previously demonstrated for single-image training . It also allows us to interpolate between single-image GANs derived from pairs ( or more ) of images . Finally , we are able to fit a new unseen image in a fraction of the time required for training a new single-image GAN , i.e.our method enables inference generation for a novel image . As far as we can ascertain , ours is the first method to perform adversarial training using hypernetworks . We provide a theoretical analysis of the proper way to perform this . This analysis demonstrates both the sufficiency of our algorithm for minimizing the objective function and the necessity of various components in our method . 2 Background . In this paper we consider the meta-learning problem of learning how to generate a variety of samples from a single image , where each individual learning problem is defined by this single input image . For this purpose , we first recall the setting of single-image generation as in [ 31 , 13 , 8 ] . 2.1 Single-Image Generation . SinGAN [ 31 ] is composed of a multi-scale residual generator G = { g1 , . . . , gk } and a patchdiscriminator D = { d1 , . . . , dk } , where gi and di are fully-convolutional networks , consisting of five layers and used for training at scale i . Given an image I , we pre-compute k scales of the image , from coarsest to finest , denoted by Ii , with height and width hi and wi , and use each Ii for training the i ’ th generator gi . The first generator g1 takes as input fixed random noise z1 ∈ R 3×h1×w1 whose coordinates are i.i.d . normally distributed , and outputs an image Î1 ∈ R 3×h1×w1 . Every other generator gi takes as input an upsampled version Î ↑ i−1 of the previous output Îi−1 , plus noise zi ∈ R 3×hi×wi ( whose coordinates are i.i.d . normally distributed ) , and recursively generates a sample at scale i as follows : Î1 : = g1 ( z1 ) : = ĝ1 ( z1 ) , Îi : = gi ( Îi−1 , zi ) : = Î ↑ i−1 + ĝi ( Î ↑ i−1 + zi ) , i > 1 ( 1 ) For each scale i ∈ [ k ] , we denote by DI , i the distribution of patches uI , i within Ii and by DÎ , i the distribution of patches uÎ , i within Îi ( for z1 , . . . , zi ∼ N ( 0 , I ) ) . The goal of this method is to train each generator gi to generate samples Îi , such that , DÎ , i and DI , i would closely match . For this task , the generators gi are progressively optimized to minimize the 1-Wasserstein distance W ( DÎ , i , DI , i ) between the distributions DÎ , i and DI , i . The 1-Wasserstein distance between two distributions D1 and D2 is defined as follows : W ( D1 , D2 ) : = max d : ∥d∥L≤1 { E u∼D1 d ( u ) − E u∼D2 d ( u ) } , ( 2 ) where ∥d∥L is the Lipschitz norm of the discriminator d. In general , computing the maximum in Eq . 2 is intractable . Therefore , [ 1 ] suggested estimating the 1-Wasserstein distance using a pseudo-metric WC ( D1 , D2 ) , where d is parameterized using a neural network from a wide class C. The method minimizes the adversarial loss , derived from Eq . 2 , Ladv ( gi , di ) : = E z1 : i [ di ( uÎ , i ) ] − E uI , i [ di ( uI , i ) ] , ( 3 ) where z1 : i = ( z1 , . . . , zi ) , uÎ , i ∼ DÎ , i and uI , i ∼ DI , i . The above objective is minimized with respect to the parameters of gi and maximized with respect to the parameters of the discriminator di , while freezing the parameters of all previous generators g1 , . . . , gi−1 . Note that Îi is given by gi according to Eq . 1 . In order to guarantee that di is of a bounded Lipschitz constant , in [ 7 ] they apply an additional gradient penalty loss to regularize the Lipschitzness of the discriminator : Llip ( di ) : = E uI , i [ ∥∇udi ( uI , i ) ∥ 2 2 ] , ( 4 ) In addition , they employ a reconstruction loss . We let z0 1 be fixed random noise , such that : Î0 1 : = ĝ1 ( z 0 1 ) , Î0i : = Î 0 , ↑ i−1 + ĝi ( Î 0 , ↑ i−1 ) , i > 1 ( 5 ) In practice , the expected values with respect to the various distributions are replaced with averages over finite sample sets . For simplicity , throughout the paper we will use expectations to avoid clutter . 2.2 Hypernetworks . Formally , a hypernetwork h ( z ; f ( I ; θf ) ) is a pair of collaborating neural networks , f and h. For an input I , network f , parameterized by a set θf of trainable parameters , produces the weights θI = f ( I ; θf ) for the primary network h. Network h takes an input z and returns an output h ( z ; θI ) that depends on both z and the task-specific input I . In practice , f is typically a large neural network and h is a small neural network . Throughout the paper , we use “ ; ” to separate the input and trainable parameters of a neural network . 3 Method . Our method solves an inherent limitation of current single-sample GAN models , which is the scaling to multi-sample learning , such that the same network can perform single-image generations for each sample . For this purpose , we adopt a hypernetwork-based modeling for the generators and discriminators involved . The hypernetwork network fg produces weights for each gi , and a hypernetwork fd produces weights for each di . In this setting , gi and di consist of the same architecture presented in Sec . 2 , and serve as the primary networks for fg and fd ( resp. ) . Two alternatives for the proposed setting are presented for completeness and are briefly discussed in Sec . 4 , ( i ) shared discriminator and ( ii ) shared feature extractor . These alternatives help in understanding our proposed approach . The full description and proofs are presented in the supplementary material . An illustration of the proposed model and the two variants are presented in Fig . 1 . 3.1 The Model . Our model consists of two main components : a hyper-generator and a hyper-discriminator ( see Fig 1 ( a ) for an illustration ) . The hyper-generator gi is a hypernetwork defined as follows : gi ( z , I ) : = gi ( z ; f i g ( I ; θfg ) ) , ( 6 ) where f ig ( I ; θfg ) is a neural network that takes an input image I and returns a vector of weights for the i ’ th generator gi . This network is decomposed into an embedding network eg that is shared among scales and a linear projection pig per scale , Eg ( I ) : = eg ( I ; θeg ) ( 7 ) f ig ( I ; θfg ) : = p i g ( Eg ( I ) ; θ i pg ) ( 8 ) The network eg is parameterized with a set of parameters θeg , and each projection p i g is parameterized with a set of parameters θipg ( for simplicity , we denote their union by θpg = ( θ i pg ) ki=1 ) . Each gi is a fully convolutional network , following Sec . 2 , whose weights are θig : = f i g ( I ; θfg ) . The overall set of trainable parameters within gi is θfg : = ( θeg , θ i pg ) ki=1 . The hyper-discriminator is defined in a similar manner : di ( u , I ) : = di ( u ; f i d ( I ; θfd ) ) ( 9 ) where f id ( I ; θθd ) is a network that takes an image I and returns a vector of weights for the i ’ th discriminator di . This network is also decomposed into an embedding network and a set of projections : Ed ( I ) : = ed ( I ; θed ) ( 10 ) f id ( I ; θfd ) : = pd ( Eg ( I ) ; θ i pd ) ( 11 ) In contrast to the generator , the hyper-discriminator works only on the last image scale . Each di is a fully convolutional network , following Sec . 2 , whose weights are θid : = f i d ( I ; θfd ) . The overall set of trainable parameters within di is θfd : = ( θed , θ i pd ) ki=1 .

Summary: The paper aims at improving single-image GAN models by training them on larger datasets. The authors use a meta-learning approach based on the hypernetworks scheme. The model consists of a hyper-generator and a hyper-discriminator. The Hyper-generator network is decomposed into an embedding network that is shared among scales and a linear projection per scale. The hyper-discriminator network is also decomposed into an embedding network and a set of projections. The model is trained with adversarial and reconstruction loss functions. Authors show that the proposed model reduce the training time per image without loss in performance. They also introduce novel capabilities, such as interpolation and feedforward modeling of novel images.