Meta Internal Learning

1 Introduction . In the field of internal learning , one wishes to learn the internal statistics of a signal in order to perform various downstream tasks . In this work , we focus on Single image GANs [ 31 , 32 , 13 , 8 ] , which present extremely impressive results in modeling the distribution of images that are similar to the input image , and in applying this distribution to a variety of applications . However , given that there is no shortage of unlabeled images , one may ask whether a better approach would be to model multiple images , and only then condition the model on a single input image . Doing so , one could ( i ) benefit from knowledge and feature sharing between the different images , ( ii ) better define the boundaries between the distribution obtained from the input image and those of other images , ( iii ) possibly avoid the costly training phase for a novel image , and instead employ feedforward inference , and ( iv ) mix different single-image models to create novel types of images . ∗Corresponding author - bensadoun @ mail.tau.ac.il 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . From the algorithmic standpoint , this multi-image capability can be attempted using various forms of conditioning . For example , one can add a one-hot vector as an input , or , more generally , a vector signature , and train multiple images using the same single image method . One can also add a complete layer of a conditioning signal to the RGB input . Alternatively , one can employ StyleGANlike conditioning and modify the normalization of the layers [ 16 ] . More generally , observing that this scenario is a meta-learning problem , one can employ methods such as MAML [ 5 ] for learning a central network and its variants per image . After performing many such attempts over a long period of time , we were unable to achieve a desirable level of performance with any of these methods . Instead , we advocate for a meta-learning solution that is based on hypernetworks [ 9 ] . Hypernetworks consist of two main components : a primary network g that performs the actual computation , and the hypernetwork f that is used for conditioning . The parameters ( weights ) of g are not learned conventionally . Instead , they are given as the output of f for the conditioned input signal . Following a single-image GAN setting with a hierarchical structure , we have two hypernetworks fg and fd , which produce the weights of the multiple generators and discriminators for input image I dynamically . Our method allows for training on multiple images at once , obtaining similar results for various applications to those previously demonstrated for single-image training . It also allows us to interpolate between single-image GANs derived from pairs ( or more ) of images . Finally , we are able to fit a new unseen image in a fraction of the time required for training a new single-image GAN , i.e.our method enables inference generation for a novel image . As far as we can ascertain , ours is the first method to perform adversarial training using hypernetworks . We provide a theoretical analysis of the proper way to perform this . This analysis demonstrates both the sufficiency of our algorithm for minimizing the objective function and the necessity of various components in our method . 2 Background . In this paper we consider the meta-learning problem of learning how to generate a variety of samples from a single image , where each individual learning problem is defined by this single input image . For this purpose , we first recall the setting of single-image generation as in [ 31 , 13 , 8 ] . 2.1 Single-Image Generation . SinGAN [ 31 ] is composed of a multi-scale residual generator G = { g1 , . . . , gk } and a patchdiscriminator D = { d1 , . . . , dk } , where gi and di are fully-convolutional networks , consisting of five layers and used for training at scale i . Given an image I , we pre-compute k scales of the image , from coarsest to finest , denoted by Ii , with height and width hi and wi , and use each Ii for training the i ’ th generator gi . The first generator g1 takes as input fixed random noise z1 ∈ R 3×h1×w1 whose coordinates are i.i.d . normally distributed , and outputs an image Î1 ∈ R 3×h1×w1 . Every other generator gi takes as input an upsampled version Î ↑ i−1 of the previous output Îi−1 , plus noise zi ∈ R 3×hi×wi ( whose coordinates are i.i.d . normally distributed ) , and recursively generates a sample at scale i as follows : Î1 : = g1 ( z1 ) : = ĝ1 ( z1 ) , Îi : = gi ( Îi−1 , zi ) : = Î ↑ i−1 + ĝi ( Î ↑ i−1 + zi ) , i > 1 ( 1 ) For each scale i ∈ [ k ] , we denote by DI , i the distribution of patches uI , i within Ii and by DÎ , i the distribution of patches uÎ , i within Îi ( for z1 , . . . , zi ∼ N ( 0 , I ) ) . The goal of this method is to train each generator gi to generate samples Îi , such that , DÎ , i and DI , i would closely match . For this task , the generators gi are progressively optimized to minimize the 1-Wasserstein distance W ( DÎ , i , DI , i ) between the distributions DÎ , i and DI , i . The 1-Wasserstein distance between two distributions D1 and D2 is defined as follows : W ( D1 , D2 ) : = max d : ∥d∥L≤1 { E u∼D1 d ( u ) − E u∼D2 d ( u ) } , ( 2 ) where ∥d∥L is the Lipschitz norm of the discriminator d. In general , computing the maximum in Eq . 2 is intractable . Therefore , [ 1 ] suggested estimating the 1-Wasserstein distance using a pseudo-metric WC ( D1 , D2 ) , where d is parameterized using a neural network from a wide class C. The method minimizes the adversarial loss , derived from Eq . 2 , Ladv ( gi , di ) : = E z1 : i [ di ( uÎ , i ) ] − E uI , i [ di ( uI , i ) ] , ( 3 ) where z1 : i = ( z1 , . . . , zi ) , uÎ , i ∼ DÎ , i and uI , i ∼ DI , i . The above objective is minimized with respect to the parameters of gi and maximized with respect to the parameters of the discriminator di , while freezing the parameters of all previous generators g1 , . . . , gi−1 . Note that Îi is given by gi according to Eq . 1 . In order to guarantee that di is of a bounded Lipschitz constant , in [ 7 ] they apply an additional gradient penalty loss to regularize the Lipschitzness of the discriminator : Llip ( di ) : = E uI , i [ ∥∇udi ( uI , i ) ∥ 2 2 ] , ( 4 ) In addition , they employ a reconstruction loss . We let z0 1 be fixed random noise , such that : Î0 1 : = ĝ1 ( z 0 1 ) , Î0i : = Î 0 , ↑ i−1 + ĝi ( Î 0 , ↑ i−1 ) , i > 1 ( 5 ) In practice , the expected values with respect to the various distributions are replaced with averages over finite sample sets . For simplicity , throughout the paper we will use expectations to avoid clutter . 2.2 Hypernetworks . Formally , a hypernetwork h ( z ; f ( I ; θf ) ) is a pair of collaborating neural networks , f and h. For an input I , network f , parameterized by a set θf of trainable parameters , produces the weights θI = f ( I ; θf ) for the primary network h. Network h takes an input z and returns an output h ( z ; θI ) that depends on both z and the task-specific input I . In practice , f is typically a large neural network and h is a small neural network . Throughout the paper , we use “ ; ” to separate the input and trainable parameters of a neural network . 3 Method . Our method solves an inherent limitation of current single-sample GAN models , which is the scaling to multi-sample learning , such that the same network can perform single-image generations for each sample . For this purpose , we adopt a hypernetwork-based modeling for the generators and discriminators involved . The hypernetwork network fg produces weights for each gi , and a hypernetwork fd produces weights for each di . In this setting , gi and di consist of the same architecture presented in Sec . 2 , and serve as the primary networks for fg and fd ( resp. ) . Two alternatives for the proposed setting are presented for completeness and are briefly discussed in Sec . 4 , ( i ) shared discriminator and ( ii ) shared feature extractor . These alternatives help in understanding our proposed approach . The full description and proofs are presented in the supplementary material . An illustration of the proposed model and the two variants are presented in Fig . 1 . 3.1 The Model . Our model consists of two main components : a hyper-generator and a hyper-discriminator ( see Fig 1 ( a ) for an illustration ) . The hyper-generator gi is a hypernetwork defined as follows : gi ( z , I ) : = gi ( z ; f i g ( I ; θfg ) ) , ( 6 ) where f ig ( I ; θfg ) is a neural network that takes an input image I and returns a vector of weights for the i ’ th generator gi . This network is decomposed into an embedding network eg that is shared among scales and a linear projection pig per scale , Eg ( I ) : = eg ( I ; θeg ) ( 7 ) f ig ( I ; θfg ) : = p i g ( Eg ( I ) ; θ i pg ) ( 8 ) The network eg is parameterized with a set of parameters θeg , and each projection p i g is parameterized with a set of parameters θipg ( for simplicity , we denote their union by θpg = ( θ i pg ) ki=1 ) . Each gi is a fully convolutional network , following Sec . 2 , whose weights are θig : = f i g ( I ; θfg ) . The overall set of trainable parameters within gi is θfg : = ( θeg , θ i pg ) ki=1 . The hyper-discriminator is defined in a similar manner : di ( u , I ) : = di ( u ; f i d ( I ; θfd ) ) ( 9 ) where f id ( I ; θθd ) is a network that takes an image I and returns a vector of weights for the i ’ th discriminator di . This network is also decomposed into an embedding network and a set of projections : Ed ( I ) : = ed ( I ; θed ) ( 10 ) f id ( I ; θfd ) : = pd ( Eg ( I ) ; θ i pd ) ( 11 ) In contrast to the generator , the hyper-discriminator works only on the last image scale . Each di is a fully convolutional network , following Sec . 2 , whose weights are θid : = f i d ( I ; θfd ) . The overall set of trainable parameters within di is θfd : = ( θed , θ i pd ) ki=1 .

Summary: The paper proposes a method to learn hypernetworks, trained on a larger dataset (500-5000 images), for single-image GANs. These hypernetworks are then used to initialize the GAN for a specific single image to speed-up training time and improve image quality/diversity. The single-image GAN follows the same architecture as SinGAN and the individual generators and discriminators for a specific image are initialized by the trained hypernetwork and then fine-tuned for the specific image. The resulting single-image GANs outperform related work (SinGAN, ConSinGAN, ...) based on SIFID, diversity, and training time.