Meta Internal Learning
Year: 2022

Abstract:
Internal learning for single-image generation is a framework where a generator is trained to produce novel images based on a single image. Since these models are trained on a single image, they are limited in their scale and application. To overcome these issues, we propose a meta-learning approach that enables training over a collection of images, in order to model the internal statistics of the sample image more effectively. In the presented meta-learning approach, a single-image GAN model is generated given an input image, via a convolutional feedforward hypernetwork f . This network is trained over a dataset of images, allowing for feature sharing among different models and for interpolation in the space of generative models. The generated single-image model contains a hierarchy of multiple generators and discriminators. Therefore, the meta-learner needs to be trained in an adversarial manner, which requires careful design choices that we justify by a theoretical analysis. Our results show that the models obtained are as suitable as single-image GANs for many common image applications, and significantly reduce training time per image, without loss in performance, and introduce novel capabilities, such as interpolation and feedforward modeling of novel images. Our code is available at: https://github.com/RaphaelBensTAU/MetaInternalLearning.

heading: 1 Introduction
text: In the field of internal learning, one wishes to learn the internal statistics of a signal in order to perform various downstream tasks. In this work, we focus on Single image GANs [31, 32, 13, 8], which present extremely impressive results in modeling the distribution of images that are similar to the input image, and in applying this distribution to a variety of applications. However, given that there is no shortage of unlabeled images, one may ask whether a better approach would be to model multiple images, and only then condition the model on a single input image. Doing so, one could (i) benefit from knowledge and feature sharing between the different images, (ii) better define the boundaries between the distribution obtained from the input image and those of other images, (iii) possibly avoid the costly training phase for a novel image, and instead employ feedforward inference, and (iv) mix different single-image models to create novel types of images.
∗Corresponding author - bensadoun@mail.tau.ac.il
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
From the algorithmic standpoint, this multi-image capability can be attempted using various forms of conditioning. For example, one can add a one-hot vector as an input, or, more generally, a vector signature, and train multiple images using the same single image method. One can also add a complete layer of a conditioning signal to the RGB input. Alternatively, one can employ StyleGANlike conditioning and modify the normalization of the layers [16]. More generally, observing that this scenario is a meta-learning problem, one can employ methods such as MAML [5] for learning a central network and its variants per image. After performing many such attempts over a long period of time, we were unable to achieve a desirable level of performance with any of these methods.
Instead, we advocate for a meta-learning solution that is based on hypernetworks [9]. Hypernetworks consist of two main components: a primary network g that performs the actual computation, and the hypernetwork f that is used for conditioning. The parameters (weights) of g are not learned conventionally. Instead, they are given as the output of f for the conditioned input signal. Following a single-image GAN setting with a hierarchical structure, we have two hypernetworks fg and fd, which produce the weights of the multiple generators and discriminators for input image I dynamically.
Our method allows for training on multiple images at once, obtaining similar results for various applications to those previously demonstrated for single-image training. It also allows us to interpolate between single-image GANs derived from pairs (or more) of images. Finally, we are able to fit a new unseen image in a fraction of the time required for training a new single-image GAN, i.e.our method enables inference generation for a novel image.
As far as we can ascertain, ours is the first method to perform adversarial training using hypernetworks. We provide a theoretical analysis of the proper way to perform this. This analysis demonstrates both the sufficiency of our algorithm for minimizing the objective function and the necessity of various components in our method.

heading: 2 Background
text: In this paper we consider the meta-learning problem of learning how to generate a variety of samples from a single image, where each individual learning problem is defined by this single input image. For this purpose, we first recall the setting of single-image generation as in [31, 13, 8].

heading: 2.1 Single-Image Generation
text: SinGAN [31] is composed of a multi-scale residual generator G = {g1, . . . , gk} and a patchdiscriminator D = {d1, . . . , dk}, where gi and di are fully-convolutional networks, consisting of five layers and used for training at scale i. Given an image I , we pre-compute k scales of the image, from coarsest to finest, denoted by Ii, with height and width hi and wi, and use each Ii for training the i’th generator gi. The first generator g1 takes as input fixed random noise z1 ∈ R 3×h1×w1 whose coordinates are i.i.d. normally distributed, and outputs an image Î1 ∈ R 3×h1×w1 . Every other generator gi takes as input an upsampled version Î ↑ i−1 of the previous output Îi−1, plus noise zi ∈ R
3×hi×wi (whose coordinates are i.i.d. normally distributed), and recursively generates a sample at scale i as follows:
Î1 := g1(z1) := ĝ1(z1) , Îi := gi(Îi−1, zi) := Î ↑ i−1 + ĝi(Î ↑ i−1 + zi) , i > 1 (1)
For each scale i ∈ [k], we denote by DI,i the distribution of patches uI,i within Ii and by DÎ,i the distribution of patches uÎ,i within Îi (for z1, . . . , zi ∼ N (0, I)). The goal of this method is to train each generator gi to generate samples Îi, such that, DÎ,i and DI,i would closely match.
For this task, the generators gi are progressively optimized to minimize the 1-Wasserstein distance W (DÎ,i,DI,i) between the distributions DÎ,i and DI,i. The 1-Wasserstein distance between two distributions D1 and D2 is defined as follows:
W (D1,D2) := max d: ∥d∥L≤1
{
E u∼D1 d(u)− E u∼D2 d(u)
}
, (2)
where ∥d∥L is the Lipschitz norm of the discriminator d.
In general, computing the maximum in Eq. 2 is intractable. Therefore, [1] suggested estimating the 1-Wasserstein distance using a pseudo-metric WC(D1,D2), where d is parameterized using a neural network from a wide class C. The method minimizes the adversarial loss, derived from Eq. 2,
Ladv(gi, di) := E z1:i [di(uÎ,i)]− E uI,i [di(uI,i)], (3)
where z1:i = (z1, . . . , zi), uÎ,i ∼ DÎ,i and uI,i ∼ DI,i. The above objective is minimized with respect to the parameters of gi and maximized with respect to the parameters of the discriminator di, while freezing the parameters of all previous generators g1, . . . , gi−1. Note that Îi is given by gi according to Eq. 1. In order to guarantee that di is of a bounded Lipschitz constant, in [7] they apply an additional gradient penalty loss to regularize the Lipschitzness of the discriminator:
Llip(di) := E uI,i
[
∥∇udi(uI,i)∥ 2
2
]
, (4)
In addition, they employ a reconstruction loss. We let z0 1 be fixed random noise, such that:
Î0 1 := ĝ1(z 0 1 ) , Î0i := Î 0,↑ i−1 + ĝi(Î 0,↑ i−1) , i > 1 (5)
In practice, the expected values with respect to the various distributions are replaced with averages over finite sample sets. For simplicity, throughout the paper we will use expectations to avoid clutter.

heading: 2.2 Hypernetworks
text: Formally, a hypernetwork h(z; f(I; θf )) is a pair of collaborating neural networks, f and h. For an input I , network f , parameterized by a set θf of trainable parameters, produces the weights θI = f(I; θf ) for the primary network h. Network h takes an input z and returns an output h(z; θI) that depends on both z and the task-specific input I . In practice, f is typically a large neural network and h is a small neural network. Throughout the paper, we use “;” to separate the input and trainable parameters of a neural network.

heading: 3 Method
text: Our method solves an inherent limitation of current single-sample GAN models, which is the scaling to multi-sample learning, such that the same network can perform single-image generations for each sample. For this purpose, we adopt a hypernetwork-based modeling for the generators and discriminators involved. The hypernetwork network fg produces weights for each gi, and a hypernetwork fd produces weights for each di. In this setting, gi and di consist of the same architecture presented in Sec. 2, and serve as the primary networks for fg and fd (resp.).
Two alternatives for the proposed setting are presented for completeness and are briefly discussed in Sec. 4, (i) shared discriminator and (ii) shared feature extractor. These alternatives help in understanding our proposed approach. The full description and proofs are presented in the supplementary material. An illustration of the proposed model and the two variants are presented in Fig. 1.

heading: 3.1 The Model
text: Our model consists of two main components: a hyper-generator and a hyper-discriminator (see Fig 1(a) for an illustration). The hyper-generator gi is a hypernetwork defined as follows:
gi(z, I) := gi(z; f i g(I; θfg )), (6)
where f ig(I; θfg ) is a neural network that takes an input image I and returns a vector of weights for the i’th generator gi. This network is decomposed into an embedding network eg that is shared among scales and a linear projection pig per scale,
Eg(I) := eg(I; θeg ) (7)
f ig(I; θfg ) := p i g(Eg(I); θ i pg ) (8)
The network eg is parameterized with a set of parameters θeg , and each projection p i g is parameterized with a set of parameters θipg (for simplicity, we denote their union by θpg = (θ i pg )ki=1). Each gi is a fully convolutional network, following Sec. 2, whose weights are θig := f i g(I; θfg ). The overall set of trainable parameters within gi is θfg := (θeg , θ i pg )ki=1. The hyper-discriminator is defined in a similar manner: di(u, I) := di(u; f i d(I; θfd)) (9) where f id(I; θθd) is a network that takes an image I and returns a vector of weights for the i’th discriminator di. This network is also decomposed into an embedding network and a set of projections:
Ed(I) := ed(I; θed) (10) f id(I; θfd) := pd(Eg(I); θ i pd ) (11)
In contrast to the generator, the hyper-discriminator works only on the last image scale. Each di is a fully convolutional network, following Sec. 2, whose weights are θid := f i d(I; θfd). The overall set of trainable parameters within di is θfd := (θed , θ i pd )ki=1.

heading: 3.2 Loss Functions
text: Our objective function is decomposed into an adversarial and a reconstruction loss function,
L(gi, di) =Ladv(gi, di) + λ1 · Llip(di) + λ2 · Lacc−rec(gi), (12)
where λ1, λ2 > 0 are two tradeoff parameters. The loss functions are described below.
Adversarial Loss Function Our adversarial loss function is defined in the following manner:
Ladv(gi, di) := EI
{
E z1:i
di(uÎ,i; f i d(I))− E
uI,i di(uI,i; f
i d(I))
}
, (13)
which is maximized by θfd and minimized by θfg . In order to suffice that di would have a bounded Lipschitz constant, we apply the gradient penalty loss function:
Llip(di) := EIEuI,i [ ∥∇udi(uI,i)∥ 2 2 ]
(14)
For a theoretical analysis of the sufficiency of these loss functions, see Sec. 4.
Reconstruction Loss Function Our method also employs a similar loss function to the reconstruction loss defined in Sec. 2. We accumulate all previous reconstruction losses for each scale:
Lacc−rec(gi) := EiLrec(Îi, Ii) (15)
Previous methods [31, 13, 8] freeze each intermediate generator gi, except for the current training scale, ensuring that each gi is independent
2. In our case, we freeze the projection of all previous scales, except the current scale. However, because eg is shared for all projections, the accumulated reconstruction loss regularizes the training of eg , by minimizing the reconstruction loss with freezed projections as well. We note that this accumulation is mostly needed for small datasets, whereas for large ones we simply compute the loss with respect to the last scale.
2In [13] they optimized each gi with its j (constant) neighboring scales.

heading: 3.3 Initialization and Optimization
text: We initialize the hypernetworks with the initialization suggested by [21]. In this initialization, the network f is initialized using the standard Kaiming He initialization [11]. Each convolutional layer in the primary networks gi and di has a
1√ cin·K·K normalization, where cin is the number of input
channels, and K ×K is the kernel size of the convolution layer.
We train the model progressively from scale 1 to scale k. As noted, during training we freeze all previous projection layers, except for the current training scale. The networks fg and fd are continuously trained across scales, where for fg we add additional projection layers for each new scale, initialized by the previous scale, while fd does not change. Each scale is trained for a constant number of iterations and optimized using the Adam [17] optimizer. Full training and experiments settings are presented in the supplementary material.

heading: 4 Theoretical Analysis
text: In this section, we analyze the soundness of our method, demonstrating its sufficiency. In the supplementary material we demonstrate the importance of the hyper-discriminator and that the generator and discriminator should inherit their parameters from two disjoint hypernetworks. For simplicity, throughout the analysis we omit the reconstruction loss (i.e., λ2 = 0), and assume that the distributions DI , DÎ,i and DI,i are supported by bounded sets. The proof for each proposition is given in the supplementary material.
We are interested in finding a hyper-generator gi for each scale, such that, for each image I , gi(·, I) would generate samples Îi whose patches uÎ,i ∼ DÎ,i are similar to the patches uI,i ∼ DI,i within Ii. Specifically, we would like to train the parameters of gi to minimize the following function:
EIWC(DÎ,i,DI,i) = EI max dI i ∈C1
{
E z1:i dIi (uÎ,i)− E uI,i dIi (uI,i) } , (16)
where Cα := C ∩ {di | ∥di∥L ≤ α}. However, to minimize this objective function directly, one needs to be able to hold a different discriminator dIi for each sample I , which is computationally expensive.
Fortunately, we can think about this expression in a different manner, as it can also be written as follows:
EIWC(DÎ,i,DI,i) = max S EI
{
E z1:i di(uÎ,i;S(I))− E uI,i
di(uI,i;S(I)) }
, (17)
where the maximum is taken over the set of mappings S from images I to parameters θI of discriminators dIi ∈ C 1. We let S∗ be a mapping that takes I and returns the parameters S∗(I) of the discriminator dIi := di(·;S ∗(I)) = argmaxdi∈C1 { Ez1:id I i (uÎ,i)− EuI,id I i (uI,i) } .
Therefore, if S∗ can be approximated by a large neural network f id(I) = f i d(I; θfd) ≈ S ∗(I), then we can approximately solve the maximization in Eq. 17 by parameterizing the discriminator with a hypernetwork di := di(u; fd(I; θfd)) and training its parameters to solve the maximization in Eq. 17 (approximately). For instance, if S∗ is a continuous function, one can approximate S∗ using a large enough neural network up to any approximation error ≤ ϵ [4, 14, 27, 24, 10, 20, 30]. This is summarized in the following proposition. Proposition 1. Assume that I ⊂ R3×h×w is compact. Let ϵ > 0 be an approximation error. Let gi(z, I) := gi(z; f i g(I; θfg )) be a hyper-generator and C a class of discriminators. Assume that S
∗ is continuous over I. Then, there is a large enough neural network f id (whose size depends on ϵ), such that, the hyper-discriminator di(u, I) := di(u; f i d(I; θfd)) satisfies:
EIWC(DÎ,i,DI,i) = max θfd Ladv(gi, di) + oϵ(1), (18)
where the maximum is taken over the parameterizations θfd of fd, such that, di(·; f i d(I; θfd)) ∈ C 1.
A proof for the existence of a continuous selector S∗(I) has been proposed [6, 25, 26] for similar settings, and the proof for Prop. 1 is provided as part of the supplementary material. According to this proposition, in order to minimize EIWC(DÎ,i,DI,i), we can simply parameterize
our discriminator with a hypernetwork di := di(u; f i d(I; θfd)) and train the hyper-generator gi to solve: minθfg maxθfd Ladv(di, gi) along with the gradient penalty loss Llip(di) to ensure that di(·; fd(I; θfd)) would have a bounded Lipschitz constant.
Put differently, in order to guarantee that the approximation error in Prop. 1 would be small, we advocate selecting the hypernetwork f id to be a large neural network. In this case, if we are able to effectively optimize θfg and θfd to solve minθfg maxθfd Ladv(di, gi) (s.t the Lipschitz constant of di is bounded), we can ensure that EIWC(DÎ,i,DI,i) is small, as desired.

heading: 4.1 Alternative Architectures
text: As presented in Sec. 3, we consider two alternative architectures (i) shared discriminator and (ii) shared feature extractor. We briefly describe each proposed variant and its limitations; the full analysis is presented in the supplementary material.
Shared Discriminator In this case, the model has two main components for each scale i: a hypergenerator gi(z, I) = gi(z; f i g(I; θfg )), along with a standard discriminator di(u) = di(u; θd) that is shared among all samples I , as illustrated in Fig. 1(b). We show that if the expected (w.r.t I ∼ DI ) distance between the distributions DÎ,i and Di is small, then the loss function Ladv(gi, di) := EI{Ez1:i di(uÎ,i)−Eui,I di(uI,i)} tends to be small. Here, Di denotes the distribution of uÎ,i ∼ DÎ,i for I ∼ DI . This proposition shows that a hyper-generator gi(·, I) that generates samples Îi whose patches are similar to samples of Di would minimize the loss function Ladv(gi, di), even though the generated samples are not conditioned on the image I . Therefore, simply minimizing the adversarial
loss would not guarantee that gi(·, I) generates samples Îi that are similar to Ii.
Shared Feature Extractor We note that as a strategy for reducing the number of trainable parameters in the whole model, one could restrict fg and fd to share their encoding component e, as illustrated in Fig. 1(c). We show two cases in which this approach fails. First, we consider the case where the model is trained using GD. In this case, GD iteratively updates (θe, θ i pg ) to minimize Ladv(gi, di) and updates (θe, θ i pd ) to maximize Ladv(gi, di) − λ1 · Llip(di). Informally, we show that θe is essentially trained to only minimize Llip(di) and that each tuple (θe, θ i pg , θipd) with di ≡ 0 is an equilibrium point. In addition, we note that Llip(di) is minimized by di ≡ 0. Therefore, it is likely that di would converge to 0 during training. This means that at some point the discriminator is ineffective. In particular, if θe = 0, then (θe, θ i pg , θipd) is an equilibrium point. We note that θe = 0 is not a desirable output of the training algorithm, since it provides a hyper-generator gi(·, I) that is independent of the input image I . Second, we consider the case where GD iteratively optimizes (θe, θ i pg ) to minimize Ladv(gi, di), θ i pd to maximize Ladv(gi, di) and (θe, θ i pg ) to minimize the loss Llip(di). We show that each tuple (θe, θ i pg , θipd) with θe = 0 is, again, an equilibrium point.

heading: 5 Experiments
text: Our experiments are divided into two parts. In the first part, we study three different training regimes of our method. First, we experiment with single-image training, in order to produce a fair comparison with existing methods. Second, we present a mini-batch training scheme, where instead of a single image, the model is trained on a fixed set of images. Lastly, we experiment with training over a full dataset that cannot fit into a single batch.
In the second part, we experiment with several applications of our method. Specifically, we study the ability of our method in the Harmonization, Editing and Animation tasks proposed by [31], as well as generating samples of arbitrary size and aspect ratio. In addition, we experiment with two new applications: image interpolations, and generation at inference time. These application are unique to multi-image training.
Due to space constraints, we focus on our novel applications, and refer the reader to the supplementary material for our full set of applications, as well as technical details, such as specific hyperparameters, GPU usage and additional experiments.
Throughout the experiments, we consider the following set of baselines: SinGAN [31], ConSinGAN [13] and HP-VAE-GAN [8]. To evaluate image generation, we use the single-image FID metric

heading: 1 0.03 0.07 0.73

heading: 2 0.04 0.07 0.66

heading: 3 0.03 0.07 0.68

heading: 4 0.04 0.08 0.70
text: (SIFID) [31]. Following [31], the metric represents the mean of minimum SIFID over 100 generated samples per image. We further compute the mean-SIFID (mSIFID), which is the mean across all generated samples of all images, without taking the minimum for each image.
As simply overfitting the training image would lead to a SIFID value of 0, a diversity measure is necessary. For this purpose we employ the diversity measure used in [31]. This measure is computed as the average standard deviation over all pixel values along the channel axis of 150 generated images.
Previous works in the field [32, 31] require training on each image independently. To enable comparison with previous work, we use the 50-image dataset of [31], denoted by Places-50 and the 50-image dataset of [12], denoted by LSUN-50. Additionally, whenever a quantitative measure is available, we present competitive results, and qualitatively our results are at least as good, if not better than those of single-image GANs. For larger datasets of up to 5000 images we perform comprehensive experiments with our proposed method.
The dataset presented by SinGAN, Places-50, consists of 50 images selected at random from subcategories of the Places dataset [38] – Mountains, Hills, Desert and Sky and the dataset presented by ConSinGAN, LSUN-50, consists of five randomly sampled images from each of the ten LSUN dataset categories. In order to evaluate our method on larger datasets, we consider three subsets of the Valleys category of the Places dataset; the first 500(V500), 2500(V2500) and 5000(V5000) (the entire category) images, and use the 100-image test set when relevant. Additionally, we consider the first 250(C250) and 500(C500) images of the Churches Outdoor category of the LSUN dataset.

heading: 5.1 Training Procedures
text: Single-Image training Our approach is first evaluated when training with a single image, as done in previous methods. Since a single function needs to be learned, a standard discriminator (i.e., not learned via hypernetwork) is used in this specific case in order to avoid redundant enlargement of the model and speed up training. Similar results are obtained using a hyper-discriminator. Tab. 1 shows that our performance is on par with current single-image models for this setting.
Single mini-batch training When introduced with multiples images, the standard discriminator, as for the baseline methods, suffers from leakage between the images in the mini-batch, i.e., the patches of the generated images are distributed as patches of arbitrary images from the batch (Sec. 4.1– Shared Discriminator). Fig. 2 illustrates this effect. To overcome this issue, we introduce a hyperdiscriminator that enables efficient learning of a different discriminator model for each image. To evaluate performance on single mini-batch learning, we randomly sampled a set of 5 images from the 50-image dataset and trained a different model for each permutation of the set of size 1 ≤ i ≤ 5. Tab. 2 show performance is good regardless of the mini-batch size, which indicates that the hypernetwork model successfully learns a different model for each input image.
Dataset training Our main contribution arises from training with large amount of data. In addition to Places-50 and LSUN-50, we trained our method on three subsets of the Valley category, as presented above – Valley500, Valley2500 and Valley5000, iterating on batches of 16 images, for 25k, 100k, 150k iterations per scale, respectively, on a single GPU. Tab. 1 and 3 show performance and training time per image. Churches250 and Churches500 were trained in a similar manner for 20k and 30k iterations per scale, respectively, and reached equal performance of 0.20, 0.27 and 0.47 for SIFID, mSIFID and diversity metrics.
As far as we are aware, our method is the first that is able to train multiple single-image models at this scale with a decent runtime. For example, training the model presented by [31] on Valley5000 with a single GPU would require the training of 5000 different and independent models, and would take approximately 200 days. Our method takes 10 days on a single GPU, and thus is faster by a factor of 20.

heading: 5.2 Applications
text: As noted above, we present our novel applications in the main text, and refer the reader to the supplementary material for applications presented in previous works.
Interpolation As our meta-learning approach learns the space of generators, and is trained on multiple images, our model is able to interpolate between different images smoothly and at different scales. Unlike common interpolation, the hierarchical structure of the model enables interpolation at different scales as follows: We start by interpolating over the latent representation eg, resulting in a new generator. Let A and B be two different images. We compute their latent representations eAg = Eg(A) and e B g = Eg(B) (resp.) and perform linear interpolation between the two, for α ∈ [0, 1], eαg = αe A g + (1− α)e B g , resulting in a new generator. We then select a primary image, A for example, and initial scale m, and perform the following steps: (i) we use eAg to generate scales 1 to m, and (ii) from scale m, we switch to eαg , and continue generating scales accordingly. The result is a mix at different patch scales, where scale 1 controls the structure of the image, and the last scale controls the finer texture of the image. Fig. 3 shows an example of a pair of images and its interpolation, where the primary image is denoted by A (top-left) and the target image by B (top-right). We show interpolation at three different scales - first (1), middle, and last, presented from top to bottom. As can be seen, interpolating at the first scale results in more structural changes, while interpolating at the middle and last scales results in more textural changes. By changing α we are able to obtain a wide gamut of intermediate options between the two images.
Feedforward generation The meta-learning approach, and the fact that our method is able to learn from a relatively large dataset such as Valley5000, introduce the ability to model a new image in one forward pass. Fig. 5 and Tab. 3 show inference results for three different models trained on Valley500, Valley2500 and Valley5000. As can be seen, it requires a significantly larger dataset than that of [31] to get the model to generalize. The network trained on Valley5000 enables modeling of a new image in a fraction of a second, and results in coherent and reasonable generated images compared with previous methods, which were unable to perform this task.

heading: 6 Limitations
text: A prominent limitation of the method is the model’s size. The hypernetwork approach leads to large projection layers, e.g. a convolution layer with a weight size of (64, 64, 3, 3) and embedding size of |e| = 512 will result in a projection layer with a weight size of |(512, 36864)| ≈ 18M parameters. This obviously affects convergence, runtime and GPU memory usage.
In Fig. 6, we quantitatively explore the out-of-distribution generalization capabilities of our feedforward method when training on the Valley5000 nature image dataset. As can be seen, for images that are completely out of domain, the generated images are not faithful to the input image. Training on a large-scale heterogeneous dataset to further improve generalization requires days of training. Until this experiment is performed, it is unclear whether the architecture has enough capacity to support this one-model-fits-all capability.

heading: 7 Related work
text: Hypernetworks, which were first introduced under this name in [9], are networks that generate the weights of a second primary network, which computes the actual task. Hypernetworks are especially suited for meta-learning tasks, such as few-shot [2] and continual learning tasks [36], due to the knowledge-sharing ability of the weight-generating network. Knowledge sharing in hypernetworks was recently used for continual learning by [36].
Predicting the weights instead of performing backpropagation can lead to efficient neural architecture search [3, 37] and hyperparameter selection [23]. In [22], hypernetworks were applied for 3D shape reconstruction from a single image. In [34] hypernetworks were shown to be useful for learning shared image representations. Note that while the name of the method introduced in [28] is reminiscent of our method, it solves a different task, with a completely different algorithm. Their method does not employ a hypernetwork to parameterize their generator (or discriminator); instead, their generator serves as a hypernetwork itself. In addition, they intend to learn the distribution of weights of high-performing classifiers on a given classification task, which is a different application.
Several GAN-based approaches were proposed for learning from a single image sample. Deep Image Prior [35] and Deep Internal Learning [33] showed that a deep convolutional network can form a useful prior for a single image in the context of denoising, super-resolution, and inpainting. SinGAN [29] uses patch-GAN [15, 29, 18, 19] to model the multiscale internal patch distribution of a single image, thus generating novel samples. ConSinGAN [12] extends SinGAN, improving quality and training time. However, these methods need to be trained on each image individually. In this work, we propose a novel approach based on hypernetworks that leverages the capabilities of single-image generation and enables efficient training on datasets of any size, while preserving the unique properties of single-image training.

heading: 8 Conclusions
text: Given the abundance of unlabeled training images, training a single image GAN is unjustifiable if viable multi-image alternatives exist. We present the first such alternative, which also opens the door to novel applications that are impossible with existing models, such as interpolation between single-image domains and feedforward modeling. From a technical perspective, we present what is to our knowledge the first adversarial hypernetwork. Working with this novel multi-network structure requires an understanding of the interplay between the components involved, and we support our method with theoretical analysis.

heading: Acknowledgments
text: This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant ERC CoG 725974). The contribution of the first author is part of a Master thesis research conducted at Tel Aviv University.

Review summaries:
The paper aims at improving single-image GAN models by training them on larger datasets. The authors use a meta-learning approach based on the hypernetworks scheme. The model consists of a hyper-generator and a hyper-discriminator. The Hyper-generator network is decomposed into an embedding network that is shared among scales and a linear projection per scale. The hyper-discriminator network is also decomposed into an embedding network and a set of projections. The model is trained with adversarial and reconstruction loss functions. Authors show that the proposed model reduce the training time per image without loss in performance. They also introduce novel capabilities, such as interpolation and feedforward modeling of novel images. 

The paper proposes a method to learn hypernetworks, trained on a larger dataset (500-5000 images), for single-image GANs. These hypernetworks are then used to initialize the GAN for a specific single image to speed-up training time and improve image quality/diversity. The single-image GAN follows the same architecture as SinGAN and the individual generators and discriminators for a specific image are initialized by the trained hypernetwork and then fine-tuned for the specific image. The resulting single-image GANs outperform related work (SinGAN, ConSinGAN, ...) based on SIFID, diversity, and training time.

This paper presents a novel single image generation framework that is able to generalize to multi-sample case, which is different from previous work where each model only works on a single sample, leveraging hyper network. The author has provided solid theoretical analysis to support their design. And the effectiveness is also validated by their produced results.