Meta Internal Learning
Authors: Raphael Bensadoun
Shir Gur
Tomer Galanti
Lior Wolf
Venue: NeurIPS 2021 Poster (2021)
Keywords: hypernetworks
meta learning
single image generation
gan
computer vision

Abstract: Internal learning for single-image generation is a framework, where a generator is trained to produce novel images based on a single image. Since these models are trained on a single image, they are limited in their scale and application. To overcome these issues, we propose a meta-learning approach that enables training over a collection of images, in order to model the internal statistics of the sample image more effectively.
In the presented meta-learning approach, a single-image GAN model is generated given an input image, via a convolutional feedforward hypernetwork $f$. This network is trained over a dataset of images, allowing for feature sharing among different models, and for interpolation in the space of generative models. The generated single-image model contains a hierarchy of multiple generators and discriminators. It is therefore required to train the meta-learner in an adversarial manner, which requires careful design choices that we justify by a theoretical analysis. Our results show that the models obtained are as suitable as single-image GANs for many common image applications, {significantly reduce the training time per image without loss in performance}, and introduce novel capabilities, such as interpolation and feedforward modeling of novel images. 

Meta Internal Learning
Raphael Bensadoun" The School of Computer Science Tel Aviv University
Shir Gur
The School of Computer Science Tel Aviv University
Tomer Galanti
Lior Wolf
The School of Computer Science Tel Aviv University
The School of Computer Science Tel Aviv University
Abstract
Internal learning for single-image generation a framework where a generator is trained to produce novel images based on single image. Since these models are trained on a single image, they are limited their scale and application. To overcome these issues, we propose meta-learning approach that enables training over a collection of images, in order to model the internal statistics of the sample image more effectively. In the presented meta-learning approach, Langle-image GAN model is generated given an input image, via a convolutional feedforward hy- pernetwork This network is trained over dataset of images, allowing for feature sharing among different models and for interpolation in the space of generative mod- els. The generated single-image model contains hierarchy of multiple generators and discriminators. Therefore, the meta-learner needs to be trained in an adversar- ial manner, which requires careful design choices that we justify by theoretical analysis. Our results show that the models obtained are as suitable as single-image GANs for many common image applications, and significantly reduce training time per image, without loss in performance, and introduce novel capabilities, such as interpolation and feedforward modeling novel images. Our code is available https:/ //g thub. com Raphaapppaaaaeeeeeee MattssS//MMMMMaeeeeteaaaa
Introduction
In the field of internal learning, one wishes to leam the internal statistics of a signal in order to perform various downstream tasks. In this work, focus Single image GANs [31, 32, 13, 8]. which present extremely impressive results in modeling the distribution of images that are similar to the input image, and in applying this distribution there is no shortage of unlabeled images, one may ask whether a better approach would be to model multiple images, and only then condition the model on a single input image. Doing so, one could (i) benefit from knowledge and feature sharing between the different images, (ii) better define the boundaries between the distribution obtained from the input image and those of other images, (iii) possibly avoid the costly training phase for novel image, and instead employ feedforward inference, and (iv) mix different single-image models to create novel types of images.
variety of applications. However, given that
Corresponding author bensadoun @ mmmil.tau.ac.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

From the algorithmic standpoint, this multi image capability can be attempted using various forms of conditioning. For example, one can add one-hot vector an input, or, more generally, vector signature, and train multiple images using the same single image method. One can also add complete layer of conditioning signal like conditioning and modify the normalization of the layers [16]. More generally. observing that this scenario a meta-learning problem, one can employ methods such as MAML [5] for learning a central network and its variants per image. After performing many such attempts over a long period of time, we were unable to achieve a desirable level of performance with any of these methods.
RGB input. Alternatively, one can employ StyleGAN-
Instead, we advocate for meta-leerrnng solution that is based on hypernetworks [9]. Hypernetworks consist of main components: a primary network that performs the actual computation, and the hypernetwork f that is used for conditioning. The parameters (weights) of are not learned conventionally. Instead, they are given as the output of f for the conditioned input signal. Following a single-image GAN setting with hierarchical structure, we have two hypernetworks fg and fa, which produce the weights of the multiple generators and discriminators for input image dynamically.
Our method allows for training on multiple images at once, obtaining similar results for various applications those previously demonstrated for single-image training. It also allows us to interpolate between single-image GANs derived from pairs (or more) of images. Finally, we are able to fit a new unseen image in a fraction of the time required for training a new angle-image GAN, i.e.our method enables inference generation for novel image.
As far as we ascertain, We provide a theoretical analysis of the proper way to perform this. This analysis demonstrates both the sufficiency of our algorithm for minimizing the objective function and the necessity of various components in our method.
is the first method perform adversarial training using hypernetworks.
2 Background
In this paper we consider the meta-learning problem of learning how generate variety of samples from a single image, where each individual learning problem is defined by this single input image. For this purpose, we first recall the setting of single-image generation as in [31, 13, 8].
2.1 Single-Image Generation
SinGAN [31] is composed of multi-scale residual generator G f91..... 9kk and patch- discriminator = {di..... dę }. where gi and d, are fully-convolutional networks, consisting of five layers and used for training at scale i. Given image I, pre-compute & scales of the image, from coarsest to finest, denoted by I,, with height and width h; and w;, and use each I; for training the i'th generator 9t-
The first generator 91 takes as input fixed random noise 21 E 1B3xhixxu whose coordinates are ii.d. normally distributed, and outputs an image I kxxii x xi, Every other generator Zi takes as input an upsampled version fi, of the previous output f,-1, plus noise 2; E R3x (whose coordinates are i.i.d. normally distributed), and recursively generates sample at scale i as follows:
i (z)) == (=1), :::9:((i-,))) = i,_ ai 2;), i>1
(I)
For each scale E [k],, we denote by Dr, the distribution of patches ur, within I, and by Dz, the distribution of patches Wi within 1; (for ..,..., .,. 24 1)), The goal of this method is train each generator 9i to generate samples i, such that, Dj, and D), would closely match
For this task, the generators are progressively optimized to minimize the 1-Wasserstein distance WDD,, Dr.;) between the distributions D and D, The I-Wasserstein distance between two distributions D, and Dr, defined as follows:
d(u) - E d(u))
((D1.D)) = max
E
(2)
d: I|d 11
Wr
where ||d]| is the Lipschitz norm of the discriminator d.

---

4.
(=(=: (5)
Val (::0
Pg
d2(
PG
(2(::03)
P
(d(-;98)
Pg
(1(::0))
(. (:))))
di (:00)
(x(:; %
(-
(b)
Figure 1: Alternative architectures for hypernetwork single image generators. (a) Our model architecture consists of two embedding networks Eg for the hyper-genenan and ed for the hyper- discriminator. The primary networks gi and d, follow Sec. 2. (b) Hyper-Generator with shared discriminator. (c) Shared feature extractor. We omit the input/output names for clarity.
In general, computing the maximum in Eq. 2 is intractable. Therefore, [1] suggested estimating the W-ssserstein distance using a pseudo-metric We(D1, D2), where d parameterized using a neural network from wide class C. The method minimizes the adversarial loss, derived from Eq. 2,
Lado de)
E [d: (wr,)) - T. [d((ur)))
where 21:1 (=1;-..; 2:), "1 Djs and uy, Dr,-- The above objective is minimized with respect to the parameters of 9i and maximized with respect to the parameters of the discriminator d,, while freezing the parameters of all previous generators -1:---: 91-1- Note that i, is given by gi according to Eq. 1. In order to guarantee that d, is a bounded Lipschitz constant, [7] they apply an additional gradient penalty loss regularize the Lipschitzness of the discriminator:
Lisp (dk) = E ([V.dd(u,,)
(4)
In addition, they employ reconstruction loss. We let of be fixed random noise, such that:
. := (1())
I- ((
i>1
(5)
In practice the expected values with respect to the various distributions are replaced with averages over finite sample sets. For simplicity, throughout the paper we will expectations avoid clutter.
2.2 Hypernetworks
Formally, hypernetwork h(2; ([; is pair of collaborating neural networks, f and h. For an input I, network S. parameterized by set 0, of trainable parameters, produces the weights 0, = = 0y) for the primury network h. Network takes an input and returns an output h12;; 0y) that depends on both and the task-specific input 7. In practice, is typically large neural network and a small neural network. Throughout the paper, we use ";" to separate the input and trainable parameters of neural network.
(/;;0y)
3 Method
Our method solves an inherent limitation of current single-sample GAN models, which is the scaling to multi-sample learning, such that the same network can perform single-image generations for each sample. For this purpose, we adopt a hypemetwork- and discriminators involved. The hypernetwork network fy produces weights for each gi. and a hypernetork fa produces weights for each d,. In this setting, 9: and d, consist of the same architecture presented in Sec. 2, and serve the primary networks for fy and fa (resp.).
modeling for the generators
Two alternatives for the proposed setting are presented for completeness and are briefly discussed in Sec. 4, (i) shared discriminator and (ii) shared feature extractor. These alternatives help in under- standing our proposed approach. The full description and proofs are presented in the supplementary material. An illustration of the proposed model and the two variants presented in Fig. 1.

---

3.1 The Model
Our model consists of two main components: Doype-ggenerato and a hyper-discriminntoor (see Fig ((a) for an illustration). The eppeer-ennrrtorr 9s is a hypermetwork defined as follows:
gu(2, 1) = gi(z; f (1;00,)
(6)
where ; is a neural network that takes an input image / and returns a vector of weights for the i'th generator Pt- This network is decomposed into embedding network eg that is shared among scales and linear projection p/ per scale,
E1(1) =e((Iiee) a
= =.
The network Eg is parameterized with of parameters e,,- and each projection p is parameterized with a set of parameters 6% (for simplicity, we denote their union by 0p = ( ) 1)) Each 9i is a Pg
fully convolutional network, following Sec. 2. whose weights are 8j := fy (1: f,,). The overall set of trainable parameters within gi is f = (Be, ) The hyper-discriminator defined similar manner:
(u, 1) == de (u; f)():
where [I Be.) network that takes an image I and returns vector of weights for the i'th discriminator d,. This network is also decomposed into embedding network and a set of projections:
Ea(1) = 6a(1;6e)) (2(T:05) == (T(F((T);
(10)
(11)
In contrast to the generator, the hyper-discriminato works only on the last image scale. Each d, is a fully convolutional network, following Sec. 2, whose weights are fj == f The overall set of trainable parameters within d, is 0fg = (ferr of V-11
3.2 Loss Functions
Our objective function is decomposed into an adversarial and a reconstruction loss function, [ d4) =adv (gr d.) + A1 (isp(d;) + X2 Luc)-ree(gs).
(12)
where \1, A2 > are two tradeoff parameters. The loss functions are described below.
Adversarial Loss Function
Our adversarial loss function is defined in the following manner:
d, (u fa(T)) E du (((ur,::((()) $2(D)) 21:3
Cadu(gia d;) E,
(13)
ar,1
which is maximized by Of2 and minimized by Oz,- In order to suffice that d; would have a bounded Lipschitz constant, we apply the gradient penalty loss function:
(dp = Ey [I (ur,((1))))
(14)
For theoretical analysis of the sufficiency of these loss functions, see Sec. 4.
Reconstruction Loss Function Our method also employs a similar loss function the recon- struction loss defined in Sec. 2. We accumulate all previous reconstruction losses for each scale:
Lace-ree (g:) = ExCrec(ft,
(15)
Previous methods [31, 13, 8] freeze each intermediate generator 9i, except for the current training scale, ensuring that each 9i is independent 2. In our case, we freeze the projection of all previous scales, except the current scale. However, because eg shared for all projections, the accumulated reconstruction loss regularizes the training of cg. by minimizing the reconstruction loss with freezed projections as well. We note that this accumulation is mostly needed for small datasets, whereas for large ones we simply compute the loss with respect to the last scale.
Pn [13] they optimized each with its (constant) neighboring scales.

---

3.3 Initialization and Optimization
We initialize the hypernetwrkss with the initialization suggested by [21]. In this initialization, the network f is initialized using the standard Kaiming He initialization [11]. Each convolutional layer in the primary networks 9; and d, has a K- K-R normalization, where Cin the number of input channels, and K x K is the kernel size of the convolution layer.
We train the model progressively from scale 1 to scale k. As noted, during training we freeze all previous projection layers, except for the current training scale. The networks fg and fa are continuously trained across scales, where for fg we add additional projection layers for each new scale, initialized by the previous scale, while fa does not change. Each scale trained for a constant number of iterations and optimized using the Adam [17] optimizer. Full training and experiments settings are presented in the supplementary material.
Theoretical Analysis
In this section, analyze the soundness of our method, demonstrating its sufficiency. In the supplementary material we demonstrate the importance of the hyper-discriminator and that the generator and discriminator should inherit their parameters from two disjoint hypernetworks. For simplicity, throughout the analysis we omit the reconstruction loss (i.e., X2 = 0), and assume that the distributions D,, D and Dr, are supported by bounded sets. The proof for each proposition is given in the supplementary material.
We are interested in finding hyper-generator for each scale, such that, for each image I, g((: would generate samples I; whose patches "f, i, are similar to the patches ur, D within 1, Specifically, we would like to train the parameters of gi to minimize the following function:
E, E,Wc(Pi, Dr,1) = E, max d/ #1.1
{ E d' (uz,)) E (ur,)) 21.3
(16)
where C == Cn {d;
||d|l2 S a²) However, minimize this objective function directly, one needs to be able to hold a different discriminator di for each sample I, which is computationally expensive.
Fortunately, we can think about this expression in a different manner, it can also be written as follows:
E, Ec DI,) = max Er{ E d((u,;; 213
E (((,, W7.
(17)
where the maximum is taken over the set of mappings S from images to parameters 0, of dis- criminators df C². We let S* be a mapping that takes 1 and returns the parameters (T) of the discriminator df == (:;
(I)) (T))
arg max4;ec {Es,. (ug,)) - Eu,, d' ((r,))).
Therefore, if S" can be approximated by a large neural network fī(1) fa(T; (ja)) R S*(7,, then we can approximately solve the maximization in Eq. 17 by parameterizing the discriminator with hypernetwork d, := du; fa(1;0 and training its parameters to solve the maximization in Eq. 17 (approximately). For instance, if S* is continuous function, one can approximate S using a large enough neural network up to any approximation error S [4, 14, 27, 24, 10, 20, 30J. This is summarized in the following proposition.
Proposition 1. Assume that C 3x3xxx compact. Let c > be an approximation error. Let Ii(2,I)= == (e fT;;; fj,)) be a hyper- hyper-generator and C a class of discriminators. Assume that S" is continuous over I. Then, there is large enough neural network få (whose size depends e), such that, the hyper-discriminator di(u, 1) == d;(u; fa))) satisfies:
E, E,We(D, J,a Dr,)) max Ludv (g:, d,) + (1))
(18)
9ia
e c.. A proof for the existence of continuous selector S*(T) has been proposed [6, 25, 26] for sim- ilar settings, and the proof for Prop. 1 is provided as part of the supplementary material. Ac- order to minimize Ey Wc j, Dr,)), we can simply parameterize
where the maximum is taken over the purameterizations Ofa of such that,
cording to this proposition, in
5

---

our discriminator with hypernetwork d; := (u; fa; f1:0 and train the hyper-generator 9: to solve: ming,, maXo,, i. Lado (d; 9i) along with the gradient penalty loss Lipp(d)) ensure that fa(:: (,,)) would have a bounded Lipschitz constant.
Put differently, in order to guarantee that the approximation error in Prop. would be small, we advocate selecting the hypernetwork få to be large neural network. In this case, if we are able to effectively optimize ff and fz to solve ming,, 5a max0,, Lado (di, 9i) (s.t the Lipschitz constant of ensure that E, (e (1,,) is small, desired.
d, is bounded),
4.1 Alternative Architectures
As presented in Sec. 3, we consider two alternative architectures (i) shared discriminator and (ii) shared feature extractor. We briefly describe each proposed variant and its limitations; the full analysis is presented in the supplementary material.
Shared Discriminator In this case, the model has two main components for each scale i: a hyper-
generator gi(2; 7) = 9ı(2; (T;; (T;))),), along with standard discriminator d; du(u) = di (w: 0.) that shared among all samples 7, as illustrated in Fig. (bb). We show that if the expected (w.r.t T ~ D)) distance between the distributions Dj. and D, is small, then the loss function Ledv (g., d,) == E, (E (((,,,))-Ev, ) tends to be small. Here, D; denotes the distribution of i Di,
for 1~D,, D,. This proposition shows that a hyper-generator 9:(:; I) that generates samples I, whose patches are similar to samples of D, would minimize the loss function Lado dz), even though the generated samples are not conditioned on the image Therefore, simply minimizing the adversarial loss would not guarantee that gi(,, : I) generates samples Í, that are similar to I,.
Shared Feature Extractor We note that as strategy for reducing the number of trainable pa- rameters in the whole model, one could restrict fg and fa to share their encoding component e, as illustrated in Fig. I(c). We show two cases in which this approach fails. First, we consider the case where the model is trained using GD. In this case, GD iteratively updates (0e:0f. to minimize
Ladv (gi, d,) and updates (Ge, (p.)) to maximize Cude(Oi: d:) - \1 Lis Informally, we show that 0g is essentially trained to only minimize Ettp(d.) and that each tuple (0e, b, 0%..) with de 0 an equilibrium point. In addition, we note that Liip (d;) minimized by d; O. Therefore, is likely that d; would converge to O during training. This means that at some point the discriminator is ineffective. In particular, if f. O, then (Bz. f%, 87,,) is an equilibrium point. We note that 6. = is not a desirable output of the traiming algorithm, since provides a is independent of the input image L. Second, (0z, 0f to minimize Ladv (ge, dc). oi., to maximize Lade (ga, d,) and (0e, &f to minimize the loss Lup (d.). We show that each tuple (fe, O2 0f with 0g 0 is, again, an equilibrium point.
g=(,, that consider the case where GD iteratively optimizes
Experiments
Our experiments are divided into two parts. In the first part, we study three different training regimes of our method. First, we experiment with aigll-image training, in order to produce a fair comparison with existing methods. Second, we present mini-batch training scheme, where instead of a single image, the model is trained on a fixed set of images. Lastly, we experiment with training over full dataset that cannot fit into a single batch.
In the second part, we experiment with several applications of our method. Specifically, we study the ability of our method in the Harmonization, Editing and Animation tasks proposed by [31], as well as generating samples of arbitrary size and aspect ratio. In addition, we experiment with two new applications: image interpolations, and generation at inference time. These application are unique to multi-image training.
Due to space constraints, we focus on our novel applications, and refer the reader the supplementary material for our full set of applications, as well as technical details, such as specific hyperparameters,
GPU usage and additional experiments.
Throughout the experiments, we consider the following set of baselines: SinGAN [31], ConSin- GAN [13] and HP-VAE-GAR [8]. To evaluate image generation, we use the single-image FID metric

---

Table Quantitative comparison on Places-50/LSUN-55,, showing SIFID, mSIFID, diversity and
training time per image (minutes). Our method shows comparable results to single-imag models in both single and datasel settings, where the overall training time per image significantly lower.
Method
SIFID +
mSIFID +
Diversity
min./image)
SinGAN [31]
0.09/0.11
0.15/0.20
0.52/0.60
60
ConSinGAN [13]
0.06/0.08
0.08/0.13
0.50/0.55
20
HP- VAE-GAN [8]
0.17/0.40
0.27/0.62
0.62/0.77
60
Ours Single
0.03/0.11
0.06/0.19
0.57/0.65
30
Ours Dataset
0.05/0.11
0.07/0.16
0.50/0.48
5
Table 2: Varying the batch size in single mini-
batch training. Both SIFID and diversity (w.rt a specific batch size) remain stable regardless of the size of the mini-batch.
Batch Size
SIFID
mSIFID4
Diversityf
(a)
(b)
(c)
0.03
0.07
0.73
Figure 2: Leakage in the multi-image training when using shared discriminator. (a) and (b) are the two training images, and (c) is generated image for the model of image (b). As can be seen, contains patches from the first image as well.
2
0.04
0.07
0.66
3
0.03
0.07
0.68
0.04
0.08
0.70
5
0.04
0.08
0.71
Table Performance on the Valley dataset, showing SIFID, mSIFID, diversity and training time image (minutes). As can be seen, inference performance increases with training data size.
Train
Test
Dataset
SIFID
MSIFID!
Diversityf
min//image]
SIFID
MSIFID
Diversity
Valley50o
0.04
0.07
0.51
4.0
0.47
2.47
0.34
Valley2500
0.05
0.08
0.52
3.5
0.43
1.86
0.37
Valleys000
0.05
0.08
0.51
3.0
0.41
1.52
0.40
(SIFID) [31]. Following [31], the metric represents the of minimum SIFID over 100 generated samples per image. We further compute the mean-SIFID (mSIFID), which is the mean across all generated samples of all images, without taking the minimum for each image.
As simply overfitting the training image would lead to a SIFID value of 0, diversity measure is necessary. For this purpose employ the diversity measure used in [31]. This measure is computed as the average standard deviation over all pixel values along the channel axis of 150 generated images.
Previous works in the field [32, 31] require training on each image independently. To enable comparison with previous work, we use 50-image dataset of [31], denoted by Places-50 and the 50-image dataset of [12), denoted by LSUN-50. Additionally, whenever quantitative measure available, we present competitive results, and qualitatively our results are at least as good, not better than those of single-imag GANs. For larger datasets of up 5000 images we perform comprehensive experiments with our proposed method.
The dataset presented by SinGAN, Places-50, consists of 50 images selected at random from sub- categories of the Places dataset [38] - Mountains, Hills, Desert and Sky and the dataset presented by ConSinGAN, LSUN-50, consists of five randomly sampled images from each of the ten LSUN dataset categories. order to evaluate our method on larger datasets, we consider three subsets of the Valleys category of the Places dataset; the first 500(V500). 2500(V2500) and 5000(V5000) (the entire category) images, and use the 100-image test set when relevant. Additionally, we consider the first 250(C250) and 500(C500) images of the Churches Outdoor category of the LSUN dataset.

---

5.1 Training Procedures
Single-Image training Our approach is first evaluated when training with single image, as done in previous methods. Since a single function needs be learned, a standard discriminator (i.e. not learned via hypernetwork) is used in this specific case in order to avoid redundant enlargement of the model and speed up training. Similar results are obtained using a hyper-diseriminator Tab. shows that our performance is on par with current single-imag models for this setting.
Single mini-batch training When introduced with multiples images, the standard discriminator, as for the baseline methods, suffers from leakage between the images in the mini-batch, i.e., the patches of the generated images are distributed as patches of arbitrary images from the batch (Sec. 4.1- Shared Discriminator). Fig. 2 illustrates this effect. To overcome this issue, we introduce a hyper- discriminator that enables efficient learning of a different diseriminator model for each image. To evaluate performance on single mini-bater learning, we randomly sampled set of 5 images from the 50-image dataset and trained a different model for each permutation of the set of size 1 S S 5. Tah. show performance is good regardless of the mini- size, which indicates that the hypernetwork model successfully learns a different model for each input image.
Dataset training Our main contribution arises from training with large amount of data. In addition to Places-50 and LSUN-50, we trained our method on three subsets of the Valley category, as presented above - Valleysoo. Valley2500 and Valleysonn. iterating on batches of 16 images, for 25k, 100k. 150k iterations per scale, respectively, on a single GPU. Tab. and 3 show performance and training time per image. Churches250 and Churches were trained in a similar manner for 20k and 30k iterations per scale, respectively, and reached equal performance of 0.20, 0.27 and 0.47 for SIFID, mSIFID and diversity metrics.
As far we aware, our method is the first that is able train multiple single-image models at this scale with a decent runtime. For example, training the model presented by [31] on Valley with single GPU would require the training of 5000 different and independent models, and would take approximately 200 days. Our method takes 10 days on a single GPU, and thus is faster by a factor of 20.
5.2 Applications
As noted above, we present our novel applications in the main text, and refer the reader to the supplementary material for applications presented in previous works.
Interpolation As our meta-learning approach learns the space of generators, and is trained on multiple images, our model is able to interpolate between different images smoothly and at different scales. Unlike common interpolation, the hierarchical structure of the model enables interpolation at different scales as follows: We start by interpolating over the latent representation eg. resulting in new generator. Let A and B be two different images. We compute their latent representations eg = (A)()) and B = (B) (resp.) and perform linear interpolation between the two, for e [0, eg = ae + (1 a)e" resulting in a new generator. We then select primary image, A for example, and initial scale m, and perform the following steps: (i) we use eg to generate scales 1 to m, and (ii) from scale m, we switch to eg. and continue generating scales accordingly. The result is mix at different patch scales, where scale 1 controls the structure of the image, and the last scale controls the finer texture of the image. Fig. 3 shows its interpolation, where the primary image is denoted by A (top-left) and the target image by B (top-right). We show interpolation at three different scales first (1), middle, and last, presented from top to bottom. As can be seen, interpolating at the first scale results in more structural changes, while interpolating at the middle and last scales results in more textural changes. By changing a able to obtain a wide gamut of intermediate options between the two images.
example of pair of images and
Feedforward generation The meta-learning approach, and the fact that our method is able to learn from a relatively large dataset such as Valleysouo, introduce the ability to model new image in one forward pass. Fig. 5 and Tab. 3 show inference results for three different models trained on Valleyson. Valley2500 and Valleys0oo. As can be seen, requires a significantly larger dataset than that of [31] to get the model to generalize. The network trained on Valleysonn enables modeling of a new image in fraction of second, and results in coherent and reasonable generated images compared with previous methods, which were unable to perform this task.
8

---

Figure 3: Interpolation in the space of generative networks. A hypernetwork is trained to produce unique Single-mmage Generators from dataset of 50 images. Top left (right) generated image from generator A (B). Each column represents different mixtures of the generators' latent representations. Each row represents injection of the mixed representation at different scales, where all previous scales use generator A representation from coarsest (top) to finest (bottom).
Real
Ours
SinGAN
ConSinGAN HP SA--AE--AAN
Editing & Harmonization
Real
Injected
Ours
Places-
Lsun- 50
Di
C250
Animation
C500
Figure 4: Left: Comparison of image generation results with single image baselines, different datasets. Right: Results of applications, trained with the Places-50 dataset. Our method allows us to manipulate images such as Editing, Harmonization and Animation at large scale, training all images at once.
Train
Test
Real
[3]]
[12]
[8]
V500
V2500
V5000
Real
V500
V2500
V5000
Figure 5: Training and testing results Valley dataset. Real images from the train/test-set respectively. Training results include SinGAN [31], ConSinGAN [12] and VAE-VAEGAA [8].
Valleysusa
MRI
Retina
Spec.
Monet
Van Gogh
Ingun
Generaled
Figure 6: Feedforward generation with the Valleysoon trained model on unseen images: Left-side image form the same distribution. Right-side images from completely different distributions.

---

Limitations
prominent limitation of the method is the model's size. The hypernetwork approach leads to large projection layers, e.g. convolution layer with weight size of (64, 6,,3,33 and embedding size of lel = 512 will result in a projection layer with a weight size of [6112, 36864) 18M parameters. This obviously affects convergence, runtime and GPU memory usage.
In Fig. 6, we quantitatively explore the out-of-distribution generalization capabilities of our feedfor- ward method when training on the Valleys000 nature image dataset. As can be seen, for images that completely out of domain, the generated images are not faithful to the input image. Training large-scale heterogeneous dataset to further improve generalization requires days of training. Until this experiment performed. is unclear whether the architecture has enough capacity support this ene-mode-ffin fits-all capability.
Related work
Hypernetworks, which were first introduced under this name in [9], are networks that generate the weights of a second primary network, which computes the actual task. Hypernetworks are especially suited for meta-learning tasks, such as few-shot [2] and continual learning tasks [36], due to the showledge-sharing ability of the weight-generating network. Knowledge sharing in hypernetworks was recently used for continual learning by [36].
Predicting the weights instead of performing backpropagation can lead to efficient neural architecture search [3, 37] and hyperparameter selection [23]. In [22], hypernetworks were applied for 3D shape reconstruction from a single image. In [34] hypernetworks were shown to be useful for learning shared image representations. Note that while the name of the method introduced in [28] is reminiscent of our method, it solves different task, with completely different algorithm. Their method does not employ hypernetwork to parameterize their generator (or discriminator): instead, their generator serves as hypermetwork itself. In addition, they intend to learn the distribution of weights of high-performing classifiers on a given classification task, which is a different application.
Several GAN-based approaches were proposed for learning from a single image sample. Deep Image Prior [35] and Deep Internal Learning [33] showed that deep convolutional network can form a useful prior for a single image in the context of denoising, super-resolution, and inpainting. SinGAN [29] uses patch-GAN [15, 29, 18, 19] to model the multiscale internal patch distribution of a single image, thus generating novel samples. ConSinGAN [12] extends SinGAN, improving quality and training time. However, these methods need to trained on each image individually. In this work, we propose a novel approach based on hypernetworks that leverages the capabilities of single-image generation and enables efficient training on datasets of any size, while preserving the unique properties of single-image training.
8 Conclusions
Given the abundance of unlabeled training images, training single image GAN unjustifiable viable multi-image alternatives exist. We present the first such alternative, which also opens the door novel applications that are impossible with existing models, such as interpolation between single-image domains and feedforward modeling. From technical perspective, we present what is to our knowledge the first adversarial hypernetwork. Working with this novel multi-network structure requires understanding of the interplay between the components involved, and we support our method with theoretical analysis.
Acknowledgments
This project has received funding from the European Research Council (ERC) under the Euro- pean Union's 's Horizon 2020 research and innovation programme (grant ERC CoG 725974). The contribution of the first author is part of a Master thesis research conducted at Tel Aviv University.
10

---

References
[1] M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein generative adversarial networks. In
D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Conference on Ma- chine Learning, volume 70 of Proceedings of Machine Learning Research, pages 214-223, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR.
[2] L. Bertinetto, J. F. Henriques, J. Valmadre, Torr, and A. Vedaldi. Learning ferw-forward one-shot learners. In Advances in Neural Information Processing Systems, pages 523-531, 2016.
[3] A. Brock, Lim, J. Ritchie, and N. Weston. SMASH:
One-shot model architecture search through hypernetworks. In International Conference on Learning Representations, 2018.
[4] G. Cybenko. Approximation by superpositions sigmoidal function. Mathematics of Control, Signals and Systems, 2(4))303-314, Dec 1989.
[5] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, pages 1 1126-1135. PMLR, 2017.
[6] T. Galanti and L. Wolf. On the modularity of hypernetworks. In Advances in Neural Information Processing Systems 33, Curran Associates, Inc., 2020.
[7] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville. Improved train- ing of wasserstein gans. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[8] S. Gur, S. Benaim, and L. Wolf. Hicrarchical patch vae-gan: Generating diverse videos from a single sample. arXiv preprint arXiv:2006.1220 12226, 2020.
[9] D. Ha, A. Dai, and Q Le. Hypernetwrrks. arXiv preprint arXiv: 1609.690996 2016.
[10] B. Hanin and M. Sellke. Approximating continuous functions by relu nets of minimal width, 2017.
[11] K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the 2015 IEEE International Confer- ence on Computer Vision (ICCV), ICCV '15, page 1026-1034, USA, 2015. IEFE Computer Society.
[12] T. Hinz, M. Fisher, O. Wang, and S. Wermter. Improved techniques for training single-image gans, 2020.
[13] T. Hinz, M. Fisher, O. Wang, and S. Wermter. Improved techniques for training single-image gans. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pages 1300-1309, 2021.
[14] K. Hornik. Approximation capabilities of multilayer feedforward networks. Neural Networks, 4:251-257, 1991.
[15] Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros. Image-to-image translation with conditional adversarial networks. In CVPR, 2017.
[16] Karras, S. Laine, and Aila. style-based generator architecture for generative adversarial networks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 4401-4410, 2019.
[17] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint 1412 69800, 2014.
[18) C. and M. Wand. Combining markov random fields and convolutional neural networks for image synthesis. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 2479-2486, 2016.
[19] C. Li and M. Wand. Precomputed real-time texture synthesis with markovian generative adversarial networks. In European conference on computer vision, pages 702-716. Springer, 2016.
[20] H. Lin and S. Jegelka. Resnet with one-neuron hidden layers is a universal approximator. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, page 6172-6181, Red Hook, NY, USA, 2018. Curran Associates Inc
11

---

[21] E. Littwin, T. Galanti, and L. Wolf. On the optimization dynamics of wide hypernetworks. arXiv preprint arXiv:2003.1219. 2020.
[22] G. Littwin and L. Wolf. Deep meta functionals for shape representation. In The IEEE Interna- lional Conference on Computer Vision (ICCV), October 2019.
[23] J. Lorraine and D. Duvenaud. Stochastic hyperparameter optimization through hypernetworks. arXiv preprint arXiv: 09919, 2018.
[24] Z. Lu, H. Pu, F. Wang, Z. Hu, and L. Wang. The expressive power of neural networks: A view from the width. In Guyon, V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 6231-
6239. Curran Associates, Inc., 2017.
[25] V. Maiorov, R. Meir, and J. Ratsaby. On the approximation of functional classes equipped with a uniform measure using ridge functions. J. Approx. Theory, 99(1):5-111) July 1999.
[26] V. Maiorov and A. Pinkus. Lower bounds for approximation by mlp neural networks. NEURO- COMPUTING, 25:81-91, 1999.
[27] H. N. Mhaskar. Neural networks for optimal approximation of smooth and analytic functions. Neural Comput., 8(1): 161))1644177, 1996.
[28] N. Ratzlaff and Fuxin. HyperGAN: A generative model for diverse, performant neural networks. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 5361-5369. PMLR, 09-15 Jun 2019.
[29] T. Rott Shaham, T. Dekel, and T. Michaeli. Singan: Learning a generative model from a single natural image. In Computer Vision (ICCV), IEEE International Conference on, 2019.
[30] I. Safran and O. Shamir. Depth-widee width tradeoffs in approximating natural functions with neural networks. In D. Precup and Y. W. Tch, editors, Proceedings of the 34th International Conference on Machine Learming, volume 70 of Proceedings of Machine Learning Research, pages 2979- 2987, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR.
[31] T. R. Shaham, T. Dekel, and T. Michaeli. Singan: Learning generative model from a single natural image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3570-4580, 2019.
[32] A. Shocher, S. Bagon, Isola, and M. Irani. Ingan: Capturing and remapping the" dna" of natural image. arXiv preprint 18Xiv:8812.002311 2018.
[33] A. Shocher, N. Cohen, and M. Irani. "zero-shot" super-resolution using deep internal learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 3118-3126, 2018
[34] V. Sitzmann, J. N. Martel, A. W. Bergman, Lindell, and G. Wetzstein. Implicit neural representations with periodic activation functions. arXiv preprint arXiv:2006.0966 2020.
[35] D. Ulyanov, A. Vedaldi, and V. Lempitsky. Deep image prior. arXiv:1711.1092. 2017.
[36] J. von Oswald, C. Henning, J. Sacramento, and B. F. Grewe. Continual learning with hypernet- works. In International Conference Learning Representations, 2020.
[37] C. Zhang, M. Ren, and R. Urtasun. Graph hypernetworks for neural architecture search. In Int. Conf. on Learning Representations, 2019.
[38] B. Zhou, A. Lapedriza, Xiao, A. Torralba, and A. Oliva. Learning deep features for scene recognition using places database. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
12


Reviews:
[{"review_id": "WJ7Grt9kv6", "reviewer": "Reviewer_FnNF", "summary": "This paper introduces an advanced version of SinGAN by incorporating the hypernetwork into the framework so that the proposed model can perform all the tasks that the original SinGAN can do, but with much faster test inference. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of employing hypernetwork to train both generators and discriminators is very interesting, and the motivation of fast inference is convincing enough for me. \n\n\nMy main concern is on the quality of the results (Figure 3 and 4). I partly agree (but with some conditions) that if there is no shortage of unlabeled images, single-image GAN is unjustifiable. This must go with conditions that the test image performance is better than or at least comparable to the original SinGAN. \n\n\nZooming at Figure 4, however, the quality of the results does not seem to match with SinGAN (even on train data with the model trained on the larger dataset!). For example, the V5000 results does not look crisp nor realistic when looking at the boundaries of the buildings. These qualitative results do not match with the quantitative results shown in Table 1 and Table 3. This needs more explanations.\n\n\nIn Table 3, using a smaller dataset gives better results on training data, and this trend is opposite in the test data. What would be the explanation for the former case? (+ why doesn’t this go along with the qualitative results in Figure 4?)\n\n\nSpeaking of the results, I don’t see the benefit of the availability to interpolate between single image GANs (Figure 3). I understand that the framework allows this to perform and indeed this is new, but cannot see what kinds of advantage it would bring by doing so. This needs more discussion, or reduce the weight it takes in the paper (Currently, it is mentioned with the other major contributions).\n\n\nIn addition, one of the contributions that the authors argue is that they are the first (as far as they know) to perform adversarial training with hypernetworks. In fact, there exists a very similar work that did use hypernetwork for generator and train it using the adversarial loss: “Generative Models as Distributions of Functions”. Although it does not use the hypernetwork scheme for both generator and discriminator, this is noteworthy. However, I do not think this compromises the novelty much since the work is quite recent and has not been published anywhere yet. \n\n\nThe organization of the paper can be also improved. For example, in my personal opinion, it looks better to spare more space in Backgound section for introducing the intuition behind the SinGAN and its characteristics than providing the technical details (like eq. (2), (3), and (4)), which are not important to understand the main idea and context. \n\n\nOverall, the idea is interesting and the advantages of using the framework are clearly seen. However, since the results do not go along with the first two benefits (Line 24 – Line 27), which are very critical, it is hard to give a strong score to the paper. \n\n---\n\nUpdate: Raised score to 6, please see the comment.\n\n---\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No comment", "review_text": "", "extra_scores": {"ethical_concerns": "No comment", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "ASk1sgbS34s", "reviewer": "Reviewer_mf7L", "summary": "The paper aims at improving single-image GAN models by training them on larger datasets. The authors use a meta-learning approach based on the hypernetworks scheme. The model consists of a hyper-generator and a hyper-discriminator. The Hyper-generator network is decomposed into an embedding network that is shared among scales and a linear projection per scale. The hyper-discriminator network is also decomposed into an embedding network and a set of projections. The model is trained with adversarial and reconstruction loss functions. Authors show that the proposed model reduce the training time per image without loss in performance. They also introduce novel\ncapabilities, such as interpolation and feedforward modeling of novel images. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**:\n\n\n- The paper is well-written, and the proposed approach is simple and intuitive. The authors motivate the problem well. \n\n\n- Experiments are thorough and demonstrate efficacy of the proposed approach qualitatively and quantitatively. The authors also define new tasks such as interpolation and feedforward modeling which enables new capacities in single-image GAN models. \n\n\n**Weaknesses**:\n\n\n- The proposed approach employs hyper-networks in SinGAN models, which is an instance of apply X to Y. Considering that hyper-networks are applied to other generative models and various discriminative models, this makes novelty of the proposed approach moderate. There are existing works such as [A] that use hyper-networks in generative models. There are also recent works such as [B, C, D] which also use hyper-networks for generation (but are not published by the submission deadline for this conference). \nHowever, the good results and improvements of the proposed approach can make up for this.    \n\n\n- The authors note the limitation of their work in terms of model size, convergence, runtime and GPU memory usage in section 6, but do not provide further details. It would be good to have comparisons with SinGAN on this.  \n\n\n[A] HCNAF: Hyper-Conditioned Neural Autoregressive Flow and its Application for Probabilistic Occupancy Map Forecasting; Oh et al.; CVPR 2020\n\n\n[B] Adversarial Generation of Continuous Images; Skorokhodov et al.; CVPR 2021\n\n\n[C] pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis; Chan et al.; CVPR 2021\n\n\n[D] HyperVAE: A minimum description length variational hyper-encoding network; Nguyen et al.; 2020\n\n------------------------------------------------------------------------------------------------------------------------------------------------\n\nUpdate after the rebuttal: I read the rebuttal and other reviews. Overall, I think the paper meets the acceptance criteria for the conference. I keep my score, and I hope the authors clarify concerns of the reviewers in the final version of the paper.  \n\n------------------------------------------------------------------------------------------------------------------------------------------------", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "9"}, "extra_text": {}}, {"review_id": "9QVbCWEr_I", "reviewer": "Reviewer_e9qp", "summary": "This paper presents a novel single image generation framework that is able to generalize to multi-sample case, which is different from previous work where each model only works on a single sample, leveraging hyper network. The author has provided solid theoretical analysis to support their design. And the effectiveness is also validated by their produced results.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. The paper is well written and easy to follow.\n2. The author has provided solid theoretical analysis of the proposed method.\n3. The idea of leverage hyper-net to generate the model weights for generator and discriminator is quite novel and intuitive.\n4. The paper has made a meaningful progress to generalize the single-sample GAN models to multi-sample case.\n5. A collection of variants (shared gen/dis hyper) are also studied to validate the effectiveness of the proposed method.\n6. The author has also provided code for reference.\n\nCons:\n1. The author has pointed out the model size might be one of the limitations. I would like to know if there is any failure case existing w.r.t the the generation quality under some circumstances? For example, does the proposed method also works for human face images, etc?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This technique might have a negative societal impact. Some people might be able to leverage this method to produce some fake images based on a given photo. The produced fake images might be harmful.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "6Pjd9v9p1Z5", "reviewer": "Reviewer_1LvZ", "summary": "The paper proposes a method to learn hypernetworks, trained on a larger dataset (500-5000 images), for single-image GANs. These hypernetworks are then used to initialize the GAN for a specific single image to speed-up training time and improve image quality/diversity. The single-image GAN follows the same architecture as SinGAN and the individual generators and discriminators for a specific image are initialized by the trained hypernetwork and then fine-tuned for the specific image. The resulting single-image GANs outperform related work (SinGAN, ConSinGAN, ...) based on SIFID, diversity, and training time.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of using a larger dataset to speed up the training of single-image GANs on a single image is promising. Using hypernetworks and meta-learning may be one suitable approach for this. The method in this paper seems to deliver promising results while only needing 5 minutes of fine-tuning for a given image. \n\nIt would be more interesting to see how well this approach performs on more complex images and how much it benefits from large-scale training in this case. SinGAN-like approaches are not good at modeling complex structures/objects and it seems to me that using hypernetworks would be most helpful in these cases. Landscapes do not contain many complex structures so it would be better to test this approach on other datasets, e.g. LSUN churches or something similar.\n\nAlso, is it necessary to train an image encoder as you do given that there are many pre-trained image encoders already available? Have you tried using a pretrained (frozen) image encoder as conditioning for your hypernetworks? \n\nWhile I get that your approach supports applications that are not supported by traditional single-image GANs (e.g. image interpolation and mixing) I think it would still be beneficial to compare against at least some of the original applications directly in the main paper (unconditional synthesis, harmonization, editing, retargeting, ...). At the moment this is all hidden in the supplementary which makes it difficult to directly qualitatively compare your approach to the baselines.\n\nI believe some (most) of section 2 (2.1 can be summarized in one paragraph like 2.2) can be moved to the supplemental to free up more space to run more experiments and evaluations. Similarlay, section 3 can be shortened (e.g. equations 9-11 are a repetition of equations 6-8).\n\nOverall, I believe this paper would benefit from a more thorough analysis of the approach on different datasets (that also contain more complex structures than landscapes), more direct comparisons to the baselines on some applications (generation, harmonization, editing, ...), and possibly running a user study to get direct comparisons between the different models besides SIFID/diversity (both of which have been shown to have weaknesses).\n\n***\nUpdate: Raised score to 6, see comment.\n***", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors address limitations and societal impacts.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Meta-review of "Meta Internal Learning"

This paper proposes a framework for single image generation from the perspective of meta-learning on larger datasets. The method uses hypernetworks as the generator and discriminator so that the weights of these networks can be conditionally adapted from the single image from a set of projections, and can be trained with adversarial and reconstruction losses. They discuss capabilities of the method such as interpolation and modeling of novel images.

Most reviewers agree that this paper is well written, easy to follow, includes solid analysis of the proposed methods, and also generally agree with the novelty. The fast inference can find important uses in real-world applications. The weaknesses are highlighted by the authors and discussed in detail with reviewers, and the authors have included several comparisons to previous work. As suggested by reviewer FnNF (who had increased their score), the authors are strongly encouraged to better reorganize the manuscript in the camera ready version, in which the authors agreed to do so.

For all these reasons stated, I recommend acceptance of this work.

Decision: Accept (Poster)