Augmentation-Interpolative AutoEncoders for Unsupervised Few-Shot Image Generation
Authors: Davis Wertheimer
Omid Poursaeed
Bharath Hariharan
Venue:  (2021)
Keywords: Interpolation
autoencoder
reconstruction
few-shot learning
few-shot image generation
generalization
augmentation

Abstract: We aim to build image generation models that generalize to new domains from few examples. To this end, we first investigate the generalization properties of classic image generators, and discover that autoencoders generalize extremely well to new domains, even when trained on highly constrained data. We leverage this insight to produce a robust, unsupervised few-shot image generation algorithm, and introduce a novel training procedure based on recovering an image from data augmentations. Our Augmentation-Interpolative AutoEncoders synthesize realistic images of novel objects from only a few reference images, and outperform both prior interpolative models and supervised few-shot image generators. Our procedure is simple and lightweight, generalizes broadly, and requires no category labels or other supervision during training.

Under review as a conference paper at ICLR 2021
AUGMENTATION INTERPOLATIVE AUTOENCODERS FOR UNSUPERVISED FEW-SHOT IMAGE GENERATION
Anonymous authors Paper under double-blind review
ABSTRACT
We aim to build image generation models that generalize to new domains from few examples. To this end, we first investigate the generalization properties of clas- sic image generators, and discover that autoencoders generalize extremely well to new domains, even when trained on highly constrained data. We leverage this in- sight to produce robust, unsupervised few-shot image generation algorithm, and introduce a novel training procedure based on recovering an image from data aug- mentations. Our Augmentaaionnllterpllatii Interpolative AutoEncoders synthesize realistic images novel objects from only a few reference images, and outperform both prior interpolative models and supervised few-shot image generators. Our pro- cedure is simple and lightweight, generalizes broadly, and requires no category labels other supervision during training.
INTRODUCTION
Modern generative models can synthesize high-quality (Karras et al. 2019; Razavi al. 2019
Zhang et al. 2018a), diverse (Ghosh et al. 2018: Mao et al. 2019: Razavi et al. 2019). and high- resolution (Brock et al., 2018; Karras al. 2017 2019) images of any class, but only given a large training dataset for these classes (Creswell et al. 2017). This requirement of large dataset is impractical in many scenarios. For example, an artist might want use image generation help create concept art of futuristic vehicles. Smartphone users may wish to animate a collection of selfies, or researchers training an image classifier might wish to generate augmented data for rare classes. These and other applications will require generative models capable of synthesizing images from large, gver-growing set of object classes. We cannot rely on having hundreds of labeled images for all of them. Furthermore, most of them will likely be unknown at the time of training.
We therefore need generative models that can train on one set of image classes, and then generalize to new class using only a small quantity of new images: few-shot image generation. Unfortu- nately, we find that the latest and greatest generative models cannol even represent novel classes in their latent space, let alone generate them demand (Figure I). Perhaps because of this gener- alization challenge, recent attempts at few-shot image generation rely on undesirable assumptions and compromises. They need impractically large labeled datasets of hundreds of classes (Edwards) & Storkey| (0066). involve substantial computation at test time (Clouâtre & Demers 2019), or are highly domain-specific, generalizing only across very similar classes (Jitkrittum et al. 2019).
In this paper, we introduce strong, efficient, unsupervised baseline for few-shot image generation that avoids all the above compromises. We leverage the finding that although the latent spaces of powerful generative models, such as VAES and GANs, do not generalize to new classes, the rep- resentations learned by autoencoders (AEs) generalize extremely well. The AEs can then be con- verted into generative models by training them to interpolate between seed images (Sainburg et al.) 2018 Berthelot et al. 2018, Beckham al.) 2019). These Interpolative AutoEncoders (IntAEs) would seem a natural fit for few-shot image generation. Unfortunately, we also find that although IntAEs can reproduce images from novel classes, the ability to interpolate between them breaks down upon leaving the training domain. To remedy this, we introduce a new training method based on data augmentation, which produces smooth, meaningful interpolations in novel domains. We demonstrate three different settings (handwritten characters, faces and general objects) that our Augmentation ciimennttioo-Interplaaatii Autoencoder (AugIntAE) achieves simple, robust, highly general, and completely unsupervised few-shot image generation.

---

Under review as a conference paper at ICLR 2021
Training Data
Novel Data
Source
PGAN
Ours
Source
PSAN
Ours
Figure 1: A basic image reconstruction task. One can either use a state-of-the-art PGAN (Karras et al., 2017). optimizing the latent code to match the generation to the input, use our simpler Ang.ntAE. Both succeed on the training set of adult faces (left), but on the novel domain of baby faces (right), the best PGAN reconstruction ages the baby. Ours is far more faithful.
2 RELATED WORK
2.1 GENERATIVE MODELING
AEs were originally intended for learned non-linear data compression, which could then be used for downstream tasks; the generator network was discarded (Kramer 1991 Hinton & Salakhutdi- nov 2006 Masci et al. 2011). VAEs do the opposite: by training the latent space toward a prior distribution, the encoder network can be discarded at test time instead. New images sampled directly from the prior (Kingma & Welling] 2013).. Subsequent models discard the encoder net- work entirely. GANs sample from noise distribution and learn generate images which fool a concurrently-trained real/fake image discriminator (Goodfellow et al,, 2014). [Bojanowski et al.) (0177) and [ooshen al. (2019) treat latent codes learnable parameters directly, and train sepa- rate sampling procedures for synthesizing the novel images.
Recent work has also seen a return to AEs conditional generators, by training reconstruction networks to interpolate smoothly between pairs or sets of seed images. This is accomplished by combining the reconstruction loss on seed images with an adversarial loss on the seed and interpo- lated images. Different forms of adversarial loss (Sainburg et al.) 2018 Berthelot et al., 2018) and interpolation (Beckham et al. 2019) have been proposed.
While all of these approaches generate new images, it is unclear if any of them can generalize to novel domains. Some results suggest the opposite: a VAE sufficiently powerful to model the training data becomes incapable of producing anything else (Bozkurt et al. (018).
2.2 FEW-SHOT IMAGE GENERATION
Current attempts at few-shot image generation span a wide range of approaches and models. Neural Statistician, an early attempt, similar to the AE in that is built for few-shot classification, and largely discards the generative capability (Edwards & Storkey 2016)) Generation-oriented iterations exist, but likewise depend on large, varied, labelled dataset for training (Hewitt al. 2018). Other approaches based on few-shot classification include generative matching networks (Bartunov) & Vetrov 2018) and adversarial meta-learnngg (Clouâtre Demers) 2019). These models also depend on heavy supervision, and are fairly complicated, involving multiple networks and training procedures working in tandem- making them potentially difficult to train in practice reliably.
Separate work has approached few-shot image generation from the side of generative modeling. Wang et al. (2018), Noguchi & Harada (2019) and Wu et al. (2018) investigate the ability of GANs to handle domain adaptation via fine-tuning thus requiring substantial computation, and more novel class examples than are available in the few-shot setting. Zhao et al. (2020) train GANs directly from few examples, though still more than are at hand for few-shot learning, and can be considered orthogonal work, as AugIntAE can serve as a useful pre- pre-trained initialization, Antoniou et al. (2017) and Liu et al. (2019) use adversarial training produce feed-forward few-shot generators. However,
2

---

Under review as a conference paper at ICLR 2021
Source:
0
2
3
4
5
6
7
0
2
3
4
5
6 7
Reconstruction:
01
0
2
3
4
5
6 6
7
0
/
233
4
5 7
Sampled images:
0
4
4
J
I
7
3
7
9
3
0
S 3
A
13
(
D
E
H 6 G
f
A
13
D
E 3
GH
Source (novel):
Reconstruction (encoder):
A
0
0
0
8
4
2
6.4
Reconstruction (oracle):
A
6
C
O
8 H
6 0
6
O
8
Figure 2: A VAE (left) and WGAN-GP (right) trained on MNIST (top row) successfully recover training images (row 2) and synthesize new ones (row 3). They struggle to represent images from novel classes (EMNIST letters, rows and 5), even when latent codes are optimized directly, as an oracle encoder (row 6). VAE reconstructions use the mean embedding directly and do not sample in latent space. Best viewed digitally.
both models still depend varied, labelled training data, and risk exhibiting the same problems as standard GANs: mode collapse and hyperparameter sensitivity (Arora al. 2017 2018).
Jitkrittum et al. (2019) introduce an algorithm for class-conditioning an unconditioned generative model. New images are produced by matching latent space batch statistics to real images from single, possibly novel class. Nguyen et al. (2017) learn individual latent codes from pretrained discriminator, while Wang et al. (2020) train latent sampling network. These approaches have little evaluation novel classes, and to what degree they generalize depends entirely the pretrained image generator. They may also require substantial test-time computation. In contrast, AugIntAEs are lightweight, train robustly, and generalize broadly from completely unlabelled data.
PROBLEM SETUP
Let X be a large, unlabelled collection of images depicting objects from set of classes C. Let X' be a very small of images as few as two belonging novel class c' of C. Our goal is train network on X which, given X', generates images clearly belonging to c. We refer to this as the network's ability to generalize to new domains (note that this usage is distinet from "generalizing" to novel images in the same domain, a much simpler task). We cannot directly adapt the network to X' using SGD, as X' contains too few images to prevent overfitting.
This is extremely difficult problem, since it is unclear if neural network trained to model the data distribution X can even represent images from different distribution, let alone sample from it. Therefore, we first examine whether existing generative models can faithfully encode novel class images in latent space. We train a VAE and a WGAN-GP (Gulrajani et al.] 2017) on MNIST handwritten digits (LeCun al. 1998), well as in Bau et al. (2019) (details in appendix). We then evaluate the ability of each generative model to recover particular images. Using the built-in VAE encoder and the WGAN-GP inversion network, we find that while both models can reconstruct training images (Fig. 2 top), the same approach fails images from novel classes in this case, EMNIST handwritten letters (Cohen al. 2017). The outputs do not much resemble the inputs; crucial semantic information is lost (Fig, 2 bottom). To discount the possibility of sub-optimal encoders, we simulate an oracle encoder, refining the latent code parameters for each image directly via SGD. These reconstructions are not much better. Fig. D demonstratte similar failure in large, state-of-the- pretrained GAN. This confirms prior findings (Bozkurt et al. 2018) that current generative approaches by default cannot images from novel classes. Generating new novel class images is simply out of the question.
encoder network that inverts the WGAN-GP
represent
Why do sophisticated generative models fail to generalize? We argue this is largely by design. Generative models such as VAEs and GANs are trained to minimize the divergence between prior distribution and learned posterior distribution, where sampling. VAEs push the learned latent posterior toward a Gaussian prior, while GANs map samples from the prior posterior distribution in image space. In both cases, latent vectors repeatedly
both approximated by repeated
3

---

Under review as a conference paper at ICLR 2021
Source:
A
C D 8 C E
E
F F
4
x
0
Reconstruction:
A
d
4
H
Nearest train image:
4
3
O
0 8 6
8
T
b
8 6600
Figure 3: Randomly chosen AE image reconstructions for unseen classes. From left to right: MNIST generalizes to EMNIST, Omniglot train generalizes to Omniglot test, and CIFAR-IO generalizes to CIFAR-100, but MNIST does not generalize trivially to white noise. Bottom row: nearest (in pixel space) training image is shown for comparison. Best viewed digitally.
Table 1: Average AE per-pixel L2 reconstruction error (normalized to [0, 255]). Rows give the training dataset, columns give the evaluation dataset, and cells measure generalization from row to column. Generally, enror is low. The bottom row contains scores for a VAE, which generalizes poorly from MNIST. Implementation details in appendix.
Omni
Omni
MNIST
EMNIST
(train)
(test)
images:
60,000
124,800
19,280
13, 180
CIFAR-10
CIFAR-TOD
MNIST
2.69
13.48
9.83
11.58
# images:
50,000
50,000
EMNIST
6,59
3,89
8.31
9.79
CIFAR-10
7.58
9.09
Omniglot (train)
14.74
21.47
2.13
8.30
CIFAR-100
9.04
7.98
Omniglot (test)
15.01
21.44
8.04
2.32
MNIST (VAE)
5.92
23.10
15.98
17.91
sampled and sent through the generator. Thus, by the time the generator is trained convergence, and the posterior approaches the prior (or vice-versa), every region of the latent space feasible under the prior will have been mapped at some point to a training image or, in the case of GANs, to an image indistinguishable from training image. This means that a properly trained VAE or GAN cannot construct reconstruct new object classes. it could, then would have been able sample such images during training which would mean it had not been properly trained at all.
4 AUGMENTATION-IITERPPO..N
AUTOENCODERS
As discussed above, minimizing the divergence between prior and posterior training distributions ensures good image synthesis, but poor generalization. The opposite could also hold: AEs do not enforce any latent distribution formally, given a network that maps an image x to latent vector 2, and a generator G mapping back to image space, refer to the function composition G(E(-)) the autoencoder. E and G are trained jointly over X to minimize the pixel reconstruction error between & E X and G(E(x)). The question of generalization becomes, what degree does trained AE maintain close proximity between x' and G(E(z') for x' which lies far from the data manifold of X?
AutoEncoders:
the data posterior, and might generalize well. More
By this measure, we find that our conjecture holds: AEs generalize surprisingly well. Examples are given in Fig. 3 demonstrating near-perfect generalization performance over three pairs of class- disjoint datasets: MNIST digits EMNIST letters, Omniglot training alphabets Omniglot testing alphabets (aake et al. 2015). and CIFAR- 10 to CIFAR-100 (Krizhevsky et al. 2009). We also quantitatively evaluate generalization (in terms of reconstruction error) between all the above pairs, as well between Omniglot and MNIST/EMNIST,, which includes domain shifts, e.g., stroke width (Table [] Reconstruction quality is high across the board, especially given that the MNIST and CIFAR- 10 networks learn only ten distinct classes! AEs exhibit very little overfitting the training domain, learning a general mapping despite heavy class constraints.
It is possible that this generalization is a result of AES simply learning identity function. Fortu- nately, this is not the case: AEs learn clear image priors. We find that our trained AEs are much more effective encoding real images than noise (see Fig. right). We also find that low- fow-frequency noise an analysis is provided appendix.
noise is encoded more faithfully than The learned AE mapping, while general, is also nontrivial.
4

---

Under review as a conference paper at ICLR 2021
Sample:
A 4
PD)
E
Z1
P1
Ludi
[0,1] 1] P2
4
P2)
E
Liyccon
A
app (1 a)p:)
4
Figure 4: Training for AugIntAE. For every training image, two sets of augmentation parameters (rotation, translation, etc.) are sampled, plus mixing coefficient a. The network must recover the mixed transformation image from the same mixture in latent space. An auxiliary adversarial discriminator improves quality.
The fact that AEs generalize suggests they are capable of acting as few-shot image generators for novel classes, given method for sampling appropriate latent vectors. One possibility is to interpolate between data points in latent space: every sampled point weighted sum of two more real points. This allows us to produce novel combinations of seed images without changing the semantic content. Unfortunately.
Interpolative AutoEncoders:
a known fact that AEs do not interpolate well, shown in Fig. [5] row 2 (Berthelot et al. 2018). Prior work (Sainburg al. 2018 Berthelot et al.] 20188] Beckham et al. 2019) addresses this by applying an adversarial loss to the interpolated images, which works well in the training domain. However, find the Interpolative AutoEncoder (IntAH) approach overly restrictive for our purposes: constrains all interpolations between arbitrary image pairs the training domain. For example, produce recognizable digits when interpolating between a 3 and 4, a semantically unintuitive result. This makes the learning process harder and causes the model learn interpolations that do not generalize. When it interpolates between letters (Fig. 5 row 3, left), we find that it does produce undesirable artifacts that look like numbers!
MNIST, IntAE must
AutoEncoders: We introduce novel training procedure for IntAEs remove such artifacts while maintaining generalizability. Instead optimizing interpolated im- ages using only GAN loss, we train the network to directly recover known, semantically interpo- lated images from the corresponding interpolated latent code. This accomplishes two things simul- tancously: first, we only interpolate between images where the interpolation makes semantic sense, since we must know the interpolated image in advance. This simplifies the learning problem sig- nificantly. Second, the model is longer constrained the training manifold when interpolating arbitrary training image pairs. The network can now learn simpler, more direct interpolations that work well on both training and novel domains.
Augmentation-Interen
Formally, suppose we have triplet of images A= f(1), B f(p2) and C = f(ap1 (1-a)p2)
where f is some unknown image generating process, P is semantic variable, and [0, 1]. Using this triplet. we train the interpolative AE to reconstruct C' by decoding the interpolated latent code of A and B. Formally, we train the encoder and the generator G by minimizing:
Lyecon ||C G(aE(A) + +1 (1-a))EBB) a(B)))
In practice, finding appropriate image triplets A, B, C' a dataset of independent images is difficult. Instead, we synthesize A, B, using affine spatial transformations, and color jitter for 3-channel im- ages. Given training image we randomly sample two sets of augmentation parameters (translation, rotation, hue, etc.). Applying each of these transformations independently to æ yields and (for example, a 10" rotation and a -5 rotation). We then sample weight weighted average of the two transformations, which we apply to x to produce (in our example, 3+ C represents 5 rotation). The Augmentaaion-Ineerpolaaw AutoEncoder (AugIntAE) is then trained to recover C' from the a-weighted interpolation between the latent embeddings for A and B. This corresponds to Equation
[0. 1] and compute
We can also augment the model with the original IntAE losses: a reconstruction loss on and B, and GAN loss Ludv the interpolated C. In practice, found that the former did not notice- ably affect performance, while the latter was helpful in reducing the blurriness of output images. Subsequent models include Tudde- The full procedure is displayed in Fig.
5

---

Under review as a conference paper at ICLR 2021
Table 2: FID score and train/test classification emor across models and datasets. Misclassification rate is measured in percentage points and captures likelihood of belonging to the wrong domain (training data). AugIntAE is superior in all contexts. GANs trained on train/test images and evalu- ated on test images are given as baselines/oracles.
WGAN-GP (train)
WGAN-GP
FID (1)
AE
IntAE
AugIntAE
(test)
MNIST + EMNIST
25.0
16.7
21.5
12.3
8.3
Omniglot (train test)
86.9
87.1
86.7
82.5
80.0
CelebA (male - female)
86.9
26.1
24.0
21.1
15,4
CIFAR-10 CIFAR-ION
54.3
53.8
52.1
47.9
42.0
WGAN-GP
WGAN-GP
Misclassification rate (4)
AE
IntAE
AugIntAE
(train)
(test)
MNIST + EMNIST
96.5
4.1
2.7
2.4
0.5
Omniglot (train - test)
78.5
9.4
10.3
7.9
24.8
CelebA (male -+ female)
89.6
10.6
29.3
8.9
8.3
CIFAR-10 + CIFAR-100
72.9
18.5
19.8
13.7
15.3
At first glance, learning the space of affine and color transformations does not appear particularly helpful for an IntAE. Very few visual relationships in the real world can be captured by these trans- formations alone. However, we find that these learned interpolations act as a powerful regularizer on the latent space, allowing AugIntAE to smoothly capture far more interesting and difficult trans- formations well, such as shape, lighting, and even 3D pose.
Few-shot generation: Once the AugIntAE is trained, we can sample novel images given only a set of seeds. Simply select random pair of images, find their latent space embeddings, sample E [0, 1]. and generate the image from the a-weighted mixture embeddings. More sophisticated sampling techniques possible, but left to future work.
5 EXPERIMENTS
All encoder/generator networks use standard 4- to 6-layer GAN architectures. We employ shallow networks to illustrate that AugIntAE itself, not network power, responsible for good performance.
We use two baseline models. The first is an AE trained without interpolation but with the auxiliary GAN loss. We use an LSGAN (Mao al., 2017) for the diseriminator, The second baseline an IntAF representing prior work: seed images are reconstructed while interpolations are trained via GAN. The GAN loss is also applied to the seed image reconstructions. The choice LSGAN discriminator means that IntAE captures two prior models: it is an LSGAN instantiation of Beck- ham et al.] (2019), and also a version of Berthelot et al. (2018) with discretized labels. We use the same data augmentation in all models and parameters large possible without introducing significant visual artifacts. Training and evaluation details for all experiments are in the appendix.
5.1 QUANTITATIVE RESULTS
We report quantitative scores for all results in Table We examine four train/test dataset pairs: two handwritten character settings and two natural image settings. For each pair we report two metries: FID score, which captures overall image quality, and test set classification rate, which captures the degree of generalization and semantic faithfulness the target domain. For the latter metric, we train separate classifier on the combined train and test datasets to distinguish training images from testing images. Generators that generalize well to the test domain should have higher rates of test-set classification, while those that do not generalize will produce images skewed toward the training domain. On both metrics we find that AugIntAE is superior to AE and IntAE baselines in all settings. We also include standard GAN models trained solely the training/testing datasets as baseline/oracle models, respectively. Our FID scores generally approach but do not exceed the oracle scores. We examine individual dataset results in detail.

---

Under review as a conference paper at ICLR 2021
Fade:
R
3
3
3
AE:
R
R
3
3
3
IntAE:
R
3
3
3
3
AugIntAE:
N
3
3
3
Figure 5: Examples of interpolation between handwritten character pairs from novel classes (EM- NIST left, Omniglot test right). AugIntAE interpolates smoothly, preserves the desired shape, and unlike the AE/InAAE baselines, produces minimal artifacts even when transformations are non-affine (far left). Best viewed digitally.
Seeds
Statistician
DAGAN
AE
IntAE
AugIntAE
C
C
9 2 5 9 8
0
- 9
1 6
5 S O
8 5
CCCCC
CCCEC
C CCCCO
5
N
0
6 5 2 3 5 5
5
7
0 0004
MUUNM
MAnnA
/ S S e e e c + T r r
e C e S S e
S S
5 5
5
5
8 5
S SSSSS
8
3 0
8 8
CEECe
C 7 P +
9
9
5
4 5
8 2
7 + L ++ Brrrr
5
5
6 8
G 0
r Hrrr
distinct set of 5 seed images (left). Neural Statistician and DAGAN overfit to the training data. The AE successfully generates letters but introduces visual artifacts. AugIntAE more effectively removes these artifacts than the baseline IntAE. Results from AE-based models are midpoint interpolations between seed pairs. Best viewed digitally.
Figure 6: Comparison few-shot generators. Each row corresponds
5.2 HANDWRITTEN CHARACTERS
Image quality: We evaluate the performance of AugIntAE on two handwritten character dataset pairs. One set of models trains on MNIST digits and evaluates on EMNIST letters, while the other transfers from the Omniglot training alphabets to the testing alphabets.
The autoencoders in both cases use the 4-layer encoder/generator of InfoGAN (Chen al. 2016). with latent dimensionality
reduced to 32, in keeping with prior IntAE work (Beckham et al.
2019 Berthelot et al., 2018).
Fig. shows example interpolations for these two contexts. AugIntAE produces better interpola- tions and removes visual artifacts, particularly discontinuities, present in the AE and IntAE images. These qualitative results are verified quantitatively in Table AugIntAE, as measured by FID score (Heusel et al., 2017), outperforms all baselines. We also measure the semantic faithfulness of the interpolations to the test seed images, by training a separate classifier to distinguish training data from testing data. AugIntAE images classified correctly more often than any baseline. These results hold not just for handwritten characters, but across all our testing regimes. In terms of both image quality and semantic fidelity, AugIntAEs effective few-shot generators.
We compare AugIntAE as a few-shot generator to two additional baselines: Neu- ral Statistician (Edwards Storkey 2016) and DAGAN (Antoniou et al. 2017). Both approaches require class labels on the training data, while ours does not. We train both models on MNIST and then attempt synthesize new images from EMNIST. Fig. makes clear that AugIntAEs generalize more broadly and are much less restricted by the narrow training domain. Neural Statistician and DAGAN overfit to the training classes, and generate number images instead of the desired letters.
Generalizability:
As a practical use case, we utilize AugIntAE for data augmentation. We train a series of classifiers on the letters in the "ByMerge" split of EMNIST (lower- and upper- case separated), each using different interpolation strategy data augmentation. Half of each training batch is augmented data, produced by interpolation between same-class images with the labels preserved and = .5. The model without augmentation use the same batch size but train for twice as long. All autoencoders used for interpolation are trained on MNIST. All augmentations provide gains, as shown in Table Table 3 including pixel-level interpolation, constrained form of MixUp (Zhang et al. (0118)). However, the largest gain is obtained by interpolating in the latent space of AugIntAE.
Data hallucination:
7

---

Under review as a conference paper at ICLR 2021
Table 3: Interpolation form of data augmentation EMNIST. Results averaged ten
runs, with 95% confidence intervals. AugIntAE provides the most effective augmentation.
Augmentation Accuracy
None 90.33 + 066 90.66 + 004
Mixup
AE
AugintAE 91.50 = .05
91.27 = 004
Seeds
None (AE)
Rotation
Translation
Scale
Skew
All (AuglinAAe)
S
S
S
S
S
S
3
S S
S
S
S S
S
S
S
S
S
S
5
5 S
S
5
S
S S S S
L H
/
/
L
L
a H
H m [
H
H
H
H
H
H
H Ph
H
H
m HA I
m 3
Pm
M
PM
m
DA T
m P
m
PM
m
m
M
PI
3.
7
T
T
7
T
7 TTT
T
W
N
N
N
N
F2
N
N
NINN N
INILNA A
N N
N
N
N
N
Figure 7: Ablation study on forms of data augmentation. Each row corresponds a distinct set of seed images (left); results are midpoint interpolations between seed pairs. Each individual form of augmentation improves over the AE baseline (fewer artifacts), but none approach the performance of using them all together (right). See, for example, the "S" in the top-right corner of cach plot. Best viewed digitally.
Ablation: Affine augmentation encompasses a range of independent transformations (rotation, translation, scale, skew) and so it is worth examining to what degree each is necessary. We train four MNIST AugIntAE models using each independent transformation as the sole augmentation technique. Sample outputs comparison. We find that interpolative training using each form of augmentation improves over the AE baseline, but the same time no individual augmentation approaches the performance of the full AugIntAE: the different augmentations act synergistically.
given in Fig. 7 along with AE and full AugIntAE baselines for
5.3 CELEB-A
We extend our results the more difficult domain of higher-resolution natural images. Our models train on the male faces from Celeb-A, scaled to 128 1 128 resolution, and generalize to female faces. Network architectures follow DCGAN (Radford et al. 2016) with an additional half-width layer (at the bottom/top of the encoder/generatoor respectively) to handle the increased resolution. Latent dimensionality 512, in keeping with prior IntAEs (Sainburg et al. 2018 Beckham et al. 2019).
The results, displayed in Fig. 8 and Table are similar to our findings for handwritten characters. AEs generalize well, but produce visual artifacts during interpolation: unrealistic. transparent re- gions around the head. IntAE produces high-saturation color artifacts. Compared to these baselines, AugIntAR removes the artifacts and restores semantic meaning to the interpolation path, even when the interpolation is non-affine (as in Fig. ())
One might wonder if interpolative sampling acts as a constraint on generated image variety. Note that the number of unique image pairs grows quadratically with the number of seed images, so that
Fade:
AE:
IntAE:
AugintAE:
Figure 8: CelebA interpolations involving simultaneous pose and lighting shifts. AE and IntAE darken the nose unrealistically (left) produce transparent artifacts (right), while AugIntAE accu- rately navigates the lighting/pose shift. Best viewed digitally.
8

---

Under review as a conference paper at ICLR 2021
Figure 9: Six seed images (left) are sufficient to produce a variety novel images (right). The fifteen displayed here represent the midpoints of the fifteen unique pairwise interpolation paths. We re-emphasize that this network was trained only on male faces. Best viewed digitally.
Fade:
IntAE:
AugintAE:
Figure 10: Examples of interpolation between image pairs from novel classes (CIFAR-100), AugIn- tAE interpolates smoothly with minimal transparency and color artifacts, Examples are hand-picke picced so that interpolation is possible. Best viewed digitally.
even with just six seeds AugIntAE can produce broad variety (Fig. 9 We conclude that AugIntAE is effective on higher-resolution color images, not just handwritten characters.
5.4 CIFAR
Finally, we evaluate our model in an extremely challenging setting: unconstrained natural images, and classes with large intra-class variation. The latter property is especially challenging for inter- polation based models (c.g., how to interpolate between a real bear and teddy bear?). We train our models on CIFAR-10 and evaluate on novel CIFAR-I00 classes. Network architecture uses DC- GAN with 512 latent dimensions. Fig. [0 displays example outputs from AE, IntAE, and AugIntAE on novel CIFAR- 100 data, and quantitative results are in Table We conclude that AugIntAE the most effective interpolative few-shot generator for natural images, though much work remains for more difficult cases (see supplementary).
6 CONCLUSION
We introduce a powerful, lightweight, and label-free method for few-shot image generation. Build- ing on the generalizability of AEs, we introduce a novel training procedure that leads higher quality interpolations. The resulting AugIntAEs are robust, generalize far more broadly than prior few-shot generators, and are practically useful for downstream tasks.
REFERENCES
Antreas Antoniou, Amos Storkey, and Harrison Edwards. Data augmentation generative adversarial networks. arXiv preprint 171 2017.

---

Under review as a conference paper at ICLR 2021
Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang. Generalization and equilibrium in generative adversarial nets (gans). In Proceedings of the 34th International Conference on Machine Learning- Gearning-Volume Volume 70, pp. 224-232. JMLR. org. 2017.
Sanjeev Arora, Andrej Risteski, and Yi Zhang. Do gans learn the distribution? some theory and empirics. In 6th International Conference on Learming Representations, ICLR 2018, 2018.
Sergey Bartunov and Dmitry Vetrov. Few-shot generative modelling with generative matching net- works. In International Conference on Artificial Intelligence and Statistics, pp. 670-678, 2018.
David Bau, Jun-Yan Yan Zhu, Jonas Wulff, William Peebles, Hendrik Strobelt, Bolei Zhou, and Antonio Torralba. Sceing what gan cannot generate. In Proceedings of the International Conference Computer Vision (ICCV), 2019.
Christopher Beckham, Sina Honari, Vikas Verma, Alex M Lamb, Farnoosh Ghadiri, R Devon Hjelm, Yoshua Bengio, and Chris Pal. On adversarial mixup resynthesis. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 4346-4357. Curran Associates, Inc., 2019.
David Berthelot, Colin Raffel, Aurko Roy, and Ian Goodfellow. Understanding and improving interpolation in autoencoders via an adversarial regularizer. arXiv preprint arXiv: 807.07553, 2018.
Piotr Bojanowski, Armand Joulin, David Lopez-Paz, and Arthur Szlam. Optimizing the latent space of generative networks. arXiv preprint arXivv1707.05776, 2017.
Alican Bozkurt, Babak Esmaeili, Dana Brooks, Jennifer Dy, and Jan lan-Willem Meent. Can vaes generate novel examples? arXiv preprini 4rXiv:/812.006624 12 2018.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint 18Xiv: 1009, 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pp. 2172-2180, 2016.
Louis Clouâtre and Mare Demers. Figr: Few-shot image generation with reptile. arXiv preprint 1rXiv:1901.02199, 2019.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and André van Schaik. Emnist: an extension of mnist to handwritten letters. arXiv preprint arXiv: (70..0:372..3333, 2017.
Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil Bharath. Generative adversarial networks: An overview. JEEE Signal Processing Magazine, 35, 10 2017.
Harrison Edwards and Amos Storkey. arXiv:1606.02168 185, 2016.
Towards
neural statistician.
arXiv preprint
Arnab Ghosh, Vivcka Kulharia, Vinay P Namboodiri, Philip HS Torr, and Puneet K Dokania. Multi- agent diverse generative adversarial networks. In Proceedings of the IEEE Conference Com- puter Vision and Pattern Recognition, pp. 8513-8521, 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of wasserstein gans. Advances neural information processing systems, pp- 5779, 2017.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Advances in Neural Information Processing Systems, pp. 6626-6637, 2017.
10

---

Under review as a conference paper at ICLR 2021
Luke B Hewitt, Maxwell I Nye, Andreea Gane, Tommi Jaakkola, and Joshua B Tenenbaum. The variational homoencoder: Learning to learn high capacity generative models from few examples. arXiv preprint arXiv: 0rXiv:1807008911 2018.
Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 3 313(5786):504-507 2006.
Yedid Hoshen, Ke Li, and Jitendra Malik. Non-adversarial image synthesis with generative la- tent nearest neighbors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.
2019.
Wittawat Jitkrittum, Patsorn Sangkloy, Muhammad Waleed Gondal, Amit Raj. James Hays, and Bernhard Schölkopf. Kernll mean matching for content addressability of GANs. In Proceedings of the 36th International Conference on Machine Learning, pp. 3140-3151, 2019.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. arXiv preprint 1rXiv: 10196, 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4401-4410, 2019.
Diederik P Kingma and Max Welling. 1rXivv1312.611 14, 2013.
Auto-encoding variational bayes. arXiv preprint
Mark A Kramer. Nonlinear principal component analysis using autoassociative neural networks AIChE journal, 37(2):233-243, 1991.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350(6266) 1332-1333. 2015.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2278- 2324, 1998.
Ming-Y Yu Liu, Xun Huang, Arun Mallya, Tero Karras, Timo Aila, Jaakko Lehtinen, and Jan Kautz. Few-shot unsupervised image-to-iman translation. arXiv preprint 1r05::1905.01723 2019.
Qi Mao, Hsin- Ying Lee, Hung-Yu Yu Tseng, Siwei Ma, and Ming- Hsuan Yang. Mode seeking genera- tive adversarial networks for diverse image synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4429-143, 2019.
Xudong Mao, Qing Li. Haoran Xie, Raymond Y.K. Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE International Confer- ence on Computer Vision (ICCV), Oct 2017.
Jonathan Masci, Ueli Meier, Dan Cireşan, and Jürgen Schmidhuber. Stacked convolutional auto- encoders for hicrarchical feature extraction. In International Conference on Artificial Neural Networks, pp. 52-59. Springer, 2011.
Anh Nguyen, Jeff Clune, Yoshua Bengio, Alexey Dosovitskiy, and Jason Yosinski. Plug & play generative networks: Conditional iterative generation of images in latent space. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4467-4477, 2017.
Atsuhiro Noguchi and Tatsuya Harada. Image generation from small datasets via batch statistics adaptation. arXiv preprint arXiv: 1904.01794417744, 2019.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, 2016.
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with va-vae-2. arXiv preprint arXiv: aurXiv:/906.00 19060000444 2019.

---

Under review as a conference paper at ICLR 2021
Tim Sainburg, Marvin Thielk, Brad Theilman, Benjamin Miglioni, and Timothy Gentner. Gener- ative adversarial interpolative autoencoding: adversarial training on latent space interpolations encourage convex latent distributions. arXiv preprint arXiv:1807.066537 2018.
Konstantin Shmelkov, Cordelia Schmid, and Karteek Alahari. How good is my gan? In Proceedings of the European Conference on Computer Vision (ECCV), pp. 213-229, 2018.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818 1818-2826, 2016.
Yaxing Wang, Chenshen Wu, Luis Herranz, Joost de Weijer, Abel Gonzalez-Garcia, and Bog- dan Raducanu. Transferring gans: generating images from limited data. In Proceedings of the European Conference Computer Vision (ECCV), pp. 218 234, 2018.
Yaxing Wang, Abel Gonzalez-Garcia, David Berga, Luis Herranz, Fahad Shahbaz Khan, and Joost van de Weijer. Minegan: Effective knowledge transfer from gans to target domains with few images. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Chenshen Wu, Luis Herranz, Xialei Liu, Joost van de Weijer, Bogdan Raducanu, et al. Memory replay gans: Learning to generate new categories without forgetting. Advances In Neural Information Processing Systems, pp. 5962-5972, 2018.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. arXiv preprint
2018a.
Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empiri- risk minimization. In 6th International Conference on Learning Representations, ICLR 2018.
OpenReview.net, 2018b.
Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training. In Advances in neural information processing systems, 2020.
12

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 ARCHITECTURES AND TRAINING
Architectures: Nearly all network backbones follow either InfoGAN, if operating on handwritten characters, or DCGAN, if operating on natural images. Celeb-A networks include an additional half- width layer the bottom of the encoder, and at the top of the generator, to account for increased image resolution. For convenience, we use the same network architecture for both AugIntAE en- coders and auxiliary GAN discriminators. The only difference that the discriminator has one output neuron in the final layer, while encoders have as many there are latent dimensions.
We Lzznnrrmlliz our latent features and perform interpolation via spherical linear interpolation (SLERP). Latent dimensionality is set to 32 for handwritten character models, which, given their 28 x 28 resolution, amounts to an almost 25-fold reduction in dimensionality. Celeb-A dimension- ality is set to 512, with 128 x 128 resolution and three color channels, producing reduction in dimensionality of almost two orders of magnitude. CIFAR images use the standard DCGAN reso- lution of 64 64 and 512 latent dimensions for almost 25-fold dimensionality reduction, similar to the handwritten character models.
VAłs, WGAN-GPs, and classifiers all the same architectures, plus necessary modifications the number of input/output neurons. WGAN-GPs draw from the same latent dimensionality as the corresponding AE models, and receive La-normalized noise samples rather than samples from standard Gaussian. VAE encoders have output dimensionality twice the above, as they predict both mean a and variance y for each coordinate. Classifiers have as many output neurons as there are classes in the given task (2 for train/test dataset classifiers, 37 for EMNIST letter classifiers) and end softmax layer.
Training: Unless stated otherwise, all networks trained on real images use the Adam optimizer with initial step size .001. Models train for 100 passes over the given dataset. Models with no adversarial component cut the learning rate by factor of 10 at epochs 35 and 70.
chose these hyperparameters without a validation set, using convergence on the training data as our sole criterion.
GAN training: To stabilize learning, adversarial models do not change their learning rate. We also rescale all adversarial gradients so that their magnitudes match those of the reconstruction loss gradient. WGAN-GPs train for twice as long as other models, but update the generator only once per five discriminator updates.
On some datasets, we found that the auxiliary LSGAN discriminator could collapse and allow the generator to "win" the adversarial game. To prevent this, introduce scaling factor & [0, 1] that updates dynamically based on discriminator performance. Specifically:
LAE = Lrecon + ks r Ladv
(2)
where
Tyccon
(3)
V Lade lg
calculated per-image, and is adjusted with each gradient update according to the following rules:
ty =
(4)
I = kt - .001 x (1 (D((reat)) D(
(5)
k+41 = maxx0, min(1, ())
(6)
This update scheme for k ensures that whenever the scores coming from the discriminator D for real and fake images are separated by less than 1 on average, decreases. The generator then focuses more the reconstruction task and becomes less competitive, allowing the discriminator network
13

---

Under review as a conference paper at ICLR 2021
Input 3 512 512 RGB image
4 4 conv. 32 BN ELU stride 2
1 X 1 conv. 64 BN ELU stride 2
4 conv. 128 ELU stride
4 4 conv. 256 BN ELU stride
1 X 1 conv. 512 BN ELU stride
4 4 conv. 1024 BN ELU stride 2
FC 512
Table 4: Inversion network architecture for PGAN.
to "catch up" until the margin of separation is greater than again. At that point A: increases back to 1, at which point the reconstruction and adversarial losses are equally weighted once more.
Augmentation Parameters: We sample data augmentation parameters p uniformly over predefined ranges of values. For handwritten character datasets, we sample rotations in the range [-20, 20). translations in the range [-4, 4), scaling in the range [.8, 1.2]. and shear in the range [-6, 6]. These values were chosen heuristically: we picked the largest values that would not occlude or obscure the character.
Natural image AugIntAEs sample from half the range of rotation and skew, and double the range of translation (though because of the higher resolution, this comes overall). These values were chosen so as not to introduce large regions of empty space into the image, while staying as large as possible. CIFAR images are symmetric-padded:: Celeb-A images are center cropped and do not require padding. Additionally, we sample a range of color jitter pa- rameters for AugIntAEs handling 3-channel RGB images. We sample wiighness/contrasssstuarrtt adjustments from the range [.75, 1.25] and hue from the range [-5%, +5%]. We again chose these values be as large possible without producing unrealistic or unparsable images.
much smaller displacement
Similar to Sainburg et al. (2018), we found that the network learned interpolations near the seed images more easily than near the midpoint of the interpolation path. We therefore biased our sampled toward the midpoint by sampling the mean of two uniform random draws.
To ensure that our baselines are trained on the same data distribution as our AugIntAEs, we use the a- a-weighted interpolated image as the training image, even when the network does not use interpola- tive training. This only applies to AE and IntAE models.
A.2 GAN INVERSION
Obtaining generalization results for GAN involves inverting the generator, which we attempt via a combination of a learned inversion network and direct SGD optimization on the latent codes for each target image. For the PGAN (fig. we use publicly available pretrained model provided by Facebook Research"] For the MNIST/EMMIST generalization experiments (fig. 2) we use our own implementations, constructed and trained using the procedure described above. The learned inver- sion network for the PGAN generator uses the architecture described in Table while the inversion network for the MNIST WGAN-GP uses the same encoder as other experiments. Both networks trained by sampling images from the generator, and attempt to reconstruct the corresponding latent code for each image using mean squared error. We use SGD with an initial step size of .0001 and momentum equal to .1. We train for 6400 iterations and cut the learning rate by factor of 10 after every 1280 iterations.
The subsequent stage of inversion involves direct refinement of the latent codes provided by the inversion network via SGD in latent space. In both cases we use 1000 iterations SGD, with a learning rate of .01 and momentum. The loss is an L1 pixel reconstruction loss, resulting in the final images displayed in figures and 2 of the main paper.
http:///yyoocchnnn///niiccchhers pytorch-gan-zoo pgan
14

---

Under review as a conference paper at ICLR 2021
Pixel-wise wise L2 error
0.4
celeba cifar
mnist omniglot
0.2
0.0
2
10
14
18
22
26
2
6
10
14
18
22
26
Frequency (initial noise resolution)
Figure 11: L2 pixel reconstruction error as function of noise frequency, j0. 1]-scaled. Dotted lines represent reconstruction loss on novel domain images (EMNIST for the MNIST model, CIFAR-100 for the CIFAR-10 10 model, etc.
A.3 QUANTITATIVE EVALUATION
Fréchet Inception Distance (FID) (Heseel al., 2017) compares the distributions embeddings of real (p)(z)) and generated (p, (z) images. Both these distributions modeled as multi-
dimensional Gaussians parameterized by their respective mean and covariance. The distance mea- sure is defined between the two Gaussian distributions as:
d((m,, Cr), (mg. Cg)) = ||m, m,,
m,, + Tr(C, CG C ((C,,,C)
(7)
where (m, Cr) and mg: C₂) denote the butions respectively (Shmelkov et al. 2018). We use port of the official implementation of FID PyTorct The default pool3 layer of the Inception-v3 network (Szegedy al. 2016) is used to extract activations. We use 5k generated images to compute the FID scores, and sample 3 uniformly distributed points along each sampled interpolation path. To ensure domain fidelity generated images, we use binary classifier to distinguish whether the images come from the train distribution or the test set. Classifiers have the same architecture as other experiments, as described Section A.I.
and covariance of real and generated image distri-
A.4 NOISE RECONSTRUCTION
We analyze the ability of a trained AE to reconstruct uniform pixel noise of different frequencies. We simulate noise frequency by sampling noise at small resolutions and scaling up the resulting map to the desired image resolution (28 28 for handwritten characters, 64 x 64 or 128 x 128 for 3- channel color images). Fig. I plots the ability of trained ABs to reproduce noise patterns over given frequencies, averaged over 1000 trials. Networks have a clear low-frequency bias though interest- ingly, the handwritten character datasets reach their minimum reconstruction error at a frequency level of 6-8, possible manifestation of learned bias toward penstroke thicknesses certain certain-sized pockets of negative space associated with handwritten characters. Most tellingly, reconstruction error for novel images (dotted lines) is significantly lower than for noise of any frequency for hand- written character models, and most frequencies for natural image models. This suggests clearly that the network has learned particular image distribution that is reflected by uniform noise an image prior.
also worth noting in what ways the AEs fail when reconstructing noise. Figs and show reconstruction attempts for random sampled noise at the given frequencies. It is clear that the hand- written character models struggle to abandon strong, learned penstroke prior. Natural image net- works are better at encoding noise, but also demonstrate clear failure mode at high frequencies where they extract low-contrast, lower-frequency patterns and ignore the higher-frequency input. The Celeb-A model attempts to compensate for this by adding high-frequency ripple artifacts, visi- ble also at some lower frequencies, probabbly reflecting learned hair prior.
15

---

Under review as a conference paper at ICLR 2021
5
97
2
8
12
14
16
18
20
22
24
26
Frequency (initial noise resolution)
Attempted reconstructions of single-channel uniform noise. First row is noise, second is the MNIST model, third the Omniglot model. Both models clearly struggle to abandon learned penstroke prior.
Figure 12:
2
8
10
12
14
16
18
20
22
24
26
Frequency (initial noise resolution)
Figure 13: Attempted reconstructions of three-channel uniform noise. First row is noise, second the Celeb-A model, third is the CIFAR-10 model. noise frequency climbs, the networks progressively abandon the input signal and attempt to extract lower-frequency pattern.
16

---

Under review as a conference paper at ICLR 2021
6
6
6
6
0
6
6 0
6
6
1
1
1
0
0
0
0
0
O
0
0
0
9
9
9
4
9
9
9
6
9
9 4 0 5
8
8
8
8
8
8
8
8
88
8
4
4
4
4
4
4
4
4
4 0 5
5
5
5
5
5
5
5
5
5
6
0
0
0
0
0
0
0
0
7
9
9
9
7
9
9
9
9
9
5
5 2
5
5
5
5
5
7
7
7
7
7
7
7
7
7 7
2
2
6
1
2
3-
2
2
Figure 14: Black digits on white background are seed images, and white digits on black background novel synthesized images. Neural statistician (left) and DAGAN (right) produce the desired behavior on training classes. Thus failures on novel classes clearly represent a failure to generalize.
A.5 FEW-SHOT GENERATION BASELINES
We reeiimllement DAGAN using the architectures from other experiments, with hyperparameters the same as for WGAN-GP training. The sole difference is that the critic network now takes in a pair of images, the number of input channels is 2 instead of 1. We implement the critic network as a WGAN-GP. Fig. 14sshows that the network converges nicely on the training data: it successfully produces novel images from the same class as the initial seed images.
Neural statistician uses a much more complex network architecture, and involves many additional hyperparameters, making direct re-implementation difficult. Instead we run the publicly available codo as-is, keeping all hyperparameters the same. We run the omniglot experiment, and simply replace the omniglot training dataset with MNIST. Fig. [1] shows that like DAGAN, the network converges nicely and produces the desired behavior on the training data. The failure of these ap- proaches on EMNIST thus truly a failure of generalization.
A.6 ADDITIONAL ILLUSTRATIVE EXAMPLES
Selected sets of interpolated image pairs from each of our four dataset regimes, demonstrating that AugIntAE. performs smooth and intuitive interpolations where AEs and IntAFs produce artifacts. Images organized in the paper, with rows in each cell correspond pixel fade, AE, IntAE, and AugIntAE. These are followed by four sets of randomly chosen pairs illustrating average-case performance, again one for each of four dataset regimes. Begins on following page.
17

---

Under review as a conference paper at ICLR 2021
Figure 15: Illustrative EMNIST pairs. Each image pair contains artifacts in the AE/IntAE models that AugIntAE is able to avoid.
18

---

Under review as a conference paper at ICLR 2021
C4
8
8
8
8
8
8
30
5
8
8
8
h
H
2
8
8
8
8
8
8
8
H
h
H
2
8
2
2
R
8
8
8
k
h
H
h
8
8
8
8
8
2
g
H
h
h
h
k
k
H
Y
Y
2
2
2
2
193
23
2
2
2
2
2
R
R
2
2
2
2
7
21
22
Figure 16: Illustrative Omniglot pairs. Each image pair contains artifacts in the AB/IntAE models that AugIntAE is able to avoid.
19

---

Under review as a conference paper at ICLR 2021
Figure 17: Illustrative Celeb-A pairs. Each image pair contains artifacts in the AE/IntAE models that AugIntAE is able to avoid. AEs generally produce transparency and silhouette artifacts around the hairline, mouth, and chin, while IntAEs produce nonsmooth interpolations, unrealistic head contours, and/or color artifacts. We note that AugIntAB is prone smoothing away fine details, sometimes to a greater degree than AE.
20

---

Under review as a conference paper at ICLR 2021
Figure 18: Illustrative CIFAR-100 pairs. AEs are largely indistinguishable from pixel fade, while IntAEs produce color and texture artifacts. AugIntAE does not always successfully preserve seman- tic content through the interpolation, but the interpolation at least usually makes sense. Of particular note are the apples on the left-hand side: rather than fade away the stem, AugIntAE model retracts it into the apple! However, again note that the more sensible AugIntAE interpolation comes at the cost of smoothing away fine details.
21

---

Under review as a conference paper at ICLR 2021
5
5
 5
SSsssssssssee
 S
S
S
m 9
Figure 19: Random EMNIST pairs. AugIntAE is superior in some cases, and gives roughly equal performance in the remainder.
22

---

Under review as a conference paper at ICLR 2021
H
C
H
K
K
X
H
X
M
*
H
15
H
Y
65
G
45
45
4
X
8
8
8
3
3
3
3
8
3
2
86
35
4
4
6
2
6
S
3
8
4
4
*
*
9
9
3
3
*
9x
9
9
9
&
&
4
A
&
Figure 20: Random Omniglot pairs. AugIntAE successfully avoids many visual artifacts, but also
tends to oversimplify complex shapes.
23

---

Under review as a conference paper at ICLR 2021
Figure 21: Random Celeb-A pairs. Pixel fade is actually reasonably strong baseline on Celeb-A because of how the images are aligned, so in many cases the models all arrive at the same reasonably good interpolation.
24

---

Under review as a conference paper at ICLR 2021
Figure 22: Random CIFAR-100 pairs. There clear room for future work some cases the AugIntAE output is unacceptably blurry, and the AugIntAE interpolation frequently fails to preserve semantic content.
25


Reviews:
[{"review_id": "xGePHk3XOWe", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The general idea of the paper is interesting: when using an AE one can use the constraint that \"interpolated images\" should also correspond to \"interpolated latent codes\". While the idea is interesting, the experimental results are not really that compelling. \n\nThe proposed approach in section 4 is an interesting and well motivated extension to Interpolative AEs. The idea is quite simple and the proposed setting to use synthesized and augmented images for the above mentioned interpolation constraint seems interesting to explore. \n\nThe main weakness of the paper are the experimental results in my view. \n\nWhile qualitatively (and potentially hand-picked) examples seem to show that the proposed approach is working well, the experiments are not sufficient to convince me as a reviewer about the power of the approach. Let me be more specific\n\n- In general, quantitative results are rare and thus it is close to impossible to assess the performance of the proposed method. In essence mostly qualitative results are shown that are obviously anecdotal only. An exception is table 2, where FID and an error is shown. While I understand the FID score, I am lacking comparisons to FID scores for these models trained to the \"same\" domain. Otherwise it is unclear how good the numbers really are. Also, the error numbers where not entirely clear to me what they correspond to.\n\n- An important ingredient and component of the approach is the way the synthesized images are obtained via augmentation. While the reader get a vague idea about what kind of augmentation is used, there is no experiment that shows which kind of augmentation is necessary and which kind of augmentation will break the system. In fact, without such an \"ablation-type\" experiment the paper is not particularly insightful. To me some experiments around this essential component the paper is incomplete and should not be accepted. \n\n- Finally, somewhat linked to the previous comment, the paper does not really show failure modes (with the somewhat too obvious failure mode given in fig 3 right) to understand the limitations of the proposed method\n\n\nSo overall the proposed method is interesting and simple (which is positive) - but the experimental results are not convincing and complete enough to justify acceptance at ICLR. \n\nUpdate after the rebuttal:\n\nThanks for the responses. Given that the other reviewers also raise serious issues I will stick with my initial rating. The paper seems not to be ready for publication at ICLR", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZRmhj5MhwtC", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "PROS:\n\n1. The work is well motivated. The topic about doing augmentation for few shot learning has potential value in real application.\n2. The authors made interesting discovery on the generalization of AE.\n3. The authors also made a good point in utilizing AE to do augmentation for few shot learning \n\nCONS:\n\nThe contributions of this work are a bit weak and not enough for ICLR. The work needs to be polished more seriously and it is not ready for publishing. The assumption/settings about the method is not clear/correct, some statements are too strong, the implementation is not clear, the settings for the experiments are omitted. Detailed concerns are listed as follows. \n\n1. What’s the relationship between the large available X and the few shot X’? Given a few shot X’, will it work if we use an arbitrarily selected large X? if no, dose there have any constrain for choosing a proper X, should be seriously analyzed. \n2. The definition of a models’ generalization in this paper seems in contrary to the existing literatures (where the VAE are claimed to generalize better than AE), a discussion on this topic is important in understanding the contribution of this work.\n3. How to obtain the reconstruction for VAE method in Figure2? Since there is a sampling process in obtaining the latent code, is the latent code obtained via z=mu(x)+sigma(x)*eps, eps~N(0,I)? If yes, what about the reconstruction with z=mu(x)? \n4. I am curious about the range of the value for latent code learned by AE, also what's the difference of the latent space between AE and AugIntAE?  \n5. The WGAN-GP on CIFAR100 as shown in Table 2 seems not trained well, by referring to [1], the FID can be as small as 15.6, however the reported performance here is 54.3, which is much higher than that in previous literature.  In section 5, the authors used a shallow network for illustration, which makes it difficult for the readers to evaluate the reliability of the experiments, it is easier if an existing model is employed, i.e. WGAN-GP.\n6. The training process for the method is very cryptic, since two datasets are mentioned in this work (one in large scale and another with few shot), which one is the case used in this work, trained jointly on these two datasets, pretrained on the large scale one and transfer to the few shot case, or only trained on the large scale dataset?\n7. How many images are used for the few shot dataset?\n8. I doubt the statement on few-shot generation is correct, once the AugIntAE is trained on some dataset, can we really apply it to “any” set of seeds (images)? This statement is really strong and the experiments didn’t support this statement properly, i.e. the model is trained  and tested on relatively related datasets.\n9. The classification error in Table 2 is confusing, e.g. how to obtain this value?\n10. Table 3 is not referred.\n11. About Table 3. How many real images (except the augmentations) are used to train the classifier? Since the augmentation is obtained from two randomly selected images, how to define the label for this augmentation? e.g. what is the label for the augmentation obtained from 3 and 4?\n12. A quantitative evaluation for the generated images in experiments of Fig.8 and Fig.9 is necessary, e.g. FID or Kernel Maximum Mean Discrepancy (KMMD). What’s the performance when compared to some recent GAN based method[2]? what’s the pros and cons by comparison? \n\n\n[1] Shmelkov K, Schmid C, Alahari K. How good is my GAN?[C]//Proceedings of the European Conference on Computer Vision (ECCV). 2018: 213-229.\n[2] Zhao S, Liu Z, Lin J, et al. Differentiable augmentation for data-efficient gan training[J]. arXiv preprint arXiv:2006.10738, 2020.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Xc0Hj7Ed1i", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#######################################\n\nSummary: \n\nThe paper investigates the generative model which generalizes to new domain with limited samples.  Authors firstly explore the current hot generative models:  VAEs and GANs, and experimentally find that both VAEs and GANs fail to learn a model which generalizes  well to novel domain.  Interestingly, AutoEncoders exhibits effective performance of the generalizability to new domain. With the encouraging insight, authors further approach Augmentation-Interpolative AutoEncoders. Specially,  the paper firstly augments the training sample to get the  input pair, and extracts the latent feature by the sharing Encoder. A weighted sum of both features is conducted to form the mixed feature, which further is taken as input for the decoder to synthesize the output sample. Differently   the paper performs the reconstruction loss between the output and the mixed input which sum the input pair with the same from to the one of the latent space.    \n\n#######################################\n\nPros:\n\n+To my background, mixing the latent space from the  input pair is new.\n\n+Unlike most papers which explore VAES and GANs, this paper focuses on AEs, and experimentally find its generalizability to new domain.\n\n+This paper is well-written, and easy to follow. The organization is clear, which firstly introduces the drawback of current hot generative models, then indicates the insight that AEs is able to generalize well to novel domain, and finally propose augmentation-Interpolative AutoEncoders.  \n\n+The framework is clear to introduce the proposed method.\n\n+Authors show quantitative and qualitative results to support the proposed method which generalizes well to new domain. \n\n#######################################\n\nCons:\n\n-The idea is little simple, and lack of enough novel. To my best knowledge, the data augmentation is able to increase the generalizability of model, and reduce the domain gap between the train and the test. In this paper, authors indicate AEs has good generalizability, which is interesting, but the proposed method is simple. In fact, for complex dataset, the visualized result is not convincing. \n\n-I am thinking the proposed techniques push the decoder remember the sum of input features.   The same form of both mixing  feature and input pair makes the decoder just remember which feature is from the input and which one form the augmented input.\n\n-Figure 1 is not convincing. Current paper[1] has shown great result on styleGAN, even target domain is from far domaon.  why authors utilize PGAN?\n\n-In this paper, Omniglot train and test are used to evaluate the generalizability. I am wondering why authors consider train and test which extremely close. Please collect me if I am wrong.\n\n-In part of experiment, authors leverage the baseline (Neural Statistician and DAGAN ) which is old. Are there latest paper?\n  \n\n[1] Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space? ICCV2019", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This meta-review is written after considering the reviews, the authors’ responses, the discussion, and the paper itself.

The paper proposes a training scheme for autoencoders, involving data augmentation and interpolation, that results in autoencoders for which interpolations in the latent space lead to meaningful interpolations in the image space. The paper notices that this property carries over reasonably well to datasets different from the training one.

The reviewers point out that the idea is interesting (R1, R2, R4) and simple (R2), but the experiments are substandard (R2, R4) and presentation is at times suboptimal (R1). Overall consensus is towards rejection. Authors addressed some of the concerns in their responses, but failed to convince the reviewers to change their evaluations.

I agree with the reviewers and recommend rejection at this point. The idea is indeed interesting and could be publishable if presented and evaluated well, but in the current manuscript the presentation is at times unclear or somewhat misleading (e.g. presenting the method as a general image generation method, not an interpolation method) and the experiments are reasonable, but not quite convincing, mainly because the architectures and the baselines are outdated (as also pointed out by R1 and R4). I encourage the authors to further improve the paper and resubmit to a different venue. 

Decision: Reject