Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The general idea of the paper is interesting: when using an AE one can use the constraint that "interpolated images" should also correspond to "interpolated latent codes". While the idea is interesting, the experimental results are not really that compelling. 

The proposed approach in section 4 is an interesting and well motivated extension to Interpolative AEs. The idea is quite simple and the proposed setting to use synthesized and augmented images for the above mentioned interpolation constraint seems interesting to explore. 

The main weakness of the paper are the experimental results in my view. 

While qualitatively (and potentially hand-picked) examples seem to show that the proposed approach is working well, the experiments are not sufficient to convince me as a reviewer about the power of the approach. Let me be more specific

- In general, quantitative results are rare and thus it is close to impossible to assess the performance of the proposed method. In essence mostly qualitative results are shown that are obviously anecdotal only. An exception is table 2, where FID and an error is shown. While I understand the FID score, I am lacking comparisons to FID scores for these models trained to the "same" domain. Otherwise it is unclear how good the numbers really are. Also, the error numbers where not entirely clear to me what they correspond to.

- An important ingredient and component of the approach is the way the synthesized images are obtained via augmentation. While the reader get a vague idea about what kind of augmentation is used, there is no experiment that shows which kind of augmentation is necessary and which kind of augmentation will break the system. In fact, without such an "ablation-type" experiment the paper is not particularly insightful. To me some experiments around this essential component the paper is incomplete and should not be accepted. 

- Finally, somewhat linked to the previous comment, the paper does not really show failure modes (with the somewhat too obvious failure mode given in fig 3 right) to understand the limitations of the proposed method


So overall the proposed method is interesting and simple (which is positive) - but the experimental results are not convincing and complete enough to justify acceptance at ICLR. 

Update after the rebuttal:

Thanks for the responses. Given that the other reviewers also raise serious issues I will stick with my initial rating. The paper seems not to be ready for publication at ICLR
review_title: Interesting and simple (which is positive) method - but experiments are lacking

final_score: rating: 4: Ok but not good enough - rejection
review_content: PROS:

1. The work is well motivated. The topic about doing augmentation for few shot learning has potential value in real application.
2. The authors made interesting discovery on the generalization of AE.
3. The authors also made a good point in utilizing AE to do augmentation for few shot learning 

CONS:

The contributions of this work are a bit weak and not enough for ICLR. The work needs to be polished more seriously and it is not ready for publishing. The assumption/settings about the method is not clear/correct, some statements are too strong, the implementation is not clear, the settings for the experiments are omitted. Detailed concerns are listed as follows. 

1. What’s the relationship between the large available X and the few shot X’? Given a few shot X’, will it work if we use an arbitrarily selected large X? if no, dose there have any constrain for choosing a proper X, should be seriously analyzed. 
2. The definition of a models’ generalization in this paper seems in contrary to the existing literatures (where the VAE are claimed to generalize better than AE), a discussion on this topic is important in understanding the contribution of this work.
3. How to obtain the reconstruction for VAE method in Figure2? Since there is a sampling process in obtaining the latent code, is the latent code obtained via z=mu(x)+sigma(x)*eps, eps~N(0,I)? If yes, what about the reconstruction with z=mu(x)? 
4. I am curious about the range of the value for latent code learned by AE, also what's the difference of the latent space between AE and AugIntAE?  
5. The WGAN-GP on CIFAR100 as shown in Table 2 seems not trained well, by referring to [1], the FID can be as small as 15.6, however the reported performance here is 54.3, which is much higher than that in previous literature.  In section 5, the authors used a shallow network for illustration, which makes it difficult for the readers to evaluate the reliability of the experiments, it is easier if an existing model is employed, i.e. WGAN-GP.
6. The training process for the method is very cryptic, since two datasets are mentioned in this work (one in large scale and another with few shot), which one is the case used in this work, trained jointly on these two datasets, pretrained on the large scale one and transfer to the few shot case, or only trained on the large scale dataset?
7. How many images are used for the few shot dataset?
8. I doubt the statement on few-shot generation is correct, once the AugIntAE is trained on some dataset, can we really apply it to “any” set of seeds (images)? This statement is really strong and the experiments didn’t support this statement properly, i.e. the model is trained  and tested on relatively related datasets.
9. The classification error in Table 2 is confusing, e.g. how to obtain this value?
10. Table 3 is not referred.
11. About Table 3. How many real images (except the augmentations) are used to train the classifier? Since the augmentation is obtained from two randomly selected images, how to define the label for this augmentation? e.g. what is the label for the augmentation obtained from 3 and 4?
12. A quantitative evaluation for the generated images in experiments of Fig.8 and Fig.9 is necessary, e.g. FID or Kernel Maximum Mean Discrepancy (KMMD). What’s the performance when compared to some recent GAN based method[2]? what’s the pros and cons by comparison? 


[1] Shmelkov K, Schmid C, Alahari K. How good is my GAN?[C]//Proceedings of the European Conference on Computer Vision (ECCV). 2018: 213-229.
[2] Zhao S, Liu Z, Lin J, et al. Differentiable augmentation for data-efficient gan training[J]. arXiv preprint arXiv:2006.10738, 2020.

review_title: Image generation for dataset with few examples

final_score: rating: 4: Ok but not good enough - rejection
review_content: #######################################

Summary: 

The paper investigates the generative model which generalizes to new domain with limited samples.  Authors firstly explore the current hot generative models:  VAEs and GANs, and experimentally find that both VAEs and GANs fail to learn a model which generalizes  well to novel domain.  Interestingly, AutoEncoders exhibits effective performance of the generalizability to new domain. With the encouraging insight, authors further approach Augmentation-Interpolative AutoEncoders. Specially,  the paper firstly augments the training sample to get the  input pair, and extracts the latent feature by the sharing Encoder. A weighted sum of both features is conducted to form the mixed feature, which further is taken as input for the decoder to synthesize the output sample. Differently   the paper performs the reconstruction loss between the output and the mixed input which sum the input pair with the same from to the one of the latent space.    

#######################################

Pros:

+To my background, mixing the latent space from the  input pair is new.

+Unlike most papers which explore VAES and GANs, this paper focuses on AEs, and experimentally find its generalizability to new domain.

+This paper is well-written, and easy to follow. The organization is clear, which firstly introduces the drawback of current hot generative models, then indicates the insight that AEs is able to generalize well to novel domain, and finally propose augmentation-Interpolative AutoEncoders.  

+The framework is clear to introduce the proposed method.

+Authors show quantitative and qualitative results to support the proposed method which generalizes well to new domain. 

#######################################

Cons:

-The idea is little simple, and lack of enough novel. To my best knowledge, the data augmentation is able to increase the generalizability of model, and reduce the domain gap between the train and the test. In this paper, authors indicate AEs has good generalizability, which is interesting, but the proposed method is simple. In fact, for complex dataset, the visualized result is not convincing. 

-I am thinking the proposed techniques push the decoder remember the sum of input features.   The same form of both mixing  feature and input pair makes the decoder just remember which feature is from the input and which one form the augmented input.

-Figure 1 is not convincing. Current paper[1] has shown great result on styleGAN, even target domain is from far domaon.  why authors utilize PGAN?

-In this paper, Omniglot train and test are used to evaluate the generalizability. I am wondering why authors consider train and test which extremely close. Please collect me if I am wrong.

-In part of experiment, authors leverage the baseline (Neural Statistician and DAGAN ) which is old. Are there latest paper?
  

[1] Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space? ICCV2019
review_title: Augmentation-Interpolative AutoEncoders for Unsupervised Few-Shot Image Generation

Meta-review: This meta-review is written after considering the reviews, the authors’ responses, the discussion, and the paper itself.

The paper proposes a training scheme for autoencoders, involving data augmentation and interpolation, that results in autoencoders for which interpolations in the latent space lead to meaningful interpolations in the image space. The paper notices that this property carries over reasonably well to datasets different from the training one.

The reviewers point out that the idea is interesting (R1, R2, R4) and simple (R2), but the experiments are substandard (R2, R4) and presentation is at times suboptimal (R1). Overall consensus is towards rejection. Authors addressed some of the concerns in their responses, but failed to convince the reviewers to change their evaluations.

I agree with the reviewers and recommend rejection at this point. The idea is indeed interesting and could be publishable if presented and evaluated well, but in the current manuscript the presentation is at times unclear or somewhat misleading (e.g. presenting the method as a general image generation method, not an interpolation method) and the experiments are reasonable, but not quite convincing, mainly because the architectures and the baselines are outdated (as also pointed out by R1 and R4). I encourage the authors to further improve the paper and resubmit to a different venue. 

Decision: Reject