Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper considers the problem of few-shot image synthesis, i.e., learning to generate samples from a distribution given only few examples. A previous approach, dubbed Implicit Maximum Likelihood Estimation (IMLE), showed that one perform few-shot synthesis by minimizing a loss that penalizes the distance between generated samples and true samples. This loss was shown to exhibit nice properties, such as convergence to the true likelihood, but operated under strong assumptions (e.g., all training images have the same likelihood). This work builds upon IMLE by proposing an adaptive version that is able to weigh examples from the training set based on their difficulty and theoretically show that their loss is a generalization of IMLE operating under weaker assumptions. Results on few-shot image synthesis benchmarks show that learning with the Adaptive IMLE can improve generation.
strength_and_weaknesses: 
**Strengths:**
- The new adaptive loss function is principled in that it can be shown in a limiting sense to converge to a maximum likelihood-based objective.
- The empirical results show quantitative and qualitative improvements in terms of generation quality and diversity of the images. To the reviewer's knowledge, comparisons against strong few-shot image synthesis baselines are considered for comparison.

**Weaknesses:**
- Some notions of the theory would be impractical to implement, such as choosing the correct weights $w_i$ in the optimization for certain distance metrics, which requires computing the volume of a neighborhood around a given datapoint.
- There are some issues in the clarity/derivations that would be good to improve upon.

clarity,_quality,_novelty_and_reproducibility: 
**Clarity:** 
Overall, I found the paper fairly easy to read, but had some questions regarding some of the proposed theory and derivations:
- There is some handwaving in the derivation from equation (3) to (4). In particular, $\tau_i$ is chosen to depend on $T_{\theta}$ prior to equation (3), but then the $\tau_i$ term is dropped in the loss function in equation (4). Can we truly treat $\tau_i$ as a constant here? 
- The derivation in Lemma 2 appears for a fixed sample $z_j$, rather than an expectation over the empirical distribution.
- There are no assumptions placed on the distribution $p_{\theta}$, which makes it hard to verify some of the results. For example, why does the limit in the third to last line in the proof of Lemma 4 exist? At a minimum, one would require continuity of $p_{\theta}$ at the datapoints to assert this.

**Novelty and significance:** 
To the reviewer’s knowledge, this extension of IMLE appears novel, but it could be that I am unaware of certain prior work. The experimental results also appear quite solid.

**General comments:**
- Ultimately, the loss that ends up being used by the authors is equation (5) versus the original IMLE equation (1). The only difference here is in taking the maximum between the distance and the threshold $\delta \tau_i$. The weights, $w_i$, however, can be computed in closed-form for certain metrics, such as the distance induced by an $\ell_p$-norm. Empirically, did it have an impact on performance if the loss with weighting $w_i$ was used versus equation (5)? 
- Along these lines, it would be nice to see some low-dimensional examples showcasing how the proposed loss actually helps in “adapting” to difficult examples. Figure 1 provides some intuition, but it would be nice to see some 2- or 3-dimensional experiments showing this more concretely.
- Since the framework allows flexibility in choosing a distance metric $d(\cdot,\cdot)$, did the authors try different metrics other than $\ell_2$-distance, such as an LPIPS metric? Did this influence performance at all?
- There is a missing equation reference in the middle of page 5 
summary_of_the_review: Overall, I think that the paper presents a nice extension of previous work in order to improve few-shot image synthesis. The experimental results seem promising, but there is some lack of polish in the presentation of the material. I am leaning more on the positive-side based on the results.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposed a generalized formulation of IMLE, Adaptive IMLE, for few-shot image synthesis. As the generator trained in GAN suffers from the mode collapse issue, i.e., GAN may overfit to a subset of training samples. Then, IMLE is proposed to avoid this issue by optimizing training images to have some similar generated images. However, as strict restrictions are applied on IMLE and may impede training efficiency in the few-shot setting, this paper proposed Adaptive IMLE by adjusting the radius w.r.t. each training data. Detailed theoretical analysis is provided, and the experimental study demonstrates its effectiveness. However, the discussion is very limited.
strength_and_weaknesses: Strength:
1.	Formulating a generalized form of techniques is essential for further development, and the few-shot learning setting is specifically studied.
2.	Clear paper writing about background introduction and method derivation & development.
3.	Consistent performance gain and promising results. Visualization are also provided.

Weakness:
1.	My biggest concern is regarding ignoring w_i in the Eq.5. The explanation seems incomplete (the unweighted objective, since tau_i and there fore w_i is fixed). Please double-check and clearly explain why w_i can be ignored.
2.	Why Prec drops slightly while FID and Rec. are improved largely? It is necessary to explain the reason. Acturally, Prec are very high and I am not sure whether it Is a good metric.
3.	Training details should be provided. Do you need additional large-scale data for pre-training? Also, do you need to find the nearest generated samples for all training data within each batch? If yes, the memory usage is pretty large and I wonder whether your approach can address this issue.
4.	The comparison between IMLE and adaptative IMLE is only on Obama and FFHQ while more experimental comparisons are performed in Table 1-2. Can you explain the reason? From my perspective, the comparison with vanilla IMLE is much more critical.
5.	For the optimization of IMLE, one weakness is that the neighbor search is centered w.r.t. each training data, then, it is possible that a very limited number of generated samples are selected in one batch. Will that be risky, and do your methods can mitigate this issue also?

clarity,_quality,_novelty_and_reproducibility: This paper provides a theoretical analysis of IMLE by providing a generalized formulation, which is important.
The pseudo-code is provided for reproducibility.
summary_of_the_review: The discussion is very limited. Please see my comments above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new objective for the problem of few-shot image synthesis based on extending implicit maximum likelihood estimation (IMLE). Specifically, they derive a data-adaptive IMLE objective for training the implicit generative model. Experiment results on six few-shot image synthesis datasets, namely Grumpy Cat, Obama, Cat, Panda, Dog, and the subset of FFHQ, demonstrate the proposed method outperforms the previous few-shot image synthesis methods.
strength_and_weaknesses: ### Strengths
* The problem setup of few-shot image synthesis is practical and interesting.
* The paper performs extensive experiments across diverse datasets.
* The paper shows solid experimental results, both quantitatively and qualitatively. 

### Weaknesses
* While the problem definition (few-shot image synthesis) is interesting in general, I am not convinced about the detailed problem setup in not using auxiliary datasets. An explanation highlighting the importance of such a setup or comparing the results with fine-tuning-based methods would be welcome.
* The paper is hard to follow. Considering that the paper proposes the generalized version of IMLE, the clarity of the paper would be much better if it explicitly mentioned the limitations of IMLE (at least in brief) in the theoretical aspect. However, the paper only mentions that “restrictive conditions need to be satisfied for the theoretical guarantees of IMLE to hold, such as requiring a uniform optimal likelihood for all data points”. Why, what, and how are such conditions restrictive, while the proposed method is not? 
*  I think the ablation study (Table 3) should be plotted (instead of Table) to compare IMLE and adaptive IMLE. Few-shot synthesis problems are generally vulnerable to the overfitting of the generator. As the proposed adaptive IMLE objective includes additional regularizations (e.g., $\min$ with $\delta \tau_i >0$)than IMLE, hence more difficult to optimize, I wonder whether the performance gain is simply from alleviating the overfitting or from indeed “better” objective design. At least, I strongly recommend authors put more details on how the ablation study is performed (e.g., “best” performance during the training is reported); but still, showing plots with respect to number of iterations would be much more welcome.
* While the proposed method shows superior performance among the reported baselines, the current manuscript misses some recent baselines (even outperforming adaptive IMLE), e.g., [1]. At least, a discussion with this baseline would strengthen the manuscript more.

[1] FakeCLR: Exploring Contrastive Learning for Solving Latent Discontinuity in Data-Efficient GANs, ECCV 2022. (arXiv: Jul 2022)

### Minor comment
* The equation number is not stated in the sentence between Eqn. 3 and Eqn. 4.

### Questions
* Why does the precision of IMLE be better than the proposed Adaptive IMLE?
* The proposed method can be applied to any few-shot synthesis problem; do the authors try to apply the method in different domains?
clarity,_quality,_novelty_and_reproducibility: **Clarity.** The clarity of the paper can be improved. For instance, a more comprehensive (theoretical) explanation of the limitation of IMLE and a comparison between IMLE and the proposed method would be helpful.

**Quality.** While the paper performs the experiments on diverse datasets, yet needs to include recent baselines that outperform the presented method. 

**Novelty.** The proposed method seems novel, yet it could be clearer due to not very good clarity.

**Reproducibility.** The supplementary material contains the code for reproducing the results, indicating good reproducibility. 
summary_of_the_review: This paper tackles the few-shot image synthesis problem by extending the IMLE objective to be “image-adaptive”. However, I feel the overall clarity could be improved. For the evaluation side, I am not fully convinced due to the missing baseline and the concerns about ablation studies. In these respects, I am currently on the negative side.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper enables training GANs with a few amount of data from scratch. It is built on top of an existing method IMLE which is improved to adaptive IMLE to avoid model collapse or model overfitting. The more generalized form of IMLE is presented which covers the original formulation as a special case. Extensive experiments on different datasets are conducted to show that the proposed method could generate higher quality and more diverse results.
strength_and_weaknesses: + Strict prove on theoretical guarantees of the generalized formulation hold under weaker condition
+ A systematic level up of vanilla IMLE algorithm
+ Obvious improvement of results under few-shot setting

A good work about making GAN training less data-hungry and potentially using less compute resources, which is applicable to personalized generation with a small library of data. Please find my minor concerns below:

To understand how the number of training data is affecting the performance, it is better to conduct an experiment by ablating this hyper-parameter and show a curve about how FID will increase with less training examples.

Just curious what is the smallest number of images that the proposed method could work? 100? 10? 1? The previous transferring learning based methods, though requiring pretraining on an external data, are more aligned with human learning process of learning new knowledge quickly after learning old knowledge and show even 1-shot setting is possible. The proposed method belongs to learning from scratch with a few amount of data. Is it limited in for example just learning 100 faces of one person (obama) instead of learning 100 faces of different persons? Essentially, GAN should be a data engine to benefit more downstream tasks. If the generated is still less diverse, it may not be quite useful as compared to transfer learning approaches.

It is suggested to include some results on non-facial data as face has been heavily explored, in order to see how robust the algorithm is cross different type of datasets.

clarity,_quality,_novelty_and_reproducibility: This work's presentation of proposed framework is clear and easy to follow. It is a more generalized version of a previous method. Reproducibility is upon the code release though the algorithm of each step is already given.
summary_of_the_review: A new perspective to tackle the few shot generation problem. Good work but I worried a bit about its performance over other non-facial dataset and eventually how useful it will be compared to transfer learning setting.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work designs a more general IMLE objective by introducing an adaptive neighbourhood radii. The theoretical derivation and the proposed curriculum learning strategy are beautiful and impressive. Some experiments are also conducted to verify the effectiveness of the proposed method.
strength_and_weaknesses: Strength:
1. The idea of introducing an adaptive radii for IMLE is interesting, which increase the degree of the vanilla IMLE.
2. Leveraging lemma 1 to construct the adaptive is very clever. It provides a very useful tool to depict the distribution of the distance between the training example and a generated sample.
3. The visual results tend to prove that this adaptive objective could produce more diverse example.

Weaknesses:
1. The written of this paper is not very clear. Beginning from lemma 3, the notation $w_i$ is frequently appeared. But there isn't any explanation about this notation, and why does introduce $w_i$?
2. I don't understand the definition of $L_{\{\tau_i\}_i}$ in the first line of the proof for lemma 3 in appendix. If possible, please provide more details.
3. The number of sample $m$ should play an important role in this work. Even this work design the curriculum strategy to avoid increasing $m$. It would be better to offer some results under different values of $m$.
4. Comparing with the vanilla IMLE, why does the proposed objective achieve better results? This paper does not clearly claim this point.
clarity,_quality,_novelty_and_reproducibility: The clarity on writing should be further improved. In my opinion, the design of the adaptive IMLE is novel.

summary_of_the_review: This work devises a theoretically sound objective to generalize the IMLE, and achieves stable and better results. Even though the writing of this paper should be improved, it is still a beautiful work.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper is proposing to extend IMLE with weights and parameters depending on input data points. The method follows replacing min with a softer version to use interchangeability of expectation and summation later taking limiting behaviour by choosing a specific function class to make sure min and softer version agrees. The method is later tested on various different benchmarks and the results were very promising. The paper was reviewed by 5 experts and received borderline reviews. Later a reviewer AC calibraiton meeting was performed and the following major points are shared by reviewers:

- Positive aspects:
  - The application is very interesting and important
  - Results are good

- Negative aspects:
  - Novelty is limited 
  - Scope is very narrow and experiments are also narrow
  - Technical correctness is not clear.

I believe the novelty issue is not a valid concern and the scope issue is solved by authors during rebuttal. However, technical correctness was remaining so I decided to read the paper in detail and checked all the proofs. I believe the paper has significant correctness issues which should have been clarified. Here are the major ones:

Before understanding the correctness issue, notice the following:
- In Lemma 1, $\Phi$ is a single function of $X$. Moreover, it is assumed to be integrable. Its proof is entirely correct and no correctness/clarity issue is there.
- In Lemma 2, The lemma is correct in its currently written form. However, there is a significant assumption hidden and not made explicit. This lemma is only correct if $\tau_i$ is independent of $z_k,x_k$ where $i \neq k$. If this is not true, probabilities and integrals are over the joint distributions not marginals. This is not made explicit anywhere in the paper, also not discussed. 
- In Lemma 3, this is rather application of Lemma 2 so it has the same assumption.

In the rest of the paper, $\tau_i$ is chosen very carefully using nearest neighbours simply invalidating the original assumption. It depends on other points. Hence, lemmas can not be applied to the remaining derivation.

Another correctness issue is caught by the reviewers. Reviewers simply noticed that $w_i$ can not be simply ignored and they are right. Authors' counter argument is the fact that both optimization problems have the same optimal solution so they are equivalent. This is only true for convex optimization since having the same global optimum point does not mean anything about the equivalence of two non-convex optimization problems. Even for convex case, it simply means they are same asymptotically which does not imply the equivalence in finite sample case. Regardless, for finite regime and non-convex case, these two optimization problems are obviously not equivalent and authors can see it by simply setting $w_i=1$ for one point and setting $\frac{1}{w_i}=10^{-20}$ for all other points and running their method. 

In summary, there are correctness issues with the paper. I believe there is a major correctness issue. Moreover, authors did not put any effort to make checking correctness easy. There are no motivation for correctness of any lemma or theorem is provided. Statements are very technical without accompanying discussion.
justification_for_why_not_higher_score: The paper has a correctness issue.
Decision: Reject