Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper concerns with uniform sampling from deep generative networks such as GANs and VAEs. The training samples of DGNs are often biased as they are obatined based on preferences, costs, or convenience that leads to DGNs producing biased examples. This paper gives a gemoetry based sampler MaGNET, that given any trained DGN, produces samples that are uniformly distributed on the learned manifold. It theoretically proves, and empirically shows that the MaGNET produces a uniform distrbution on the manifold regardless of the training set distribution. The theoretical proofs require that the DGNs only comprise continuous piecewise affine (CPA) non-linearities, such as ReLu, absolute value, max-pooling. The three main contributions of the paper are as following:
(a) It characterizes the transformation incurred by a density distribution when composed with a CPA mapping.
(b) It derives an analytical sampling strategy that allows to obtain a uniform distribution on a manifold that is continuous and piecewise affine.
(c) It provides multiple numerical experiments validating the gains of their proposed method MaGNET.

main_review: Strengths:
(a) Given any trained DGN, the paper gives a novel theoretical method to produce samples that are uniformly distributed on the learned manifold, regardless of the training set distribution. The approach is novel and solves the problem elegantly. 
(b) It proves the proposed method for a mild assmption that DGN only comprises continuous piecewise affine (CPA) non-linearities, such as ReLu, absolute value, max-pooling. 
(c) It gives convincing experiments on synthetic dataset showing that regardless of the training set distribution their MaGNET approach produces samples that are uniformly distributed.

Weakness: The paper needs improvement in writing. 
(a) In section 3.2, the notation J_s(z_i) is used without explaining it. "compute the per-region slope matrices A_i = J_s(z_i)". Please define the notation and explain how to compute the slope matrices. 
(b) A high level proof of the main theorem 2, in the main paper will help the reader understand the theorem better. 
(c) the x-axis values of the two plots in Figure 3 are different by order of 100, it does not seem correct. 
summary_of_the_review: The paper provides a novel provable method to produce samples that are uniformly distributed on the learned manifold, regardless of the training set distribution. The method is also well proven empirically through numerous experiments. The proposed method help address the problem fairness in samples produced from DGNs trained on not-so-well-representative training datasets.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a uniform sampling technique for deep generative networks (DGNs) inspired by the probabilistic change of variables formula. The technique works with any already trained DGN and does not involve any further training. (Though it does require back propagation w.r.t. the input $x$.) In essence, the algorithm works by drawing many samples N >> K from the DGN, then sampling from these $N$ samples with probability inversely related to their pushforward density (as computed by the change-of-variables formula).

main_review: Applying the change of variables formula to augment sampling from DGNs is a novel idea. Moreover, the method is interesting and theoretically well-motivated. Finally, the sampling algorithm itself is very straightforward.

However, I take issue with the framing of the algorithm in the abstract and introduction. Namely, the authors use the colloquial understanding of “uniform” side-by-side with the differential geometric / measure theoretic understanding of “uniform”. For example, in the abstract, the authors state that 1) many generative models today are trained on non-uniform data, which has “potential implications for fairness, data augmentation, anomaly detection, domain adaptation, and beyond,” and 2) their algorithm “produces a uniform distribution on the manifold regardless of the training set distribution”. This creates the false impression that the present technique is capable of neutralizing the negative "implications" on fairness, data augmentation, etc etc. This juxtaposition may imply parity between the two definitions of "uniform" to the inattentive reader. While the authors do emphasize the difference between the term in these two contexts much later in the paper, I feel that it is not appropriate for the abstract to mislead in this manner. Again, I only have this issue with the framing of the technique, not the technique itself.

WIth regard to the uniform sampling property of MaGNET, I have two concerns about the practicality of the method. 
1. The authors have touched upon this, but uniform sampling from the data manifold does not imply uniform sampling of attributes. This is exacerbated when the model has not fully learned the manifold. Therefore MaGNET’s sampling is only as “uniform” as the DGN and the data manifold itself is.
2. Since the DGN can only be trained on the training data distribution, sample quality will vary across the true data manifold. Namely, sample quality will likely correlate with density w.r.t. the training data distribution. Therefore, I imagine that sampling uniformly will reduce sample quality overall. This seems to be corroborated by qualitative comparison of original v. MaGNET samples in the paper figures.

Computationally, the authors demonstrate in Appendix D that sampling with $N$ past 250k does not affect the Precision-Recall metric, but I could not find what $N$ is in the experiments shown. And since each image sample requires computing the Jacobian of the DGN w.r.t. its input, I wonder what is the approximate computation time needed to sample $N=250,000$ times for each of the models.

summary_of_the_review: This method itself is novel and interesting, and warrants acceptance into the conference. However, the current wording of the title and abstract can be misleading, and should be edited to remove confusion.

Pros:
+ Theoretically motivated
+ Algorithmically simple

Cons:
- Seems computationally expensive
- Oversells the capabilities of the technique in the abstract, namely: mathematically uniform is implied to mean semantically uniform
- Needs proofreading
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a sampling method called MaGNET for generative models which aims at sampling data from the latent distribution of the images uniformly.
main_review: This paper proposes a sampling method that aims at providing uniform samples from the latent space for deep generative models. Providing uniform samples from the latent space is very important, but the manuscript requires more quantitative results to support their claims of uniform samples:

Major concerns: 
1. How does the MaGNET sampling affect the quality of the generated images? Common GANs literature provides some quantitative evaluation metrics (inception score, FID score, KID score) as a justification for their proposed methods, but this submission does not provide any justification using the prevalent evaluation metrics. The visual quality of some generated images using the MaGNET method is poorer than using the original GAN alone (Figure 7 &9) In addition, in addition, the provided samples of the generated images (Figure 15 - 20) are not clear enough to provide a good justification for their visual quality.

2. It is hard to tell whether the proposed method truly improves the uniformity of the sampled data (in Figure 22):
 (1) (Gender) it seems that the MaGNET-style reduces gender bias but the MaGNET-pixel increases gender bias
 (2) (Hair) due to the low clarity of this subfigure, it is hard to draw any conclusion
 (3) (Glasses) it seems that the MaGNET-pixel improves the occurrence of sunglasses and the improvement of MaGNET-style is limited
 (4) (Age) it seems that both methods have some improvement in age
 (5) (Emotion) the improvement of this subfigure is hard to tell (both improves in fear, but decrease in disgust)
 (6) (Accessaries) it seems that both methods (MaGNET-pixel and MaGNET-style) can improve the occurrence of headwear
But all the results are qualitative, the reviewer would like to see some quantitative results, such as how close the new distribution sampled using the MaGNET methods is to the uniform distribution, under the same categories (Gender, Hair, Glasses, Age, Emotion, Accessaries) provided by the authors comparing to the previous methods.

Minor comments:
1. What is the per-region slope matrices $\mathbf{A}_i = \mathbf{J_S}(\mathbf{z}_i)$ in the sampling procedure of the MaGNET?
2. The resolution of some figures (Figure 5, 8, 15 - 20) are too low, especially in Figure 15 - 20 where the quality of generated samples is hard to verify.


summary_of_the_review: This paper does not provide enough experimental results (both qualitatively and quantitively) to support their claims.
review_title: null

Meta-review: The paper proposes a simple method for uniform sampling from generative manifold using change of variables formula. The method works by first sampling a much larger number of samples (N) from uniform distribution in the latent space and then does sampling by replacement (using probability proportional to change in volume) to generate a smaller number of final samples (k << N) that are seen as approximately sampled from a uniform distribution from the generative manifold. 

Reviewers had some questions/concerns about the confusing language in the abstract and introduction around the use of the term "uniform" which the authors have addressed satisfactorily. Authors have also provided results on quality (FID metric) of the generated samples as asked by the reviewers. 

While the proposed method is rather simple, has high computational cost, and novelty is marginal (as noted by two of the reviewers), reviewers agree it is above the acceptance bar.
Decision: Accept (Poster)