Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: This manuscript presents ADIS-GAN (affine disentangled GAN), a method for learning disentangled affine transformations of images such as rotation, zoom, and translation. The authors enforce this disentanglement using a regularizer that penalizes the difference between affine transformations on latent representations. The method is difficult to follow with many critical details left out. Without a complete description, it’s impossible to fully understand and evaluate this work. Some specific comments and questions follow below.
1.	How is the latent vector, C, used? How does it interact with W_basis and W_real? Must it be 5-d to match with rotation, zoom, and translation? How are additional factors of variability encoded?
2.	How is W_basis learned?
3.	What is Norm_AL
4.	What are the encoder/generator/adversary model architectures and how were they trained? For that matter, what is the encoder listed in algorithm 1? I can’t find any reference to it in the text.
5.	How do you ensure that affine transformations of the latent vector actual produce affine transformations of the image? In Figure 5, it looks like the faces aren’t simply being transformed by zoom, because other attributes clear change. In column 5 from the right, the top face has short hair, an exposed ear, and visible teeth. The bottom face, in contrast, has long hair, covered ear, and no visible teeth. This does not appear to be a simple zoom transformation.

Things that would improve my score:
1.	Fully explain the method, neural network architectures, and training parameters.
2.	Explain how affine transformations of the latent vector are forced to produce matching affine transformations of the image.

review_title: Description of method incomplete

final_score: rating: 4: Ok but not good enough - rejection
review_content: Paper summary:

This paper seems to be about aligning latent variables in a GAN with certain affine transformations of the image. This is done by adding an additional "affine regularization" on top of the InfoGAN formulation. This affine regularization seems to be the L2 distance between the randomly sampled c (interpretable latent variables?) and some transformed c'. How the transformed c' is computed was not understood by the reviewer, but it seems to depend on a set of explicitly chosen affine transformations. The affine transformations specified are rotation, horizontal and vertical zoom, horizontal and vertical translation. The authors experiment on MNIST, dSprites, four shapes, and CelebA data. Quantitative metrics are reported for dSprites.

Strong points:

Not many papers tackling disentanglement in GANs.

The paper seems to show nice visual results and seems to be working at recovering the selected affine transformations of images.

Weak points:

(My overall concerns are about the additional supervision required by the method and the clarity of the writing, in particular the method itself.)

The paper mentions this is a self-supervised approach, but it seems closer to a supervised approach where the affine loss can only help recover specific latent factors that must be known beforehand. Would ADIS-GAN be capable of discovering unknown latent factors?

The paper repeatedly claims in the text and in Table 1 that previous works do not have axis-alignment or scalability. Does ADIS-GAN guarantee alignment in some way that is more reliable than in previous works? Isn't the scalability aspect (modeling horizontal and vertical zoom) explicitly encoded by ADIS-GAN, whereas previous works attempt to learn this from data?

I could not follow the exposition in the paper. For instance:
  - There seems to be a complete change in notation from Sections 1-3 to Section 4. No mention of the previously defined x_real, x_fake, z, or c are used in Section 4, where the core contribution is described. 
  - Many sentences end in "see Appendix", but I have looked at the Appendix and still come away confused..
  - Why is M 3x3?
  - Where do the A_{ij}'s come from in equation 9? I guessed it might be elements of c, but there are 5 elements in c and 6 A_{ij}'s.
  - There seems to be derivations of a maximum likelihood estimate for constructing c', but these depend on knowing what the A_{ij}'s are.
  - The purpose of Norm_{AL/LA} is never explicitly mentioned, though I imagine it's for scaling the interval [-1, 1] to specific bounds so that M can be applied?
  - What is Encoder in Algorithm 1 and why does it return a linear operator? (Is this different from the encoder in InfoGAN which outputs a distribution over the latent variables?)

Additional feedback:

 - There is mention that ADIS-GAN used categorical codes as well. I assume these do not contribute to the affine loss?

 - There are multiple rows in Table 2 that reference models not explained in the text. e.g. references for HFVAE and ChyVAE, and what is "InfoGAN(modified)"?

 - Please use \citep to include brackets around your citations, when not explicitly using them in a sentence.
review_title: review

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a method that can explicitly learn the affine transformations via disentangled representations in a self-supervised manner. The proposed ADIS-GAN learns to extract affine parameters by adding an affine regularizer on the top of InfoGAN. It seems to me that the main contribution of the paper is to learn the explicit parameterization of the affine transformation, such as rotation, zoom, and translation, by maximum likelihood estimation.  Experiments show that the proposed method is successful in estimating individual parameters of the affine transformation. 

**Strengths**
+ The paper represents the first method that can learn explicit parameters of the affine transformation in a self-supervised manner.
+ The experiments show that the proposed method can learn a disentangled representation with explicit affine parameters. 

**Weakness** 
- Experiments are not convincing enough. The paper only contains experiments on simple/synthetic datasets. The benefits of the proposed method are not completely clear. For the dSprites dataset, the proposed method does not outperform InfoGAN-CR, while InfoGAN-CR can perform disentanglement beyond affine transformations. For scalability, the paper only presents one example for zoom. It is also unclear how to merge zoom parameters into one. It is also helpful to give real applications that can benefit from the explicit disentanglement. 
- Writing of the paper can be improved. For example, Section 4.1 needs improvements. The notations are not completely clear, and their roles in the whole algorithm are not clear. It would be better to strengthen the connection of Equation (7) to the affine regularizer loss defined at the end of Section 4. 
- The factorization of the affine matrix and derivation of the maximum likelihood estimator are straightforward, making the paper's technical contribution less significant. 

**Minor issues**
- The paper emphasizes scalability, but it is not completely clear what scalability the proposed method achieves and how it is achieved. For example, is it possible to merge the translation parameters into a single one? If so, how to do it, and how well the combined parameter performs?
- It is unclear whether the proposed method can factor out skews. Figure 1 shows examples for skews, but the paper does not present how the skew parameters are estimated. 
- In Appendix A, the paper assumes the noise statistics for all parameters A_{ij}. It seems unrealistic. 
- The citations are correctly formatted. The parentheses are missing for citations.

**Post-rebuttal**

Thank the authors for the rebuttal. It addresses parts of the raised issues. However, my rating keeps the same after reading the rebuttal and other reviews because
1. the contribution and utility of the proposed method are not significant;
2. the writing needs improvement; and
3. the experiments are not convincing enough, and its advantages over previous methods are not clear. 
review_title: Some interesting results, but the benefits of the explicit parameterization are not clear 

Meta-review: Most of the reviewers had serious problems with clarity to start out. 
The authors have addressed some, but not all of these problems. 

More importantly, there were issues of significance and experimental evaluation.
I concur with r4 on the experimental evaluation. 
I think if you're going to explicitly specialize toward disentangling affine transform parameters, 
that's fine, but then you're in application-paper land, and I think there needs to be more of an attempt to show
that it will work "in the wild". 
For this reason, and for the general reason that reviewers unanimously voted to reject, I am recommending rejection.
Decision: Reject