Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: EDIT: Updated score to weak Accept in lieu of author's response.  See below for more details.

The authors propose a GAN architecture that aims to align the latent representations of the GAN with different interpretable degrees of freedom of the underlying data (e.g., size, pose).  While the text, motivation, and experiments are fairly clear, there are some spelling/grammar mistakes throughout, and the draft could use a solid pass for overall clarity.

While the idea of using the log trace covariance as a regularizer for the manifold is certainly interesting, it seems fairly incremental upon previous work (i.e., DMWGAN).  Even modulo the work being incremental, I still have concerns regarding the comparison to baselines/overall impact, and thus I suggest a Weak Rejection.

Table 1 seems to indicate that the author's proposed method is on par with or worse than every method compared against except for 3D chair (bright).  Additionally, the lack of comparison against DMWGAN for every task (except the first) is a bit concerning, considering its similarity to the proposed method.  If the authors could check DMWGAN's performance for all of their tasks and report it, I would be more likely to raise my score.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The paper suggests performing simultaneous manifold learning and alignment by multiple generators that share common weight matrices and a constructed inverse map that is instantiated by a single encoder. The method utilizes a special regularizer to guide the training. It has been empirically well tested on a multi-manifold learning task, manifold alignment, feature disentanglement, and style transfer.
Overall, this is an interesting idea with a motivated approach, however, I would like several points to be addressed before I could increase a score.
1. It seems that the method makes direct use of the number of classes in the datasets used. How would it fare compared to other models when the number of manifolds is not known (e.g. CelebA dataset)?
2. In MADGAN (Ghosh et al., 2017) generators share first layers which possibly makes them not independent as claimed in the paper, thus it is worth checking if MADGAN exhibits any kind of manifold alignment and could be a baseline for disentanglement with multiple generators.
3. There are hyperparameters \lambda and \mu for the regularizers in the model. It would be helpful to study their effect of different values on the training and encoding.
4. Is there a reason for DMWGAN/InfoGAN scores being omitted in Table 1?

Minor remark - there are a number of typos in the text.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: A paper's idea is to train joint Wasserstein GAN for k datasets given in R^n (corresponding, e.g., to different classes of objects), together with manifold alignment of k manifolds. 

The idea is to align points whose corresponding latent points are the same. This induces a natural constraint: k encoder functions (inverse of generators) should be consistent with each other. This is done by adding a regularization term. A paper demonstrates a clear motivation and a working solution for the problem. Experiments are convincing.

The only question is that the regularization term forces something stronger than just consistency of encoders. It seems, the requirement that "all  tangential components of biases are the same" means that k images of latent space (under k generator functions) are either coincide or non-intersecting. This is much stronger than just consistency, which is the weakest part of the approach.
review_title: Official Blind Review #2

Meta-review: This work proposes a GAN architecture that aims to align the latent representations of the generator with different interpretable degrees of freedom of the underlying data (e.g., size, pose).

Reviewers found this paper well-motivated and the proposed method to be technically sound. However, they cast some doubts about the novelty of the approach, specifically with respect to DMWGAN and MADGAN. The AC shares these concerns and concludes that this paper will greatly benefit from an additional reviewing cycle that addresses the remaining concerns. 

Decision: Reject