Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes to relax the assumption of disentangled representation and encourage the model to learn linearly manipulable representations. 
The paper assumes that the latent canonicalizers are predefined for each task and that it is possible to obtain the ground-truth image of different latent canonicalizations. I find these assumptions too strong for the task of learning disentangled representation. 
Firstly, most prior works such as beta-vae, info-gan do not assume that the factors / canonicalizers are known beforehand. In fact, this is a very difficult part of learning disentangled representation. Secondly, if it is possible to obtain the ground-truth image of different latent canonicalizations, you can simply train a network to predict the canonicalizations by simple supervised learning. Hence, these overly simplified and unrealistic assumptions make the task too trivial. 
The proposed method is very simple and frames the problem basically as a supervised learning problem. Although experiments show that learning such representations are beneficial for low-shot setting of SVHN, it is not clear whether such improvement generalizes to more realistic datasets such as ImageNet. If the goal is to learn representation for low-shot setting, the method needs to be compared with other representation learning methods such as jigsaw[1], colorization[2] and rotation[3].

[1] Noroozi, Mehdi, and Paolo Favaro. "Unsupervised learning of visual representations by solving jigsaw puzzles." European Conference on Computer Vision. Springer, Cham, 2016.
[2] Zhang, Richard, Phillip Isola, and Alexei A. Efros. "Colorful image colorization." European conference on computer vision. Springer, Cham, 2016.
[3] Gidaris, Spyros, Praveer Singh, and Nikos Komodakis. "Unsupervised representation learning by predicting image rotations." arXiv preprint arXiv:1803.07728 (2018).
review_title: Official Blind Review #4

final_score: rating: 8: Accept
review_content: This paper proposes a method for unsupervised learning  of data representations that can be manipulated to remove factors of variation via linear transformations. These transformations are called canonicalizations in the paper. The canonicalizations are trained such that images for arbitrary values of the corresponding factor of variation are transformed into images with a fixed, canonical, value for that factor. The paper proposes a model architecture based on a denoising autoencoder, where the canonicalizations are applied to the encoded representation. It also proposes a loss function and sampling scheme for training the model. The paper demonstrates the method on the dSprites dataset, showing that it can effectively learn linear canonicalizations, and that multiple of these canonicalizations can be applied to the same image representation. The paper goes on to test the method on a digit classification task, where the model is trained to learn a representation in a simulator for SVHN data where the transformations to be canonicalized can be controlled, and  used to train a classifier on unseen real data from the SVHN test set.

I think this paper should be accepted as it proposes a novel idea, which does not seem too difficult to reproduce, describes a simulator for synthetic data for digit recognition, and proposes it as a benchmark for learning representations, and provides experimental results that help in better understanding the representation learned by the model.


A couple things I thought were missing in the paper:

Did you try applying the classification loss to both the encoded representation and the canonicalized representation at the same time?

For Figure 2, Did you try applying canonicalizations in different orders? Do they give the same results?

Instead of trying to learn idempotency by gradient descent, you could try to parametrize the canonicalizations with a matrix X, such that C =  X (X^T X)^{-1} X^T. C will be idempotent (although restricted to be symmetric). There might be other constructions that are more efficient and less restrictive.

I'm not sure I understand the PCA figures. Can you please explain how the first principal component was used to generate them?


Minor comments:

* "data  tripets" on page 2
* Figure 5 should appear after Figure 4.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: 
1. I had hard time to understand latent canonicalization. Do you mean that each latent variable is fixed to a value, such that this factor of variation is disabled? Are the canonicalizers pre-specified using meta-labels? Are they updated/learned during model training?   More explanation of canonicalization is needed. Perhaps an example in linear algebra is needed.

2. The learning of the proposed model relies on meta-data description such that the learning is supervised. Can the method be applicable to situations where no meta-data and no class labels are available? 
  
3. How can the proposed method be generalized to non-image data? The experiments were only done on simple image datasets. I am wondering this method can be applied to other complex datasets whose latent factors are unknown. 

4. I do not understand this: "to fit well the method overfitting rate" in Section 3.3.

Minors:
(1) than -> that 
(2) Eq. (3): is there a superscription "(j)" on z_canon in decoder?
review_title: Official Blind Review #2

Meta-review: This paper proposes a method to allow models to generalize more effectively through the use of latent linear transforms.

Overall, I think this method is interesting, but both R2 and R4 were concerned with the experimental evaluation being too simplistic, and the method not being applicable to areas where a good simulator is not available. This seems like a very valid concern to me, and given the high bar for acceptance to ICLR, I would suggest that the paper is not accepted at this time. I would encourage the authors to continue with follow-up experiments that better showcase the generality of the method, and re-submit a more polished draft to a conference in the near future.
Decision: Reject