Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to learn natural transformations in datasets, with manifold auto encoder (MAE), where the underlying identity-preserving transformations are not easily identifiable. By using Lie group operator, this problem reduces to learning paths/motion in the latent space of MAE. The main challenge in training MAE is how to choose a transformation pair, a point and its identity preserving transformed counterpart, in an unsupervised way. This paper introduces the idea of using penultimate layer  of a pretrained classifier on Imagenet to choose such point pairs and learn such MAE more effectively. The proposed method is validated on three datasets. 
main_review: 
My main concern is with the backbone chosen (MAE) in this paper and its training, to demonstrate the idea of learning manifold transformation effectively. Three-stage training phase for MAE is already something and on top of that, this paper introduces to exploit a pretrained Resnet classifier and a coefficient encoder network with Laplace distribution prior on the coefficients. As a result, the overall approach errs on the side of being a bit complicated. 

Presentation:
I found page 5 quite hard to follow. As a result, I don't understand the fine tuning phase described as such on page 6. 'The fine-tuning phase adapts the autoencoder latent space to fit the learned transformations in the event that that pretrained latent space does not match the desired data manifold' I understand this without coefficient encoder network but not sure what is being matched after the introduction of coefficient encoder network. 

Also, why name Section 3 as Methods? if named MAE, it will help to delineate your contributions from MAE clearly.

The experimental setup is ok although I don't understand why choose only two baselines, the latest being from 2017. 
summary_of_the_review: I give marginally above for now but I will revisit it based on other reviews and discussions. The problem of learning manifold transformation is well motivated. Experiments seem ok. The idea to use pre-trained classifier to fit MAE latent space traversal is sensible.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors introduce an approach for learning identity preserving transformations from data. First the low-dimensional manifold structure of the data is learned using an autoencoding neural network, then the transformations (the Lie group operators and their coefficients of combination) that map between perceptually similar image pairs are learned. The main contribution of the paper is that the learned operators can transform the data semantically. This is a challenging since 1) semantic transformation labels are not usually available; and 2) not all semantic transformations make sense for all elements of the dataset. The authors address these challenges using two auxiliary networks. The first identifies perceptually similar image pairs in the dataset and the second identifies where particular transformations are likely to be used.
main_review: #### Strengths
The authors consider the important and challenging task of learning identity preserving transformations. Much of the previous work considers physical transformations (e.g., rotations, scaling) rather than semantic transformations which is arguable more challenging due to the lack of transformation labels (consider citing exception [1]). The challenge is compounded further by the presence of multiple classes in the dataset since not all transformations make sense for all classes. The proposed solutions for these challenges is interesting and the empirical analysis shows that they are also effective.

[1] Ham, Jihun, and Daniel D. Lee. "Separating pose and expression in face images: a manifold learning approach." Neural Information Processing-Letters and Reviews 11.4 (2007): 91-100.

#### Weaknesses
It is difficult for me to reconcile the use of Lie operators which assume some smoothness in the data manifold and the statement: “However, semantic transformations of interest may not be exhibited through pixel similarity nor through unsupervised autoencoder feature similarity”. Moreover, the NN samples shown in Figure 9 seem to say otherwise. Specifically, the hair style between Latent NN examples appears more similar than those of the Perceptual NN.

I'm not convinced the use of CAE and beta-VAE as baselines is appropriate. Comparison against more recent work in disentangled representation learning (e.g., [2]) would be more convincing. I would have also liked to see a comparison against another approach that uses Lie operators (e.g., [1, 3])

A common dataset for analysis of disentangled representation learning algorithms is [4]. I would have liked to see results on this dataset.

[2] Yu Deng, Jiaolong Yang, Dong Chen, Fang Wen, Xin Tong; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 5154-5163

[3] Connor, Marissa, and Christopher Rozell. "Representing closed transformation paths in encoded network latent space." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 04. 2020.

[4] Locatello, Francesco, et al. "Challenging common assumptions in the unsupervised learning of disentangled representations." international conference on machine learning. PMLR, 2019.

summary_of_the_review: The proposed work is interesting and addresses an important challenge in representation learning. I'm not convinced the baselines or datasets used in the empirical analysis are most appropriate for this work.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The manuscript proposes a method for learning group action on data using features from classification networks. The idea is natural and easy to follow. 
main_review: The idea of using features from classification networks is quite interesting. The proposed way of learning local transport also makes sense. However,  the way of MAE is an ill-posed decomposition of encoding-decoding and group actions. The additional step of combining MAE together with another encoder network for local operation makes the decomposition even more ill-posed. It would be nice to have a better explanation of why the desired decomposition (i.e. latent variables, group action, local action regions) can be obtained. 
summary_of_the_review: In fact, I found the proposed method is quite interesting.  A better explanation of why the proposed ill-posed decomposition can provide the desired solution would make the idea more convincing.
review_title: null

Meta-review: The paper proposes a method for learning identity-preserving transformations through a set of learned Lie group operators. It builds upon previous work ( (Connor & Rozell, 2020; Connor et al., 2021) addressing two points: (i) how to select semantically related pairs of points, (ii) how to identify which operators are appropriate for a given local region of the manifold. Authors use nearest neighbors computed via the penultimate layers of a pretrained network to address (i), and learn a separate network q(c|z) that predicts the coefficients given a latent input z. Reviewers have two main concerns with the paper -- limited novelty over earlier work that learns the Lie group operators; and the complicated nature of the method which needs training in three stages and uses a pretrained ResNet for finding nearest neighbors. Lack of comparison with relevant baselines is also pointed out by the reviewers. Given these issues, the paper is unfortunately not suitable for publication in ICLR at this point.
Decision: Reject