Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Pros:
Figure 4. (a) is informative, which clearly illustrates the invertible learning process. 

The authors display the failure cases in Fig 4. (b), which is helpful for other researchers working on this filed. However, more analyses and discussion regarding the failure cases are suggested. 

My major concerns are listed as follows.

1, the contribution of this paper seems to be over-claimed. This paper attempts to learn an NLDR without loss of information; however, it is clear that the method in this paper also loses some information in the second step. Moreover, this paper seems to be a combination of previous works (LIS + sparse coordinate transformation), and thus it is important to clearly state the real contributions.

2, the presentation of this paper needs to be improved as many technical details are omitted. For example, in introduction, authors do not well describe how to preserve topology and geometry. This is very important as the preservation of topological and geometric properties of complex structures is a very difficult task [1], while at the same time, low-dimensional structures usually have sophisticated geometric and topological structures [2]. 

[1] Cohen, et al. "A general theory of equivariant CNNs on homogeneous spaces." NIPS. 2019.
[2] Wakin, et al. “The multiscale structure of non-differentiable image manifolds”. 	In Proceedings o fSPIE, the International Society for Optical Engineering, pages 59141B–1, 	2005.
	
3, The network mentioned in Fig. 2 is not clear. For example, what is the main difference between the blue and red arrows, the difference between the solid lines and the dash lines, and the difference between the gray rectangles and the white rectangles? Moreover, as this network is based on ML-Enc, without introducing the structure of ML-Enc, it is hard to clearly illustrate the network structure of the proposed method. 

4, Figure 4 is important as it visualizes the embeddings of different methods; however, there isn’t any discussion related to figure 4. In addition, in figures 4 (c)-(d), I cannot see the advantage of the proposed model over the comparted ones. 

5, There is an interesting phenomenon that the L-1 th layer preserves the most NLDR results, which should be elaborated.


6. there are some typos in this paper, e.g., a NLDR -> an NLDR, and the image quality like figure 1 can be enhanced. 

review_title: This paper proposes an invertible manifold learning (inv-ML) method. It first uses a homeomorphic sparse coordinate transformation to find a low-dimensional representation without loss of topological information. Second, a linear compression is performed on the learned sparse codding to get a trade-off between the target dimension and the incurred information loss. Experiments are conducted on seven datasets to evaluate the reconstruction performance and the generalization ability.

final_score: rating: 4: Ok but not good enough - rejection
review_content: In this paper, the authors propose a novel manifold learning method, via adding a locally isometric smoothness constraint, which preserves topological and geometric properties of data manifold. Empirical results demonstrate the efficacy of their approach. The authors also show that the reliability of tangent space approximated by its local neighborhood is essential to the success of manifold learning approaches.

Overall I found the ideas in the paper somewhat interesting. Many aspects were unclear to me (refer below).

1) In the abstract, the authors claim that in their first step there is no loss of topological information. How did the authors measure this i.e. the sparse coordinate transformation step does not result in any loss of topological information ? Do the results in the paper demonstrate this ? 

2) I was not clear to me as to how the authors in their approach are preserving topology and in general geometry of the space while reducing the dimension of the space simultaneously. I would like the authors to add a section on this and/or improve the clarity of their presentation. 

3) What is the intuition for the two step process of their approach ? It definitely fits the encoder-decoder framework but is there some other reasoning behind this approach ?

4) In section 3.3, the authors introduce the orthogonal loss wherein they force the weight matrices to be orthogonal. After training, are the weights orthogonal or how close to orthogonal are the weights ? Can we possibly add some other constraint in place of this ? 

5) Given the complexity of the model and the different loss functions associated with the objective, how much overhead is involved in model training and execution. Do the authors plan to share any time complexity results so that we can compare training/execution times with other state-of-the-art models ?

6) In the appendix section A.1, what is the intuition behind the exact mathematical form of the expressions used for the Trust and Cont metrics ? I was curious with regards to the exact mathematical expressions used i.e. the different constants and factors involved.

7) I did not quite understand the notation of Figure 2. What is the meaning of the different colored arrows and boxes indicate and how are they used ? There is not discussion on this as well. Similarly for Figure 4 for which there is no analysis or discussion. I would like the authors to include these without which the notation is cumbersome to follow. It is surprising that there is no additional discussion or analysis of the different figures included in the paper, given the captions are not sufficient by themselves.

8) How are the authors deciding on the values of the different hyper-parameters i.e. gamma and alpha in the Appendix ? Overall it felt very heuristic and I did not quite understand how the authors decided on the values used. 

9) Some of the loss based definitions in Section 3.3 are unclear to me. In case the authors used some standard/other loss functions from some paper, they could have added those definitions in the main paper or the Appendix etc. I found that the paper was not self contained as such and I had to navigate to the references quite a bit.

I found quite a few typos in the submitted draft. Kindly proofread and correct these. Overall I felt that the paper does not quite achieve lossless compression as well as there is only marginal improvement in terms of results when compared against other approaches. The approach presented in the paper in not clear in many parts.
review_title: Review

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary:
Paper presents a novel encoder-decoder framework for invertible dimensionality reduction. It is composed of multiple stages of homeomorphic embedding, sparse representation, linear compression and an inverse (reconstruction) process to learn invertible non-linear representations. Proposed idea is indeed novel and interesting actualization of geometry preserving dimension reduction shown in Figure 1. 

Strengths:
Proposed methods combines multiple ideas work on structure-preserving manifold learning, invertible and distance-preserving sparse representation learning.
Each of the steps above are achieved by NN structure and novel loss functions that impose orthogonality, sparsity and isometry constraints and so on.
Empirical results on synthetic and real-world datasets support the approach and shown efficacy of the method. Ablation studies on adding different components show need of each aspect.

Weakness: 
Invertible mapping learned is computed explicitly but can also be learned end-to-end during training. Is there a reason why the prior is preferred. 
Paper doesn't not address convergence aspect of the training and how it affects empirical results. Sparsity count (s) for representation is still heuristic and choice is not obvious. Can the RIP property provide a lower bound for choosing s.

Recommendation:
Paper is a clear accept as it introduces a novel method achieve Figure 1 using NN. It is well written and technically sound and qualitative results demonstrate effectiveness of the technique in terms of SOTA results.
review_title: Presents a novel method for an invertible non-linear dimensionality reduction with strong empirical results.

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper presents a nonlinear dimensionality reduction (NLDR) method which is claimed to be invertible. The method inserts an existing idea (LIS) to a two-stage neural network implementation. The proposed method is tested with several known benchmark datasets as well as three synthetic datasets.

The paper currently is not enough for acceptance.

First of all, the problem in attack is unclear. By title, abstract and introduction, it reads that the target is a lossless NLDR method such that the data objects can be perfectly reconstructed from the subspace. However, this is not true in the experiment part. Clearly the loss is not zero after reconstruction. On the other hand, without perfect reconstruction, the proposed method is just another approximation and I don't see major contribution. So I believe the term "invertibility" is different from common understanding and needs more careful definition.

In the definition of "Topology preserving dimension reduction", it requires the data set is sampled from a d-dimensional compact manifold with d<<m. This is too ideal and the assumption often does not hold. For example, when the data has m-dimensional noise, then the so-called invertibility does not exist.

From Fig.4, I don't see any advantage of using i-ML-Enc for visualization. Clearly the classes are more mixed by using the proposed method.

No need to refer to the LIS paper. The LIS loss is just about local distance preserving, which is well known in Multidimensional Scaling (MDS) literature.

It is unknown how to determine the hyperparameters, e.g., gamma^{(l)} and mu^{(l)}.

It is unknown how to determine network architecture for various data types.

The writing is fairly good. The main problem is that the paper is not self-contained. Unless the readers dive into the references, it is hard to understand the methodology, especially many symbols and acronyms.
Some detailed comments:
* All equations are not numbered.
* Some presentation is vague and unexplained. For example, what do dash lines mean in Fig.1? At the end of Section 2, what do you mean "their performance is not impresssive"?
* I cannot find the definitions of d_X and d_Z in the LIS-loss

review_title: Really invertible?

Meta-review: 
While reviewers find the ideas in the paper interesting, they also raise several major concerns.
In particular, R1 and R4 find the claims of "invertible" and "lossless" to be potentially misleading.
The bijective property is achieve on the first stage (L-1 layers) due to a sequence of one-to-one mappings, as is done in previous work (e.g. i-RevNet)  so the novelty is limited.  As stated by R3,  since the paper is a combination of previous methods, the writing should be substantially improved to clarify what the real, new contributions are. The interpretation of the results (e.g. Figure 4) should also be better explained.
Decision: Reject