Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: N/A
review_title: N/A

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper presents an approach to non-linear kernel dimensionality reduction with a trace norm regularizer in the feature space. The authors proposed an iterative minimization approach in order to obtain a local optimum of a relaxed problem. 
The paper contains errors and the experimental evaluation is not convincing. Only old techniques are compared against in very toy datasets. 

The authors claim state-of-the-art, however, the oil dataset is not a real benchmark, and the comparisons are to very old approaches. 
The experimental evaluation should demonstrate robustness to more complex noise and outliers, as this was one of the motivations in the introduction.

The authors do not address the out-of-sample problem. This is a problem of kernel-based methods vs LVMs, and thus should be address here.


The paper contains errors:

- The last paragraph of section 1 says that this paper proposes a closed form solution to robust KPCA. This is simply wrong, as the proposed approach consists of iteratively solving iterativey a set of closed form updates  and Levenberg-Marquard optimizationd. This is not any more closed form!

- In the same paragraph (and later in the text) the authors claim that the proposed approach can be trivially generalized to incorporate other cost functions. This is not true, as in general there will be no more inner loop closed form updates and the authors will need to solve a much more complex optimization problem. 

- The third paragraph of section 2 claims that this paper presents a novel energy minimization framework to solve problems of the general form of eq. (2). However, this is not what the authors solve at the end. They solve a different problem that has been subject to at least two relaxations. It is not clear how solving for a local optima of this double relaxed problem is related to the original problem they want to solve. 

- The paper says that Geiger et al defined non linearities on a latent space of pre-defined dimensionality. This is wrong. This paper discovers the dimensionality of the latent space by means of a regularizer that encourages the singular values to be sparse. Thus, it does not have a fixed dimensionality, the latent space is just bounded to be smaller or equal than the dimensionality of the original space. 


It is not clear to me why the author say for LVMs such as GPLVM that "the latent space is learned a priority with clean training data". One can use different noise models within the GP framework. Furthermore, the proposed approach assumes Gaussian noise (see eq. 6), which is also the trivial case for GP-based LVMs.  


It is not clear what the authors mean in the paper by "pre-training" or saying that techniques do not have a training phase. KPCA is trained via a closed-form update, but there is still training.  

review_title: This paper presents an approach to non-linear kernel dimensionality reduction with a trace norm regularizer in the feature space. The authors proposed an iterative minimization approach in order to obtain a local optimum of a relaxed problem.  The paper contains errors and the experimental evaluation is not convincing. Only old techniques are compared against in very toy datasets. 

final_score: rating: 3: Clear rejection
review_content: This paper considers an alternate formulation of Kernel PCA with rank constraints incorporated as a regularization term in the objective. The writing is not clear. The focus keeps shifting from estimating “causal factors”, to nonlinear dimensionality reduction to Kernel PCA to ill-posed inverse problems. The problem reformulation of Kernel PCA uses somewhat standard tricks and it is not clear what are the advantages of the proposed approach over the existing methods as there is no theoretical analysis of the overall approach or empirical comparison with existing state-of-the-art.  

- Not sure what the authors mean by “causal factors”. There is a reference to it in Abstract and in Problem formulation on page 3 without any definition/discussion.

- In KPCA, I am not sure why one is interested in step (iii) outlined on page 2 of finding a pre-image for each

- Authors outline two key disadvantages of the existing KPCA approach. The first one, that of low-dimensional manifold assumption not holding exactly, has received lots of attention in the machine learning literature. It is common to assume that the data lies near a low-dimensional manifold rather than on a low-dimensional manifold. Second disadvantage is somewhat unclear as finding “a data point (pre-image) corresponding to each projection in the input space” is not a standard step in KPCA. 

- On page 3, you never define $\mathcal{X} \times N$, $\mathcal{Y} \times N$, $\mathcal{H} \times N$. Clearly, they cannot be cartesian products. I have to assume that notation somehow implies N-tuples. 

- On page 3, Section 2, $\mathcal{X}$ and $\mathcal{Y}$ are sets. What do you mean by $\mathcal{Y} \ll \mathcal{X}$

- On page 5, $\mathcal{S}^n$ is never defined. 

- Experiments: None of the standard algorithms for matrix completion such as OptSpace or SVT were considered 

- Experiments: There is no comparison with alternate existing approaches for Non-rigid structure from motion.  

- Proof of the main result Theorem 3.1: To get from (16) to (17) using the Holder inequality (as stated) one would end up with a term that involves sum of fourth powers of weights w_{ij}. Why would they equal to one using the orthonormal constraints? It would be useful to give more details here, as I don’t see how the argument goes through at this point. 
review_title: Not clear

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes a nonlinear regularizer for solving ill-posed inverse problems. The latent variables (or causal factors) corresponding to the observed data are assumed to lie near a low dimensional subspace in an RKHS induced by a predetermined kernel. The proposed regularizer can be seen as an extension of the linear low-rank assumption on the latent factors. A nuclear norm penalty on the Cholesky factor of the kernel matrix is used as a relaxation for the dimensionality of the subspace. Empirical results are reported on two tasks involving linear inverse problems -- missing feature imputation, and estimating non-rigid 3D structures from a sequence of 2D orthographic projections -- and the proposed method is shown to outperform linear low-rank regularizer. 

The clarity of the paper has scope for improvement (particularly, Introduction) - the back and forth b/w dimensionality reduction techniques and inverse problems is confusing at times. Clearly defining the ill-posed inverse problem first and then motivating the need for a regularizer (which brings dimensionality reduction techniques into the picture) may be a more clear flow in my opinion. 

The motivation behind relaxation of rank() in Eq 1 to nuclear-norm in Eq 2 is not clear to me in this setting. The relaxation does not yield a convex problem over S,C (Eq 5) and also increases the computations (Algo 2 needs to do full SVD of K(S) every time). The authors should discuss pros/cons over the alternate approach that fixes the rank of C (which can be selected using cross-validation, in the same way as $\tau$ is selected), leaving just the first two terms in Eq 5. For this simpler objective, an interesting question to ask would be -- are there kernel functions for which it can solved in a scalable manner? 

The proposed alternating optimization approach in the current form is computationally intensive and seems hard to scale to even moderate sized data -- in every iteration one needs to compute the kernel matrix over S and perform full SVD over the kernel matrix (Algo 2). Empirical evaluations are also not extensive -- (i) the dataset used for feature imputation is old and non-standard, (ii) for structure estimation from motion on CMU dataset, the paper only compares with linear low-rank regularization, (iii) there is no comment/study on the convergence of the alternating procedure (Algo 1). 






review_title: Lacking in several aspects; limited novelty

Meta-review: There is complete consensus among the reviewers that the KPCA formulation in this paper needs better motivation; that the paper has technical errors, and the experimental evaluation is not convincing. As such the paper is not up to ICLR standards. The authors are encouraged to revise the paper based on feedback from the reviews.
Decision: Reject