Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes and analyzes a new loss function for linear autoencoders (LAEs) whose minima directly recover the principal components of the data. The core idea is to simultaneously solve a set of MSE LAE problems with tied weights and increasingly stringent masks on the encoder/decoder matrices. My intuition is that the weights that touch every subproblem are the most motivated to find the largest principal component, the weights that touch all but one find the next largest, and so forth; I found this idea clever and elegant.

That said, I lean towards rejection, because the paper does not do a very good job of demonstrating the practical or theoretical utility of this approach. As I see it, there are two main claims that one could make to motivate this work:
1. This is a practical algorithm for doing PCA.
2. This is a step towards better understanding (and perhaps improving) nonlinear autoencoders, which do things that PCA can't.
Claim (2) might be compelling, but the authors do not make it, and it isn't self evident.

I do not find claim (1) convincing on the basis of the evidence presented. PCA is an extremely well studied problem, with lots of good solutions such as randomized SVD (Halko et al., 2009). A possible advantage of using LAEs to address the PCA problem is that they play nicely with SGD, but again, the claim that the SGD-LAE approach is superior to, say, randomized SVD on a data subsample requires evidence. Also, even if one buys the claim that LAEs are a good way to solve PCA, one can always recover the eigenvectors/eigenvalues by a final decomposition step; the authors claim that an advantage of their approach is that it does not require such "bells and whistles", but this seems like a pretty minor consideration; implementing the proposed loss function seems at least as complicated as making a call to an SVD solver, and it's hard for me to imagine a situation where the cost of that final SVD isn't dominated by the cost of solving the MSE LAE problem.

In summary, I think this paper proposes a clever and elegant solution to a problem that doesn't seem to be very important. I can't recommend acceptance unless the authors can come up with a stronger argument for why it's not just interesting but also useful.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper proposes a new loss function for performing principal component analysis (PCA) using linear autoencoders (LAEs). With this new loss function, the decoder weights of LAEs can eventually converge to the exact ordered unnormalized eigenvectors of the sample covariance matrix. The main contribution is to add the identifiability of principal components in PCA using LAEs and. Two empirical experiments were done to show the effectiveness of proposed loss function on one synthetic dataset and the MNIST dataset. 
Overall, this paper provides a nontrivial contribution for performing principal component analysis (PCA) using linear autoencoders (LAEs), with this new novel loss function. This paper is well presented.
There are some issues to be addressed:
1. The output matrix is constrained to be the same size of the input, which is scarcely seen in practical applications.
2. Literature on (denoising) auto-encoder can be reviewed more thoroughly.
3. Comparison with state-of-the-art auto-encoder can be provided to demonstrate the effectiveness of the proposed algorithm.
4. It is better to explain the meaning of each variable when it first appears, e.g., , the projection matrices A and B, and Variable A* in theorem 2.
5. In the experiment part, in both the Synthetic Data or MNIST, the size of each data set is relatively small. It's better to add experimental results on big data sets with larger dimension.
6. In order to better show the effectiveness of the new loss function, you can add some comparative test for different choice of compressed dimension p.
7. There are some typos, such as ‘faila’ in the second line of the second paragraph in the INTRODUCTION. 

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper proposes a new loss function to compute the exact ordered eigenvectors of a dataset. The loss is motivated from the idea of computing the eigenvectors sequentially. However doing so would be computationally expensive, and the authors show that the loss function they propose (sum of sequential losses) has the same order (constant less than 7) of computational complexity as using the squared loss. A proof of the correctness of the algorithm is given, along with experiments to verify its performance.

The loss function proposed in the paper is useful, and the decomposition in Lemma 1 shows that it is not computationally expensive. While the writing of the proofs of the theorems is clear, I find it hard to understand the flow of the paper. It would help if the authors could summarize the argument of the proof at the start of Sec 4. Along a similar vein, it would also help if the authors could describe in words the significance / claim of every theorem.

The repetition in stating the theorems can be avoided. The main result (Theorem 2) is stated twice.
review_title: Official Blind Review #2

Meta-review: Quoting from R3: "This paper proposes and analyzes a new loss function for linear autoencoders (LAEs) whose minima directly recover the principal components of the data. The core idea is to simultaneously solve a set of MSE LAE problems with tied weights and increasingly stringent masks on the encoder/decoder matrices."  With two weak acceptance recommendations and a recommendation for rejection, this paper is borderline in terms of its scores.

The approach and idea are interesting.  The main shortcoming of the paper, as highlighted by the reviewers, is that the approach and theoretical analysis are not properly motivated to solve an actual problem faced in real-world data.  The approach does not provide a better algorithm for recovering the eigenvectors of the data, nor is it proposed as part of a learning framework to solve a real-world problem.  Experiments are shown on synthetic data and MNIST.  As a stand-alone theoretical result, it leaves open questions as to the proposed utility.
Decision: Reject