Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes an auto-encoder pre-training approach for regularising the neural network parameters, which can be used in many different existing neural models. The proposed approach is build based on the unsupervised auto-encoder pre-training and the orthogonality constraints. A number of classical applications are shown to be improved using the proposed models.



Strengths of the paper:

1. The proposed approach is a general approach, easy to integrate into the existing models.

2. The experimental results demonstrate the gains in accuracy for original and new tasks below for a wide range of applications



Weaknesses of the paper:

1. The writing of the paper needs to be improved. Many typos and informal/improper places can be easily found, e.g

   - Left double quotation marks in latex should be $``$, rather than $”$.

   - Paragraph before Eq (9), "equation 10 " should be "equation 8 ", since equation 10 has no $d_{m+1}{′}$.

   - $L^2$ and $L2$, $L_{rr}$ and $L_{RR}$, are confused in this paper.  
   - CNNs, Autoencoders, and GANs, although are well-known, should cite with the corresponding references.

2. The proposed approach is very simple, which is incremented from (Bansal et al., 2018), hence the contribution is limited.

3. The motivation of the proposed models but not utilize other models for the problem is not clear.

4. My main concern is the organisation of this paper, where the current version is not suitable for general readers in different domains. The preliminary knowledge of the proposed model is unclear. I see at least two key points need to be discussed at the beginning of the paper, i.e. regularisers, and unsupervised auto-encoder pertaining, however, are not well illustrated. 

5. The model section should be organised with subsections, where the differences of the preliminary models and the proposed approaches should be explicitly indicated.

6. What is missing when doing the orthogonality constraint, comparing with other regularisation approaches, should be discussed.

7. Comparing the proposed $L_{RR}$ (Eq. (3)) with Eq. (1) (Bansal et al., 2018), the only difference is the weighting/regularizing of parameters by the input data $d_{m}^{I}$. However, the input space could be arbitrarily in value scales, Eq. (3) could punish the large/sharp values of the input space, which may not be desirable for some tasks where the input spaces are very sparse and discrete.  

Overall, this is good work that uses a simple method to improve the generalising capabilities of many models, but there is a number of weaknesses as indicated above.

*************
Updates: Thanks for the authors' response. However, not of all my concerns are well-addressed. Based on the current methodology and the novelty of this paper, I remain my overall rating of this paper unchanged.


review_title: A good work that uses a simple method to improve the generalising capabilities, but need to be improved.

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: Quality

The paper is well written and mathematically precise. Statements are supported through helpful illustrations and the appendix provides the reader with sufficient information about the experimental setup and evaluations.

Clarity

The approach is clearly motivated and is described in full mathematical detail. The authors also show a highly efficient way to integrate the actually computational very expensive loss into neural networks. The derivation is moderately easy to follow for readers with a background in linear algebra.

Originality

The proposed is, while inspired by previous orthogonalization approaches, a novel idea and its relation to previous work is discussed appropriately.

Significance

Unsupervised pretraining in itself has a large significance for deep learning, even though it lost in popularity due to other approaches that achieved a similar result without the extra preparation phase of the neural network training. It is important and useful to keep the research in this area alive and the authors contributed very valuable knowledge with this paper.

Pros:

-- very promising approach for pretraining of neural networks

-- well written paper with good illustrations

-- large experimental evaluation

Cons

-- derivation only for networks without activation function

-- evaluation very much focused on image tasks

-- Figure 5: legend hardly readable (kind of true for many figures)

-- source code not referenced (might be due to remain anonymous during review)
review_title: An interesting and well described approach for unsupervised pretraining of neural networks

final_score: rating: 3: Clear rejection
review_content: The authors present an approach to pre-training of an ANN which utilizes a purportedly novel approach.  This approach aims to be "data aware" by incorporating the individual data points in the orthogonality constraint in the loss function.

The paper is well-written and his mathematically rigorous.  The mathematics appears to be correct.  It is a concept paper, which I appreciate.

The paper puts the proposed approach in juxtaposition with the greedy, layer-by-layer approach.

The authors show improved performance on a handful of tasks against some baseline tasks.  The baselines were, mostly, well selected with a major exception (see below).

The authors provide some experimental data analysis showing that the features learned by the hidden layers in the proposed approach are more easily interpretable by humans.  Furthermore, the features learned by the baseline features appear to be noisy whereas the features learned by the proposed approach capture the structure seen in the data.  Finally, the authors have some interesting results indicating that the features learned by this approach lead to greater generalization or transferability between tasks/data sets.

My largest critique of this work is the lack of discussion/comparison with the long-standing auto-encoder approaches which can already compute PCA/SVD/orthogonal basis vectors utilizing a single hidden layer.  Are these methods unrelated in some way?  If so, why?  This work should be tied back to these approaches.

Another critique of this work is the lack of future work directions.  The authors offered no questions about the proposed approach which were generated during their line of inquiry.  No analysis of where the proposed approach might be inferior to existing approaches was provided or discussed.
review_title: Review of Reviving Autoencoder Pretraining.

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes to use orthogonal weight constraints for autoencoders. The authors demonstrate that under orthogonal weights (hence invertible), more features could be extracted. The theory is conducted under linear cases while the authors claim it can be applied to more complicated scenarios such as higher dimension and with nonlinearity. The experiments demonstrate the performance of proposed model on classification tasks and generative tasks. Several baselines are compared.

The paper is poorly written. It is full of inconsistent and irrelevant claims. The method is not clarified. All experiments are in low quality. 

+ves: 

+ This paper discusses its connection to several of topics such as mutual information, greedy learning, SVD etc.

 
Concerns: 

- No novelty. Using orthogonal weight regularization has been widely studied.

- The theory does not apply to higher dimensional cases or nonlinear cases. The discussion seems trivial. There are no connection of corresponding theory and the model in the experiments.

- Throughout the paper, the "pretraining" process is not clarified. 

- The authors claims applying their method to GAN but I don't see how they combine their model with it. 

- The classification and generation experiment results are not convincing. In Figure 6, 7, the difference are in range of error bar. In Figure 8, 9, there is no advantage from proposed method.

=====POST-REBUTTAL COMMENTS========

I would like to thank the authors for their response. The authors have clarified the method in their response. I appreciate all the experimental details in the appendix. I tend to agree this is a promising idea and worth explored.

However, this paper clearly cannot be accepted in its current form. The paper is poorly organized and poorly written. The method in the paper needs to be clarified. Their theory goes nowhere and proves nothing. In terms of the experimental results, the authors choose unclear baselines (which they claim state-of-the-art) and report improvement in terms of percentage increase (percentage over percentage). None of this is convincing to me.

I slightly increased my score.




review_title: Lack of novelty

Meta-review: This work presents a practical unsupervised pretraining strategy that does not require layer-wise training stages. Clearly this is an area that has lot of potential and the work seems to head in the right direction. 
However, despite a very positive review, I share the same concerns raised by the remaining 3 reviewers. Better motivation and clarity is needed and, considering the proposed approach, a much more thorough comparison and analysis of the theoretical advantages and guarantees of such an approach.
A main argument is that the method can handle arbitrary networks due to the way it is implemented, it is however not clear how practical that would be and how that would work in presence of non-linearities. 
Experiments require also additional work to be presented with clear and standard baselines, not by presenting SOTA on arbitrary tasks. This seemed to be a main concern of the reviewers.
Decision: Reject