Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors propose an alternating minimization framework for training autoencoders and encoder-decoder networks. The central idea is that a single encoder-decoder network can be cast as an alternating minimization problem. Each minimization problem is not convex but is quasi-convex and hence one can use stochastic normalized gradient descent to minimize w.r.t. each variable. This leads to the proposed algorithm called DANTE which simply minimizes w.r.t. each variable using stochastic normalized gradient algorithm to minimize w.r.t. each variable The authors start with this idea and introduce a generalized ReLU which is specified via a subgradient function only whose local quasi-convexity properties are established. They then extend these idea to multi-layer encoder-decoder networks by performing greedy layer-wise training and using the proposed algorithms for training each layer. The ideas are interesting, but I have some concerns regarding this work.

Major comments:

1. When dealing with a 2 layer network where there are 2 matrices W_1, W_2 to optimize over. It is not clear to me why optimizing over W_1 is a quasi-convex optimization problem? The authors seem to use the idea that solving a GLM problem is a quasi-convex optimization problem. However, optimizing w.r.t. W_1 is definitely not a GLM problem, since W_1 undergoes two non-linear transformations one via \phi_1 and another via \phi_2. Could the authors justify why minimizing w.r.t. W_1 is still a quasi-convex optimization problem?

2. Theorem 3.4, 3.5 establish  SLQC properties with generalized RELU activations. This is an interesting result, and useful in its own right. However, it is not clear to me why this result is even relevant here. The main application of this paper is autoencoders, which are functions from R^d -> R^d. However, GLMs are functions from R^d ---> R. So, it is not at all clear to me how Theorem 3.4, 3.5 and eventually 3.6 are useful for the autoencoder problem that the authors care about. Yes they are useful if one was doing 2-layer neural networks for binary classification, but it is not clear to me how they are useful for autoencoder problems.

3. Experimental results for classification are not convincing enough. If, one looks at Table 1. SGD outperforms DANTE on ionosphere dataset and is competent with DANTE on MNIST and USPS. 

4. The results on reconstruction do not show any benefits for DANTE over SGD (Figure 3). I would recommend the authors to rerun these experiments but truncate the iterations early enough. If DANTE has better reconstruction performance than SGD with fewer iterations then that would be a positive result.
review_title: Some interesting ideas. But, not sure if they are applicable to the autoencoder problem and it is not clear if it outperforms SGD.

final_score: rating: 6: Marginally above acceptance threshold
review_content: After reading the rebuttal:

The authors addressed some of my theoretical questions. I think the paper is borderline, leaning towards accept.

I do want to note my other concerns:

I suspect the theoretical results obtained here are somewhat restricted to the least-squares, autoencoder loss.  

And note that the authors show that the proposed algorithm performs comparably to SGD, but not significantly better. The classification result (Table 1) was obtained on the autoencoder features instead of training a classifier on the original inputs. So it is not clear if the proposed algorithm is better for training the classifier, which may be of more interest.

=============================================================

This paper presents an algorithm for training deep neural networks. Instead of computing gradient of all layers and perform updates of all weight parameters at the same time, the authors propose to perform alternating optimization on weights of individual layers. 

The theoretical justification is obtained for single-hidden-layer auto-encoders. Motivated by recent work by Hazan et al 2015, the authors developed the local-quasi-convexity of the objective w.r.t. the hidden layer weights for the generalized RELU activation. As a result, the optimization problem over the single hidden layer can be optimized efficiently using the algorithm of Hazan et al 2015. This itself can be a small, nice contribution.

What concerns me is the extension to multiple layers. Some questions are not clear from section 3.4:
1. Do we still have local-quasi-convexity for the weights of each layer, when there are multiple nonlinear layers above it? A negative answer to this question will somewhat undermine the significance of the single-hidden-layer result.

2. Practically, even if the authors can perform efficient optimization of weights in individual layers, when there are many layers, the alternating optimization nature of the algorithm can possibly result in overall slower convergence. Also, since the proposed algorithm still uses gradient based optimizers for each layer, computing the gradient w.r.t. lower layers (closer to the inputs) are still done by backdrop, which has pretty much the same computational cost of the regular backdrop algorithm for updating all layers at the same time. As a result, I am not sure if the proposed algorithm is on par with / faster than the regular SGD algorithm in actual runtime. In the experiments, the authors plotted the training progress w.r.t. the minibatch iterations, I do not know if the minibatch iteration is a proxy for actual runtime (or number of floating point operations).

3. In the experiments, the authors found the network optimized by the proposed algorithm generalize better than regular SGD. Is this result consistent (across dataset, random initializations, etc), and can the authors elaborate the intuition behind?

review_title: an attempt of new training method for DNNs

final_score: rating: 7: Good paper, accept
review_content: In this paper an alternating optimization approach is explored for training Auto Encoders (AEs).
The authors treat each layer as a generalized linear model, and suggest to use the stochastic normalized GD of [Hazan et al., 2015] as the minimization algorithm in each (alternating) phase.
Then they apply the suggested method to several single layer and multi layer AEs, comparing its performance to standard SGD. The paper suggests an interesting approach and provides experimental evidence for its usefulness, especially for multi-layer AEs.


Some comments on the theoretical part:
-The theoretical part is partly misleading. While it is true that every layer can be treated a generalized linear model, the SLQC property only applies for the last layer.
Regarding the intermediate layers, we may indeed treat them as generalized linear models, but with non-monotone activations, and therefore the SLQC property does not apply.
The authors should mention this point.

-Showing that generalized ReLU is SLQC with a polynomial dependence on the domain is interesting. 

-It will be interesting if the authors can provide an analysis/relate to some theory related to alternating minimization of bi-quasi-convex objectives. Concretely: Is there any known theory for such objectives? What guarantees can we hope to achieve?


The extension to muti-layer AEs makes sense and seems to works quite well in practice.

The experimental part is satisfactory, and seems to be done in a decent manner. 
It will be useful if the authors could relate to the issue of parameter tuning for their algorithm.
Concretely: How sensitive/robust is their approach compared to SGD with respect to hyperparameter misspecification.

review_title: Interesting approach to training Autoencoders

Meta-review: Pros:
+ Interesting alternative algorithm for training autoencoders

Cons:
- Not a lot of practical value because DANTE does not outperform SGD in terms of time or classification performance using autoencoder features.

This is an interesting and well-written paper that doesn't quite meet the threshold for ICLR acceptance. If the authors can find use cases where DANTE has demonstrable advantages over competing training algorithms, I expect the paper would be accepted.

Decision: Reject