Title: Learning One-hidden-layer Neural Networks with Landscape Design
Authors: Rong Ge
Jason D. Lee
Tengyu Ma
Venue: ICLR 2018
Keywords: theory
non-convex optimization
loss surface

Abstract: We consider the problem of learning a one-hidden-layer neural network: we assume the input x is from Gaussian distribution and the label $y = a \sigma(Bx) + \xi$, where a is a nonnegative vector and  $B$ is a full-rank weight matrix, and $\xi$ is a noise vector. We first give an analytic formula for the population risk of the standard squared loss and demonstrate that it implicitly attempts to decompose a sequence of low-rank tensors simultaneously. 
	
Inspired by the formula, we design a non-convex objective function $G$ whose landscape is guaranteed to have the following properties:	

1. All local minima of $G$ are also global minima.
2. All global minima of $G$ correspond to the ground truth parameters.
3. The value and gradient of $G$ can be estimated using samples.
	
With these properties, stochastic gradient descent on $G$ provably converges to the global minimum and learn the ground-truth parameters. We also prove finite sample complexity results and validate the results by simulations. 

TL;DR: The paper analyzes the optimization landscape of one-hidden-layer neural nets and designs a new objective that provably has no spurious local minimum. 

Review discussion:
[ =========================== REVISION ===============================================================]
I am satisfied with the answers to my questions. The paper still needs some work on clarity, and authors defer the changes to the next version (but as I understood, they did no changes for this paper as of now), which is a bit frustrating.
However I am fine accepting it.
[ ============================== END OF REVISION =====================================================]
This paper concerns with addressing the issue of SGD not converging to the optimal parameters on one hidden layer network for a particular type of data and label (gaussian features, label generated using a particular function that should be learnable with neural net). Authors demonstrate empirically that this particular learning problem is hard for SGD with l2 loss (due to apparently bad local optima) and suggest two ways of addressing it, on top of the known way of dealing with this problem (which is overparameterization).
First is to use a new activation function, the second is by designing a new objective function that has only global optima and which can be efficiently learnt with SGD
Overall the paper is well written. The authors first introduce their suggested loss function and then go into details about what inspired its creation. I do find interesting the formulation of population risk in terms of tensor decomposition, this is insightful
My issues with the paper are as follows:
- The loss function designed seems overly complicated.
On top of that authors notice that to learn with this loss efficiently, much larger batches had to be used. I wonder how applicable this in practice - I frankly didn't see insights here that I can apply to other problems that don't fit into this particular narrowly defined framework
- I do find it somewhat strange that no insight to the actual problem is provided (e.g. it is known empirically but there is no explanation of what actually happens and there is a idea that it is due to local optima), but authors are concerned with developing new loss function that has provable properties about global optima. Since it is all empirical, the first fix (activation function) seems sufficient to me and new loss is very far-fetched.
- It seems that changing activation function from relu to their proposed one fixes the problem without their new loss, so i wonder whether it is a problem with relu itself and may be other activations funcs, like sigmoids will not suffer from the same problem
- No comparison with overparameterization in experiments results is given, which makes me wonder why their method is better.
Minor: fix margins in formula 2.7.
Thanks for the review and comments.
Response to your questions:
--- "empirical fix seems sufficient": We do consider the proposal of the empirical fix as part of the contribution of the paper. It's driven by the theoretical analysis of the squared loss (more intuition below), and it's novel as far as we know.
That said, we think it's also valuable to pursue a provably nice loss function without bad local minima. Note that there is no known method to empirically verify whether a function has no bad local minima, and such statements can only be established by proofs.
Admittedly our first-cut design of the loss function is complicated and sub-optimal in terms of sample complexity, we do hope that our technique can inspire better loss function and model design in the future.
--- "does sigmoid suffer from the same problem?": we did experiment with the sigmoid activation and found that the sigmoid activation function also has bad local minima. We will add this experiment to the next version of the paper. We conjecture that the activation h_2 +h_4 or the activation 1/2 |z|  has no spurious local minima.
--- "actual insights about the landscape": Our insights is that the squared loss objective function is trying to perform an infinite number of tensor decomposition problems simultaneously, and the mixing of all of these problems very likely creates bad local minima, as we empirically observed. The intuition behind the empirical fix is that removing some of these tensor decomposition problems would make the landscape simpler and nicer.
--- "comparison with over-parameterization":  over-parameterization is indeed a powerful way to remove the bad local minima, and it gives models with good prediction.
But it doesn't recover the parameters of the true model because the training parameters space is larger. Our method is guaranteed to recover the true parameters of the model, which in turns guarantees a ``complete" generalization to any unseen examples, even including e.g., adversarial examples or test examples drawn from another distribution. In this sense, our approaches (both the empirical fix and theoretical one) return solutions with stronger guarantees than over-parameterization can provide.
Of course, it's also a very important open problem to understand better other alternatives to landscape design such as over-parametrization.
Thanks for the review again!
We apologize that we didn't know that the paper was expected to be updated. We just added the results for sigmoid that answers the question "does sigmoid suffer from the same problem?" as we claimed in the response before. Please see page 9, figure 2 in the current version.
We will revise the paper with more intuitions/explanations as promised in the previous response as soon as possible.
This paper studies the problem of learning one-hidden layer neural networks and is a theory paper. A well-known problem is that without good initialization, it is not easy to learn the hidden parameters via gradient descent. This paper establishes an interesting connection between least squares population loss and Hermite polynomials. Following from this connection authors propose a new loss function. Interestingly, they are able to show that the loss function globally converges to the hidden weight matrix. Simulations confirm the findings.
Overall, pretty interesting result and solid contribution. The paper also raises good questions for future works.
For instance, is designing alternative loss function useful in practice?
In summary, I recommend acceptance. The paper seems rushed to me so authors should polish up the paper and fix typos.
Two questions:
1) Authors do not require a^* to recover B^*. Is that because B^* is assumed to have unit length rows? If so they should clarify this otherwise it confuses the reader a bit.
2) What can be said about rate of convergence in terms of network parameters? Currently a generic bound is employed which is not very insightful in my opinion.
Thanks for the comments.
Regarding the questions:
1) Yes, since we mostly focus on the ReLU activation, we assume that the rows of B^* have unit norms. For ReLU activation, the row norms are inherently unidentifiable.
2) The technical version of the landscape analysis in Theorem B.1 specifies the precise dependencies of the landscape properties on the dimension, etc. To get a convergence rate, one can combine our Theorem B.1 with an analysis of gradient descent or other algorithms on non-convex functions. The best-known analysis for SGD in Ge et al. 2015 does not specify the precise polynomial dependencies. Since developing stochastic algorithms (beyond SGD) with lower iteration complexity is an active area of research, the best-known convergence rate is constantly changing.
This paper proposes a tensor factorization-type method for learning one hidden-layer neural network. The most interesting part is the Hermite polynomial expansion of the activation function. Such a decomposition allows them to convert the population risk function as a fourth-order orthogonal tensor factorization problem. They further redesign a new formulation for the tensor decomposition problem, and show that the new formulation enjoys the nice strict saddle properties as shown in Ge et al. 2015. At last, they also establish the sample complexity for recovery.
The organization and presentation of the paper need some improvement.
For example, the authors defer many technical details. To make the paper accessible to the readers, they could provide more intuitions in the first 9 pages.
There are also some typos: For example, the dimension of a is inconsistent. In the abstract, a is an m-dimensional vector, and on Page 2, a is a d-dimensional vector. On Page 8, P(B) should be a degree-4 polynomial of B.
The paper does not contains any experimental results on real data.
Thanks for the comments. We will add more intuitions in the paper and fix the typos.
The high-level intuition is that the squared loss objective function is trying to perform an infinite number of tensor decomposition problems simultaneously and the mixing of all of these problems very likely creates bad local minima, as we empirically observed.
Thus we design an objective function that selects only two of these tensor decompositions problems, which empirically removes the bad local minima.
Finally, we design an objective function that resembles more a 4th order tensor decomposition objective in Ge et al. (2015), which are known to be good.

Meta-review:
role: PC
data: comment: I recommend acceptance based on the reviews. The paper makes novel contributions to learning one-hidden layer neural networks and designing new objective function with no bad local optima.

 There is one point that the paper is missing. It only mentions Janzamin et al in the passing. Janzamin et al propose using score function framework for designing alternative objective function. For the case of Gaussian input that this paper considers, the score function reduces to Hermite polynomials. Lack of discussion about this connection is weird. There should be proper acknowledgement of prior work. Also missing are some of the key papers on tensor decomposition and its analysis

I think there are enough contributions in the paper for acceptance irrespective of the above aspect.  

Decision: Accept (Poster)