Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: [ =========================== REVISION ===============================================================]
I am satisfied with the answers to my questions. The paper still needs some work on clarity, and authors defer the changes to the next version (but as I understood, they did no changes for this paper as of now), which is a bit frustrating. However I am fine accepting it.
[ ============================== END OF REVISION =====================================================]

This paper concerns with addressing the issue of SGD not converging to the optimal parameters on one hidden layer network for a particular type of data and label (gaussian features, label generated using a particular function that should be learnable with neural net). Authors demonstrate empirically that this particular learning problem is hard for SGD with l2 loss (due to apparently bad local optima) and suggest two ways of addressing it, on top of the known way of dealing with this problem (which is overparameterization). First is to use a new activation function, the second is by designing a new objective function that has only global optima and which can be efficiently learnt with SGD

Overall the paper is well written. The authors first introduce their suggested loss function and then go into details about what inspired its creation. I do find interesting the formulation of population risk in terms of tensor decomposition, this is insightful

My issues with the paper are as follows:
- The loss function designed seems overly complicated. On top of that authors notice that to learn with this loss efficiently, much larger batches had to be used. I wonder how applicable this in practice - I frankly didn't see insights here that I can apply to other problems that don't fit into this particular narrowly defined framework
- I do find it somewhat strange that no insight to the actual problem is provided (e.g. it is known empirically but there is no explanation of what actually happens and there is a idea that it is due to local optima), but authors are concerned with developing new loss function that has provable properties about global optima. Since it is all empirical, the first fix (activation function) seems sufficient to me and new loss is very far-fetched. 
- It seems that changing activation function from relu to their proposed one fixes the problem without their new loss, so i wonder whether it is a problem with relu itself and may be other activations funcs, like sigmoids will not suffer from the same problem
- No comparison with overparameterization in experiments results is given, which makes me wonder why their method is better.

Minor: fix margins in formula 2.7. 



review_title: Overall an interesting heuristic approach to solving SGD problems on one particular type of data, with limited practical applications

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a tensor factorization-type method for learning one hidden-layer neural network. The most interesting part is the Hermite polynomial expansion of the activation function. Such a decomposition allows them to convert the population risk function as a fourth-order orthogonal tensor factorization problem. They further redesign a new formulation for the tensor decomposition problem, and show that the new formulation enjoys the nice strict saddle properties as shown in Ge et al. 2015. At last, they also establish the sample complexity for recovery.

The organization and presentation of the paper need some improvement. For example, the authors defer many technical details. To make the paper accessible to the readers, they could provide more intuitions in the first 9 pages.

There are also some typos: For example, the dimension of a is inconsistent. In the abstract, a is an m-dimensional vector, and on Page 2, a is a d-dimensional vector. On Page 8, P(B) should be a degree-4 polynomial of B.

The paper does not contains any experimental results on real data.
review_title: An interesting tensor factorization-type method for learning one hidden-layer neural network

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: This paper studies the problem of learning one-hidden layer neural networks and is a theory paper. A well-known problem is that without good initialization, it is not easy to learn the hidden parameters via gradient descent. This paper establishes an interesting connection between least squares population loss and Hermite polynomials. Following from this connection authors propose a new loss function. Interestingly, they are able to show that the loss function globally converges to the hidden weight matrix. Simulations confirm the findings.

Overall, pretty interesting result and solid contribution. The paper also raises good questions for future works. For instance, is designing alternative loss function useful in practice? In summary, I recommend acceptance. The paper seems rushed to me so authors should polish up the paper and fix typos.

Two questions:
1) Authors do not require a^* to recover B^*. Is that because B^* is assumed to have unit length rows? If so they should clarify this otherwise it confuses the reader a bit.
2) What can be said about rate of convergence in terms of network parameters? Currently a generic bound is employed which is not very insightful in my opinion.


review_title: Interesting paper and solid contribution: Accept.

Meta-review: I recommend acceptance based on the reviews. The paper makes novel contributions to learning one-hidden layer neural networks and designing new objective function with no bad local optima.

 There is one point that the paper is missing. It only mentions Janzamin et al in the passing. Janzamin et al propose using score function framework for designing alternative objective function. For the case of Gaussian input that this paper considers, the score function reduces to Hermite polynomials. Lack of discussion about this connection is weird. There should be proper acknowledgement of prior work. Also missing are some of the key papers on tensor decomposition and its analysis

I think there are enough contributions in the paper for acceptance irrespective of the above aspect.  
Decision: Accept (Poster)