Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper analyzed the asymptotic training error of a simple regression model trained on the random features for a noisy autoencoding task and proved that a mixture of nonlinearities can outperform the best single nonlinearity on such tasks.

Comments:
1.The paper is well written and provides sound derivation for the theories.

2. Since this area is out of my expertise, I’m not sure whether merely extending the work of Pennington & Worah (2017) to non-Gaussian data distributions is significant enough or not.

3. Except for Fig 4, the other figures seem out of the context. There is no explanation for the purpose of those figures in the main contents. It is a bit hard for the audience to figure out what to look at in the figures or what the figures try to prove. 

4. In “..., and our analysis actually extends to general such distributions, ... ”, “general” should be “generalize”.

5. In “And whether these products generate a medical diagnosis or a navigation decision or some other important output, ..”, “whether” should be “no matter”.

6. “..., they may not be large in comparison to the number of constraints they are designed asked satisfy.” should be “...  they are designed to satisfy”.

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper investigates the asymptotic spectral density of a random feature model F(Wx + B).  This is an extension of existing result that analyzed a model without the bias term, i.e., F(WX). This extension requires a modification of the proof technique. In addition to that, it analyzed a mixture of linear and non-linear activation functions, and show that mixture is better than single nonlinearity in terms of expected training error for ridge regression estimators.

Pros:
- This paper investigates an interesting problem and it successfully extends the existing work. The theoretical curve well matches the simulated curve.
- The finding that mixture of nonlinearities gives better expected training error is interesting.

Cons:
- The extension to the model with bias seems a bit incremental. In practice, we may consider an input with additional constant feature, X <- [X,1], to deal with both models in a unified manner. There should be more discussion about why this kind of trivial argument cannot be applied in the analysis.
- The effect of mixture of activation functions is investigated in the "training error," but I don't see much significance on investigating the training error thoroughly. Instead, people are interested in the test error. I guess there does not appear such a trade-off for the test error and the linear activation function would be always better because the true function is the linear model. Hence, more expositions about why the training error is investigated should be provided.

More minor comment:
- I guess the definition of Etrain  (Eq.(17)) requires an expectation with respect to the training data.
- Assumptions of the activation function f should be provided; is it just assumed to be differentiable?, ReLU is included?
- The definition of G(\gamma) in page 6 had better to be consistent to that in previous pages.


review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: In this work, the authors focus on the high-dimensional regime in which both the dataset size and the number of features tend to infinity. They analyze the performance of a simple regression model trained on the random features and revealed several interesting and important observations. I think it is a solid work and vote for acceptance. 
Pros: 
(1) This paper has a solid theoretical foundation. Although I have not checked in detail, I think the deduction is clear and the contribution is well-established.
(2) It extends some traditional bounds to more general cases. I think it will provide useful guidance to real applications, such as the network design in deep learning.
(3) The authors have explained the results in a clear way. Thus, it will benefit the following readers and give deep insights about the related research areas.
Minor comments:
(1) I think some assumptions should be explained. For example, why the authors focus only on linear model. Due to the simplicity or the requirement from real applications?
(2) More experimental results on large data sets should be added to validate the effectiveness.
review_title: Official Blind Review #1

Meta-review: In this work, the authors focus on the high-dimensional regime in which both the dataset size and the number of features tend to infinity. They analyze the performance of a simple regression model trained on the random features and revealed several interesting and important observations.

Unfortunately, the reviewers could not reach a consensus as to whether this paper had sufficient novelty to merit acceptance at this time. Incorporating their feedback would move the paper closer towards the acceptance threshold.
Decision: Reject