Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies the generalization in an analytically solvable setup: a two-layer network with random features on the first layer and learnable features on the second layer, under squared loss and logistic loss. The authors derive asymptotic expressions for the train and test errors of the above model trained on a dataset with block covariance matrix with tunable saliency (for input) and noise (for mapping). In experiments, the authors studied and compared how the data structure and loss function affect the double descent, generalization, and also test it in the case of MNIST and CIFAR10 datasets.
main_review: On the theoretical side, I did not check the proof of the theorems, but I believe that the derivation is correct and reasonable. I think the theoretical contribution is novel. For clarity, it would make the paper much more insightful if the authors can provide clear intuition about the cause of the double descent using the main analytical result, and why in noiseless scenario with logistic loss this phenomenon disappear.

On the experiment side, I think the experiment is well-conducted. 

The authors note that their results appear to be in stark conflict with those of (Hui and Belkin, 2020), which train on large-scale models. Could the authors address the following question: to which extend can the results in this two-layer network generalize to larger models and more complex datasets?

Overall, I think the paper is well-written, novel, and makes a step that offers interesting technique to address the generalization of NNs on structured datasets.
limitations_and_societal_impact: The authors address the limitation that their analytical results are based on non-rigorous tools from statistical physics. It would be better if the authors can also discuss how their results and insights can generalize over more complex models and datasets.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper used the replica method to derive the high dimensional asymptotics of the test error of random features model in the strong and weak features model with square loss and logistic loss. The observations drawn from plotting the analytical prediction are the following: 1. When the parameters vector of the target function is aligned with the covariance structure of the covariates, the generalization error is smaller (given fixed N/D, P/D, lambda, loss function). 2. In the aligned settings, the test error of using the logistic loss is smaller than that of using the square loss (in a specific setting). This paper also performed experiments on parity MNIST and CIFAR10(2), and the experimental result coincides with the results with idealized assumptions. 
main_review: Overall, this is a decent result. The theoretical part which is the replica calculation seems like a straightforward generalization of previous results (it didn’t appear in the literature though). The strong and weak features model setup in the proportional limit has not been fully investigated before, so drawing some useful conclusions from the analytical prediction would be very interesting. 

An issue with this paper is that it drew conclusions based on a very specific choice of parameters and specific settings. It is not clear whether this is representative enough to lead to the conclusions. 
1) In the figures, this paper just chooses N = D. It is not clear whether the conclusion still holds for some larger N/D (in practice, the sample size is several times larger than the dimension). In the appendix, there seem to be more experiments for other N/D, but the author didn’t use these experimental results to support the conclusions drawn in this paper. The reviewer suggests the author draw necessary figures for other N/D to demonstrate that the conclusions consistently hold. 
2) This paper just fixes lambda to be a small parameter like 10^{-3} or 10^{-4}. However, it is not clear whether one should compare logistic/square loss by choosing a vanishing regularization. Another scenario would be to choose the optimal lambda that minimizes the test error and then compare the logistic/square loss. 

Other issues: 
1. In Eq (8), please explain what is x_i. Is it the eigenvectors of top PCs, or the coordinates of the covariates? 
limitations_and_societal_impact: In general, the reviewer suggests the authors consider more scenarios and perform a more thorough investigation to support their conclusions.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors study the loss achieved by random feature models that learns from simple (but structured) data.
The main result consists in the comparison of two losses (square and logistic) under three settings in the data structure (misaligned, isotropic and aligned that roughly represent the difficulty of the task).
The paper is well written and presents a thorough analysis of the related works. 
main_review: The paper uses standard and new tools of statistical physics to address the problem summarized above.

The comparison between the two losses is very interesting. A direct comparison between the results would be unfair and the authors make nice considerations to convince of the benefits of logistic loss. Considerations that are confirmed in the experimental setup on real data.

In the end, the authors conclude that their results may be limited to the random feature models as there is a contrast with the results of ref.[51] about DNNs. I do not understand why the authors did not include this analysis in their paper using the framework of ref.[32]. I strongly believe that it would be more natural to answer these questions in this paper instead of waiting for future works. 
add 
In the end, the role of the data structure seems to play a minor role in the paper (and in the results). Apart from a shift of the curves and a counter-intuitive result on the regularization effect of misalignment in the square loss, I do not see many considerations on this aspect.
Is it because the model is still too simple? The author should extend their discussion and maybe analysis to highlight the importance of data structure. 

My impression is that, although it is pointing in the right direction, the paper needs more results to meet the acceptance bar.

################# after rebuttal #################

Thanks to the authors for the clarification and the promised changes.

The authors are indeed right about ref.32, the analysis I suggest is more complex than what appeared in the first place. Nevertheless, it may be worth integrating their analytical result with some empirical ones and give a partial answer to the questions they asked. I still believe it is more natural to provide an answer (although partial) in this paper, instead of waiting for future works. 

limitations_and_societal_impact: It is a simple yet instructive model.
No societal impact is foreseen.
review_title: null

Meta-review: The paper presents an interesting analytical result on generalization which all three reviewers considered worthwhile.
Based on the given scores (5,6,7) and the fact that the machine learning field would likely benefit from some more theoretical contributions I recommend acceptance as a poster presentation. 
Decision: Accept (Poster)