reviews: 
Comments: This paper focuses on ways that non-differentiability may affect the training of neural networks. Specifically, the paper considers one layer fully-connected networks with ReLU activation functions, lasso and ridge regression. The paper shows three settings under which the training of neural network may have undesirable behavior. Specifically, the convergence rate of non-asymptotic NGD (non-gradient descent, as opposed to GD, gradient descnet), the lasso problem and the edge-of-stability problem.
Strengths: The paper is mostly well-written, both in terms of language and, flow and literature review. It is also sound. The paper may motivate research towards the training of non-differentiable neural networks.
Weaknesses: The main motivation for the work is that the theoretical results for gradient-descent methods on smooth functions may not hold in the case of non-smooth functions (second paragraph of the introduction), even though GD methods are used in non-differentiable cases in practice. This motivation, while true, is not too strong. As the authors acknowledge, it is a common violation in practical machine learning, and its success is undeniable. Therefore, I think the work would benefit significantly from having more significant, large-scale examples, where we see the non-differentiability as a clear problem. Considering in the examples non-asymptotic and constant learning rate settings is also distracting, as this way we can not distinguish if it the asymptotic and diminishing learning rate settings would hold the same problems.

The three specific arguments for drawbacks of non-differentiability are too disconnected and the paper also does not go too deep into each of these problems, and the results are too superficial and lacking a significant novelty. However, I do think each of the problems could be interested on its own, and could deserve more exploration from the authors. Specifically, while the authors point out problems, they do not investigate the impact of obvious solutions: if the problem is the constant learning rate, we can just make it diminishing; if the problem is non-differentiability, for instance from the ReLU layer or the lasso penalty, we can smoothen the function at points.

Finally, even though the work calls attention for three problems within the training of non-differential neural networks, investigating specific solutions, or at least directions for solving the problems raised, would make the paper much stronger. It could also allow for more specific and interesting conclusions than the one presented (which is that attention to the problem is required).

Minor and detailed comments:
- in the abstract, "under-estimate" is ambiguous;
- in the abstract, "unreliable" is also not clear;
- in the preliminaries, the learning rate $\alpha$ should be defined when it appears. the vectors $z_i$ are also not defined, even though we can understad their dimensions for the dimensions of the matrix $\mathbf{Z}$. The response vector is defined bold but appears un-bolded in the expression. Its dimension should also be $N \times 1$, not $P \times 1$.
- in the preliminaries, if it is argued that the differences between subgradient and standard gradient play a key role in the analysis, subgradient must be introduced, defined and discussed.
- the results in section 3 may be too unsurprising, as the authors acknowledge after discussing the experiments whose results appear in figure 1.
- the results in section 4 are more interesting. Nevertheless, the constant learning rate plays a key role in the contributions, and even in the differentiable case, non-convergence can happen with non-diminishing learning rates.
- the results in section 5 are too disconnected from the previous and makes the reading not transition smoothly.
- in section 5, the result cited in the proof holds for decreasing learning rates.
Recommendation: 3: reject, not good enough
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 3 good
Clarity: 3 good