reviews: 
Comments: This paper studies how non-differentiability affects the optimization trajectory of convex non-smooth minimization problems. The analysis comes from three perspectives: the convergence behavior, the algorithm’s ability to reach the optimal point, and the effect of non-differentiability on the phenomenon of the edge of stability. In general, the paper argues that non-differentiability introduces a different dynamic of optimization, and calls for a more fine-grained analysis to deal with non-differentiability.
Strengths: 1. The topic considered in this paper is interesting. Indeed, non-differentiability deviates from traditional regulatory conditions in convex optimization, and it is a mystery why blindly applying SGD on e.g. ReLU neural network can achieve a good performance.

2. The paper is well-written. The topics of the paper are decoupled nicely into three different sections, and each section is written clearly.
Weaknesses: 1. Objectives are too simple and not representative of neural networks. The paper considers a one-layer linear model with an added ReLU activation at the end, which is an extremely simple version of neural networks. It further assumes that the labels $\mathbf{y}$ contain only non-positive entries, which is almost never the case in real neural network training. In particular, the two conditions combined enforces the neural network to almost never fit all training labels.

2. The argument that the GD dynamic differs on differentiable and non-differentiable objectives is not strong. In particular, the contradiction is constructed by running experiments on the simple objectives discussed above, and no rigorous of the whole claim is provided.

3. The paper argues on Page 4 that ”it is consistently observed that the optimal parameter values ... non-differentiability for ReLU activations”. This argument is wrong since zeros of the parameters in the neural networks are not necessarily points at non-differentiability, which depends both on the neural network inputs and the bias.

4. The argument in the last paragraph on Page 4 is also not strong since Tibshirani 2015 only proves the upper bound on the number of steps for subgradient methods. Saying the upper bound of one algorithm is different than the upper bound of the other algorithm is not sufficient to say that the two algorithms have different convergent behavior since the slower convergence rate might be an artifact of proof.

5. The conclusion in Section 4 of the paper does not significantly deviate from previous viewpoints. In summary, the paper shows that the subgradient gradient method does not give sparse solutions for LASSO, which is a known fact. The paper also does not theoretically extend this argument to more complicated objectives such as neural networks.

6. In Section 5, the author argues that EoS have different behavior under non-differentiability, and uses Proposition 4 as a support. This is insufficient since the boundedness of the gradients is not related to either the progressive sharpening or the stabilization of the sharpness right above $2/\alpha$.

7. In the first paragraph of Page 8, the paper tries to use the setting in Eq. 12 to experimentally contradict Ma et al. (2022). However, Eq. 12 deviates a lot from the practical neural network setup and conclusions on Eq. 12 may not directly apply to neural networks.
Recommendation: 3: reject, not good enough
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 2 fair
Clarity: 3 good