Kati8/ICLR_2023_2025 huggingface dataset:
authors: Siddharth Krishna Kumar
reviews: 
Comments: This paper discusses three ways that non-differentiability impacts network training: by affecting the convergence rate, causing unreliable LASSO solutions, and breaking results from edge-of-stability analysis.
Strengths: 1. The paper is clearly structured.
2. Non differentiability is an interesting and important component of modern neural networks.
Weaknesses: 1. The authors imply that small VGG16 network weights are close to the "domain of non-differentiability." This is flawed, as weights and activations are not the same -- if for example the bias of the linear layer is a very large constant vector, then we have differentiability even from small weights.
2. The authors generally frame their analysis as "practical networks do not satisfy theoretical ssumption (e.g., L-smooth loss functions), therefore they don't apply." This is a fundamental misunderstanding of the importance of theory. Theory tends to study a "toy" scenario, and derive results that are insightful even if they don't directly apply.
3. Generally, the paper consists of a few loosely connected ideas, and does not provide sufficient analysis for a conference at ICLR's level.
4. Presented theoretical results are very simple.
Recommendation: 3: reject, not good enough
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 2 fair
Clarity: 2 fair
reviews: 
Comments: This paper tries to investigate how non-differentiability affects machine learning algorithms. Several issues are raised when non-differentiability is present, including convergence rate under smooth conditions, convergence for LASSO, and reason for the edge of stability. Dozens of existing works are used to explain these issues.
Strengths: This paper studies non-differentiability in deep learning, which is an important issue.
Weaknesses: 1. I am really confused about what the algorithm "NGD" is. The authors say it is "Non-differentiable gradient descents" without providing any concrete definition (how is $\tilde{\nabla}$ defined? Is it subgradient? If it depends on the problem, what is the concrete definition of it in Figure 1?). This makes it hard to evaluate this paper, as I do not know if it is the version of gradient applied to deep learning practice. Also, I would like to remind the authors that "NGD" is usually referred to as "Natural Gradient Descent", and I suggest the authors change the name.

2. The presentation of this paper needs to be polished. As an example, it is quite weird that the left subfigure of Figure 1 is not in a log scale, since there are only two or three points in the figure much above 0. Also, the subfigure even appears to be a screenshot with inconsistent color with the background. Also, on page 5, I do not think it is a proper way to use discussions in a forum to support claims.

3. Is Prop 1 correct? I do not check the reference, but at first glance, this appears to be strange, as intuitively it is possible that at $\theta_0$ the gradient is quite large and the iterations jump outside the unit ball.

4. This paper extensively cites conclusions from other papers without any further explanation. For example, 
> Since the loss function described in equation (6) satisfies the Lipschitz condition (see equation (5)) with L = λ1, we have from Boyd et al. (2003) (Section 2 on the Constant Step Size) that...

5. Many connections are drawn based on imprecise observations/statements. For example, in the texts below,
> we know from Figure 2 that the optimal weights lies close to points of non-differentiability
How "close" is needed to make the Taylor expansion invalid?



**Typos**

1. The first line under Eq. (3): y is a $N\times 1$ ...?

2. Proof of Proposition 1: missing "."
Recommendation: 3: reject, not good enough
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 1 poor
Clarity: 1 poor