Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors study the theoretical properties of diagonally grouped linear neural networks, and they show that gradient descent over the squared regression loss, without any explicit regularization, biases towards solutions with a group sparsity structure. They conduct some experiments to verify their conclusion.  My main concern is the implications of these results on the modern neural networks are unclear. The reason is that the relationship between the diagonally grouped linear neural networks and the widely used deep neural networks in practice are unclear. Thus, the contribution of this paper is limited.
strength_and_weaknesses: Strength:

1. This paper is well written and easy to read. 

2. Some theoretical results in this paper are interesting. 


Weaknesses:

1. As we know deep learning theory studies aim at understanding the modern neural networks in practice. The authors need to discuss the relationship between their results, such as the group sparsity structure,  and the observations of deep neural networks in practice.

2. In this paper, the authors focus on the diagonally grouped linear neural network, compared with the existing work, the authors need to explain more on the relationships between their structure and the ones in the modern neural networks. The grouped CNNs the authors mentioned are not the widely used neural network in practice. The authors are also recommended to specify the relationship between the structure of their model and grouped CNNs, such as where one can be reduced to another by using some specific settings.
The above limitation makes the contribution of this paper limited. 

3. Can the authors show some experimental results to demonstrate that their findings also exist in the neural networks in real applications.


clarity,_quality,_novelty_and_reproducibility: Please see my comments above. 
summary_of_the_review:  My main concern is the implications of these results on the modern neural networks are unclear, which weakens the contributions of this paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper studies the implicit bias of gradient descent on diagonal linear networks.
It has been lately shown that parametrizing $x$ as $u \odot v$ in $y \approx A (u \odot v)$ induces sparsity in the gradient flow iterates on the least squares loss.
In the paper a special architecture is proposed so that gradient descent imposes **group sparsity** on the weights x$.
The architecture is a so-called *diagonally-grouped linear neural network* (DGLNN)
An analysis is provided under the restrictive assumption of orthogonal design (Theorem 2).
For $L$ groups, the proposed parametrization uses two set of variables: scalar ones $u_l$ encoding the magnitude, and vector ones $v_l$ encoding the direction.

Gradient flow of the overparametrized objective (Eq 3) is analyzed.
Section 4 is devoted to a practical version of the gradient flow, gradient descent with weight normalization.
Guaranties similar to the sparse recovery literature are derived under subGaussianity of the noise (Corollary 1)
strength_and_weaknesses: Strengths:
- this is a very active area of research, the contribution is relevant
- even though the assumptions are strong, there are not so different from classical ones in sparse recovery literature and are a first step to a better understanding of the phenomenon at stake

Weaknesses:
- the assumption is very strong and unlikely to hold in practice. Testing the behavior of the proposed algorithm on real data to assess the sensitivity to this assumption would be of great interest.
- comparison with explicit regularization methods is also expected. Implicit bias of mirror descent [2] for group elastic net could also be studied (if the elastic net regularization parameter is sufficiently low, the solution is the same as for group Lasso [2]), as well as iterative regularization methods based on primal dual procedures that can handle non strongly convex bias [4].
clarity,_quality,_novelty_and_reproducibility: ## Experiments:
- The authors use Rademacher random variables for X. It seems to me that using Gaussian entries is more standard. Random Gaussian matrices are also supposed to satisfy the Assumptions of the paper. For the rebuttal I'd like to see the same experiment with a random Gaussian matrix.
- "The entries on the support are all 1". Can the author provide the Signal to Noise Ratio, $||Xw^\star||/||\xi||$? That will be much more informative than saying $\sigma=0.5$.

## References:
- The following preprint would be relevant (it appeared on 31 August 2022, so it is of course not a flaw that it was not included in the paper): [1]

## Clarification.
- The authors state: "This reparameterization is equivalent to a diagonal linear network, as shown in Figure 1a." > For me Figure 1a use the parametrization $w = u \odot v$, that induces the same bias, but is not equal to the w= $u^N - v^N$ one. Where is $N$ in Figure 1a ? Where is the substraction?

## Maths:
- In section 3.2, $u^\star_{min}$ is potentially 0. Then in theorem 3, you need both $\delta$'s to be equal to 0?


[1] *Incremental Learning in Diagonal Linear Networks*, R. Berthier, https://arxiv.org/abs/2208.14673
[2] *Exact Regularization of Polyhedral Norms*, SIAMOPT 2010, F. Schopfer
[3] *Characterizing implicit bias in terms of optimization geometry*, ICML 2018, Gunasekar, Lee, Soudry, Srebro.
[4] *Iterative regularization for convex regularizers*, AISTATS 2021, C Molinari, M Massias, L Rosasco, S Villa
summary_of_the_review: Interesting idea and result, but valdiation fails to compare to the wealth of existing methods for sparse recovery, including implicit ones.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper studies "implicit regularization", a phenomena in gradient-based optimization algorithms in which the local minima of unregularized cost functions contain properties that one would expect only by adding a regularization term, e.g., sparsity being the most popular one. More precisely, they proposed a reparametrization in which the solution shows "group sparsity" structure, which is often obtained by having a elastic net regularization.
strength_and_weaknesses: Strength
======

- Excellent contextualization of the problem being investigated.
- Thorough analysis of the proposed reparametrization and the reasons behind its implicit regularization structure.
- Great numerical analysis showcasing the theorectical development.
- The analysis in Theorem 1 showing that the gradient dynamics in DGLNN cannot be emulated by mirror descent is interesting and appreciated.

Weaknesses
=========

- No major weaknesses.

Minor Comments
=============

- If I were the authors I would be more specific in the title of the paper.  Maybe find a way to introduce "Neural Networks" in the title as it is the model being studied here?
clarity,_quality,_novelty_and_reproducibility: Well-written and well-motivated paper.  Novelty is significant as the authors discover a particular neural net reparametrization that leads to group sparsity implicit regularization.  Results are reproducible as the code has been shared in the supplementary material.  I would kindly request the authors to make the code available in a public platform such as GitHub in case the paper gets accepted.
summary_of_the_review: Well-organized paper with potentially impactful results.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper studies the implicit bias of gradient descent on diagonal linear networks.
A special architecture is proposed so gradient descent imposes group sparsity on the network weights.
An analysis is performed under the (strong) assumption that the features are orthogonal.
Gradient flow of the overparametrized objective is analyzed, and then a practical version of the gradient flow is introduced (gradient descent with weight normalization).
Guaranties similar to the sparse recovery literature are derived under sub-Gaussianity of the noise.
justification_for_why_not_higher_score: Comparison with explicit regularization methods could be reinforced.
Similarly, the practical implications are not direct for standard NN architectures. 
justification_for_why_not_lower_score: It is a well-organized and well written paper with potentially impactful results.
Decision: Accept: poster