Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper discussed the convergence of the gradient flows of two-layer neural networks, while they claim the convergence of non-convex gradient flows to global optimum which is different from existing works. The theoretical analysis framework seems to be novel and some sufficient condition is proposed to support the strong theoretical results.
main_review: Strengths:
1. Clear presentation and strong logicality;
2. The corresponding non-convex max-margin problem with the following dual reformulation is clear (maybe mathematically correct);
3. The geometry discussion of the neural gradient flow is clear, especially based on Figures 1-4.

summary_of_the_review: Although I am not an expert on this topic, the paper is well-written and the novelty of the idea seems to be clear after careful reading and thinking.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work analyzes the training dynamics of two-layer ReLU networks applied to separable problem data, based on the equivalent convex formulation by Pilanci, Ergen 2020. The main result states that the gradient-flow training dynamics provably converges to a "maximum-margin classifier". 
main_review: Overall, I feel that the technique and the result are novel and of interest to the community. While other approaches, such as NTK or mean-field, have established global convergence guarantees, those results are asymptotic in the size of the neural network and do not provide the interpretation of the max-margin classifier. I feel that these results make the line of work on the convex formulation more complete, and will serve to further encourage the line of work analyzing neural networks based on some hidden form of convexity.

As an aside, it would be great if the authors can comment on the extendability of this approach to the setup of three-layer ReLU, as studied in Ergen and Pilanci 2021. 
summary_of_the_review: Establishing training guarantees of two-layer ReLU networks based on convex formulation of Pilanci, Ergen 2020 is sufficiently novel and interesting.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies the subgradient flows when training a two-layer ReLU neural network. To this end, the non-convex max-margin problem is reformulated as a convex optimization problem. The authors then analyze the dual extreme points of the convex formulation and show the implicit regularization of unregularized gradient flow as convex regularization. Then, for the binary classification problem, it is proven that the KKT points of the non-convex max-margin problem correspond to the KKT points of the convex max-margin problem if the direction is dual feasible. The paper then demonstrates that this is the case under some conditions on spike-free matrices and orthogonal separable data. Finally it is shown that if the dataset is orthogonal separable and initialized sufficiently close to zero, the limiting point of the gradient flow is the global minimizer of the max-margin problem.
main_review: The main contribution of the paper is to prove that the non-convex gradient flows for the max-margin problem converges to the KKT points of the convex max-margin problem. This is a significant result which may be of interest to parts of the community.

On the other hand, the structure of the paper can be improved. For instance, Section 2.2. is labelled "Outline of our contributions" but this section already contains some preliminary results, e.g. the characterization of the dual extreme points as well as the discussion of the implicit regularization of unregularized gradient flow, while the main contributions are not discussed here. 

Moreover, I found Section 5 a bit hard to follow since the purpose of the individual Lemmas is not made clear directly in the section. The flow of the section could be improved by providing additional information about which role they play in the overall proofs.

Finally, the abstract says that the paper presents numerical results verifying the predictions of their theory. However, those experiments are only contained in the Appendix.

Overall, while the paper contains some interesting results, overall the presentation and flow can be improved. Therefore I am leaning towards rejection of the paper.



--- 
After rebuttal: The authors made several improvements to the flow and structure of the paper, as suggested in my review. Therefore I have increased my score.
summary_of_the_review: The paper contains some interesting results which may be of interest to the community, but the presentation and flow of the paper can be improved.
review_title: null

Meta-review: The papers makes progress on the important question of implicit bias in gradient based neural learning. Remarkably they derive reasonable conditions for global optimality.
Decision: Accept (Poster)