Venue: NeurIPS 2022 Workshop NeurReps

Reviews:

final_score: rating: 7: Technically solid paper, with moderate-to-high impact, and good-to-excellent evaluation and reproducibility.
review_content: summary: There is growing interest in a detailed theoretical understanding of how the Riemannian geometry induced by neural networks affects learning. It is intuitive that local amplification of areas (as represented by the volume element) and a local flattening of the space (quantified by the Ricci curvature scalar) in the vicinity of a decision boundary would improve discrimination tasks. This work aims to characterize how these geometric quantities of the representational manifold change over training. 
strengths_and_weaknesses: This extended abstract makes a valuable contribution to the topic. Comprehensively, I consider this a strong paper that is well presented and makes a persuasive argument for expanding upon this work in the future. The appendix provides a thorough analytic derivation of the relevant geometric quantities. One contribution of value is the generalization of the spherical symmetry of the Ricci scalar for infinite-width shallow networks to arbitrary (smooth) activation functions. This paper makes a strong argument that the training of neural networks for simple tasks indeed locally expands areas and reduces curvature near decision boundaries. One weakness is there is no mention of temporal dynamics of the Riemannian geometry throughout learning, or a quantitative characterization of the convergence toward the final final state.
questions: Suggestions: There are a few minor typos throughout the text, and the figures need to be improved for readability. Figure 3 in particular is quite messy, and should be reworked. 
limitations: As previously mentioned, since the study claims to characterize "how" training shapes the Riemannian curvature, there should be some mention of the convergence toward the final state. Importantly, there should also be an attempt to estimate the statistical error. 
recommended_decision: 3: Accept
review_title: A valuable contribution toward understanding the Riemmanian geometry induced by neural networks. 

final_score: rating: 8: Technically strong paper with novel ideas and high-to-excellent impact, and excellent evaluation and reproducibility.
review_content: summary: The paper studies the geometry of infinite-width shallow neural networks, in particular how classification tasks impacts the curvature and the volume element of the Riemannian metric pulled-back from the feature space to the input space. With some assumptions, closed-form expressions are obtained for the metric, the Ricci scalar and the volume elements. The theoretical results are confirmed empirically.
strengths_and_weaknesses: The paper uses Riemannian geometry to better understand the training of shallow neural networks. It is both novel and highly significant for geometers and machine learners. It is is written clearly with well designed experiments. Only minors details could be improved such as justifying the definition of the metric, or explaining the soundness of some hypothesis.
questions: (1) The feature map is divided by the number of weights in the hidden layer: is it really correct in theory? In general, the feature map are only defined as $\phi(x) = w^{\top} x + b$, I understand that dividing by $n$ would ultimately help proving the central limit theorem. Yet, would you reach the same conclusion without dividing the metric by $n$? If not, can you justify the division?

(2) The number of features is defined to be always greater than the number of inputs: what could potentially happen in cases where $n \geq d$? 

(3) Will the authors consider extending their work to non-shallow neural networks? Would the results likely be influenced by increasing the number of layers in a neural network?
limitations: No limitations to note
recommended_decision: 3: Accept
review_title: Excellent paper with significant impact to understand the theory of neural networks using geometry.

final_score: rating: 7: Technically solid paper, with moderate-to-high impact, and good-to-excellent evaluation and reproducibility.
review_content: summary: This papers studies Riemannian properties (curvature and volume element) of the metric induced on the manifold defined as the image of a subset of the input space by the neural network. The results from the literature that ground this study are recalled, and some of the computations are detailed in appendix. These concern first finitely wide neural networks. In this case symmetries of the metric allow explicit computations of the curvature tensor and Ricci scalar. Then, infinitely wide networks are considered, and again the Ricci scalar can be computed thanks to spherical symmetry. Finally, numerical results from toy datasets are given and show that the volume element grows near boundary decisions while curvature decreases.
strengths_and_weaknesses: The paper is well written and gives a concise but clear presentation of the necessary material as well as the contributions. The results seem well proven by the numerical experiments but maybe more interpretation/discussion could give a better grasp of the significance of these results.

questions: The authors consider smooth activation functions. What could be done for RELU functions? As the behavior of neural networks is similar with both smooth and non smooth activation functions, what tools could we use to analyse jointly both cases?
The title and abstract are not clear: the verb "to shape" is hard to understand in this context if the reader is more familiar with shape analysis. Maybe "to model" would be more understandable? I only understood the meaning of the title after reading the end of the introduction.
As the color map changes, it is hard to see that the volume element actually increases on Figure 1, maybe keeping the color map fixed for the three images could make it more obvious?
limitations: The experiments are performed on rather small learning tasks.
Moreover, the analysis and computations of curvature are restricted to shallow networks, whereas the real challenge is to understand the benefits of depth, as this is the key to the revolution in machine learning.
recommended_decision: 3: Accept
review_title: Concise well written paper on the behavior of curvature when training neural networks

Decision: Accept (Poster)