Hide & Seek: Transformer Symmetries Obscure Sharpness & Riemmanian Geometry Finds It

Authors: Marvin F. da Silva, Felix Dangel, Sageev Oore
Primary area: learning theory
Keywords: generalization, symmetry, sharpness, flatness, riemannian geometry, loss landscape

Abstract:
The concept of sharpness has been successfully applied to traditional architectures like MLPs and CNNs to predict their generalization.
  For transformers, however, recent work reported weak correlation between flatness and generalization. We argue that existing sharpness measures fail for transformers, because they have much richer symmetries in their attention mechanism that induce directions in parameter space along which the network or its loss remain identical.
  We posit that sharpness must account fully for these symmetries, and thus we redefine it on a quotient manifold that results from quotienting out the transformer symmetries, thereby removing their ambiguities.
  Leveraging tools from Riemannian geometry, we propose a fully general notion of sharpness, in terms of a geodesic ball on the symmetry-corrected quotient manifold. In practise, we need to resort to approximating the geodesics. Doing so up to first order yields existing adaptive sharpness measures, and we demonstrate that including higher-order terms is crucial to recover correlation with generalization.
  We present results on diagonal networks with synthetic data, and show that our geodesic sharpness reveals the correlation for real-world transformers on ImageNet.

Decision: reject
Reviewer scores: 6, 6, 8, 3, 6

Reviews and discussion:
<thread>
creation date:
1730315464964

thread_type:
official_review

<thread_object>
creator: reviewer_qb83

<content>
summary:
This paper generalizes the (adaptive) sharpness concept by considering both the first-order perturbation and the (second-order) curvature of the loss landscape, presented under a Riemannian geodesic framework. It aims at solving the inconsistency in the literature on the correlation between sharpness and generalization, and addresses the failure of existing sharpness in measuring the generalization of Transformers with larger symmetry, explained by the scaling/rescaling symmetry of the parameter space. The symmetry is formalized as the quotient group. Experiments on both a toy diagonal network and fine-tuned vision transformers show that the geodesic sharpness has a significant correlation with the generalization gap.

strengths:
The motivation is strong, the theoretical discussion is rich and sound, and the curvature-aware sharpness significantly depends on the generalization gap. I appreciate the insights of the Attention symmetry and the strong theoretical connections.

weaknesses:
- My main concern is the practical effectiveness of the proposed method: does it really improve existing adaptive sharpness? In Figure 4 the improvements are not quite visible, and the parameter $\tau$ is neither explained nor aligned by column.
- The experiments are not described in detail, which affects reproducibility. For self-containedness, the method used in the experiment from Wortsman et al. (2022) could be explained better in detail. For example, what are the hyperparameters, how are the models trained, what is the precise algorithm, etc.?
- It is questionable whether the attention's symmetry argued by the author is indeed the practical reason rather than a hypothesis. It would be helpful if the author could give a stronger argument or some illustrations. To give an example, ensure that the scaling symmetry causes irregularity in pre-trained models by measuring the conditional number, and modulating out this symmetry indeed corrects the sharpness---or any other way you prefer.
- I believe the paper's writing could benefit greatly by being more concise. Many sentences in the introduction seem to wander around and are not precisely to the point.
- Minor issues: Figures 2, 3, and 4 are not referenced.

questions:
- I don't understand the role of the $\tau$ variable in the result figures. For example, could you explain how it is chosen, and how it relates to the results?
- I don't understand why in Figure 3 (left), the average-case adaptive sharpness is claimed to reveal a correlation with the test loss (generalization gap). To me, it looks not obvious, and the dependency is reversed.
- $c$ inconsistently appears in Appendix D. Do the experiments of geodesic sharpness also choose a vector $c$ by taking $c=|w|$ to normalize each parameter tensor? If not how to guarantee fair comparison? In Appendix D $c$ is preset, but I don't understand whether the geodesic is invariant over scaling?
- I wonder if normalizing QK embeddings will fix the concern in the paper that Transformers have GL group symmetry? It is shown in practice that it stabilizes training (such as [1]). [2] also uses spherical projection for theoretical convenience. Or if not, are there alternative ways to address the scaling symmetry?

[1] Scaling rectified flow transformers for high-resolution image synthesis. Esser et al. 2024.
[2] The emergence of clusters in self-attention dynamics. Geshkovski et al. 2024.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730644638402

thread_type:
official_review

<thread_object>
creator: reviewer_yq4w

<content>
summary:
This paper presents a novel analysis of sharpness in machine learning, with a particular focus on its implications for generalization. The authors argue that existing sharpness measures are inadequate due to their failure to account for the symmetries which arise in parameter space, especially the complex symmetries inherent in transformer architectures. To address this limitation, they leverage tools from Riemannian geometry to propose a refined definition of sharpness (their so-called geodesic sharpness) that explicitly removes these symmetries by instead focusing on the quotient space. This construction draws upon concepts from the theory of quotient manifolds. The authors validate their theoretical framework through experiments on both toy models and vision transformer models trained on ImageNet.

strengths:
I found this to be an interesting paper. The authors explore the notion and role of sharpness in machine learning and shed light on how sharpness is affected by the symmetries which arise in parameter space for these large machine learning models. They use interesting ideas from Riemannian geometry and provide a clear, detailed analysis of difficult ideas. The ideas are original and they provide a creative way and mathematically sound way to define a new notion of sharpness (i.e., geodesic sharpness) which mitigates this issue of symmetry in parameter space.

weaknesses:
Based on my understanding of the author's work, here are the weaknesses I see

There are a few conceptual gaps for me. While the use of quotient spaces to address parameter symmetries is conceptually nice, certain aspects of their construction are unclear to me. In particular, the choice of Riemannian metric on the total space $\bar{\mathcal{M}}$ appears to be crucial, as exemplified by the metric defined in equation (15) for attention layers.  This metric is not unique or canonical. Would similar results hold for any metric, except I suppose for the standard Euclidean one? It is unclear whether the proposed sharpness measure is invariant under different choices. One could conceive of a metric on $\bar{\mathcal{M}}$ that yields a different value for geodesic sharpness. Is that right? Or should all measures for geodesic sharpness be invariant under the R. metric of the total space as well? This technique is marketed in the introduction/conclusion as a "one-size-fits-all" technique for appropriately measuring sharpness but this choice of total R. metric doesn't feel that way to me. To address this concern could the authors
 1. Explain the rationale behind choosing the specific metric in equation (15)
 2. Provide some discuss around how sensitive the results are to different choices of Riemannian metrics on the total space
 3. Clarify if there's a principled way to select an "optimal" metric for a given architecture.
These points would help evaluate whether the method is truly "one-size-fits-all" as claimed.

Also the details of how to actually compute the geodesic sharpness for the attention layer example (Sec 5.2) is lacking, see line 468-469. It's not clear to me what that means to "...plug Eq. 16 into Eq. 17 and solve the resulting optimization problem using SGD". Would it be possible to include the details for that computation and optimization solution details/setup? To address this concern, could the authors
 1. provide the explicit form of the optimization problem after plugging Eq. 16 into Eq. 17 and clarify the specific SGD algorithm used to solve it and hyperparameters used, and
 2. provide any additional constraints or modifications needed to ensure the optimization remains on the manifold

It would be nice to see more experiments or at least some more computational examples. The paper only addresses, in a careful way, very simple diagonal networks and attention layers in transformers. Nothing in between. The experiments are limited to correlational studies. It would be good to have more architectures and datasets to help convince the reader of this new technique. To address this, the authors could
 1. test this measure of geometric sharpness on intermediate architectures between diagonal networks and transformers, such as MLPs or CNNs; or ideally, architectures that are more closely comparable with other measures of sharpness in the literature.
 2. apply the technique to different datasets beyond ImageNet
 3. conduct ablation studies to isolate the impact of different components of the 'geodesic sharpness' measure

Also, the final takeaway message from Figure 4 (see line 488) is that the "...geodesically sharpest models studied on ImageNet are those that generalise best." This seems counter to the typically held belief that less sharp or more flat models generalize better. It would be nice to have more experiments or theory to explain why we see the opposite behavior here than previously in the literature. Or it would be nice to recreate the experiments in the literature which demonstrate that less sharp models generalize better (e.g. the original SAM paper and followups) and show that the geodesic sharpness behaves in the opposite way. To address this concern, could the authors
 1. provide a theoretical explanation or justification as to why 'geodesic sharpness' might behave differently from traditional sharpness measures
 2. conduct a direct comparison with previous sharpness measures on the same models and datasets. This, in particular, would be very nice. And
 3. discuss potential implications of this result for providing better understanding of generalization in deep learning

There are also some parts of the exposition that I found confusing or misleading (in either the concepts or lack of details in plots) and quite a large number of typos which made the paper hard to follow at times. I've outlined these points in the questions section below.

questions:
- line 112. in contribution (d) you claim that Figure 1 shows that there is a strong correlation between generalization and sharpness. Is this a typo? Can you better explain the connection between Figure 1 and generalization?
- In general, the paper appears to only address symmetries that arise as group actions of GL. Is it obvious that this captures all possible symmetries in the parameter space? What is the effect of your construction of the geodesic sharpness when not all symmetries are removed? 
- in contribution (f) and in the paper you make a comparison with the CLIP  experiments from Andriushchenko et al. However, those models were not trained to convergence and one could argue that the role of sharpness in generalization is only evident for well-tune and converged models. Does that also apply to your measure of geodesic sharpness?
- line 279, why denote the points in total space by $\bar{x}^{(')}$. I'm confused by this notation.
- line 280, is this a typo? Do you mean x,y for points in the quotient space?
- line 297. Can you clarify the details in the section "Linear embedding space". When you say "On the outer most layer, we are given a linear Euclidean space E…" what does this mean? Do you mean the $d$ parameters relating to the outermost layer of the network? Also, why define the loss $\overline{\overline{\ell}}$ just on these outer layer parameters? How are $\overline{\overline{\ell}}$ and $\overline{\ell}$ related to each other?
- then on line 304, you reference the "Riemannian generalization" The R. generalization of what? The gradient?
- line 318, i mentioned this above, but here you reference endowing the total space $\overline{\mathcal{M}}$ with a smooth inner product. Yes. it is possible to endow any smooth manifold with a Riemannian metric. However, this R. metric is not unique. You reference defining it in Appendix B.4 but that just points to a definition of what a Riemannian metric is. This is confusing and crucial aspect of this work. 
- Section 4. When defining the geodesic sharpness, I'm surprised to not see mention of the exponential map to define geodesics in the manifold using vectors in the tangent space. Would it not be possible to reformulate your construction looking only at vectors in the tangent space? And restrict yourself to vectors of a certain size in order to keep them within a fixed ball when projected to the manifold. 
- In Figure 3, you plot sharpness vs test loss for diagonal models. But only for 50 of the 200 models from the experiment setup? Why only 50 and why those?
- How natural or reasonable is Assumption 5.1? There is a low-rank bias in deep learning when training overparameterized models. Can you provide a reference that this assumption is usually satisfied in multi-head attention layers for default choices of $d_v, d_k$?
- line 453: Is it possible to endow the total space with any smooth Riemannian metric? Why this one? Admittedly I'm less familiar with the reference (Absil et al 2008) but it would be nice to see a computation verifying this is indeed a Riemannian metric and why this inner product was chosen.
- line 514: Can you clarify what you mean when you say "..attention weights approach being singular" here? How are you defining singular. Also you mention a relaxation parameter but I'm not clear on the role this 'relaxation parameter' plays. You indicate that "..in practise we found out results were robust to this parameter". Can you explain more what robustness analysis you conducted to justify this claim.

typos/nits
 - (typo) Do you mean $L_{\mathbb{S}}$ not $L_{S}$  in equations (1) and (2)? same in equation (3)
 - (nit/personal preference) := is more common that :- for definitional equivalence
 - (typo) line 229. Do you mean $\overline{\overline{f}}$ is symmetric…?
 - the Figures have been resized and the font is very small, difficult to read
 - line 434: what do you mean that the GL(h) symmetry is 'dealt' by geodesic sharpness?
 - line 463: 15 should be (15)
</content>
</thread_object>

</thread>
<thread>
creation date:
1730714635396

thread_type:
official_review

<thread_object>
creator: reviewer_kb1x

<content>
summary:
This work introduces a new approach to measuring model sharpness using Riemannian geometry. This approach takes into account the symmetries in model parameters, removing ambiguities that these symmetries can cause. As a result, it provides a clearer and more reliable sharpness measure compared to previous methods.

strengths:
1. This work takes a broader view of parameter symmetries and introduces a more robust metric for measuring sharpness.

2. The theoretical framework and assumptions are well-grounded, and the experimental results are both strong and consistent.

weaknesses:
1. $\textbf{Geodesic sharpness}$ in section 5.2 is abbreviated. Providing a clear, explicit formula for geodesic sharpness would improve comprehension.

2. The discussion of geodesic sharpness in transformers (Section 5.3) is brief. Including a description of geodesic sharpness for transformers, or a brief explanation of how adaptive sharpness could be applied to each layer, would make the analysis more thorough.

3. The paper aims to propose a metric that surpasses previous methods. Therefore, I would expect comprehensive experimental results demonstrating both the effectiveness of the proposed approach and its superiority over existing methods. While the experiments provide sufficient evidence of the method’s effectiveness, the comparative analysis with previous methods seems limited.

Minor points:

$\bullet$ Example 3.2 "(Self-attention Vaswani et al. (2017))" $\to$ "Example 3.2 (Self-attention (Vaswani et al., 2017))".

$\bullet$ $\textbf{Geodesics}$ in section 5.2: "... the geodesics of metric 15" $\to$ "... the geodesics of metric (15)".

questions:
1. In Case A), the second-order term in Equation (12) is omitted because it is considered small compared to the first-order term. However, I wonder if this omission might be overly convenient. Since there is no indication of how small $\rho$ can be, I’m unsure about the relative scale of the first and second orders. Would it be possible to retain the second-order term and, perhaps, combine the results of both cases to reach the conclusion?

2. Could you provide further clarification on why "adaptive sharpness is more appropriate" in Section 5.3?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730716824979

thread_type:
official_review

<thread_object>
creator: reviewer_vdtg

<content>
summary:
This paper proposes a method for measuring the sharpness of neural networks, called geodesic sharpness, which takes into account the symmetries present in the network architecture. The authors argue that previous sharpness measures fail to accurately capture the sharpness of transformers because they do not account for the rich symmetries present in the attention mechanism. The authors clearly motivate the need for a new sharpness measure that is invariant to these symmetries. However, the paper need to be polished more to fix typo and clearer explanation, especially experiments.

strengths:
From theoretical aspects, the authors' main theoretical contribution is the application of Riemannian geometry to the study of neural network parameter space symmetry. They propose to use the geometry of the quotient manifold, which is obtained by removing the symmetries from the parameter space, to define a sharpness measure that is invariant to these symmetries. This approach is general and can be applied to a wide range of symmetries. The authors show that ignoring the curvature introduced by the symmetries leads to traditional adaptive sharpness measures. 

From experiment aspects, yo validate their approach, the authors conduct experiments on both synthetic and real-world data. They analytically derive the geodesic sharpness for diagonal networks and show that it correlates strongly with generalization. They also apply their method to large vision transformers and find that geodesic sharpness has a stronger correlation with generalization than any previously reported measure, both for in-distribution and out-of-distribution settings.

The main strength of the paper is its novel and principled approach with the use of Riemannian geometry to account for symmetries in sharpness measures, which can apply to more problem like equivariant neural functional network.

weaknesses:
The paper, while presenting a promising theoretical approach to measuring sharpness, suffers from certain shortcomings related to the experimental validation of the proposed method and the potential computational cost involved:

-   **Limited empirical support:** The experiments conducted on real-world transformers lack breadth.It would be good if the authors can demonstrated in the natural language task too, like language modeling with wikitext103.

-   **Effect of assumptions:** The paper makes an assumption regarding the full column rank of weight matrices in attention layers. A relaxation parameter is introduced to handle potential violations of this assumption, but the impact of this parameter on the results is not thoroughly investigated.

-   **Computational burden:** The paper acknowledges the potential computational demands of calculating geodesic sharpness, especially for large models. However, it lacks a detailed analysis of the computational cost, including aspects like time complexity, memory requirements, and actual runtime measurements. Such an analysis would enable a better understanding of the feasibility of employing geodesic sharpness in practical scenarios, particularly when dealing with large-scale models.

Addressing these weaknesses would bolster the paper's contributions and enhance the understanding of the interplay between sharpness and generalization in the context of transformer models. I willing to increase the score more if the authors can provide more convincing analysis on these weaknesses.

questions:
1. "RIEMMANIAN" -> "RIEMANNIAN" in title
2. ",and" -> ", and" in line 39
3.  In my opinion, the contributions can be collated, where (b)-(c) and (d)-(e)-(f) are collated into two contributions: the development of geodesic sharpness and experiments for it.
4. Diagonal network experiment:  it is not clear to me what Figure 3 is showing. What is the meaning of the x axis and y axis? What is the ideal result? Why only show 50 data points when you have 200 model trained? Why the points in adaptive average case aligned in different direction in comparison with two other cases? I suggest adding more detailed captions and axis labels to Figure 3, and include explanations for these points in the corresponding text.
5.  Assumption 5.1: Does Assumption 5.1 are too strong? What if the assumption is violated? Can we still calculated the result but with wrong value, or not able to calculate it at all? It would be good if the author can provide the evident and explanation of effect of adding $\epsilon I_h$ into $GH$ since there have not any for the assumption. Also, including a discussion of the implications of violating Assumption 5.1 and provide empirical evidence for the effects of adding the $\epsilon I_h$ term.
6. Can the authors describe in detail how you treat the multi-layer schema of transformers model? It is not clear at this time. It would be good if the author can provide the diagram illustrating the approach for multi-layer transformers.
7. Experiment diversity: it would be good if we can demonstrated in the natural language task too, like language modeling with wikitext103, to ensure that it works across domains.
8.  In order to find the geodesic approximation, you need to solve the optimization using SGD, what is the quantitative performance, i.e. wall clock time or time/memory complexity of this approach in comparison with adaptive sharpness? How does it scale with large models like LLMs?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730755212185

thread_type:
official_review

<thread_object>
creator: reviewer_tckk

<content>
summary:
This paper proposes a more general notion of sharpness measurement based on a geodesic ball on the symmetry-corrected quotient manifold, which accounts for the symmetry equivalence of the original network. Experiments with Diagonal Networks on synthetic data and transformers on the ImageNet dataset show that Geodesic Sharpness has a stronger correlation with the model’s generalization performance compared to Adaptive Sharpness.

strengths:
- The paper introduces a novel approach to sharpness by leveraging quotient manifolds, reflecting the symmetry of the parameter space. This effectively addresses a gap in previous work that has not considered network symmetry.
- The methodology is theoretically well-founded, employing tools from Riemannian geometry to incorporate network symmetry to the sharpness measure in a principled way.
- Geodesic sharpness is shown to achieve a higher Kendall Tau correlation with the relevant metrics than Adaptive sharpness in both synthetic and ImageNet experiments.

weaknesses:
The experimental section should be strengthened. Assessing the correlation between the proposed sharpness and the generalization of transformers trained on other tasks, such as time series forecasting or language modeling, would further support the findings.

questions:
- Could you include additional experiments on real-world data beyond vision transformers? This would enhance the experiment effectiveness and demonstrate the predictive capacity of the proposed Geodesic sharpness across a broader range of tasks.
- Could you provide a time complexity analysis of the proposed method relative to Adaptive sharpness?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733158079354

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Rebuttal and Discussion Period

comment:
As the discussion period draws to a close, we once again thank all the reviewers for their thoughtful comments. We are happy to see that the reviewers unanimously appreciated the originality of our idea (tcKK, VDTG, KB1X, Yq4W, qb83); the principled nature and theoretical richness and soundness of our approach (tcKK, VDTG, KB1X, Yq4W, qb83); and the generality of our approach (VDTG, KB1X,qb83). 
The most common recommendations were to add additional experimental results (especially on language models) and discuss optimization and complexity: we have provided both of these in detail, along with other important additions (discussion and experiments on choice of metric, and ablations, for example). In all cases, the new results and analyses support our proposed approach.

We therefore believe we have addressed all of the reviewers’ concerns; for your convenience, we summarize below the new sections and appendices that we added in response to reviewers’ suggestions, along with corresponding main changes made during discussion period and their location in the paper.

**New results for language models (all reviewers)**:[__Section 5.3.2__] We added a section to the paper with results obtained using the fine-tuned BERT models setup from Andriushchenko et al. (2023). We again recover a stronger correlation with generalization than that observed with adaptive sharpness alone.

**Optimization and complexity (tcKK, VDTG, Yq4W)**: [__Appendix C__]  Several reviewers mentioned that a more detailed discussion of the optimization method and its computational complexity was missing. This is now included in an appendix in the paper, Appendix C, showing the feasibility of our approach.

**Illustration of the role of symmetry (qb83)**: [__Appendix D.1__] One reviewer suggested the paper could benefit from an example illustrating the role of symmetry in sharpness and its relationship with generalization. We added a small example (__Figure 7__) extending Figure 1, showing how in a simple setting accounting for parameter symmetry regularizes to some extent the loss landscape and provides a clearer signal as to the generalizing performance of a network.

**Completing the diagonal network analysis (KB1X)**: [__Appendix E__] An appendix with the analysis of geodesic sharpness in diagonal network in the fullest generality was added.

**Reduction of geodesic sharpness to adaptive sharpness and different choices for metrics (qb83, Yq4W)**: [__Appendix F__] The previous discussion on how and when geodesic sharpness reduces to adaptive sharpness was rightly pointed out as being somewhat confusing. This discussion was reworked and expanded and can now be found in [__Appendix F.1__]. 
One reviewer mentioned that there was some ambiguity in the choice of metric. While the symmetry compatibility criterion restricts the possible choices of metrics immensely, it does not uniquely determine it. We conducted additional experiments with the one other symmetry-compatible metric we are aware off that has commonly been used in the literature. This discussion can be found in [__Appendix F.2__]. This remained consistent with our overall results.

**Ablation studies (Yq4W)**: [__Appendix G__] We added an appendix with ablation studies investigating (and subsequently demonstrating) the importance of both the second-order weight corrections (given by the Christoffel symbols), and the use of the symmetry-compatible norm. 


**Assumption 5.1 and the relaxation hyperparameter (VDTG,Yq4W)**: [__Appendix H__] Questions were raised both about the naturalness of assumption 5.1 and the impact that the relaxation parameter has. [__Appendix H.1__] discusses the former, and [__Appendix H.2__] shows the impact the relaxation parameter has on results. Overall, assumption 5.1 is reasonable, and the relaxation parameter does not impact results significantly, while improving numerical stability.
 
We believe that this addresses all raised concerns by the reviewers (thank you to all of them for their insightful feedback!), and, in sum, it demonstrates the effectiveness, feasibility, and significance of our novel approach to measuring sharpness.

PS. Since writing this summary, two reviewers (VDTG,tckk) have acknowledged our rebuttals so far and subsequently raised their scores. Since all reviewers shared the same set of 2 primary concerns (experiments and computational feasibility), and since (VDTG, tckk) responded that we did satisfy those primary issues and subsequently raised their scores, we very much hope that we have indeed satisfied these concerns for all of the reviewers, along with our other rebuttal points.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes to use Riemannian geometry to measure the sharpness of neural networks which explicitly accounts for some of the symmetries in the network by measuring sharpness in the quotient space.  The authors then show that in certain settings this measure of sharpness correlates with model generalization.  The reviewers largely found the concept of the paper interesting and appreciated the theoretical development.  However, the reviewers also noted several weaknesses.  In particular the the reviewers asked for more extensive experimental evaluation, which the authors responded to by adding further experiments on language models and ablation studies.  Along with this there were questions raised by several reviewers regarding the implications of the experimental results.  Notably, the correlation between the sharpness measure and model generalization is often negative (thought the sign of the correlation also appears to potentially switch between experiments), which is counter to the typical intuition for the motivation of studying sharpness.

Overall, the majority of the reviews are somewhat borderline along with one additional positive review and one additional negative review.  Unfortunately I am inclined to side with the negative review and recommend that the paper not be published at this time.  While the proposed measure of sharpness appears to provide stronger correlation with the generalization performance of the overall model, the fact that basic aspects of the correlation such as the difference in the sign of the correlation that occur between experiments somewhat muddies the overall message.  This phenomenon is perhaps a more general property of sharpness in general, but I believe the work would be better served by a revision which can provide a more complete message as to the benefits and interpretation of the measure as a predictor of network generalization.  Indeed, the authors already suggest at potential directions along these lines in the discussion such as considering the properties of the data or incorporating further symmetries into the metric, and I would encourage the authors to further refine the overall implications/interpretation of the proposed metric in future versions of the manuscript.