Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: Based on the Euler-Lagrangian framework, this manuscript develops the theoretical method to model the learning dynamics of DNNs as symmetry breaking of the likelihood by the kinetic energy. 
The model is expressed as the time evolution of Noether-current.
As a result, the author found that the learning dynamics of batch normalization are similar to RMSProp from the viewpoint of symmetry breaking.
Their findings deepen our understanding of batch normalization.
This study is unique not only because of the findings on batch normalization mechanics but also because it links the symmetry-breaking process of the likelihood function with the learning theory.
main_review: The study about elucidating implicit adaptive optimization from the viewpoint of symmetry is excellent and unique, so this manuscript has originality and significance worthy of publication.
On the other hand, I can not understand some points related to the effectiveness of the theory as below.

The first concern is that there may be few cases in DNNs where the symmetry of the potential function is known.
For example, in their theory building on BatchNorm, the authors state:
"We investigate this physical system because the learning dynamics
264 of the most successful deep learning systems are exactly analogous to this system. For example, BatchNorm layers, ubiquitous in modern successful deep learning architectures, introduce scale symmetry in the parameter space q of the potential (loss) function."
However, I can not understand that BatchNormalization introduce scale invariance when it is subjected to an activation function such as sigmoid? (I can understand this for if the parameters are the regression coefficients of linear regression.)
Also, I understand that the space q is a parameter of the BatchNorm layer when considering the dynamics of BatchNorm.
And I understand that for the parameters of the other layers, the potential function is not symmetric to the scale transformation of these parameters.
Therefore, the symmetry of the parameters of the other layers is unknown.
From these understandings, I think it does not seem appropriate to compare the dynamics of RMSProp with those of DNN, since the space q in RMSProp is all the parameters of DNN.
By the way, isn't "f((1+s)q)=q" in L263 "f((1+s)q)=f(q)"?

The second is about the difficulty of the Lagrangian approach.
In general, it seems difficult to construct a Lagrangian that models the dynamics of a dissipative system.
I understood that the Bregman Lagrangian was constructed based on the Lagrangian of a known dissipative system called damped oscillation.
Therefore, when the time evolution of the error has a plateau, which is often the case in DNN training, the learning dynamics cannot be a damped oscillation, and that is why Bregman Lagrangian cannot be used.
Therefore, I could not understand the validity of theoretical analysis assuming damped oscillations.


There are some minor concern as below:


Around L168: Since q is supposed to be a generalized coordinate system, I think that authors should not make the assumption that $q = \dot{q}$ and $Q = \dot{Q}$.
In that case, I think that the premise of the discussion in this vicinity is no longer valid.

L119: There is no definition of bra-ket which represents the inner product.

L238: ",," might be typo

L275: "as$m" might be "as $m"

L295: "| as" might be "as"

L331, L332: There is no apparent definition of "unconstrained filter norm" and "constrained filter norm".
                    I think it is friendly for reader to describe these definition more clearly.
limitations_and_societal_impact: I have described the concerns about limitation in Main Review.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors identify a "kinetic asymmetry" in the learning dynamics under gradient flow, where the kinetic energy is not invariant to the symmetries of the potential energy. An example is scaling: even when the network function is invariant to scaling or ReLUs or batch norm inputs, the learning dynamics are not. They derive a genealized version of Noether's theorem which takes this asymmetry into account and use it to study the implict adaptive optimization exhibited by batch normalization, showing that it is the same as that of RMSprop.

main_review: The topic and techniques of this paper are well outside of my comfort zone so I can only provide a high-level overview. From this high level, I find the premise of the paper exciting. The interaction between Lagrangian dynamics and optimization in machine learning is not new (e.g., the cited work of Wibisono, Wilson, and Jordan), but connections to  symmetry breaking and Noether's theorem, at least in the context of deep learning, seem to be original. The demonstrated consequence that the dynamics induced by batch normalization are the same as those of RMSprop is very cute. The paper is clearly written (although with quite a few misprints) and the provided examples are helpful.

My main objection is towards the generality of results and the corresponding claims. For example, the statement that "kinetic symmetry breaking is a key design principle to the success of a learning system" could be mollified. Symmetry breaking is certainly a beautiful perspective but it is one out of many.

While the theory and the symmetry-broken Noether's law hold generally, the only example in the paper is the scale symmetry exhibited by ReLU activations and batch norm (since scaling does not preserve kinetic energy). Could the authors suggest or speculate other interesting symmetries in deep learning that do not preserve kinetic energy? Another question is about using your insights opportunely---since learning dynamics is not invariant under symmetries, symmetries may be used to improve performance or accelerate training. On a related note: is there a connection to quiver representations of neural nets https://arxiv.org/abs/2007.12213, https://arxiv.org/abs/2107.02550?

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper considers the role of symmetries in the learning dynamic by considering a continuous time formulation of the learning process in Lagrangian formalism. In such formulation there are "kinetic" and "potential" components to the dynamics, the former arising from the learning rules employed during optimization and the latter generated by the loss function landscape. Authors argue that for a large class of practically relevant network architectures there are symmetries that are present in the "potential" component, but not in the "kinetic" part.

Starting with Bergman Lagrangian formulation of the learning dynamics, authors derive the equations for time evolution of the Noether's charge associated with symmetry transformation (assumed to be present) in the network architecture.

For the case on scale symmetry introduces by BatchNormalization, authors observe that in case of training with gradient descent + momentum + weight decay; the training dynamics effectively reduces to a learning dynamics with an adaptive scale where the scale is related to the associated Noether's charge. This suggests that kinetic asymmetry induces an implicit adaptive property to the optimization symmetry. A connection between RMSProp and implicit adaptation is established based on continuous time learning model.

Lastly, authors test theoretical predictions by training a set of models verifying theoretical predictions on Tiny ImageNet dataset. For comparison, the adaptive rescaling effect is artfully removed by renormalizing the parameters throughout training. 
main_review: The role of symmetries in deep learning is an active research area. The paper studies a relevant problem of investigating how symmetry properties of the model, combined with asymmetry of the learning rules, give rise to non-trivial adaptive learning dynamics. To the best of my knowledge the presented results are novel and interesting. Connections between architecture and model components are often highly important in practical settings for guiding the design decisions and understanding the interactions between architectures and optimization.

The paper is clearly written and cites relevant related works. One caveat related to the presentation of the related work is that it does not acknowledge the fact that Noether's theorem has been used in recent works (e.g. reference [14]). The hypotheses are adequately tested. The work reads complete and insightful.

limitations_and_societal_impact: The work does not have negative societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The gradient descent with a momentum can be regarded as the Newtonian equation with damping. The output of the batch normalization is invariant to the scale of the input (i.e., a scale symmetry). When a BN is inserted after a convolution, the whole network is invariant to the weight scale. A typical gradient descent algorithm ignores (i.e., breaks) this symmetry. Thanks to this symmetry breaking, the learning rate is implicitly adapted, and the adaptation is equivalent to RMSProp's.

main_review: Firstly, the main result is very interesting. The perspective from the symmetry breaking is unique and provides a new insight to the reason why BN works. The reviewer still has several questions about experiment and relationship with physics.

1) In the experiment, the VGG with BN is compared with a "constrained filter norm" variant, whose weight is normalized after each update. Even then, the gradient and the momentum term are calculated under the assumption of no constraints. This gap might affect the performance negatively. Hence, the performance degradation of the "constrained filter norm" variant might be insufficient to prove the effectiveness of the implicit adaptation by BN.

2) How necessary is the perspective of the Noether's theorem and symmetry breaking? While a physical analogy is well known to the GD with momentum, the equivalence between (17) and (18) might be derived without it.

3) Some prior works such as [34] are introduced in Section 5. In particular, [34] also provided the theoretical explanation of the reason why BN works. The relationship with and difference from prior works should be described in detail.

4) The title might be excessive advertising. The theoretical analysis does not describe the general role of the symmetry breaking in deep learning, but it is applicable only to BN

5) At line 62, the sentence of BN's symmetry might have a typo. Correctly, it might be BN((1+s)qx)=BN(qx).
limitations_and_societal_impact: The theoretical analysis is only applicable to the batch normalization, while the title seems to be more general. No negative societal impact.
review_title: null

Meta-review: Following in the tradition of physics, the paper considers the implications of differentiable symmetries exhibited by a given learning task for the dynamics of the associated learning process. Although similar ideas have previously appeared in the literature ([14], in particular), an explicit study of the suggested Lagrangian framework is conceptually new. The reviewers found this framework interesting and insightful, and appreciated the unique perspective it provides on the connection between batch normalization and the popular optimization method RMSProp.
Decision: Accept (Poster)