SlowGuess/NeurIPS_2023_OCR huggingface dataset:
There exist no more independent conserved functions. We now come to the core of the analysis, which consists in actually computing  $\mathrm{Lie}(W_{\phi})$  as well as its traces  $\mathrm{Lie}(W_{\phi})(\theta)$  in the matrix factorization case. The crux of the analysis, which enables us to fully work out theoretically the case  $q = 2$ , is that  $W_{\phi}$  is composed of linear vector fields (that are explicitly characterized in Proposition H.2 in Appendix H), the Lie bracket between two linear fields being itself linear and explicitly characterized with skew matrices, see Proposition H.3 in Appendix H. Eventually, what we need to compute is the dimension of the trace  $\mathrm{Lie}(W_{\phi})(U,V)$  for any  $(U,V)$ . We prove the following in Appendix H
Proposition 4.3. If  $(U;V) \in \mathbb{R}^{(n + m)\times r}$  has full rank noted  $\mathbf{rk}$ , then:  $\dim (\operatorname{Lie}(W_{\phi})(U;V)) = (n + m)r - (2r + 1 - \mathbf{rk}) / 2$ .
With this explicit characterization of the trace of the generated Lie algebra and Proposition 4.2, we conclude that Proposition 4.1 has indeed exhausted the list of independent conservation laws.
Corollary 4.4. If  $(U;V)$  has full rank, then all conserved functions are given by  $\Psi : (U,V) \mapsto U^{\top}U - V^{\top}V$ . In particular, there exist no more independent conserved functions.
4.2 Numerical guarantees in the general case
The expressions derived in the previous section are specific to the linear case  $q = 2$ . For deeper linear networks and for ReLU networks, the vector fields in  $W_{\phi}$  are non-linear polynomials, and computing Lie brackets of such fields can increase the degree, which could potentially make the generated Lie algebra infinite-dimensional. One can however use Lemma 3.2 and stop as soon as  $\dim((W_{\phi})_k(\theta))$  stagnates. Numerically comparing this dimension with the number  $N$  of independent conserved functions known in the literature (predicted by Proposition 4.1) on a sample of depths/widths of small size, we empirically confirmed that there are no more conservation laws than the ones already known for deeper linear networks and for ReLU networks too (see Appendix J for details). Our code is open-sourced and is available at [18]. It is worth mentioning again that in all tested cases  $\phi$  is polynomial, and there is a maximum set of conservation laws that are also polynomial, which are found algorithmically (as detailed in Section 2.5).
In this article, we proposed a constructive program for determining the number of conservation laws. An important avenue for future work is the consideration of more general classes of architectures, such as deep convolutional networks, normalization, and attention layers. Note that while we focus in this article on gradient flows, our theory can be applied to any space of displacements in place of  $W_{\phi}$ . This could be used to study conservation laws for flows with higher order time derivatives, for instance gradient descent with momentum, by lifting the flow to a higher dimensional phase space. A limitation that warrants further study is that our theory is restricted to continuous time gradient flow. Gradient descent with finite step size, as opposed to continuous flows, disrupts exact conservation. The study of approximate conservation presents an interesting avenue for future work.
The work of G. Peyré was supported by the European Research Council (ERC project NORIA) and the French government under management of Agence Nationale de la Recherche as part of the "Investissements d'avenir" program, reference ANR-19-P3IA-0001 (PRAIRIE 3IA Institute). The work of R. Gribonval was partially supported by the AllegroAssai ANR project ANR-19-CHIA-0009. We thank Thomas Bouchet for introducing us to SageMath, as well as Léo Grinsztajn for helpful feedbacks regarding the numerics. We thank Pierre Ablin and Raphael Barboni for comments on a draft of this paper. We also thank the anonymous reviewers for their fruitful feedback.
[1] S. ARORA, N. COHEN, N. GOLOWICH, AND W. HU, A convergence analysis of gradient descent for deep linear neural networks, arXiv preprint arXiv:1810.02281, (2018).
[2] S. ARORA, N. COHEN, AND E. HAZAN, On the optimization of deep networks: Implicit acceleration by overparameterization, in Int. Conf. on Machine Learning, PMLR, 2018, pp. 244-253.
[3] S. AZULAY, E. MOROSHKO, M. S. NACSON, B. E. WOODWORTH, N. SREBRO, A. GLOBERSON, AND D. SOUDRY, On the implicit bias of initialization shape: Beyond infinitesimal mirror descent, in Proceedings of the 38th International Conference on Machine Learning, M. Meila and T. Zhang, eds., vol. 139 of Proceedings of Machine Learning Research, PMLR, 18–24 Jul 2021, pp. 468–477.
[4] B. BAH, H. RAUHUT, U. TERSTIEGE, AND M. WESTDICKENBERG, Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizers, Information and Inference: A Journal of the IMA, 11 (2022), pp. 307-353.
[5] M. BELKIN, D. HSU, S. MA, AND S. MANDAL, Reconciling modern machine-learning practice and the classical bias-variance trade-off, Proc. of the National Academy of Sciences, 116 (2019), pp. 15849-15854.
[6] B. BONNARD, M. CHYBA, AND J. ROUOT, Geometric and Numerical Optimal Control - Application to Swimming at Low Reynolds Number and Magnetic Resonance Imaging, SpringerBriefs in Mathematics, Springer Int. Publishing, 2018.
[7] A. B. BROWN, Functional dependence, Transactions of the American Mathematical Society, 38 (1935), pp. 379-394.
[8] L. CHIZAT AND F. BACH, Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss, in Conf. on Learning Theory, PMLR, 2020, pp. 1305-1338.
[9] S. S. DU, W. HU, AND J. D. LEE, Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced, Adv. in Neural Inf. Proc. Systems, 31 (2018).
[10] S. GUNASEKAR, J. LEE, D. SOUDRY, AND N. SREBRO, Characterizing implicit bias in terms of optimization geometry, in Int. Conf. on Machine Learning, PMLR, 2018, pp. 1832-1841.
[11] S. GUNASEKAR, B. E. WOODWORTH, S. BHOJANAPALLI, B. NEYSHABUR, AND N. SREBRO, Implicit regularization in matrix factorization, Adv. in Neural Inf. Proc. Systems, 30 (2017).
[12] G. GLUCH AND R. URBANKE, Noether: The more things change, the more stay the same, 2021.
[13] A. ISIDORI, Nonlinear system control, New York: Springer Verlag, 61 (1995), pp. 225-236.
[14] Z. Ji, M. DUDÍK, R. E. SCHAPIRE, AND M. TELGARSKY, Gradient descent follows the regularization path for general losses, in Conf. on Learning Theory, PMLR, 2020, pp. 2109-2136.
[15] Z. Ji AND M. TELGARSKY, Gradient descent aligns the layers of deep linear networks, arXiv preprint arXiv:1810.02032, (2018).
[16] D. KUNIN, J. SAGASTUY-BRENA, S. GANGULI, D. L. YAMINS, AND H. TANAKA, Neural mechanics: Symmetry and broken conservation laws in deep learning dynamics, arXiv preprint arXiv:2012.04728, (2020).
[17] Z. LI, T. WANG, J. D. LEE, AND S. ARORA, Implicit bias of gradient descent on reparametrized models: On equivalence to mirror descent, in Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, eds., vol. 35, Curran Associates, Inc., 2022, pp. 34626-34640.
[18] S. MARCOTTE, R. GRIBONVAL, AND G. PEYRE, Code for reproducible research. Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flows, Oct. 2023.
[19] H. MIN, S. TARMOUN, R. VIDAL, AND E. MALLADA, On the explicit role of initialization on the convergence and implicit bias of overparametrized linear networks, in Int. Conf. on Machine Learning, PMLR, 2021, pp. 7760-7768.
[20] W. F. Newns, Functional dependence, The American Mathematical Monthly, 74 (1967), pp. 911-920.
[21] B. NEYSHABUR, Implicit regularization in deep learning, arXiv preprint arXiv:1709.01953, (2017).
[22] B. NEYSHABUR, R. TOMIOKA, AND N. SREBRO, In search of the real inductive bias: On the role of implicit regularization in deep learning, arXiv preprint arXiv:1412.6614, (2014).
[23] E. NOETHER, Invariante variationsprobleme, Nachrichten von der Gesellschaft der Wissenschaften zu Göttingen, Mathematisch-Physikalische Klasse, 1918 (1918), pp. 235-257.
[24] A. M. SAXE, J. L. McCLELLAND, AND S. GANGULI, Exact solutions to the nonlinear dynamics of learning in deep linear neural networks, arXiv preprint arXiv:1312.6120, (2013).
[25] S. SHALEV-SHWARTZ AND S. BEN-DAVID, Understanding machine learning: From theory to algorithms, Cambridge university press, 2014.
[26] D. SOUDRY, E. HOFFER, M. S. NACSON, S. GUNASEKAR, AND N. SREBRO, The implicit bias of gradient descent on separable data, The Journal of Machine Learning Research, 19 (2018), pp. 2822-2878.
[27] P. STOCK AND R. GRIBONVAL, An Embedding of ReLU Networks and an Analysis of their Identifiability, Constructive Approximation, (2022). Publisher: Springer Verlag.
[28] S. TARMOUN, G. FRANCA, B. D. HAEFFELE, AND R. VIDAL, Understanding the dynamics of gradient flow in overparameterized linear models, in Int. Conf. on Machine Learning, PMLR, 2021, pp. 10153-10161.
[29] THE SAGE DEVELOPERS, SageMath, the Sage Mathematics Software System (Version 9.7), 2022. https://www.sagemath.org.
[30] C. ZHANG, S. BENGIO, M. HARDT, B. RECHT, AND O. VINYALS, Understanding deep learning requires rethinking generalization, in Int. Conf. on Learning Representations, 2017.
[31] B. ZHAO, I. GANEV, R. WALTERS, R. YU, AND N. DEHMAMY, Symmetries, flat minima, and the conserved quantities of gradient flow, arXiv preprint arXiv:2210.17216, (2022).