SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Condition (9) holds for classical losses  $\ell$  (e.g. quadratic/logistic losses), as shown in Lemma C.2 in Appendix C. Note that the additional hypothesis of pairwise distinct hyperplanes for the two-layer ReLU case is a generic hypothesis and is usual (see e.g. the notion of twin neurons in [27]). The tools from Appendix C extend Theorem 2.14 beyond (deep) linear and shallow ReLU networks. An open problem is whether the conclusions of Lemma 2.13 still hold for deep ReLU networks.
Theorem 2.14. Under the same assumptions as in Lemma [2.13] we have that for linear neural networks, for all  $\theta \in \Theta \coloneqq \mathbb{R}^D$ :
$$
W _ {\theta} ^ {g} = W _ {\phi} (\theta). \tag {10}
$$
The same result holds for two-layer ReLU networks with  $\phi$  from Example 2.11 and  $\Theta$  the (open) set of all parameters  $\theta$  such that hidden neurons are associated to pairwise distinct "hyperplanes".
This means as claimed that for linear and two-layer ReLU networks, being locally conserved on  $\Theta$  for any data set exactly means being conserved on  $\Theta$  through the finite-dimensional functional space  $W_{\phi} \subseteq \mathcal{C}^{1}(\Theta, \mathbb{R}^{D})$ . This motivates the following definition
Definition 2.15. A real-valued function  $h$  is a conservation law of  $\phi$  if it is conserved through  $W_{\phi}$ .
Proposition 2.5 yields the following intermediate result.
Proposition 2.16.  $h\in \mathcal{C}^1 (\Omega ,\mathbb{R})$  is a conservation law for  $\phi$  if and only if
$$
\nabla h (\theta) \perp \nabla \phi_ {j} (\theta), \forall \theta \in \Omega , \forall j \in \{1, \dots , d \}.
$$
Thanks to Theorem 2.14, the space  $W_{\phi}$  defined in (8) introduces a much simpler proxy to express  $W_{\theta}^{g}$  as a trace of a subset of  $\mathcal{C}^1(\Theta, \mathbb{R}^D)$ . Moreover, when  $\phi$  is  $\mathcal{C}^\infty$ ,  $W_{\phi}$  is a finite-dimensional space of infinitely smooth functions on  $\Theta$ , and this will be crucial in Section 4.1 to provide a tractable scheme (i.e. operating in finite dimension) to compute the maximum number of independent conservation laws, using the Lie algebra computations that will be described in Section 3.
Example 2.17. Revisiting Example [2.8] the function to minimize is factorized by the reparametrization  $\phi : (u \in \mathbb{R}, v \in \mathbb{R}) \mapsto uv \in \mathbb{R}$  with  $\theta := (u, v)$ . We saw that  $h((u, v)) := u^2 - v^2$  is conserved: and indeed  $\langle \nabla h(u, v), \nabla \phi(u, v) \rangle = 2uv - 2vu = 0$ ,  $\forall (u, v)$ .
In this simple example, the characterization of Proposition 2.16 gives a constructive way to find such a conserved function: we only need to find a function  $h$  such that  $\langle \nabla h(u,v), \nabla \phi(u,v) \rangle = \langle \nabla h(u,v), (v,u)^{\top} \rangle = 0$ . The situation becomes more complex in higher dimensions, since one needs to understand the interplay between the different vector fields in  $W_{\phi}$ .
2.5 Constructibility of some conservation laws
Observe that in Example 2.17 both the reparametrization  $\phi$  and the conservation law  $h$  are polynomials, a property that surprisingly systematically holds in all examples of interest in the paper, making it possible to algorithmically construct some conservation laws as detailed now.
By Proposition 2.16, a function  $h$  is a conservation law if it is in the kernel of the linear operator  $h \in \mathcal{C}^1(\Omega, \mathbb{R}) \mapsto (\theta \in \Omega \mapsto (\langle \nabla h(\theta), \nabla \phi_i(\theta) \rangle)_{i=1,\dots,d})$ . Thus, one could look for conservation laws in a prescribed finite-dimensional space by projecting these equations in a basis (as in finite-element methods for PDEs). Choosing the finite-dimensional subspace could be generally tricky, but for the linear and ReLU cases all known conservation laws are actually polynomial "balancedness-type conditions" [1,2,9], see Section 4. In these cases, the vector fields in  $W_\phi$  are also polynomials (because  $\phi$  is polynomial, see Theorem C.4 and Theorem C.5 in Appendix C), hence  $\theta \mapsto \langle \nabla h(\theta), \nabla \phi_i(\theta) \rangle$  is a polynomial too. This allows us to compute a basis of independent polynomial conservation laws of a given degree (to be freely chosen) for these cases, by simply focusing on the corresponding subspace of polynomials. We coded the resulting equations in SageMath, and we found back on selected examples (see Appendix I) all existing known conservation laws both for ReLU and linear networks. Open-source code is available at [18].
2.6 Independent conserved functions
Having an algorithm to build conservation laws is nice, yet how can we know if we have built "all" laws? This requires first defining a notion of a "maximal" set of functions, which would in some sense be independent. This does not correspond to linear independence of the functions themselves (for instance, if  $h$  is a conservation law, then so is  $h^k$  for each  $k \in \mathbb{N}$  but this does not add any other constraint), but rather to pointwise linear independence of their gradients. This notion of independence is closely related to the notion of "functional independence" studied in [7, 20]. For instance, it is shown in [20] that smooth functionally dependent functions are characterized by having dependent gradients everywhere. This motivates the following definition.
Definition 2.18. A family of  $N$  functions  $(h_1, \dots, h_N)$  conserved through  $W \subset \mathcal{C}^1(\Omega, \mathbb{R}^D)$  is said to be independent if the vectors  $(\nabla h_1(\theta), \dots, \nabla h_N(\theta))$  are linearly independent for all  $\theta \in \Omega$ .
An immediate upper bound holds on the largest possible number  $N$  of functionally independent functions  $h_1, \ldots, h_N$  conserved through  $W$ : for  $\theta \in \Omega \subseteq \mathbb{R}^D$ , the space  $\operatorname{span}\{\nabla h_1(\theta), \ldots, \nabla h_N(\theta)\} \subseteq \mathbb{R}^D$  is of dimension  $N$  (by independence) and (by Proposition 2.5) orthogonal to  $W(\theta)$ . Thus, it is necessary to have  $N \leq D - \dim W(\theta)$ . As we will now see, this bound can be tight under additional assumptions on  $W$  related to Lie brackets (corresponding to the so-called Frobenius theorem). This will in turn lead to a characterization of the maximum possible  $N$ .
3 Conservation Laws using Lie Algebra
The study of hyper-surfaces trapping the solution of ODEs is a recurring theme in control theory, since the existence of such surfaces is the basic obstruction of controllability of such systems [6]. The basic result to study these surfaces is the so-called Frobenius theorem from differential calculus (See Section 1.4 of [13] for a good reference for this theorem). It relates the existence of such surfaces, and their dimensions, to some differential condition involving so-called "Lie brackets"  $[u,v]$  between pairs of vector fields (see Section 3.1 below for a more detailed exposition of this operation). However, in most cases of practical interest (such as for instance matrix factorization), the Frobenius theorem is not suitable for a direct application to the space  $W_{\phi}$  because its Lie bracket condition is not satisfied. To identify the number of independent conservation laws, one needs to consider the algebraic closure of  $W_{\phi}$  under Lie brackets. The fundamental object of interest is thus the Lie algebra generated by the Jacobian vector fields, that we recall next. While this is only defined for vector fields with stronger smoothness assumption, the only consequence is that  $\phi$  is required to be infinitely smooth, unlike the loss  $\ell (\cdot ,y)$  and the model  $g(\cdot ,x)$  that can be less smooth. All concretes examples of  $\phi$  in this paper are polynomial hence indeed infinitely smooth.
Notations Given a vector subspace of infinitely smooth vector fields  $W \subseteq \mathcal{X}(\Theta) \coloneqq \mathcal{C}^{\infty}(\Theta, \mathbb{R}^{D})$ , where  $\Theta$  is an open subset of  $\mathbb{R}^{D}$ , we recall (cf Proposition 2.5) that its trace at some  $\theta$  is the subspace
$$
W (\theta) := \operatorname {s p a n} \left\{\chi (\theta): \chi \in W \right\} \subseteq \mathbb {R} ^ {D}. \tag {11}
$$
For each open subset  $\Omega \subseteq \Theta$ , we introduce the subspace of  $\mathcal{X}(\Omega)$ :  $W_{|\Omega} \coloneqq \{\chi_{|\Omega} : \chi \in W\}$ .
3.1 Background on Lie algebra
A Lie algebra  $A$  is a vector space endowed with a bilinear map  $[\cdot, \cdot]$ , called a Lie bracket, that verifies for all  $X, Y, Z \in A$ :  $[X, X] = 0$  and the Jacobi identity:  $[X, [Y, Z]] + [Y, [Z, X]] + [Z, [X, Y]] = 0$ .
For the purpose of this article, the Lie algebra of interest is the set of infinitely smooth vector fields  $\mathcal{X}(\Theta)$ , endowed with the Lie bracket  $[\cdot, \cdot]$  defined by
$$
[ \chi_ {1}, \chi_ {2} ]: \quad \theta \in \Theta \mapsto [ \chi_ {1}, \chi_ {2} ] (\theta) := \partial \chi_ {1} (\theta) \chi_ {2} (\theta) - \partial \chi_ {2} (\theta) \chi_ {1} (\theta), \tag {12}
$$
with  $\partial \chi (\theta)\in \mathbb{R}^{D\times D}$  the jacobian of  $\chi$  at  $\theta$ . The space  $\mathbb{R}^{n\times n}$  of matrices is also a Lie algebra endowed with the Lie bracket  $[A,B]\coloneqq AB - BA$ . This can be seen as a special case of (12) in the case of linear vector fields, i.e.  $\chi (\theta) = A\theta$ .
Generated Lie algebra Let  $A$  be a Lie algebra and let  $W \subset A$  be a vector subspace of  $A$ . There exists a smallest Lie algebra that contains  $W$ . It is denoted  $\operatorname{Lie}(W)$  and called the generated Lie algebra of  $W$ . The following proposition [6, Definition 20] constructively characterizes  $\operatorname{Lie}(W)$ , where for vector subspaces  $[W, W'] := \{[\chi_1, \chi_2] : \chi_1 \in W, \chi_2 \in W'\}$ , and  $W + W' = \{\chi_1 + \chi_2 : \chi_1 \in W, \chi_2 \in W'\}$ .
Proposition 3.1. Given any vector subspace  $W \subseteq A$  we have  $\operatorname{Lie}(W) = \bigcup_{k} W_{k}$  where:
$$
\left\{ \begin{array}{l l} W _ {0} & := W \\ W _ {k} & := W _ {k - 1} + [ W _ {0}, W _ {k - 1} ] f o r k \geq 1. \end{array} \right.
$$
We will see in Section 3.2 that the number of conservation laws is characterized by the dimension of the trace  $\operatorname{Lie}(W_{\phi})(\theta)$  defined in (11). The following lemma (proved in Appendix D) gives a stopping criterion to algorithmically determine this dimension (see Section 3.3 for the algorithm).
Lemma 3.2. Given  $\theta \in \Theta$ , if for a given  $i$ ,  $\dim W_{i+1}(\theta') = \dim W_i(\theta)$  for every  $\theta'$  in a neighborhood of  $\theta$ , then there exists a neighborhood  $\Omega$  of  $\theta$  such that  $W_k(\theta') = W_i(\theta')$  for all  $\theta' \in \Omega$  and  $k \geq i$ , where the  $V_i$  are defined by Proposition 3.1. Thus  $\operatorname{Lie}(W)(\theta') = W_i(\theta')$  for all  $\theta' \in \Omega$ . In particular, the dimension of the trace of  $\operatorname{Lie}(W)$  is locally constant and equal to the dimension of  $W_i(\theta)$ .
3.2 Number of conservation laws
The following theorem uses the Lie algebra generated by  $W_{\phi}$  to characterize the number of conservation laws. The proof of this result is based on two successive uses of the Frobenius theorem and can be found in Appendix E (where we also recall Frobenius theorem for the sake of completeness).
Theorem 3.3. If  $\dim(\operatorname{Lie}(W_{\phi})(\theta))$  is locally constant then each  $\theta \in \Omega \subseteq \mathbb{R}^D$  admits a neighborhood  $\Omega'$  such that there are  $D - \dim(\operatorname{Lie}(W_{\phi})(\theta))$  (and no more) independent conserved functions through  $W_{\phi|_{\Omega'}}$ , i.e., there are  $D - \dim(\operatorname{Lie}(W_{\phi})(\theta))$  independent conservation laws of  $\phi$  on  $\Omega'$ .
Remark 3.4. The proof of the Frobenius theorem (and therefore of our generalization Theorem 3.3) is actually constructive. From a given  $\phi$ , conservation laws are obtained in the proof by integrating in time (i.e. solving an advection equation) the vector fields belonging to  $W_{\phi}$ . Unfortunately, this cannot be achieved in closed form in general, but in small dimensions, this could be carried out numerically (to compute approximate discretized laws on a grid or approximate them using parametric functions such as Fourier expansions or neural networks).
A fundamental aspect of Theorem 3.3 is to rely only on the dimension of the trace of the Lie algebra associated with the finite-dimensional vector space  $W_{\phi}$ . Yet, even if  $W_{\phi}$  is finite-dimensional, it might be the case that  $\operatorname{Lie}(W_{\phi})$  itself remains infinite-dimensional. Nevertheless, what matters is not the dimension of  $\operatorname{Lie}(W_{\phi})$ , but that of its trace  $\operatorname{Lie}(W_{\phi})(\theta)$ , which is always finite (and potentially much smaller than  $\dim \operatorname{Lie}(W_{\phi})$  even when the latter is finite) and computationally tractable thanks to Lemma 3.2 as detailed in Section 3.3. In section 4.1 we work out the example of matrix factorization, a non-trivial case where the full Lie algebra  $\operatorname{Lie}(W_{\phi})$  itself remains finite-dimensional.
Theorem [3.3] requires that the dimension of the trace at  $\theta$  of the Lie algebra is locally constant. This is a technical assumption, which typically holds outside a set of pathological points. A good example is once again matrix factorization, where we show in Section [4.1] that this condition holds generically.
3.3 Method and algorithm, with examples
Given a reparametrization  $\phi$  for the architectures to train, to determine the number of independent conservation laws of  $\phi$ , we leverage the characterization 3.1 to algorithmically compute  $\dim(\operatorname{Lie}(W_{\phi})(\theta))$  using an iterative construction of bases for the subspaces  $W_{k}$  starting from  $W_{0} \coloneqq W_{\phi}$ , and stopping as soon as the dimension stagnates thanks to Lemma 3.2. Our open-sourced code is available at [18] and uses SageMath. As we now show, this algorithmic principle allows to fully work out certain settings where the stopping criterion of Lemma 3.2 is reached at the first step ( $i = 0$ ) or the second one ( $i = 1$ ). Section 4.2 also discusses its numerical use for an empirical investigation of broader settings.
Example where the iterations of Lemma 3.2 stop at the first step. This corresponds to the case where  $\operatorname{Lie}W_{\phi}(\theta) = W_1(\theta) = W_0(\theta) \coloneqq W_{\phi}(\theta)$  on  $\Omega$ . This is the case if and only if  $W_{\phi}$  satisfies that
$$
[ \chi_ {1}, \chi_ {2} ] (\theta) := \partial \chi_ {1} (\theta) \chi_ {2} (\theta) - \partial \chi_ {2} (\theta) \chi_ {1} (\theta) \in W _ {\phi} (\theta), \quad \text {f o r a l l} \chi_ {1}, \chi_ {2} \in W _ {\phi} \text {a n d a l l} \theta \in \Omega . \tag {13}
$$
i.e., when Frobenius Theorem (see Theorem E.1 in Appendix E) applies directly. The first example is a follow-up to Example 2.11.
Example 3.5 (two-layer ReLU networks without bias). Consider  $\theta = (U,V)$  with  $U\in \mathbb{R}^{n\times r}$ ,  $V\in \mathbb{R}^{m\times r}$ ,  $n,m,r\geq 1$  (so that  $D = (n + m)r$ ), and the reparametrization  $\phi (\theta)\coloneqq (u_i v_i^\top)_{i = 1,\dots ,r}\in$ $\mathbb{R}^{n\times m\times r}$ , where  $U = (u_{1};\dots ;u_{r})$  and  $V = (v_{1};\dots ;v_{r})$ . As detailed in Appendix F.1, since  $\phi (\theta)$  is a collection of  $r$  rank-one  $n\times m$  matrices,  $\dim (W_{\phi}(\theta)) = \mathrm{rank}\partial \phi (\theta) = (n + m - 1)r$  is constant on the domain  $\Omega$  such that  $u_{i},v_{j}\neq 0$ , and  $W_{\phi}$  satisfies (13), hence by Theorem 3.3 each  $\theta$  has a neighborhood  $\Omega^{\prime}$  such that there exists  $r$  (and no more) independent conserved function through  $W_{\phi | \Omega '}$ . The  $r$  known conserved functions [9] given by  $h_i:(U,V)\mapsto \| u_i\| ^2 -\| v_i\| ^2,i = 1,\dots ,r,$  are independent, hence they are complete.
Example where the iterations of Lemma 3.2 stop at the second step (but not the first one). Our primary example is matrix factorization, as a follow-up to Example 2.10.
Example 3.6 (two-layer linear neural networks). With  $\theta = (U,V)$ , where  $(U\in \mathbb{R}^{n\times r},V\in \mathbb{R}^{m\times r})$  the reparameterization  $\phi (\theta)\coloneqq UV^{\top}\in \mathbb{R}^{n\times m}$  (here  $d = nm$ ) factorizes the functions minimized during the training of linear two-layer neural networks (see Example 2.10). As shown in Appendix I, condition (13) is not satisfied when  $r > 1$  and  $\max (n,m) > 1$ . Thus, the stopping criterion of Lemma 3.2 is not satisfied at the first step. However, as detailed in Proposition H.3 in Appendix H.1  $(W_{\phi})_{1} = (W_{\phi})_{2} = \mathrm{Lie}(W_{\phi})$ , hence the iterations of Lemma 3.2 stop at the second step.
We complete this example in the next section by showing (Corollary 4.4) that known conservation laws are indeed complete. Whether known conservation laws remain valid and/or complete in this settings and extended ones is further studied in Section 4 and Appendix 5.
3.4 Application: recasting over-parameterized flows as low-dimensional Riemannian flows
As we now show, one striking application of Theorem 3.3 (in simple cases where  $\dim(W_{\phi}(\theta)) = \dim(\operatorname{Lie}W_{\phi}(\theta))$  is constant on  $\Omega$ , i.e.,  $\operatorname{rank}(\partial \phi(\theta))$  is constant on  $\Omega$  and  $W_{\phi}$  satisfies (13)) is to fully rewrite the high-dimensional flow  $\theta(t) \in \mathbb{R}^D$  as a low-dimensional flow on  $z(t) := \phi(\theta(t)) \in \mathbb{R}^d$ , where this flow is associated with a Riemannian metric tensor  $M$  that is induced by  $\phi$  and depends on the initialization  $\theta_{\mathrm{init}}$ . We insist on the fact that this is only possible in very specific cases, but this phenomenon is underlying many existing works that aim at writing in closed form the implicit bias associated with some training dynamics (see Section 1 for some relevant literature. Our analysis sheds some light on cases where this is possible, as shown in the next proposition.
Proposition 3.7. Assume that  $\operatorname{rank}(\partial \phi(\theta))$  is constant on  $\Omega$  and that  $W_{\phi}$  satisfies (13). If  $\theta(t) \in \mathbb{R}^D$  satisfies the ODE (2) where  $\theta_{init} \in \Omega$ , then there is  $0 < T_{\theta_{init}}^{\star} \leq T_{\theta_{init}}$  such that  $z(t) \coloneqq \phi(\theta(t)) \in \mathbb{R}^d$  satisfies the ODE
$$
\dot {z} (t) = - M \left(z (t), \theta_ {\text {i n i t}}\right) \nabla f (z (t)) \quad \text {f o r a l l} t \in \left[ 0, T _ {\theta_ {\text {i n i t}}} ^ {\star}\right), \text {w i t h} z (0) = \phi \left(\theta_ {\text {i n i t}}\right), \tag {14}
$$
where  $M(z(t),\theta_{init})\in \mathbb{R}^{d\times d}$  is a symmetric positive semi-definite matrix.
See Appendix G for a proof. Revisiting Example 3.5 leads to the following analytic example.
Example 3.8. Given the reparametrization  $\phi : (u \in \mathbb{R}^*, v \in \mathbb{R}^d) \mapsto uv \in \mathbb{R}^d$ , the variable  $z := uv$  satisfies (14) with  $M(z, \theta_{\mathrm{init}}) = \|z\|_{\delta} \mathrm{I}_d + \|z\|_{\delta}^{-1} z z^\top$ , with  $\|z\|_{\delta} := \delta + \sqrt{\delta^2 + \|z\|^2}$ ,  $\delta := 1/2(u_{\mathrm{init}}^2 - \|v_{\mathrm{init}}\|^2)$ .
Another analytic example is discussed in Appendix G. In light of these results, an interesting perspective is to better understand the dependence of the Riemannian metric with respect to initialization, to possibly guide the choice of initialization for better convergence dynamics.
Note that the metric  $M(z, \theta_{\mathrm{init}})$  can have a kernel. Indeed, in practice, while  $\phi$  is a function from  $\mathbb{R}^D$  to  $\mathbb{R}^d$ , the dimensions often satisfy  $\operatorname{rank} \partial \phi(\theta) < \min(d, D)$ , i.e.,  $\phi(\theta)$  lives in a manifold of lower dimension. The evolution (14) should then be understood as a flow on this manifold. The kernel of  $M(z, \theta_{\mathrm{init}})$  is orthogonal to the tangent space at  $z$  of this manifold.
4 Conservation Laws for Linear and ReLU Neural Networks
To showcase the impact of our results, we show how they can be used to determine whether known conservation laws for linear (resp. ReLU) neural networks are complete, and to recover these laws algorithmically using reparametrizations  $\phi$  adapted to these two settings. Concretely, we study the conservation laws for neural networks with  $q$  layers, and either a linear or ReLU activation, with an emphasis on  $q = 2$ . We write  $\theta = (U_1, \dots, U_q)$  with  $U_i \in \mathbb{R}^{n_i - 1 \times n_i}$  the weight matrices and we assume that  $\theta$  satisfies the gradient flow [2]. In the linear case the reparametrization is  $\phi_{\mathrm{Lin}}(\theta) := U_1 \dots U_q$ . For ReLU networks, we use the (polynomial) reparametrization  $\phi_{\mathrm{ReLu}}$  of [27, Definition 6], which is defined for any (deep) feedforward ReLU network, with or without bias. In the simplified setting of networks without biases it reads explicitly as:
$$
\phi_ {\mathrm {R e L u}} \left(U _ {1}, \dots , U _ {q}\right) := \left(U _ {1} [:, j _ {1} ] U _ {2} [ j _ {1}, j _ {2} ] \dots U _ {q - 1} [ j _ {q - 2}, j _ {q - 1} ] U _ {q} [ j _ {q - 1}, : ]\right) _ {j _ {1}, \dots , j _ {q - 1}} \tag {15}
$$
with  $U[i,j]$  the  $(i,j)$ -th entry of  $U$ . This covers  $\phi(\theta) := (u_j v_j^\top)_{j=1}^r \in \mathbb{R}^{n \times m \times r}$  from Example 2.11. Some conservation laws are known for the linear case  $\phi_{\mathrm{Lin}}$  [1, 2] and for the ReLu case  $\phi_{\mathrm{ReLu}}$  [9].
Proposition 4.1 ([129]). If  $\theta \coloneqq (U_1, \dots, U_q)$  satisfies the gradient flow (2), then for each  $i = 1, \dots, q-1$  the function  $\theta \mapsto U_i^\top U_i - U_{i+1} U_{i+1}^\top$  (resp. the function  $\theta \mapsto \operatorname{diag}(U_i^\top U_i - U_{i+1} U_{i+1}^\top))$  defines  $n_i \times (n_i + 1)/2$  conservation laws for  $\phi_{\mathrm{Lin}}$  (resp.  $n_i$  conservation laws for  $\phi_{\mathrm{ReLu}}$ ).
Proposition 4.1 defines  $\sum_{i=1}^{q-1} n_i \times (n_i + 1)/2$  conserved functions for the linear case. In general, they are not independent, and we give below in Proposition 4.2 for the case of  $q = 2$ , the exact
number of independent conservation laws among these particular laws. Establishing whether there are other (previously unknown) conservation laws is an open problem for  $q > 2$ . We already answered negatively to this question in the two-layer ReLu case without bias (See Example 3.5). In the following Section (Corollary 4.4), we show the same result in the linear case  $q = 2$ . Numerical computations suggest this is still the case for deeper linear and ReLU networks as detailed in Section 4.2.
4.1 The matrix factorization case  $(q = 2)$
To simplify the analysis when  $q = 2$ , we rewrite  $\theta = (U,V)$  as a vertical matrix concatenation denoted  $(U;V)\in \mathbb{R}^{(n + m)\times r}$ , and  $\phi (\theta) = \phi_{\mathrm{Lin}}(\theta) = UV^{\top}\in \mathbb{R}^{n\times m}$ .
How many independent conserved functions are already known? The following proposition refines Proposition 4.1 for  $q = 2$  by detailing how many independent conservation laws are already known. See Appendix H.1 for a proof.
Proposition 4.2. Consider  $\Psi : \theta = (U; V) \mapsto U^\top U - V^\top V \in \mathbb{R}^{r \times r}$  and assume that  $(U; V)$  has full rank noted  $\mathbf{rk}$ . Then the function  $\Psi$  gives  $\mathbf{rk} \cdot (2r + 1 - \mathbf{rk}) / 2$  independent conserved functions.