SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Arthur Jacot
Courant Institute of Mathematical Sciences
New York, NY 10012
arthur.jacot@nyu.edu
Previous work [Jac23] has shown that DNNs with large depth  $L$  and  $L_{2}$ -regularization are biased towards learning low-dimensional representations of the inputs, which can be interpreted as minimizing a notion of rank  $R^{(0)}(f)$  of the learned function  $f$ , conjectured to be the Bottleneck rank. We compute finite depth corrections to this result, revealing a measure  $R^{(1)}$  of regularity which bounds the pseudo-determinant of the Jacobian  $|Jf(x)|_{+}$  and is subadditive under composition and addition. This formalizes a balance between learning low-dimensional representations and minimizing complexity/irregularity in the feature maps, allowing the network to learn the 'right' inner dimension. Finally, we prove the conjectured bottleneck structure in the learned features as  $L \to \infty$ : for large depths, almost all hidden representations are approximately  $R^{(0)}(f)$ -dimensional, and almost all weight matrices  $W_{\ell}$  have  $R^{(0)}(f)$  singular values close to 1 while the others are  $O(L^{-\frac{1}{2}})$ . Interestingly, the use of large learning rates is required to guarantee an order  $O(L)$  NTK which in turns guarantees infinite depth convergence of the representations of almost all layers.
The representation cost  $R(f;L) = \min_{\theta :f_{\theta} = f}\| \theta \| ^2$  [DKS21] can be defined for any model and describes the bias in function space resulting from the minimization of the  $L_{2}$ -norm of the parameters. While it can be computed explicitly for linear networks [DKS21] or shallow nonlinear ones [Bac17], the deep non-linear case remains ill-understood [JGHG22].
Previous work [Jac23] has shown that the representation cost of DNNs with homogeneous nonlinearity  $\sigma$  converges to a notion of rank over nonlinear functions  $\lim_{L\to \infty}\frac{R(f;L)}{L}\rightarrow R^{(0)}(f)$ . Over a large set of functions  $f$ , the limiting representation cost  $R^{(0)}(f)$  was proven the so-called Bottleneck (BN) rank  $\operatorname{Rank}_{BN}(f)$  which is the smallest integer  $k$  such that  $f:\mathbb{R}^{d_{in}}\to \mathbb{R}^{d_{out}}$  can be factored  $f = \mathbb{R}^{d_{in}}\xrightarrow{g}\mathbb{R}^k\xrightarrow{h}\mathbb{R}^{d_{out}}$  with inner dimension  $k$  (it is conjectured to match it everywhere). This suggests that large depth  $L_{2}$ -regularized DNNs are adapted for learning functions of the form  $f^{*} = g\circ h$  with small inner dimension.
This can also be interpreted as DNNs learning symmetries, since a function  $f: \Omega \to \mathbb{R}^{d_{out}}$  with symmetry group  $G$  (i.e.  $f(g \cdot x) = f(x)$ ) can be defined as mapping the inputs  $\Omega$  to an embedding of the modulo space  $\Omega / G$  and then to the outputs  $\mathbb{R}^{d_{out}}$ . Thus a function with a lot of symmetries will have a small BN-rank, since  $\mathrm{Rank}_{BN}(f) \leq \dim(\Omega / G)$  where  $\dim(\Omega / G)$  is the smallest dimension  $\Omega / G$  can be embedded into.
A problem is that this notion of rank does not control the regularity of  $f$ , but results of [Jac23] suggest that a measure of regularity might be recovered by studying finite depths corrections to the  $R^{(0)}$  approximation. This formalizes the balance between minimizing the dimension of the learned features and their complexity.
Another problem is that minimizing the rank  $R^{(0)}$  does not uniquely describe the learned function, as there are many fitting functions with the same rank. Corrections allow us to identify the learned function amongst those.
Finally, the theoretical results and numerical experiments of [Jac23] strongly suggest a bottleneck structure in the learned features for large depths, where the (possibly) high dimensional input data is mapped after a few layers to a low-dimensional hidden representation, and keeps the approximately same dimensionality until mapping back to the high dimensional outputs in the last few layers. We prove the existence of such a structure, but with potentially multiple bottlenecks.
We analyze the Taylor approximation of the representation cost around infinite depth  $L = \infty$ :
$$
R (f; \Omega , L) = L R ^ {(0)} (f; \Omega) + R ^ {(1)} (f; \Omega) + \frac {1}{L} R ^ {(2)} (f; \Omega) + O (L ^ {- 2}).
$$
The first correction  $R^{(1)}$  measures some notion of regularity of the function  $f$  that behaves subadditively under composition  $R^{(1)}(f \circ g) \leq R^{(1)}(f) + R^{(1)}(g)$  and under addition  $R^{(1)}(f + g) \leq R^{(1)}(f) + R^{(1)}(g)$  (under some constraints), and controls the Jacobian of  $f$ :  $\forall x, 2\log |Jf(x)|_{+} \leq R^{(1)}(f)$ , where  $|\cdot|_{+}$  is the pseudo-determinant, the product of the non-zero singular values.
This formalizes the balance between the bias towards minimizing the inner dimension described by  $R^{(0)}$  and a regularity measure  $R^{(1)}$ . As the depth  $L$  grows, the low-rank bias  $R^{(0)}$  dominates, but even in the infinite depth limit the regularity  $R^{(1)}$  remains relevant since there are typically multiple fitting functions with matching  $R^{(0)}$  which can be differentiated by their  $R^{(1)}$  value.
For linear networks, the second correction  $R^{(2)}$  guarantees infinite depth convergence of the representations of the network. We recover several properties of  $R^{(2)}$  in the nonlinear case, but we also give a counterexample that shows that norm minimization does not guarantee convergence of the representation, forcing us to look at other sources of bias.
To solve this issue, we show that a  $\Theta (L^{-1})$  learning rate forces the NTK to be  $O(L)$  which in turn guarantees the convergence as  $L\to \infty$  of the representations at almost every layer of the network.
Finally we prove the Bottleneck structure that was only observed empirically in [Jac23]: we show that the weight matrices  $W_{\ell}$  are approximately rank  $R^{(0)}(f)$ , more precisely  $W_{\ell}$  has  $R^{(0)}(f)$  singular values that are  $O(L^{-\frac{1}{2}})$  close to 1 and all the other are  $O(L^{-\frac{1}{2}})$ . Together with the  $O(L)$  NTK assumption, this implies that the pre-activations  $\alpha_{\ell}(X)$  of a general dataset at almost all layers is approximately  $R^{(0)}(f)$ -dimensional, more precisely that the  $k + 1$ -th singular value of  $\alpha_{\ell}(X)$  is  $O(L^{-\frac{1}{2}})$ .
The representation cost has mostly been studied in settings where an explicit formula can be obtained, such as in linear networks [DKS21], or shallow nonlinear networks [Bac17], or for deep networks with very specific structure [OW22, LJ22]. A low rank phenomenon in large depth  $L_{2}$ -regularized DNNs has been observed in [TVS22].
Regarding deep fully-connected networks, two reformulations of the representation cost optimization have been given in [JGHG22], which also shows that the representation becomes independent of the width as long as the width is large enough.
The Bottleneck structure that we describe in this paper is similar to the Information Bottleneck theory [TZ15]. It is not unlikely that the bias towards dimension reduction in the middle layers of the network could explain the loss of information that was observed in the first layers of the network in [TZ15].
In this paper, we study fully connected DNNs with  $L + 1$  layers numbered from 0 (input layer) to  $L$  (output layer). The layer  $\ell \in \{0,\dots ,L\}$  has  $n_\ell$  neurons, with  $n_0 = d_{in}$  the input dimension and  $n_L = d_{out}$  the output dimension. The pre-activations  $\tilde{\alpha}_{\ell}(x)\in \mathbb{R}^{n_{\ell}}$  and activations  $\alpha_{\ell}(x)\in \mathbb{R}^{n_{\ell}}$  are defined by
$$
\begin{array}{l} \alpha_ {0} (x) = x \\ \tilde {\alpha} _ {\ell} (x) = W _ {\ell} \alpha_ {\ell - 1} (x) + b _ {\ell} \\ \alpha_ {\ell} (x) = \sigma (\tilde {\alpha} _ {\ell} (x)), \\ \end{array}
$$
for the  $n_{\ell} \times n_{\ell-1}$  connection weight matrix  $W_{\ell}$ , the  $n_{\ell}$ -dim bias vector  $b_{\ell}$  and the nonlinearity  $\sigma: \mathbb{R} \to \mathbb{R}$  applied entry-wise to the vector  $\tilde{\alpha}_{\ell}(x)$ . The parameters of the network are the collection of all connection weights matrices and bias vectors  $\theta = (W_1, b_1, \ldots, W_L, b_L)$ . The network function  $f_{\theta}: \mathbb{R}^{d_{in}} \to \mathbb{R}^{d_{out}}$  is the function that maps an input  $x$  to the pre-activations of the last layer  $\tilde{\alpha}_L(x)$ .
We assume that the nonlinearity is of the form  $\sigma_{a}(x) = \left\{ \begin{array}{ll} x & \text{if } x \geq 0 \\ ax & \text{otherwise} \end{array} \right.$  for some  $\alpha \in (-1,1)$  (yielding the ReLU for  $\alpha = 0$ ), as any homogeneous nonlinearity  $\sigma$  (that is not proportional to the identity function, the constant zero function or the absolute function) matches  $\sigma_{a}$  up to scaling and inverting the inputs.
The functions that can be represented by networks with homogeneous nonlinearities and any finite depth/width are exactly the set of finite piecewise linear functions (FPLF) [ABMM18, HLXZ18].
Remark 1. In most of our results, we assume that the width is sufficiently large so that the representation cost matches the infinite-width representation cost. For a dataset of size  $N$ , a width of  $N(N + 1)$  suffices, as shown in [JGHG22] (though a much smaller width is often sufficient).
2.1 Representation Cost
The representation cost  $R(f; \Omega, L)$  is the minimal squared parameter norm required to represent the function  $f$  over the input domain  $\Omega$ :
$$
R(f;\Omega ,L) = \min_{\theta :f_{\theta |\Omega} = f_{|\Omega}}\| \theta \|^{2}
$$
where the minimum is taken over all weights  $\theta$  of a depth  $L$  network (with some finite widths  $n_1, \ldots, n_{L-1}$ ) such that  $f_\theta(x) = f(x)$  for all  $x \in \Omega$ . If no such weights exist, we define  $R(f; \Omega, L) = \infty$ .
The representation cost describes the natural bias on the represented function  $f_{\theta}$  induced by adding  $L_{2}$  regularization on the weights  $\theta$ :
$$
\min _ {\theta} C (f _ {\theta}) + \lambda \| \theta \| ^ {2} = \min _ {f} C (f) + \lambda R (f; \Omega , L)
$$
for any cost  $C$  (defined on functions  $f:\Omega \mapsto \mathbb{R}^{d_{out}}$ ) and where the minimization on the right is over all functions  $f$  that can be represented by a depth  $L$  network with nonlinearity  $\sigma$ .
For any two functions  $f, g$ , we denote  $f \to g$  the function  $h$  such that  $g = h \circ f$ , assuming it exists, and we write  $R(f \to g; \Omega, L) = R(h; f(\Omega), L)$ .
Remark 2. The representation cost also describes the implicit bias of networks trained with the cross-entropy loss  $\mathrm{[SHN^{+}18,GLSS18,CB20]}$
3 Representation Cost Decomposition
Since there are no explicit formula for the representation cost of deep nonlinear networks, we propose to approximate it by a Taylor decomposition in  $1 / L$  around  $L = \infty$ . This is inspired by the behavior of the representation cost of deep linear networks (which represent a matrix as a product  $A_{\theta} = W_{L}\dots W_{1}$ ), for which an explicit formula exists [DKS21]:
$$
R (A; L) = \min  _ {\theta : A = A _ {\theta}} \| \theta \| ^ {2} = L \| A \| _ {2 / L} ^ {2 / L} = L \sum_ {i = 1} ^ {\operatorname {R a n k} A} s _ {i} (A) ^ {2 / L},
$$
where  $\| \cdot \|_p^p$  is the  $L_p$ -Schatten norm, the  $L_p$  norm on the singular values  $s_i(A)$  of  $A$ .
Approximating  $s^{\frac{2}{L}} = 1 + \frac{2}{L} \log s + \frac{2}{L^2} (\log s)^2 + O(L^{-3})$ , we obtain
$$
R (A; L) = L \mathrm {R a n k} A + 2 \log | A | _ {+} + \frac {1}{2 L} \left\| \log_ {+} A ^ {T} A \right\| ^ {2} + O (L ^ {- 2}),
$$
where  $\log_{+}$  is the pseudo-log, which replaces the non-zero eigenvalues of  $A$  by their log.
We know that gradient descent will converge to parameters  $\theta$  representing a matrix  $A_{\theta}$  that locally minimize the loss  $C(A) + \lambda R(A;L)$ . The approximation  $R(A;L)\approx L\mathrm{Rank}A$  fails to recover the local minima of this loss, because the rank has zero derivatives almost everywhere. But this problem is alleviated with the second order approximation  $R(A;L)\approx L\mathrm{Rank}A + 2\log |A|_{+}$ . The minima can then be interpreted as first minimizing the rank, and then choosing amongst same rank solutions the matrix with the smallest pseudo-determinant. Changing the depth allows one to tune the balance between minimizing the rank and the regularity of the matrix  $A$ .
3.1 First Correction: Regularity Control
As a reminder, the dominating term in the representation cost  $R^{(0)}(f)$  is conjectured in [Jac23] to converge to the so-called Bottleneck rank  $\mathrm{Rank}_{BN}(f;\Omega)$  which is the smallest integer  $k$  such that  $f$  can be decomposed as  $f = \Omega \xrightarrow{g} \mathbb{R}^k \xrightarrow{h} \mathbb{R}^{d_{out}}$  with inner dimension  $k$ , and where  $g$  and  $h$  are FPLF. A number of results supporting this conjecture are proven in [Jac23]: a sandwich bound
$$
\operatorname {R a n k} _ {J} (f; \Omega) \leq R ^ {(0)} (f; \Omega) \leq \operatorname {R a n k} _ {B N} (f; \Omega)
$$
for the Jacobian rank  $\mathrm{Rank}_J(f;\Omega) = \max_x\mathrm{Rank}Jf(x)_{|T_x\Omega}$ , and three natural properties of ranks that  $R^{(0)}$  satisfies:
1.  $R^{(0)}(f\circ g;\Omega)\leq \min \left\{R^{(0)}(f),R^{(0)}(g)\right\} ,$
2.  $R^{(0)}(f + g;\Omega)\leq R^{(0)}(f) + R^{(0)}(g),$
3.  $R^{(0)}(x\mapsto Ax;\Omega) = \mathrm{Rank}A$  for any full dimensional and bounded  $\Omega$
These results imply that for any function  $f = \phi \circ A \circ \psi$  that is linear up to bijections  $\phi, \psi$ , the conjecture is true  $R^{(0)}(f; \Omega) = \operatorname{Rank}_{BN}(f; \Omega) = \operatorname{Rank}A$ .
The proof of the aforementioned sandwich bound in [Jac23] actually proves an upper bound of the form  $L\mathrm{Rank}_{BN}(f;\Omega) + O(1)$  thus proving that the  $R^{(1)}$  term is upper bounded. The following theorem proves a lower bound on  $R^{(1)}$  as well as some of its properties:
Theorem 3. For all inputs  $x$  where  $\operatorname{Rank} Jf(x) = R^{(0)}(x)$ ,  $R^{(1)}(f) \geq 2\log |Jf(x)|_{+}$ , furthermore:
1. If  $R^{(0)}(f \circ g) = R^{(0)}(f) = R^{(0)}(g)$ , then  $R^{(1)}(f \circ g) \leq R^{(1)}(f) + R^{(1)}(g)$ .
2. If  $R^{(0)}(f + g) = R^{(0)}(f) + R^{(0)}(g)$ , then  $R^{(1)}(f + g) \leq R^{(1)}(f) + R^{(1)}(g)$ .
3. If  $P_{\mathrm{Im}A^T}\Omega$  and  $A\Omega$  are  $k = \mathrm{Rank}A$  dimensional and completely positive (i.e. they can be embedded isometrically into  $\mathbb{R}_+^m$  for some  $m$ ), then  $R^{(1)}(x \mapsto Ax; \Omega) = 2\log |A|_+$ .
Notice how these properties clearly point to the first correction  $R^{(1)}(f)$  measuring a notion of regularity of  $f$  instead of a notion of rank. One can think of  $L_{2}$ -regularized deep nets as learning functions  $f$  that minimize
$$
\min _ {f} C (f (X)) + \lambda L R ^ {(0)} (f) + \lambda R ^ {(1)} (f).
$$
The depth determines the balance between the rank regularization and regularity regularization. Without the  $R^{(1)}$ -term, the above minimization would never be unique since there can be multiple functions  $f$  with the same training outputs  $f(X)$  with the same rank  $R^{(0)}(f)$ .
Under the assumption that  $R^{(0)}$  only takes integer value the above optimization can be rewritten as
$$
\min  _ {k = 0, 1, \ldots} \lambda L k + \min  _ {f: R ^ {(0)} (f) = k} C (f (X)) + \lambda R ^ {(1)} (f).
$$
(a) Parameter norm and depth
(b) Bottleneck structure at different depths.
Figure 1: (a) Plot of the parameter norm at the end of training  $(\lambda = 0.001)$  over a range of depths, colored according to the rank (# of sing. vals above 0.1) of the weight matrices  $W_{L/2}$  in the middle of the network, and marked with a dot ',' or cross 'x' depending on whether the final train cost is below or above 0.1. The training data is synthetic and designed to have an optimal rank  $k^* = 2$ . We see different ranges of depth where the network converges to different rank, with larger depths leading to smaller rank, until training fails and recover the zero parameters for  $L > 25$ . Within each range the norm  $\|\theta\|^2$  is well approximated by a affine function with slope equal to the rank. (b) Plot of the singular values of  $W_\ell$  throughout the networks for 4 trials, we see that the bottleneck structure remains essentially the same throughout each range of depth, with only the middle low-rank part growing with the depth.
Every inner minimization for a rank  $k$  that is attained inside the set  $\{f : R^{(0)}(f) = k\}$  corresponds to a different local minimum. Note how these inner minimization do not depend on the depth, suggesting the existence of sequences of local minima for different depths that all represent approximately the same function  $f$ , as can be seen in Figure 1. We can classify these minima according to whether they recover the true BN-rank  $k^*$  of the task, underestimate or overestimate it.
In linear networks [WJ23], rank underestimating minima cannot fit the data, but it is always possible to fit any data with a BN-rank 1 function (by mapping injectively the datapoints to a line and then mapping them nonlinearly to any other configuration). We therefore need to also differentiate between rank-underestimating minima that fit or do not fit the data. The non-fitting minima can in theory be avoided by taking a small enough ridge (along the lines of [WJ23]), but we do observe them empirically for large depths in Figure 1.
In contrast, we have never observed fitting rank-underestimating minima, though their existence was proven for large enough depths in [Jac23]. A possible explanation for why GD avoids these minima is their  $R^{(1)}$  value explodes with the number of datapoints  $N$ , since these network needs to learn a space filling surface (a surface of dimension  $k < k^*$  that visits random outputs  $y_i$  that are sampled from a  $k^*$ -dimensional distribution). More precisely Theorem 2 of [Jac23] implies that the  $R^{(1)}$  value of fitting BN-rank 1 minima explodes at a rate of  $2\left(1 - \frac{1}{k^*}\right)\log N$  as the number of datapoints  $N$  grows, which could explain why we rarely observe such minima in practice, but another explanation could be that these minima are very narrow, as explained in Section 4.1.
In our experiments we often encountered rank overestimating minima and we are less sure about how to avoid them, though it seems that increasing the depth helps (see Figure 1), and that SGD might help too by analogy with the linear case [WJ23]. Thankfully overestimating the rank is less problematic for generalization, as supported by the fact that it is possible to approximate BN-rank  $k^*$  with a higher rank function with any accuracy, while doing so with a low rank function requires a pathological function.
3.2 Second Correction
We now identify a few properties of the second correction  $R^{(2)}$ :
Proposition 4. If there is a limiting representation as  $L \to 0$  in the optimal representation of  $f$ , then  $R^{(2)}(f) \geq 0$ . Furthermore:
1. If  $R^{(0)}(f \circ g) = R^{(0)}(f) = R^{(0)}(g)$  and  $R^{(1)}(f \circ g) = R^{(1)}(f) + R^{(1)}(g)$ , then  $\sqrt{R^{(2)}(f \circ g)} \leq \sqrt{R^{(2)}(f)} + \sqrt{R^{(2)}(g)}$ .
2. If  $R^{(0)}(f + g) = R^{(0)}(f) + R^{(0)}(g)$  and  $R^{(1)}(f + g) = R^{(1)}(f) + R^{(1)}(g)$ , then  $R^{(2)}(f + g) \leq R^{(2)}(f) + R^{(2)}(g)$ .
3. If  $A^p\Omega$  is  $k = \mathrm{Rank}A$ -dimensional and completely positive for all  $p \in [0,1]$ , where  $A^p$  has its non-zero singular taken to the  $p$ -th power, then  $R^{(2)}(x \mapsto Ax; \Omega) = \frac{1}{2}\left\| \log_+A^TA\right\|^2$ .
While the properties are very similar to those of  $R^{(1)}$ , the necessary conditions necessary to apply them are more restrictive. There might be case where the first two terms  $R^{(0)}$  and  $R^{(1)}$  do not uniquely determine a minimum, in which case the second correction  $R^{(2)}$  needs to be considered.
In linear networks the second correction  $R^{(2)}(A) = \frac{1}{2}\left\| \log_{+}A^{T}A\right\|^{2}$  plays an important role, as it bounds the operator norm of  $A$  (which is not bounded by  $R^{(0)}(A) = \mathrm{Rank}A$  nor  $R^{(1)}(A) = 2\log |A|_{+}$ ), thus guaranteeing the convergence of the hidden representations in the middle of network. We hoped at first that  $R^{(2)}$  would have similar properties in the nonlinear case, but we were not able to prove anything of the sort. Actually in contrast to the linear setting, the representations of the network can diverge as  $L\to \infty$ , which explains why the  $R^{(2)}$  does not give any similar control, which would guarantee convergence.
3.3 Representation geodesics
One can think of the sequence of hidden representations  $\tilde{\alpha}_1, \ldots, \tilde{\alpha}_L$  as a path from the input representation to the output representation that minimizes the weight norm  $\| W_\ell \|^2$  required to map from one representation to the next. As the depth  $L$  grows, we expect this sequence to converge to a form of geodesic in representation space. Such an analysis has been done in [Owh20] for ResNet where these limiting geodesics are continuous.
Two issues appear in the fully-connected case. First a representation  $\tilde{\alpha}_{\ell}$  remains optimal after any swapping of its neurons or other symmetries, but this can easily be solved by considering representations  $\tilde{\alpha}_{\ell}$  up to orthogonal transformation, i.e. to focus on the kernels  $K_{\ell}(x,y) = \tilde{\alpha}_{\ell}(x)^{T}\tilde{\alpha}_{\ell}(y)$ . Second the limiting geodesics of fully-connected networks are not continuous, and as such they cannot be described by a local metric.
We therefore turn to the representation cost of DNNs to describe the hidden representations of the network, since the  $\ell$ -th pre-activation function  $\tilde{\alpha}^{(\ell)}:\Omega \to \mathbb{R}^{n_{\ell}}$  in a network which minimizes the parameter norm must satisfy
$$
R (f; \Omega , L) = R (\tilde {\alpha} _ {\ell}; \Omega , \ell) + R (\sigma (\tilde {\alpha} _ {\ell}) \rightarrow f; \Omega , L - \ell).
$$
Thus the limiting representations  $\tilde{\alpha}_p = \lim_{L\to \infty}\tilde{\alpha}_{\ell_L}$  (for a sequence of layers  $\ell_L$  such that  $\lim_{L\to \infty}\ell_L / L = p\in (0,1)$ ) must satisfy
$$
R ^ {(0)} (f; \Omega) = R ^ {(0)} (\tilde {\alpha} _ {p}; \Omega) = R ^ {(0)} (\sigma (\tilde {\alpha} _ {p}) \to f; \Omega)
$$
$$
R ^ {(1)} (f; \Omega) = R ^ {(1)} (\tilde {\alpha} _ {p}; \Omega) + R ^ {(1)} (\sigma (\tilde {\alpha} _ {p}) \to f; \Omega)
$$
Let us now assume that the limiting geodesic is continuous at  $p$  (up to orthogonal transformation, which do not affect the representation cost), meaning that any other sequence of layers  $\ell_L'$  converging to the same ratio  $p \in (0,1)$  would converge to the same representation. The taking the limits with two sequences  $\lim \frac{\ell_L}{L} = p = \lim \frac{\ell_L'}{L}$  such that  $\lim \ell_L' - \ell_L = +\infty$  and taking the limit of the equality
$$
R (f; \Omega , L) = R (\tilde {\alpha} _ {\ell_ {L}}; \Omega , \ell_ {L}) + R (\sigma (\tilde {\alpha} _ {\ell_ {L}}) \to \tilde {\alpha} _ {\ell_ {L} ^ {\prime}}; \Omega , \ell_ {L} ^ {\prime} - \ell_ {L}) + R (\sigma (\tilde {\alpha} _ {\ell_ {L} ^ {\prime}}) \to f; \Omega , L - \ell_ {L} ^ {\prime}),
$$
we obtain that  $R^{(0)}(\sigma (\tilde{\alpha}_p)\to \tilde{\alpha}_p;\Omega) = R^{(0)}(f)$  and  $R^{(1)}(\sigma (\tilde{\alpha}_p)\to \tilde{\alpha}_p;\Omega) = 0$ . This implies that  $\sigma (\tilde{\alpha} (x)) = \tilde{\alpha} (x)$  at any point  $x$  where  $\mathrm{Rank}Jf(x) = R^{(0)}(f;\Omega)$ , thus  $R^{(0)}(id;\tilde{\alpha}_p(\Omega)) = R^{(0)}(f;\Omega)$  and  $R^{(1)}(id;\tilde{\alpha}_p(\Omega)) = 0$  if  $\mathrm{Rank}Jf(x) = R^{(0)}(f;\Omega)$  for all  $x\in \Omega$ .
3.3.1 Identity
When evaluated on the identity, the first two terms  $R^{(0)}(id;\Omega)$  and  $R^{(1)}(id;\Omega)$  describe properties of the domain  $\Omega$ .