SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Agustinus Kristiadi
Felix Dangel
Vector Institute, University of Tubingen akristiadi,fdangel@vectorinstitute.ai
University of Tübingen, Tübingen AI Center philipp.hennig@uni-tuebingen.de
Model reparametrization, which follows the change-of-variable rule of calculus, is a popular way to improve the training of neural nets. But it can also be problematic since it can induce inconsistencies in, e.g., Hessian-based flatness measures, optimization trajectories, and modes of probability densities. This complicates downstream analyses: e.g. one cannot definitively relate flatness with generalization since arbitrary reparametrization changes their relationship. In this work, we study the invariance of neural nets under reparametrization from the perspective of Riemannian geometry. From this point of view, invariance is an inherent property of any neural net if one explicitly represents the metric and uses the correct associated transformation rules. This is important since although the metric is always present, it is often implicitly assumed as identity, and thus dropped from the notation, then lost under reparametrization. We discuss implications for measuring the flatness of minima, optimization, and for probability-density maximization. Finally, we explore some interesting directions where invariance is useful.
Neural networks (NNs) are parametrized functions. Since it is often desirable to assign meaning or interpretability to the parameters (weights) of a network, it is interesting to ask whether certain transformations of the parameters leave the network invariant—equivalent in some sense. Various notions of invariance have been studied in NNs, in particular under weight-space symmetry [5, 14, 20, 42, 53] and reparametrization [20, 21, 30, 32, 71]. The former studies the behavior of a function  $\mathcal{L}(\theta)$  under some invertible  $T: \Theta \times \mathcal{G} \to \Theta$  where  $\mathcal{G}$  is a group; for any  $\theta$ , the function  $\mathcal{L}$  is invariant under the symmetry  $T$  if and only if  $\mathcal{L}(\theta) = \mathcal{L}(T(\theta, g))$  for all  $g \in \mathcal{G}$ . For example, normalized NNs are symmetric under scaling of the weights, i.e.  $\mathcal{L}(\theta) = \mathcal{L}(c\theta)$  for all  $c > 0$  [7]—similar scale-symmetry also presents in ReLU NNs [61]. Meanwhile, invariance under reparametrization studies the behavior of the NN when it is transformed under the change-of-variable rule
of calculus: Given a transformed parameter  $\psi = \varphi(\theta)$  under a bijective differentiable  $\varphi : \Theta \to \Psi$  that maps the original parameter space onto another parameter space, the function  $\mathcal{L}(\theta)$  becomes  $\hat{\mathcal{L}}(\psi) \coloneqq \mathcal{L}(\varphi^{-1}(\psi))$ . Note their difference (see Fig. 1): In the former, one works on a single parameter space  $\Theta$  and a single function  $\mathcal{L}$ —the map  $T$  acts as a symmetry of elements of  $\Theta$  under the group  $\mathcal{G}$ . In contrast, the latter assumes two parameter spaces  $\Theta$  and  $\Psi$  which are connected by  $\varphi$  and hence two functions  $\mathcal{L}$  and  $\hat{\mathcal{L}}$ .
Figure 1: The difference between the symmetry (top) and reparametrization problems (bottom).  $\Theta$  and  $\Psi$  are two different parameter spaces.
(a) Optimization
(b) Maximum Density
(c) Flatness Measure
Figure 2: Invariance is retained (bottom) if the Riemannian metric is explicitly tracked and one transforms geometric objects such as vectors, covectors, and the metric itself properly under a reparametrization  $\varphi : \theta \mapsto \psi$ . (a) Gradient descent's trajectories are therefore invariant under a change of parametrization. (b) Invariance of the modes of a probability density function is an inherent property under a natural base measure induced by the metric. (c) When the Hessian  $\pmb{H}$  at a critical point is seen as a linear map  $\pmb{E}$  with the help of the metric, its determinant is invariant.
We focus on the latter. While in this scenario, we have  $\hat{\mathcal{L}} (\psi) = \mathcal{L}(\theta)$  whenever  $\psi = \varphi (\theta)$ , it is well-known that downstream quantities such as optimization trajectories [55, 71], the Hessian [20, Sec. 5], and probability densities over  $\pmb{\theta}$  [59, Sec. 5.2.1.4] are not invariant under model reparametrization. These non-invariances are detrimental since an arbitrary reparametrization can affect the studied quantities and thus consistency cannot be guaranteed—parametrization muddles the analysis.
For instance, because of this, (i) one cannot relate Hessian-based sharpness measures with generalization and the correlation analyses between them [27, 33, 73] become meaningless, (ii) a good preconditioner cannot be guaranteed to be good anymore for optimizing the reparametrized model—this is one reason why invariant methods like natural gradient are desirable [37, 50, 58, 62], and (iii) under a reparametrization, a prior density might put low probability in a region that corresponds to the high-probability region in the original space, making posterior inference pathological [25].
To analyze this issue, in this work we adopt the framework of Riemannian geometry, which is a generalization of calculus studying the intrinsic properties of manifolds. "Intrinsic" in this context means that objects such as functions, vectors, and tensors defined on the manifold must be independent of how the manifold is represented via a coordinate system, and conserved under a change of coordinates [47]. The parameter space of a neural network, which is by default assumed to be  $\Theta = \mathbb{R}^d$  with the Euclidean metric and the Cartesian coordinates, is a Riemannian manifold—model reparametrization is such a change in the coordinate system.
Why, then, can reparametrizations bring up the aforementioned inconsistencies? In this work, we discuss this discrepancy. We observe that this issue often arises because the Riemannian metric is left implicit, and dropped when computing downstream quantities such as gradients, Hessians, and volumes on the parameter space. This directly suggests the solution: Invariance under reparametrization is guaranteed if the Riemannian metric is not just implicitly assumed, but made explicit, and it is made sure that the associated transformation rules of objects such as vectors, covectors, and tensors are performed throughout. We show how these insights apply to common cases (Fig. 2).
**Limitation** Our work focuses on the reparametrization consistency of prior and future methods that leverage geometric objects such as gradients and the Hessian. Thus, we leave the geometric analysis for invariance under symmetry as future work. In any case, our work is complimentary to other works that analyze invariance under symmetry [17, 22, 43, 67, etc]. Moreover, this work's focus is not to propose "better" methods e.g. for better preconditioners or better generalization metrics. Rather, we provide a guardrail for existing and future methods to avoid pathologies due to reparametrization.
This section provides background and notation on relevant concepts of neural networks and Riemannian geometry. For the latter, we frame the discussion in terms of linear algebra as close as possible to the notation of the ML community: We use regular faces to denote abstract objects and bold faces for
their concrete representations in a particular parametrization. E.g., a linear map  $A$  is represented by a matrix  $\mathbf{A}$ , a point  $z$  is represented by a tuple of numbers  $\theta$ , an inner product  $G(z)$  at  $z$  is represented by a matrix  $\mathbf{G}(\theta)$ . The same goes for the differential  $\nabla$ ,  $\nabla$ ; and the gradient  $\operatorname{grad}$ ,  $\operatorname{grad}$ .
2.1 Some notation for neural networks
The following concepts are standard, introduced mostly to clarify notation. Let  $f: \mathbb{R}^n \times \mathbb{R}^d \to \mathbb{R}^k$  with  $(\pmb{x},\pmb{\theta}) \mapsto f(\pmb{x};\pmb{\theta})$  be a model with input, output, and parameter spaces  $\mathbb{R}^n$ ,  $\mathbb{R}^k$ ,  $\mathbb{R}^d$ . Let  $\mathcal{D} := \{(x_i,y_i)\}_{i=1}^m$  be a dataset and write  $X := \{x_i\}_{i=1}^m$  and  $Y := \{y_i\}_{i=1}^m$ . We moreover write  $f(\pmb{X};\pmb{\theta}) := \operatorname{vec}(\{f(x_i;\pmb{\theta})\}_{i=1}^m) \in \mathbb{R}^{mk}$ . The standard way of training  $f$  is by finding a point  $\pmb{\theta}^* \in \mathbb{R}^d$  s.t.  $\pmb{\theta}^* = \arg \min_{\pmb{\theta} \in \mathbb{R}^d} \sum_{i=1}^m \ell(y_i, f(x_i;\pmb{\theta})) =: \arg \min_{\pmb{\theta} \in \mathbb{R}^d} \mathcal{L}(\pmb{\theta})$ , for some loss function  $\ell$ . If we add a weight decay  $\gamma/2\|\pmb{\theta}\|_2^2$  term to  $\mathcal{L}(\pmb{\theta})$ , the minimization problem has a probabilistic interpretation as maximum a posteriori (MAP) estimation of the posterior density  $p(\pmb{\theta} | \mathcal{D})$  under the likelihood function  $p(Y | f(\pmb{X};\pmb{\theta})) \propto \exp(-\sum_{i=1}^m \ell(y_i, f(x_i;\pmb{\theta}))$  and an isotropic Gaussian prior  $p(\pmb{\theta}) = \mathcal{N}(\pmb{\theta} | 0, \gamma^{-1}\pmb{I})$ . We denote the MAP loss by  $\mathcal{L}_{\mathrm{MAP}}$ . In our present context, this interpretation is relevant because this probabilistic interpretation implies a probability density, and it is widely known that a probability density transforms nontrivially under reparametrization.
A textbook way of obtaining  $\pmb{\theta}^{*}$  is gradient descent (GD): At each time step  $t$ , obtain the next estimate  $\pmb{\theta}^{(t + 1)} = \pmb{\theta}^{(t)} - \alpha \nabla \mathcal{L}|_{\pmb{\theta}^{(t)}}$ , for some  $\alpha >0$ . This can be considered as the discretization of the gradient flow ODE  $\dot{\pmb{\theta}} = -\nabla \mathcal{L}|_{\pmb{\theta}}$ . Among the many variants of GD is preconditioned GD which considers a positive-definite matrix field  $\pmb{R}$  on  $\mathbb{R}^d$ , yielding  $\dot{\pmb{\theta}} = -\pmb{R}(\pmb{\theta})^{-1}\nabla \mathcal{L}|_{\pmb{\theta}}$ .
2.2 The geometry of the parameter space
Remark 1. We restrict our discussion to global coordinate charts since they are the default assumption in neural networks, and in a bid to make our discussion clear for people outside of the differential geometry community. We refer the curious reader to Appendix D for a discussion on local coordinate charts. Note that, our results hold in the general local-chart formulation.
The parameter space  $\mathbb{R}^d$  is the canonical example of a smooth manifold. We can impose upon this manifold a (global) coordinate chart: a homeomorphism that represents a point with an element of  $\mathbb{R}^d$ . The standard choice is the Cartesian coordinate system  $\theta : \mathbb{R}^d \to \Theta \cong \mathbb{R}^d$ , which we have used in the previous section. That is, the image of  $\theta$  uniquely represents elements of  $\mathbb{R}^d$  given a point  $z \in \mathbb{R}^d$ , we write  $\theta = \theta(z)$  for its coordinate representation under  $\theta$ .
The choice of a coordinate system is not unique. Any homeomorphism  $\psi : \mathbb{R}^d \to \Psi \cong \mathbb{R}^d$  can be used as an alternative coordinate chart. For instance, the polar coordinates can be used to represent points in  $\mathbb{R}^d$  instead. Crucially, the images of any pair of coordinates must be con
Figure 3: The implicit geometric assumption on the parameter space  $\mathbb{R}^d$ .  $\theta, \psi$  are two different (global) coordinates on  $\mathbb{R}^d$ .
nected through a diffeomorphism (a smooth function with a smooth inverse)  $\varphi :\Theta \to \Psi$ . Such a map is called a change of coordinates or reparametrization since it acts on the parameter space.
Example 2 (Reparametrization). Reparametrization is ubiquitous in machine learning:
(a) Mean-Field Variational Inference. Let  $q(\pmb{x}; \pmb{\theta})$  be a variational approximation, which is often chosen to be  $\mathcal{N}(\pmb{x} \mid \pmb{\mu}, \mathrm{diag}(\pmb{\sigma}^2))$ , i.e.  $\pmb{\theta} = \{\pmb{\mu} \in \mathbb{R}^d, \pmb{\sigma}^2 \in \mathbb{R}_{>0}^d\}$ . Common choices of reparametrization of  $\pmb{\sigma}^2$  include the log-space [52] or softplus [12] parametrization.
(b) Weight normalization. Given a NN  $f_{\theta}$ , WeightNorm [69] applies the reparametrization  $\psi = r\eta$  where  $r \in \mathbb{R}_{>0}$  and  $\pmb{\eta} \coloneqq \frac{\pmb{\theta}}{\|\pmb{\theta}\|} \in \mathbb{S}^{d-1}$ . This is akin to the polar coordinates. (Note that we assume  $\pmb{\theta} \in \mathbb{R}^d \setminus \{\mathbf{0}\}$  since otherwise  $\varphi$  is not a global diffeomorphism.)
At each point  $z \in \mathbb{R}^d$ , there exists a vector space  $T_z\mathbb{R}^d$  called the tangent space at  $z$ , consisting of the so-called tangent vectors. An important example of a tangent vector is the gradient vector  $\operatorname{grad}\mathcal{L}|_z$
of  $\mathcal{L}:\mathbb{R}^d\to \mathbb{R}$  at  $z$ . The dual space  $T_z^*\mathbb{R}^d$  of  $T_{z}\mathbb{R}^{d}$  is referred to as the cotangent space and consists of linear functionals of  $T_{z}\mathbb{R}^{d}$ , called tangent covectors. An example of a tangent covector is the differential  $\nabla \mathcal{L}|_z$  of  $\mathcal{L}$  at  $z$ . Under a coordinate system, one can think of both tangent vectors and covectors as vectors in the sense of linear algebra, i.e., tuples of numbers.
One can take an inner product of tangent vectors by equipping the manifold with a Riemannian metric  $G$ , which, at each point in  $\mathbb{R}^d$  is represented by a positive-definite  $d \times d$  matrix  $G$  whose coefficients depend on the choice of coordinates. With the help of the metric, there is an isomorphism between  $T_z\mathbb{R}^d$  and  $T_z^*\mathbb{R}^d$ . In coordinates, it is given by the map  $T_z\mathbb{R}^d \to T_z^*\mathbb{R}^d: \boldsymbol{v} \mapsto G\boldsymbol{v}$  and its inverse  $T_z^*\mathbb{R}^d \to T_z\mathbb{R}^d: \boldsymbol{\omega} \mapsto G^{-1}\boldsymbol{\omega}$ . One important instance of this isomorphism is the fact that  $\operatorname{grad} \mathcal{L}|_z$  is represented by  $G^{-1}\nabla \mathcal{L}$  in coordinates. The natural gradient is a direct consequence of this fact. In practice, it is standard to assume the Cartesian coordinates on  $\mathbb{R}^d$ , implying  $G \equiv I$ . We have thus  $\operatorname{grad} \mathcal{L} \equiv \nabla \mathcal{L}$ , which only seems trivial at first sight, but reveals the relationship between the tangent vector on the l.h.s. and the cotangent vector on the r.h.s.
Remark 3. Another common way to define a NN's parameter space is to define a manifold of probabilistic models  $M \coloneqq \{p(\mathbf{Y} \mid f(\mathbf{X};\boldsymbol{\theta})): \boldsymbol{\theta} \in \mathbb{R}^d\}$  and assume that  $p(\mathbf{Y} \mid f(\mathbf{X};\boldsymbol{\theta})) \mapsto \boldsymbol{\theta}$  is the coordinate chart [19, 55, etc.]. The problem with this construction is that there is no bijection  $\mathbb{R}^d \to M$  in general. Indeed, the Jacobian of the map  $f(\mathbf{X};\cdot): \mathbb{R}^d \to M$  is in practice surjective everywhere due to overparametrization [34, 78]. Therefore, one cannot define a proper metric on the parameter space  $\mathbb{R}^d$  that corresponds to a metric on the distribution space  $M$  [46, Prop. 13.9]. For instance, using this construction, the Fisher metric is singular for overparametrized NNs [10, 34, 74, 78] and thus not a valid Riemannian metric. The pseudoinverse has been used to handle this but is mostly limited to theoretical analyses [10, 74]: As far as we know, in practice, damping—which breaks the interpretation of the Fisher on  $\mathbb{R}^d$  as the pullback metric from  $M$ —is de facto for handling this due to its numerical stability [4].
By detaching from the distribution space, the definition used in this work does not have this issue. It enables more general constructions of metrics in the parameter space since any positive-definite matrix is admissible. E.g. one is free to add damping or use any approximation to the Fisher—our results still apply to this case. Thus, our construction is closer to practice.
2.2.1 Transformation rules
Differential geometry is the study of coordinate-independent objects: Geometric objects must be invariant under change-of-coordinates in the sense that any pair of representations must refer to the same (abstract) object. Suppose  $\varphi : \Theta \to \Psi$  is a reparametrization, with  $\psi = \varphi(\theta)$ . Coordinate independence is encoded in the following transformation rules, which involve the Jacobian  $J(\theta) = (\partial \psi_i / \partial \theta_j)$  of  $\varphi$ , and its inverse  $J^{-1}(\psi) = (J(\theta))^{-1} = (\partial \theta_i / \partial \psi_j)$  the Jacobian of  $\varphi^{-1}$ . (Color codes are used for clarity when different objects are present in a single expression later on.)
(a) A function  $h:\Theta \to \mathbb{R}$  in  $\theta$  -coordinates transforms into  $\hat{h} = h\circ \varphi^{-1}$  in  $\psi$  -coordinates.
(b) A tangent vector  $v$  in  $\theta$ -coordinates transforms into  $J(\theta)v$ . In particular, a gradient vector  $\operatorname{grad} \mathcal{L}|_{\theta}$  transforms into  $J(\theta) \operatorname{grad} \mathcal{L}|_{\theta}$ .
(c) A tangent covector  $\omega$  in  $\theta$ -coordinates transforms into  $J^{-1}(\psi)^{\top}\omega$ . In particular, given a transformed function  $\mathcal{L} \circ \varphi^{-1}: \Psi \to \mathbb{R}$ , we have  $\nabla (\mathcal{L} \circ \varphi^{-1})|_{\psi} = J^{-1}(\psi)^{\top}\nabla \mathcal{L}|_{\varphi^{-1}(\psi)}$ , which we recognize as the standard chain rule.
(d) A metric  $G(\theta)$  becomes  $J(\theta)^{-\top}G(\theta)J(\theta)^{-1} = J^{-1}(\psi)^{\top}G(\psi)J^{-1}(\psi)$ . In general, this rule applies to any tensor that takes tangent vectors as its arguments, e.g. a bilinear map.
The following examples show how these rules ensure the invariance of geometric objects.
Example 4. Let  $h, v, \omega, G$  respectively be a function, vector, covector, and metric in  $\theta$ -coordinates at a point  $\theta = \theta(z)$ , and let  $\varphi: \Theta \to \Psi$  be a reparametrization.
(a) Let  $\psi \coloneqq \varphi (\pmb {\theta})$  and  $\hat{h}\coloneqq h\circ \varphi^{-1}$  be  $\pmb {\theta}\in \Theta$  and  $h$  expressed in  $\psi$  coordinates, respectively. Then,  $\hat{h} (\psi) = h(\varphi^{-1}(\psi)) = h(\pmb {\theta})$  . That is, the actions of  $h$  and  $\hat{h}$  agree in both coordinates and thus they represent the same abstract function on  $\mathbb{R}^d$
(b) The action of  $\omega$  on  $\pmb{v}$  in  $\theta$ -coordinates is given by the product  $\omega^{\top} \pmb{v}$ . Let  $\hat{\pmb{v}} := J(\pmb{\theta}) \pmb{v}$  and  $\hat{\omega} := J(\pmb{\theta})^{-\top} \pmb{\omega}$  be the transformed vector and covectors. Then,
$$
\hat {\omega} ^ {\top} \hat {v} = (J (\theta) ^ {- \top} \omega) ^ {\top} (J (\theta) v) = \omega^ {\top} v. \tag {1}
$$
That is, both  $\omega$  and  $\hat{\omega}$  are the representations of the same linear functional;  $\pmb{v}$  and  $\hat{\pmb{v}}$  are the representations of the same tangent vector.
(c) Let  $\hat{G} \coloneqq J^{-\top} G J^{-1}$  be the transformed metric in the  $\psi$ -coordinates. Then,
$$
\hat {v} ^ {\top} \hat {G} \hat {v} = (\mathcal {J} v) ^ {\top} \mathcal {J} \mathcal {G} J ^ {- 1} (\mathcal {J} v) = v ^ {\top} G v,
$$
and thus the transformation rules ensure that inner products are also invariant.
Finally, we say that an ODE's dynamics is invariant if the trajectory in parametrization corresponds to the trajectory in another parametrization. Concretely, a trajectory  $(\pmb{\theta}_t)_t$  in  $\Theta$  is invariant if under the reparametrization  $\varphi : \Theta \to \Psi$ , the transformed trajectory  $(\psi_t)_t$  is related to  $(\pmb{\theta}_t)_t$  by  $\pmb{\theta}_t = \varphi^{-1}(\psi_t)$  for each  $t$ . See Fig. 2a for an illustration.
3 Neural Networks and Reparametrization
We discuss three aspects of the parameter space under reparametrization, as illustrated in Fig. 2. First, we address the non-invariance of Hessian-based flatness measures [e.g., 20], and show how taking the metric into account provides invariance. Second, we show that the invariance property often cited in favor of natural gradient is not unique, but an inherent property of any gradient descent algorithm when considering its ODE. Finally, we show that modes of probability density functions on the parameter space are invariant when the Lebesgue measure is generalized using the metric.
3.1 Invariance of the Hessian
A growing body of literature connects the flatness of minima found by optimizers to generalization performance [8, 13, 23, 27, 36, 38, 48, 53]. However, as Dinh et al. [20] observed, this association does not have a solid foundation since standard sharpness measures derived from the Hessian of the loss function are not invariant under reparametrization.
From the Riemannian-geometric perspective, the Hessian of a function  $\mathcal{L}$  (or  $\mathcal{L}_{\mathrm{MAP}}$  or any other twice-differentiable function) on  $\theta$ -coordinates is represented by a  $d\times d$  matrix with coefficients  $\pmb{H}_{ij}(\pmb {\theta})\coloneqq \frac{\partial^2\mathcal{L}}{\partial\theta_i\partial\theta_j} (\pmb {\theta}) - \sum_{k = 1}^d\Gamma_{ij}^k (\pmb {\theta})\frac{\partial\mathcal{L}}{\partial\theta_k} (\pmb {\theta})$ , for any  $\pmb {\theta}\in \Theta$  and  $i,j = 1,\ldots ,d$ , where  $\Gamma_{ij}^{k}$  is a particular three-dimensional array. While it might seem daunting, when  $\pmb{\theta}$  is a local minimum of  $\mathcal{L}$ , the second term is zero since the partial derivatives of  $\mathcal{L}$  are all zero at  $\pmb{\theta}$ . Thus,  $\pmb {H}(\pmb {\theta})$  equals the standard Hessian matrix  $(\partial^2\mathcal{L} / \partial \theta_i\partial \theta_j)$  at  $\pmb {\theta}\in \arg \min \mathcal{L}$ .
Considering the Hessian as a bilinear function that takes two vectors and produces a number, it follows the covariant transformation rule, just like the metric (see Appendix B.1 for a derivation): Let  $\varphi : \Theta \to \Psi$  and  $\psi = \varphi(\theta)$ . The Hessian matrix  $H(\theta)$  at a local minimum in  $\theta$ -coordinates transforms into  $\hat{H}(\psi) = J^{-1}(\psi)^{\top} H(\varphi^{-1}(\psi)) J^{-1}(\psi)$  in  $\psi$ -coordinates—this is correctly computed by automatic differentiation, i.e. by chain and product rules. But, while this gives invariance of  $H$  as a bilinear map (Example 4c), the determinant of  $H$ —a popular flatness measure—is not invariant because
$$
(\det  \hat {\boldsymbol {H}}) (\boldsymbol {\psi}) := \det  \left(\boldsymbol {J} ^ {- 1} (\boldsymbol {\psi}) ^ {\top} \boldsymbol {H} (\varphi^ {- 1} (\boldsymbol {\psi})) \boldsymbol {J} ^ {- 1} (\boldsymbol {\psi})\right) = (\det  \boldsymbol {J} ^ {- 1} (\boldsymbol {\psi})) ^ {2} (\det  \boldsymbol {H} (\varphi^ {- 1} (\boldsymbol {\psi}))),
$$
and so in general, we do not have the relation  $(\operatorname{det}\dot{\pmb{H}}) = (\operatorname{det}\pmb{H}) \circ \varphi^{-1}$  that would make this function invariant under reparametrization (Example 4a).
The key to obtaining invariance is to employ the metric  $G$  to transform the bilinear Hessian into a linear map/operator on the tangent space. This is done by simply multiplying  $H$  with the inverse of the metric  $G$ , i.e.  $E := G^{-1}H$ . The determinant of  $E$  is thus an invariant quantity. To show this, under  $\varphi$ , the linear map  $E$  transforms into
$$
\begin{array}{l} \hat {\boldsymbol {E}} (\boldsymbol {\psi}) = \left(\boldsymbol {J} ^ {- 1} (\boldsymbol {\psi}) ^ {\top} \boldsymbol {G} \left(\varphi^ {- 1} (\boldsymbol {\psi})\right) \boldsymbol {J} ^ {- 1} (\boldsymbol {\psi})\right) ^ {- 1} \boldsymbol {J} ^ {- 1} (\boldsymbol {\psi}) ^ {\top} \boldsymbol {H} \left(\varphi^ {- 1} (\boldsymbol {\psi})\right) \boldsymbol {J} ^ {- 1} (\boldsymbol {\psi}) \\ = (\boldsymbol {J} ^ {- 1} (\psi)) ^ {- 1} \boldsymbol {G} (\varphi^ {- 1} (\psi)) \boldsymbol {H} (\varphi^ {- 1} (\psi)) \boldsymbol {J} ^ {- 1} (\psi), \\ \end{array}
$$
due to the transformation of both  $G$  and  $H$ . Hence,  $\operatorname{det} E$  transforms into