Year: 2021

Abstract:
The Ricci flow is a method of manifold surgery, which can trim manifolds to more regular. However, in most cases, the Rich flow tends to develop singularities and lead to divergence of the solution. In this paper, we propose linearly nearly Euclidean metrics to assist manifold micro-surgery, which means that we prove the dynamical stability and convergence of such metrics under the Ricci-DeTurck flow. From the information geometry and mirror descent points of view, we give the approximation of the steepest descent gradient flow on the linearly nearly Euclidean manifold with dynamical stability. In practice, the regular shrinking or expanding of Ricci solitons with linearly nearly Euclidean metrics will provide a geometric optimization method for the solution on a manifold.

heading: 1 INTRODUCTION
text: In general relativity (Wald, 2010), a complete Riemannian manifold (M, g) endowed with a linearly nearly flat spacetime metric gij is considered for linearized gravity to solve the Newtonian limit. The form of this metric is gij = ηij + γij , where ηij represents a flat Minkowski metric whose background is special relativity and γij is small from ηij . An adequate definition of “smallness” in this context is that the components of γij are much smaller than 1 in some global inertial coordinate system of ηij . Now, let us step out of the physical world and give a similar metric gij = δij + γij in Riemannian n-manifold (Mn, g), i.e. the linearly nearly Euclidean metric, where δij represents a flat Euclidean metric and γij is small from δij .
A natural problem for such a linearly nearly Euclidean metric is: how does the metric evolve over time with respect to the Ricci flow while ensuring the constant topological structure? Let us review some stability analyses of different manifolds along with the Ricci flow.
For the Riemannian n-dimensional manifold (Mn, g) that is isometric to the Euclidean ndimensional space (Rn, δ), Schnürer et al. (Schnürer et al., 2007) have showed the stability of Euclidean space under the Ricci flow for a small C0 perturbation. Koch et al. (Koch & Lamm, 2012) have given the stability of the Euclidean space along with the Ricci flow in the L∞-Norm. Moreover, for the decay of the L∞-Norm on Euclidean space, Appleton (Appleton, 2018) has given the proof of a different method. Considering the stability of integrable and closed Ricci-flat metrics, Sesum (Sesum, 2006) has proved that the convergence rate is exponential because the spectrum of the Lichnerowicz operator is discrete. Furthermore, Deruelle et al. (Deruelle & Kröncke, 2021) have proved that an asymptotically locally Euclidean Ricci-flat metric is dynamically stable under the Ricci flow, for an L2 ∩ L∞ perturbation on non-flat and non-compact Ricci-flat manifolds. If we embed a Riemannian n-dimensional manifold in the neural network, then we are training the neural network on the dynamic manifold. The most famous method for training neural networks on manifolds is the natural gradient (Amari, 1998). However, for the Riemannian manifold corresponding to the KL divergence (representing the natural gradient), its stability and convergence are still unknown (Martens, 2020). As a way of manifold evolution, Ricci flow seems to be an excellent choice to ensure that neural networks are trained on dynamic and stable manifolds (Glass et al., 2020; Jejjala et al., 2020). But the research on the relationship between the two has not yet sprouted.
In this paper, we consider a complete Riemannian n-dimensional manifold (Mn, g) endowed with linearly nearly Euclidean metrics g(t) = δ + γ(t). First of all, we prove the stability of linearly nearly Euclidean manifolds under the Ricci-DeTurck flow in the L2-Norm if initial metrics are
integrable and linearly stable, i.e. has a manifold structure of finite dimension. We mean that any Ricci-DeTurck flow which starts from near g exists for all time and converges to a linearly nearly Euclidean metric near g. Note that we use the Einstein summation convention and denote generic constants by C or C1.
Furthermore, we define and construct linearly nearly Euclidean manifolds based on information geometry and mirror descent algorithm. Based on a symmetrized convex function, we obtain the linearly nearly Euclidean divergence which is used to calculate the steepest descent gradient in linearly nearly Euclidean manifolds. Experimentally, when we use the approximated steepest descent gradient flow to learn several neural networks on classification tasks, we observe the evolution of its metric is consistent with the micro-surgery process under the Ricci-DeTurck flow.

heading: 2 RICCI FLOW
text: Let us introduce a partial differential equation, the Ricci flow, without explanation. The concept of the Ricci flow first published by Hamilton (Hamilton et al., 1982) on the manifold M of a timedependent Riemannian metric g(t) with the initial metric g0:
∂ ∂t g(t) = −2 Ric(g(t))
g(0) = g0
(1)
where Ric denotes the Ricci curvature tensor whose definition can be found in Appendix A.
The purpose of the Ricci flow is to prove Thurston’s Geometrization Conjecture and Poincaré Conjecture because the Ricci flow is like a surgical scalpel, trimming irregular manifolds into regular manifolds to facilitate observation and discussion (Sheridan & Rubinstein, 2006).
In general, in order to possess good geometric and topological properties, we expect the metric to become converge and round with the help of the Ricci flow. “become round” means that the solution will not shrink to a point but converge to a constant circle. However, in most cases, we do not even know the convergence of the solution and whether the solution will develop a singularity. Next, we will discuss these issues for brevity.

heading: 2.1 SHORT TIME EXISTENCE
text: To show that there exists a unique solution for a short time, we must check if the system of the Ricci flow is strongly parabolic.
Theorem 1 When u :M× [0, T )→ E is a time-dependent section of the vector bundle E whereM is some Riemannian manifold, if the system of the Ricci flow is strongly parabolic at u0 then there exists a solution on some time interval [0, T ), and the solution is unique for as long as it exists.
Proof. The proofs can be found in (Ladyzhenskaia et al., 1988).
Definition 1 The Ricci flow is strongly parabolic if there exists δ > 0 such that for all covectors ϕ 6= 0 and all symmetric hij = ∂gij(t)∂t 6= 0, the principal symbol of −2 Ric satisfies
[−2 Ric](ϕ)(h)ijhij = gpq (ϕpϕqhij + ϕiϕjhpq − ϕqϕihjp − ϕqϕjhip)hij > δϕkϕkhrshrs.
Since the inequality cannot always be satisfied, the Ricci flow is not strongly parabolic, which makes us unable to prove the existence of the solution based on Theorem 1.
It is possible to understand which parts have an impact on its non-parabolic by the linearization of the Ricci curvature tensor.
Lemma 1 The linearization of −2 Ric can be rewritten as D[−2 Ric](h)ij = gpq∇p∇qhij +∇iVj +∇jVi +O(hij)
where Vi = g pq
( 1
2 ∇ihpq −∇qhpi
) .
(2)
Proof. The proofs can be found in Appendix B.1.
The term O(hij) will have no contribution to the principal symbol of −2 Ric, so ignoring it will not affect our discussion of this problem. By carefully observing the above equation, one finds that the impact on the non-parabolic of the Ricci flow comes from the terms in V , not the term gpq∇p∇qhij . The solution is followed by the DeTurck Trick (DeTurck, 1983) that has a timedependent reparameterization of the manifold:
∂ ∂t ḡ(t) = −2 Ric(ḡ(t))− L ∂ϕ(t) ∂t ḡ(t)
ḡ(0) = ḡ0 + d, (3)
where d is a symmetric (0,2)-tensor on M. See Appendix B.2 for details. By choosing ∂ϕ(t)∂t to cancel the effort of the terms in V , the reparameterized Ricci flow is strongly parabolic. Thus, one can say that the Ricci-DeTurck flow has a unique solution, the pullback metric, for a short time.

heading: 2.2 CURVATURE EXPLOSION AT SINGULARITY
text: In this subsection, we will present the behavior of the Ricci flow in finite time and show that the evolution of the curvature is close to divergence. The core demonstration is followed with Theorem 4, which requires some other proof as a foreshadowing.
Theorem 2 Given a smooth Riemannian metric g0 on a closed manifoldM, there exists a maximal time interval [0, T ) such that a solution g(t) of the Ricci flow, with g(0) = g0, exists and is smooth on [0, T ), and this solution is unique.
Proof. The proofs can be found in (Sheridan & Rubinstein, 2006).
Theorem 3 LetM be a closed manifold and g(t) a smooth time-dependent metric onM, defined for t ∈ [0, T ). If there exists a constant C <∞ for all x ∈M such that∫ T
0 ∣∣∣∣ ∂∂tgx(t) ∣∣∣∣ g(t) dt ≤ C, (4)
then the metrics g(t) converge uniformly as t approaches T to a continuous metric g(T ) that is uniformly equivalent to g(0) and satisfies
e−Cgx(0) ≤ gx(T ) ≤ eCgx(0).
Proof. The proofs can be found in Appendix B.3.
Corollary 1 Let (M, g(t)) be a solution of the Ricci flow on a closed manifold. If |Rm |g(t) is bounded on a finite time [0, T ), then g(t) converges uniformly as t approaches T to a continuous metric g(T ) which is uniformly equivalent to g(0).
Proof. The bound on |Rm |g(t) implies one on |Ric |g(t). Based on Equation (1), we can extend the bound on | ∂∂tg(t)|g(t). Therefore, we obtain an integral of a bounded quantity over a finite interval is also bounded, by Theorem 3.
Theorem 4 If g0 is a smooth metric on a compact manifoldM, the Ricci flow with g(0) = g0 has a unique solution g(t) on a maximal time interval t ∈ [0, T ). If T <∞, then
lim t→T ( sup x∈M |Rmx(t)| ) =∞. (5)
Proof. For a contradiction, we assume that |Rmx(t)| is bounded by a constant. It follows from Corollary 1 that the metrics g(t) converge uniformly in the norm induced by g(t) to a smooth metric g(T ). Based on Theorem 2, it is possible to find a solution to the Ricci flow on t ∈ [0, T ) because the smooth metric g(T ) is uniformly equivalent to initial metric g(0).
Hence, one can extend the solution of the Ricci flow after the time point t = T , which is the result for continuous derivatives at t = T . This tell us that the time T of existence of the Ricci flow has not been maximal, which contradicts our assumption. In other words, |Rmx(t)| is unbounded. As approaching the singular time T , the Riemann curvature |Rm |g(t) becomes no longer convergent and tends to explode.

heading: 3 EVOLUTION OF LINEARLY NEARLY EUCLIDEAN METRICS
text: Next, this paper will focus on linearly nearly Euclidean metrics, proving that them can have a good performance in terms of stability, i.e., the convergence of a Ricci-DeTurck flow ḡ(t) to a linearly nearly Euclidean metric ḡ(∞). Before that, we have to construct a family ḡ0 of linearly nearly Euclidean reference metrics such that ∂∂t ḡ0(t) = O((ḡ(t)− ḡ0(t)) 2). Let
F = { ḡ(t) ∈Mn ∣∣ 2 Ric(ḡ(t)) + L ∂ϕ(t) ∂t ḡ(t) = 0 }
be the set of stationary points under the Ricci-DeTurck flow. We are able to establish a manifold
F̃ = F ∩ U (6)
where U is an L2-neighbourhood of integral ḡ0. Before starting the discussion about long time stability of linearly nearly Euclidean metrics, we need some prior knowledge about short time existence (Appendix D and Appendix E). In particular, we yield
Lemma 2 Let ḡ(t) be a Ricci–DeTurck flow on a maximal time interval t ∈ (0, T ) in an L2 neighbourhood of ḡ0. We have the following estimate such that:∥∥∥∥ ∂∂td0(t) ∥∥∥∥ L2 ≤ C ∥∥∥∇ḡ0(t) (d(t)− d0)∥∥∥2 L2 .
Proof. Let {e1(t), e2(t), . . . , en(t)} be a family of L2-orthonormal bases of the kernel kerL2 such that ∂∂tei(t) depends linearly on ∂ ∂td0(t). For an isomorphism orthogonal projection Π : Tḡ0F̃ → kerL2 , by the Hardy inequality (Minerbe, 2009), one has similar proofs by referring the details (Deruelle & Kröncke, 2021).
Theorem 5 Let (Mn, ḡ0) be a linearly nearly Euclidean n-manifold which is linearly stable and integrable. Furthermore, there exists a constant αḡ0 such that
(∆d(t) + Rm(ḡ0) ∗ d(t), d(t))L2 ≤ −αḡ0 ∥∥∇ḡ0h∥∥2
L2
for all ḡ(t) ∈ F̃ which is as in (6).
Proof. The similar proofs can be found in (Devyver, 2014) with some minor modifications. Due to the linear stability requirement of linearly nearly Euclidean manifolds in Definition 8, −Lḡ0 is non-negative. Then there exists a positive constant αḡ0 such that
αḡ0 (−∆d(t), d(t))L2 ≤ (−∆d(t)− Rm(ḡ0) ∗ d(t), d(t))L2 .
By Taylor expansion, one repeatedly uses elliptic regularity and Sobolev embedding (Pacini, 2010) to obtain the estimate.
Corollary 2 Let (Mn, ḡ0) be a linearly nearly Euclidean n-manifold which is integrable. For a Ricci–DeTurck flow ḡ(t) on a maximal time interval t ∈ [0, T ], if it satisfies ‖ḡ(t) − ḡ0‖L∞ < where > 0, then there exists a constant C < ∞ for t ∈ [0, T ] such that the evolution inequality satisfies
‖d(t)− d0‖2L2 ≥ C ∫ T
0
∥∥∥∇ḡ0(t) (d(t)− d0)∥∥∥2 L2 dt.
Proof. Based on Equation (16), we know ∂
∂t (d(t)− d0) =∆(d(t)− d0) + Rm ∗(d(t)− d0)
+ Fḡ−1 ∗ ∇ḡ0(d(t)− d0) ∗ ∇ḡ0(d(t)− d0) +∇ḡ0 ( GΓ(ḡ0) ∗ (d(t)− d0) ∗ ∇ ḡ0(d(t)− d0) ) .
Followed by Lemma 2 and Theorem 5, we further obtain ∂
∂t ‖d(t)− d0‖2L2 =2 (∆(d(t)− d0) + Rm ∗(d(t)− d0), d(t)− d0)L2 + ( Fḡ−1 ∗ ∇ḡ0(d(t)− d0) ∗ ∇ḡ0(d(t)− d0), d(t)− d0 ) L2
+ ( ∇ḡ0 ( GΓ(ḡ0) ∗ (d(t)− d0) ∗ ∇ ḡ0(d(t)− d0) ) , d(t)− d0 ) L2
+ ( d(t)− d0, ∂
∂t d0(t) ) L2 + ∫ M (d(t)− d0) ∗ (d(t)− d0) ∗ ∂ ∂t d0(t)dµ
≤− 2αḡ0 ∥∥∇ḡ0 (d(t)− d0)∥∥2L2
+ C ‖(d(t)− d0)‖L∞ ∥∥∇ḡ0 (d(t)− d0)∥∥2L2
+ ∥∥∥∥ ∂∂td0(t) ∥∥∥∥ L2 ‖d(t)− d0‖L2
≤ (−2αḡ0 + C · ) ∥∥∇ḡ0 (d(t)− d0)∥∥2L2 .
Let be small enough that −2αḡ0 + C · < 0 holds, we can find ∂
∂t ‖d(t)− d0‖2L2 ≤ −C ∥∥∇ḡ0 (d(t)− d0)∥∥2L2 holds.
Theorem 6 Let (Mn, ḡ0) be a linearly nearly Euclidean n-manifold which is linearly stable and integrable. For every 1 > 0, there exists a 2 > 0 satisfying: For any metric ḡ(t) ∈ BL2(ḡ0, 2), there is a complete Ricci–DeTurck flow (Mn, ḡ(t)) starting from ḡ(t) converging to a linearly nearly Euclidean metric ḡ(∞) ∈ BL2(ḡ0, 1). Note thatBL2(ḡ0, ) is the -ball with respect to theL2-Norm induced by ḡ0 and centred at ḡ0.
Proof. By Lemma 4, one can find so small 2 > 0 such that d(t) ∈ BL2(0, 2) holds. By Lemma 2 and Corollary 2, we have
‖d0(T )‖L2 ≤ C ∫ T
1 ∥∥∥∥ ∂∂td0(t) ∥∥∥∥ L2 dt
≤ C ∫ T
1 ∥∥∇ḡ0 (d(t)− d0(t))∥∥2L2 dt ≤ C ‖d(1)− d0(1)‖2L2 ≤ C‖d(1)‖ 2 L2 ≤ C · ( 2) 2 .
Furthermore, we obtain ‖d(T )− d0(T )‖L2 ≤ ‖d(1)− d0(1)‖L2 ≤ C · 2. By the triangle inequality, we get
‖d(T )‖L2 ≤ C · ( 2) 2 + C · 2. Followed by Corollary 4 and Lemma 2, such T should be pushed further outward, because
lim t→+∞ sup ∥∥∥∥ ∂∂td0(t) ∥∥∥∥ L2 ≤ lim t→+∞ sup ∥∥∇ḡ0 (d(t)− d0(t))∥∥2L2 = 0.
Thus, as t approaches to +∞, ḡ(t) converges to ḡ(∞) = ḡ0 + d0(∞). By the Euclidean Sobolev inequality (Minerbe, 2009), d(t)− d0(t) converges to 0 as t goes to +∞,
lim t→+∞ ‖d(t)− d0(t)‖L2 ≤ limt→+∞C ∥∥∇ḡ0 (d(t)− d0(t))∥∥L2 = 0.
We now conclude a result for linearly nearly Euclidean manifolds under the Ricci-DeTurck flow, which ensures a infinite time existence.

heading: 4 GRADIENT FLOW WITH LINEARLY NEARLY EUCLIDEAN METRICS
text: Consequently, we have clarified the convergence of linearly nearly Euclidean manifolds under the Ricci-DeTurck flow. Furthermore, we will consider the solution of gradient flow with linearly nearly Euclidean metrics, which will allow us to observe the neural network behavior for back-propagation in the linearly nearly Euclidean manifold. Empirically, we introduce information geometry (Amari & Nagaoka, 2000; Amari, 2016) and mirror descent algorithm (Bubeck et al., 2015) to construct the gradient flow with the help of divergences.

heading: 4.1 LINEARLY NEARLY EUCLIDEAN DIVERGENCE
text: From the perspective of information geometry and mirror descent algorithm, the metric ḡ can be deduced by the divergence that needs to satisfy certain criteria (Basseville, 2013). We now consider two nearby points P and Q in a manifoldM, where these two points are expressed in coordinates as ξP and ξQ, where ξ is a column vector. Moreover, the divergence is defined as half the square of an infinitesimal distance between two sufficiently close points in Definition 2.
Definition 2 D[P : Q] is called a divergence when it satisfies the following criteria:
(1)D[P : Q] ≥ 0, (2)D[P : Q] = 0 when and only when P = Q, (3) When P andQ are sufficiently close, by denoting their coordinates by ξP and ξQ = ξP + dξ, the Taylor expansion of D is written as
D[ξP : ξP + dξ] = 1
2 ∑ i,j ḡij(ξP )dξidξj +O(|dξ|3),
and metric ḡij is positive-definite, depending on ξP .
In order to construct a linearly nearly Euclidean metric, according to Definition 2, one can construct the divergence to obtain the expression of the metric indirectly. And the advantage is that the constructed divergence can be used to calculate the gradient flow under linearly nearly Euclidean metrics. Therefore, with the assist of Definition 3, we introduce a symmetrized convex function to construct the needed divergence:
φ(ξ) = ∑ i 1 τ2 log 1 2 (exp(τξi) + exp(−τξi)) = ∑ i 1 τ2 log (cosh(τξi)) (7)
where τ is a constant parameter.
Definition 3 The Bregman divergence (Bregman, 1967) DB [ξ : ξ′] is defined as the difference between a convex function φ(ξ) and its tangent hyperplane z = φ(ξ′) + (ξ− ξ′)∇φ(ξ′), depending on the Taylor expansion at the point ξ′:
DB [ξ : ξ ′] = φ(ξ)− φ(ξ′)− (ξ − ξ′)∇φ(ξ′).
Theorem 7 For a convex function φ defined by Equation (7), the linearly nearly Euclidean divergence between two points ξ and ξ′ is
DLNE [ξ ′ : ξ] = ∑ i [ 1 τ2 log cosh(τξ′i) cosh(τξi) − 1 τ (ξ′i − ξi) tanh(τξi) ]
(8)
where the Riemannian metric is ḡij(ξ(t)) = δij − [ tanh(τξ) tanh(τξ)> ] ij
= 1− tanh(τξ1(t)) tanh(τξ1(t)) · · · − tanh(τξ1(t)) tanh(τξn(t))... . . . ... − tanh(τξn(t)) tanh(τξ1(t)) · · · 1− tanh(τξn(t)) tanh(τξn(t))  . (9) Proof. The proofs can be found in Appendix C.1.
Based on Theorem 7, the form of the metrics ḡ(t) constructed by the linearly nearly Euclidean divergence is consistent with the definition of linearly nearly Euclidean metrics, as long as we adjust parameter τ to satisfy Definition 7. Moreover, we have also proven that the linearly nearly Euclidean divergence satisfies the criteria of divergence followed by Definition 2.

heading: 4.2 WEAK APPROXIMATION OF THE GRADIENT FLOW
text: By the linearly nearly Euclidean divergence, we can perform micro-surgery on the neural manifold under the gradient descent. Specifically, we dynamically consider the gradient flow toward the optimal descent direction on a manifold endowed with linearly nearly Euclidean metrics.
Lemma 3 The steepest descent gradient flow measured by the linearly nearly Euclidean divergence is defined as
∂̃ξ = ḡ −1(t)∂ξ = [ δij − tanh(τξ(t)) tanh(τξ(t))> ]−1 ∂ξ. (10)
Proof. The proofs can be found in Appendix C.2.
However, Lemma 3 involves inversion, which greatly consumes computing resources. In particular, we propose two methods for approximating the gradient flow: weak approximation and strong approximation respectively.
For the weak approximation of the gradient flow, we put forward higher requirements for this metric on the basis of Definition 7, which requires that the linearly nearly Euclidean metric is also a strictly diagonally-dominant matrix based on Corollary 3.
Corollary 3 The weak approximation of the gradient flow measured by the linearly nearly Euclidean divergence is defined as
∂̃ξ ≈ [ δij + tanh(τξ(t)) tanh(τξ(t)) >] ∂ξ (11) if the metric satisfies strictly diagonally-dominant.
Proof. The proofs can be found in Appendix C.3.

heading: 4.3 STRONG APPROXIMATION WITH NEURAL NETWORKS
text: Bypassing the requirement of weak approximation in Corollary 3, our goal is to approximate the gradient flow, ḡ−1(t)∂ξ in Lemma 3, from the assist of multi-layer perceptron (MLP) neural network. Using the neural network, we can easily yield the strong approximation of the gradient flow because a neural network with a single hidden layer and a finite number of neurons can be used to approximate a continuous function on compact subsets (Jejjala et al., 2020), which is stated by the universal approximation theorem (Cybenko, 1989; Hornik, 1991).
As an n × n symmetric matrix, ḡ(t) can be decomposed in terms of the combination of entries P and A, where P is the entries made up of the elements of the lower triangular matrix that contains n(n− 1)/2 real parameters and A is the entries made up of the elements of the diagonal matrix that contains n real parameters. During the training in Figure 1, g̃(t) can be used as strong approximation of ḡ−1(t) in the gradient flow.
L = ‖I − ḡ(t)g̃(t)‖2. (12)
5 EXPERIMENT
CIFAR datasets. The two CIFAR datasets Krizhevsky et al. (2009) consist of natural color images with 32×32 pixels, respectively 50,000 training and 10,000 test images, and we hold out 5,000
training images as a validation set from the training set. CIFAR10 consists of images organized into 10 classes and CIFAR100 into 100 classes. We adopt a standard data augmentation scheme (random corner cropping and random flipping) that is widely used for these two datasets. We normalize the images using the channel means and standard deviations in preprocessing.
Settings. We set total training epochs as 200 where the learning strategy is lowered by 10 times at epoch 80, 150, and 190, with the initial 0.1. The learning strategy is a weight decay of 0.0001, a batch size of 128, SGD optimization. On CIFAR10 and CIFAR100 datasets, we apply ResNet20, ResNet32, ResNet44, ResNet56 and ResNet110 models (He et al., 2016) to observe the evolution of neural manifold, i.e., the convergence of metrics depended on time. As far as we define the metric ḡ(t), we can use the length |ds2| = √∑ i,j ḡij(t)dξidξj to intuitively reflect the change of metrics.
Specifically, we define a ball whose radius is equal to |ds2|:
Br(t) := r = √∑
i,j
ḡij(t)dξidξj  . (13) Details. We embed the linearly nearly Euclidean manifold into a neural network, which means that a neural networks uses Corollary 3 for back-propagation. No other parts of the neural network need to be modified.
Neural Network Behavior. By observing the change of the ball in Figure 2, we can know the change of the metric. Through simple observation, metrics ḡ(t) on CIFAR10 converge in about 100 epochs and metrics ḡ(t) on CIFAR100 converge in about 150 epochs. For CIFAR10, metrics ḡ(t) in ResNet32 and ResNet44 seem to converge the fastest. For CIFAR100, metrics ḡ(t) in ResNet110 seem to converge the fastest. In general, experiments show that all metrics in neural manifolds converges to a linearly nearly Euclidean metric. It is consistent with the evolution of Ricci-DeTurck flow in Section 3.
Remark. For a neural network that is specified by connection weights, the set of all such weighs forms a manifold. When we use the gradient flow to learn a neural network (ξ(t) is composed of weights), we observe the evolution of its metric is consistent with the micro-surgery process under the Ricci-DeTurck flow. Consequently, the training of a neural manifold is also a surgery, i.e., the manifold is gradually regular, whose process is stable and eventually converges.

heading: 6 CONCLUSION
text: In this paper, we have analysed the evolution of linearly nearly Euclidean metrics under the RicciDeTurck flow, including proof of convergence in short and infinite time. Furthermore, we construct a linearly nearly Euclidean metric with the assist of the information geometry and use it as a springboard to reach the approximation of gradient flow. In view of the convergence and stability of metrics, the neural network trained by the approximated gradient flow allows us to observe the relevance between Ricci flow and neural network behavior under the manifold microsurgery. We hope that this paper will open an exciting future direction which will use Ricci flow to assist neural network training in a manifold.

heading: A DIFFERENTIAL GEOMETRY
text: 1. Riemann curvature tensor (Rm) is a (1,3)-tensor defined for a 1-form ω:
Rlijkωl = ∇i∇jωk −∇j∇iωk
where the covariant derivative of F satisfies
∇pF j1...jli1...ik = ∂pF j1...jl i1...ik + l∑ s=1 F j1...q...jli1...ik Γ js pq − k∑ s=1 F j1...jli1...q...ikΓ q pis .
In particular, coordinate form of the Riemann curvature tensor is:
Rlijk = ∂iΓ l jk − ∂jΓlik + Γ p jkΓ l ip − Γ p ikΓ l jp.
2. Christoffel symbol in terms of an ordinary derivative operator is:
Γkij = 1
2 gkl(∂igjl + ∂jgil − ∂lgij).
3. Ricci curvature tensor (Ric) is a (0,2)-tensor:
Rij = R p pij .
4. Scalar curvature is the trace of the Ricci curvature tensor:
R = gijRij .
5. Lie derivative of F in the direction dϕ(t)dt :
L dϕ(t) dt F =
( d
dt ϕ∗(t)F ) t=0
where ϕ(t) :M→M for t ∈ (− , ) is a time-dependent diffeomorphism ofM toM.

heading: B PROOF FOR THE RICCI FLOW
text: B.1 PROOF FOR LEMMA 1
Definition 4 The linearization of the Ricci curvature tensor is given by
D[Ric](h)ij = − 1
2 gpq(∇p∇qhij +∇i∇jhpq −∇q∇ihjp −∇q∇jhip).
Proof. Based on Appendix A, we have
∇q∇ihjp = ∇i∇qhjp −Rrqijhrp −Rrqiphjm.
Combining with Definition 4, we can obtain the deformation equation because of∇kgij = 0,
D[−2Ric](h)ij =gpq∇p∇qhij +∇i ( 1
2 ∇jhpq −∇qhjp
) +∇j ( 1
2 ∇ihpq −∇qhip
) +O(hij)
=gpq∇p∇qhij +∇iVj +∇jVi +O(hij).
B.2 DESCRIPTION OF THE DETURCK TRICK
Using a time-dependent diffeomorphism ϕ(t), we express the pullback metrics ḡ(t):
g(t) = ϕ∗(t)ḡ(t)
is a solution of the Ricci flow. Based on the chain rule for the Lie derivative in Appendix A, we can calculate
∂ ∂t g(t) =
∂ (ϕ∗(t)ḡ(t))
∂t
=
( ∂ (ϕ∗(t+ τ)ḡ(t+ τ))
∂τ
) τ=0
= ( ϕ∗(t) ∂ḡ(t+ τ)
∂τ
) τ=0 + ( ∂ (ϕ∗(t+ τ)ḡ(t)) ∂τ ) τ=0
= ϕ∗(t) ∂
∂t ḡ(t) + ϕ∗(t)L ∂ϕ(t) ∂t ḡ(t).
With the help of Equation (1), for the reparameterized metric, we have
∂ ∂t g(t) = ϕ∗(t) ∂ ∂t ḡ(t) + ϕ∗(t)L ∂ϕ(t) ∂t ḡ(t) = −2 Ric(ϕ∗(t)ḡ(t)) = −2ϕ∗(t) Ric(ḡ(t)).
The diffeomorphism invariance of the Ricci curvature tensor is used in the last step. The above equation is equivalent to
∂ ∂t ḡ(t) = −2 Ric(ḡ(t))− L ∂ϕ(t) ∂t ḡ(t).
B.3 PROOF FOR THEOREM 3
Considering any x ∈M, t0 ∈ [0, T ), V ∈ TxM, we have∣∣∣∣log(gx(t0)(V, V )gx(0)(V, V ) )∣∣∣∣ = ∣∣∣∣∫ t0
0
∂ ∂t [log gx(t)(V, V )] dt ∣∣∣∣ = ∣∣∣∣∣ ∫ t0
0
∂ ∂tgx(t)(V, V )
gx(t)(V, V ) dt ∣∣∣∣∣ ≤ ∫ t0
0
∣∣∣∣ ∂∂tgx(t) ( V |V |g(t) , V |V |g(t) )∣∣∣∣ dt ≤ ∫ t0
0 ∣∣∣∣ ∂∂tgx(t) ∣∣∣∣ g(t) dt
≤ C. By exponentiating both sides of the above inequality, we have
e−Cgx(0)(V, V ) ≤ gx(t0)(V, V ) ≤ eCgx(0)(V, V ). This inequality can be rewritten as
e−Cgx(0) ≤ gx(t0)(V, V ) ≤ eCgx(0)(V, V ) because it holds for any V . Thus, the metrics g(t) are uniformly equivalent to g(0).
Now, we have the well-defined integral: gx(T )− gx(0) = ∫ T
0
∂ ∂t gx(t)dt.
We say that this integral is well-defined because of two reasons. Firstly, as long as the metrics are smooth, the integral exists. Secondly, the integral is absolutely integrable. Based on the norm inequality induced by g(0), one has
|gx(T )− gx(t)|g(0) ≤ ∫ T t ∣∣∣∣ ∂∂tgx(t) ∣∣∣∣ g(0) dt.
For each x ∈M, the above integral will approach to zero as t approaches T . BecauseM is compact, the metrics g(t) converge uniformly to a continuous metric g(T ) which is uniformly equivalent to g(0) onM. Moreover, we can show that
e−Cgx(0) ≤ gx(T ) ≤ eCgx(0).

heading: C PROOF FOR THE INFORMATION GEOMETRY
text: C.1 PROOF FOR THEOREM 7
The linearly nearly Euclidean divergence can be defined between two nearby points ξ and ξ′, where the first derivative of the linearly nearly Euclidean divergence w.r.t. ξ′ is:
∂ξ′DLNE [ξ ′ : ξ]
= ∑ i [ ∂ξ′ 1 τ2 log cosh(τξ′i)− ∂ξ′ 1 τ2 log cosh(τξi)− 1 τ ∂ξ′(ξ ′ i − ξi) tanh(τξi) ] = ∑ i ∂ξ′ 1 τ2 log cosh(τξ′i)− 1 τ tanh(τξ).
The second derivative of the linearly nearly Euclidean divergence w.r.t. ξ′ is:
∂2ξ′DLNE [ξ ′ : ξ] = ∑ i ∂2ξ′ 1 τ2 log cosh(τξ′i).
We deduce the Taylor expansion of the linearly nearly Euclidean divergence at ξ′ = ξ:
DLNE [ξ ′ : ξ] ≈ DLNE [ξ : ξ] + (∑ i ∂ξ′ 1 τ2 log cosh(τξ′i)− 1 τ tanh(τξ) )> ∣∣∣∣ ξ′=ξ dξ
+ 1
2 dξ> (∑ i ∂2ξ′ 1 τ2 log cosh(τξ′i) )∣∣∣∣ ξ′=ξ dξ
= 0 + 0 + 1
2τ2 dξ>∂
[ ∂ cosh(τξ)
cosh(τξ)
] dξ
= 1
2τ2 dξ>
∂2 cosh(τξ) cosh(τξ)− ∂ cosh(τξ)∂ cosh(τξ)>
cosh2(τξ) dξ
= 1
2τ2 dξ>
( ∂2 cosh(τξ)
cosh(τξ) − τ2
[ sinh(τξ)
cosh(τξ)
] [ sinh(τξ)
cosh(τξ)
]>) dξ
= 1
2 ∑ i,j δij − [ tanh(τξ) tanh(τξ)> ] ij dξidξj .
C.2 PROOF FOR LEMMA 3
We would to know in which direction minimizes the loss function with the constraint of the linearly nearly Euclidean divergence, so that we do the minimization:
dξ∗ = arg min dξ s.t. DLNE [ξ:ξ+dξ]=c L(ξ + dξ)
where c is the constant. The loss function descends along the manifold with constant speed, regardless the curvature.
Now, we write the minimization in Lagrangian form. Combined with Theorem 7, the linearly nearly Euclidean divergence can be approximated by its second order Taylor expansion. Approximating L(ξ + dξ) with it first order Taylor expansion, we get:
dξ∗ = arg min dξ L(ξ + dξ) + λ (DLNE [ξ : ξ + dξ]− c)
≈ arg min dξ
L(ξ) + ∂ξL(ξ)>dξ + λ
2 dξ>ḡij(t)dξ − cλ.
To solve this minimization, we set its derivative w.r.t. dξ to zero:
0 = ∂
∂dξ L(ξ) + ∂ξL(ξ)>dξ +
λ 2 dξ>
[ δij − tanh(τξ(t)) tanh(τξ(t))> ] dξ − cλ
= ∂ξL(ξ) + λ [ δij − tanh(τξ(t)) tanh(τξ(t))> ] dξ
dξ = − 1 λ
[ δij − tanh(τξ(t)) tanh(τξ(t))> ]−1 ∂ξL(ξ)
where a constant factor 1/λ can be absorbed into learning rate. Up to now, we get the optimal descent direction, i.e., the opposite direction of gradient which takes into account the local curvature defined by [ δij − tanh(τξ(t)) tanh(τξ(t))> ]−1 .
C.3 PROOF FOR COROLLARY 3
Definition 5 ForA ∈ Rn×n,A is called as the strictly diagonally-dominant matrix when it satisfies∣∣aii∣∣ > n∑ j=1,j 6=i ∣∣aij∣∣, i = 1, 2, . . . , n. Definition 6 IfA ∈ Rn×n is a strictly diagonally-dominant matrix, thenA is a nonsingular matrix together.
Subsequently, we can consider the inverse matrix of the metric ḡ(t). Due to the strictly diagonally-dominant feature in Definition 5 and Definition 6, we can approximate[ δij − tanh(τξ(t)) tanh(τξ(t))> ]−1 . As for we can also ignore the fourth-order small quantity∑
O(ρaρbρcρd) because of the characteristic of the strictly diagonally-dominant, so that[ δij − tanh(τξ(t)) tanh(τξ(t))> ] [ δij + tanh(τξ(t)) tanh(τξ(t)) >] =
1− ρ1ρ1 −ρ1ρ2 · · ·−ρ2ρ1 1− ρ2ρ2 · · ·... ... . . .  1 + ρ1ρ1 ρ1ρ2 · · ·ρ2ρ1 1 + ρ2ρ2 · · ·... ... . . .  =  1− ∑ O(ρaρbρcρd) ρ1ρ2 − ρ1ρ2 − ∑ O(ρaρbρcρd) · · · −ρ2ρ1 + ρ2ρ1 − ∑ O(ρaρbρcρd) 1− ∑ O(ρaρbρcρd) · · ·
... ...
. . .  ≈ I. Note that the Euclidean metric δij is equal to the identity matrix I .

heading: D ANALYSIS ON LINEARLY NEARLY EUCLIDEAN METRICS
text: Let us give the definition of linearly nearly Euclidean metrics without further explanation:
Definition 7 A complete Riemannian n-manifold (Mn, g0) is said to be linearly nearly Euclidean with one end of order τ > 0 if there exists a compact set K ⊂ M, a radius r, a point x in M and a diffeomorphism satisfying φ :M\K → (Rn\B(x, r))/SO(n), where B(x, r) is the ball and SO(n) is a finite group acting freely on Rn\{0}, then∣∣∂k(φ∗γ0)∣∣δ = O(r−τ−k) ∀k ≥ 0 (14) holds on (Rn\B(x, r))/SO(n). g0 can be linearly decomposed into a form containing the Euclidean metric δ:
g0(t) = δ + γ0(t). (15)
In this paper, we consider the linear stability and integrability of the initial metric g0. Fortunately, similar to the proof process of (Koiso, 1983; Besse, 2007), we can proceed that g0 is integral and linearly stable.
Definition 8 A complete linearly nearly Euclidean n-manifold (Mn, g0) is said to be linearly stable if the L2 spectrum of the Lichnerowicz operator Lg0 := ∆g0 + 2 Rm(g0)∗ is in (−∞, 0] where ∆g0 is the Laplacian, when Lg0 acting on dij satisfies
Lg0(d) = ∆g0d+ 2 Rm(g0) ∗ d = ∆g0d+ 2 Rm(g0)ikljdmng km 0 g ln 0 .
Definition 9 A n-manifold (Mn, g0) is said to be integrable if a neighbourhood of g0 has a smooth structure.
E SHORT TIME CONVERGENCE IN THE L2-NORM
For convenience, we rewrite the Ricci-DeTurck flow (3) in terms of the difference d(t) := ḡ(t)− ḡ0, such that
∂ ∂t d(t) = ∂ ∂t ḡ(t) = −2 Ric(ḡ(t)) + 2 Ric(ḡ0) + L ∂ϕ′(t) ∂t ḡ0 − L ∂ϕ(t) ∂t ḡ(t)
= ∆d(t) + Rm ∗d(t) + Fḡ−1 ∗ ∇ḡ0d(t) ∗ ∇ḡ0d(t) +∇ḡ0 ( GΓ(ḡ0) ∗ d(t) ∗ ∇ ḡ0d(t) ) ,
(16) where the tensors F and G depend on ḡ−1 and Γ(ḡ0). Note that ḡ0 is a linearly nearly Euclidean metric which satisfies the above formula, where d0(t) = ḡ0(t) − ḡ0, so that d(t) − d0(t) = ḡ(t) − ḡ0(t) holds. Note that ‖ · ‖L2 or ‖ · ‖L∞ denotes the L2-Norm or L∞-Norm with respect to the metric ḡ0.
Lemma 4 Let (Mn, ḡ0) be a complete linearly nearly Euclidean n-manifold. If ḡ(0) is a metric satisfying ‖ḡ(0) − ḡ0‖L∞ < where > 0, then there exists a constant C < ∞ and a unique Ricci–DeTurck flow ḡ(t) that satisfies
‖ḡ(t)− ḡ0‖L∞ < C‖ḡ(0)− ḡ0‖L∞ < C · .
If a Ricci-DeTurck flow in BL∞(ḡ0, ) for t ≥ 1, there exist constants such that ∥∥∇k (ḡ(t)− ḡ0)∥∥L∞ < C(k) , ∀k ∈ N. Proof. The similar statement for the case of negative Einstein metrics is given in (Bamler, 2010). The proofs can be translated easily to the case of linearly nearly Euclidean metrics by referring the details (Bamler, 2011).
Lemma 5 Let (Mn, ḡ0) be a linearly nearly Euclidean n-manifold. For a Ricci–DeTurck flow ḡ(t) on a maximal time interval t ∈ [0, T ), if it satisfies ‖ḡ(0) − ḡ0‖L∞ < where > 0, then there exists a constant C <∞ for t ∈ (0, T ) such that
‖ḡ(t)− ḡ0‖L2 < C. (17)
Proof. Based on Lemma 4, we can consider about ‖ḡ(t) − ḡ0‖L2 . Let κ be a function such that κ = 1 on B(x, r), κ = 0 onMn\B(x, 2r) and |∇κ| ≤ 2/r where x ∈Mn and a radius r.
Followed by Equation (16), we obtain
∂
∂t ∫ M |d(t)|2κ2dµ ≤2 ∫ M 〈 ∆d(t), κ2d(t) 〉 dµ+ C‖Rm ‖L∞ ∫ M |d(t)|2κ2dµ
+ C‖d(t)‖L∞ ∫ M |∇d(t)|2κ2dµ+ ∫ M 〈∇(GΓ ∗ d ∗ ∇d), d〉κ2dµ
≤− 2 ∫ M |∇d(t)|2κ2dµ+ C ∫ M |∇d(t)||d(t)||∇κ|κdµ
+ C (ḡ0) ∫ M |d(t)|2κ2dµ+ C‖d(t)‖L∞ ∫ M |∇d(t)|2κ2dµ
≤(−2 + C · + C1) ∫ M |∇d(t)|2κ2dµ+ C (ḡ0) ∫ M |d(t)|2κ2dµ
+ 1
C1 ∫ M |d(t)|2|∇κ|2dµ
≤ ( C (ḡ0) + 2
C1r2 )∫ B(x,2r) |d(t)|2dµ.
Note that we can always find a suitable C1 to make the above formula true. By integration in time t, we can further obtain∫
M |d(t)|2κ2dµ ≤ ∫ M |d(0)|2κ2dµ+ ( C (ḡ0) + 2 C1r2 )∫ t 0 ∫ B(x,2r) |d(s)|2dµds <∞.
Consequently, we can find a finite ball that satisfies this estimate.
Corollary 4 Based on Lemma 5, we further have
sup ∫ M |d(t)|2κ2dµ <∞. (18)
Proof. We obtain
sup ∫ M |d(t)|2κ2dµ ≤ sup ∫ M |d(0)|2κ2dµ
+N ( C (ḡ0) + 2
C1r2 )∫ t 0 sup ∫ M |d(s)|2κ2dµds,
where each ball of radius 2r on M can be covered by N balls of radius r because (Mn, ḡ0) is linearly nearly Euclidean. By the Gronwall inequality, we have
sup ∫ M |d(t)|2κ2dµ ≤ exp ( N ( C (ḡ0) + 2 C1r2 ) t ) sup ∫ M |d(0)|2κ2dµ.
For the L2-Norm, the Ricci-DeTurck flow in linearly nearly Euclidean manifolds has a solution for a short time.

Review summaries:
The paper concerns the Ricci flow and surgery to handle singularities occuring during the Ricci flow. The authors propose linearly nearly Euclidean metrics that they prove are stable under the Ricii-DeTurck flow. The authors use this to approximate steepest descent gradient flows in information geometry and state that they obtain a new method for geometric optimization on manifolds.

This paper investigates linearly nearly Euclidean metrics on Riemannian manifolds under the Ricci flow. Analysis, focusing on the stability and convergence of the evolution of such metrics, is presented. In addition, the utility of the analysis for measuring the approximation of gradient flow in the context of training neural networks is demonstrated.