On Non-Linear operators for Geometric Deep Learning
Authors: Grégoire Sergeant-Perthuis
Jakob Maier
Joan Bruna
Edouard Oyallon
Venue: NeurIPS 2022 Accept (2022)
Keywords: Manifold
Deep learning
Diffeomorphisms

Abstract: This work studies operators mapping vector and scalar fields defined over a manifold $\mathcal{M}$, and which commute with its group of diffeomorphisms $\text{Diff}(\mathcal{M})$. We prove that in the case of scalar fields $L^p_\omega(\mathcal{M,\mathbb{R}})$, those operators correspond to point-wise non-linearities, recovering and extending known results on $\mathbb{R}^d$. In the context of Neural Networks defined over $\mathcal{M}$, it indicates that point-wise non-linear operators are the only universal family that commutes with any group of symmetries, and justifies their systematic use in combination with dedicated linear operators commuting with specific symmetries. In the case of vector fields $L^p_\omega(\mathcal{M},T\mathcal{M})$, we show that those operators are solely the scalar multiplication. It indicates that $\text{Diff}(\mathcal{M})$ is too rich and that there is no universal class of non-linear operators to motivate the design of Neural Networks over the symmetries of $\mathcal{M}$.

On Non-Linear operators for Geometric Deep Learning
Grégoire Sergeant-Perthuis
Jakob Maier INRIA, DI/ENS, PSL Paris
Univ. Artois, UR 2462, Laboratoire de Mathématiques de Lens (LML)
F-62300 Lens, France & OURAGAN team, Inria Paris & IMJ-PRG Paris, France. gregoireeerprdgmail.commm com
Joan Bruna
Edouard Oyallon
Courant Institute of Mathematical Sciences New York University New York
MLIA Machine Learning and Information Access Sorbonne Université, CNRS, ISIR, F-75005 Paris, France
Abstract
This work studies operators mapping vector and scalar fields defined over a man- ifold M, and which commute with its group of diffeomorphisms Dif We prove that in the case of scalar fields I (MM, IR, those operators correspond to point-wise non-linearities, recovering and extending known results on Rd. In the context of Neural Networks defined over M,, it indicates that point-wise non- linear operators are the only universal family that commutes with any group of symmetries, and justifies their systematic use in combination with dedicated lin- ear operators commuting with specific symmetries. In the case of vector fields (M,, MM, we show that those operators are solely the scalar multiplication. It indicates that Dif is too rich and that there is no universal class of non-linear operators to motivate the design of Neural Networks over the symmetries of M.
Introduction
Given physical domain M and measurements f .M -> y observed over it, one is often interested in processing intrinsic information from f. i.c. consistent with the symmetries of the domain. Let M denote an operator, it can be seen as a non-linear operator acting on measurements, In words, two measurements f. = g.f are related by a symmetry of the domain, like a rigid motion on an observed molecular compound, we would like our processed data M(f) and Mf) to be related by the same symmetry thus that M(g.f) with the symmetry transformation of the domain. The study of operators that satisfy such symmetry constraints has played long and central role in the history of physics and mathematics, motivated by the inherent symmetries of physical laws. More recently, such importance has also extended to the design of machine learning systems, where symmetries improve the sample complexity [2] For instance, Convolutional Neural Networks build translation symmetry, whereas Graph Neural Networks build permutation symmetry, amongst other examples coined under the 'Geometric Deep Learning" umbrella 5 4
(f) equivalently that M commutes
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Lie groups of transformations are of particular interest, because there exists a precise and systematic framework build such intrinsic operators. Indeed, for locally compact group G, it possible to define a Haar measure which is invariant to the action of G [2] then a simple filtering along the orbit of G allows to define class of linear operators that commute with the group action. Examples of locally compact groups are given by specific Lie groups acting on RR such as the translations or the rotations (IR). Often these Lie groups G only act on manifold M, and one tries to average along the orbit induced by G. Note that it is possible, beyond invariance, to linearize more complex groups of variability like diffeomorphisms Dif 7
While the description of such linear intrinsic structures is of central mathematical importance and forms the basis of Representation theory [30], in itself is not sufficient to bear fruit in the context of Representation learning using Neural Networks [12]. Indeed, linear operators do not have the capacity to extract rich information needed to solve challenging high-dimensional learning problems. It is therefore necessary to extend the systematic construction and classification of intrinsic operators to the non-linear case.
With that purpose in mind, our work aims at studying the class of (non-linear) operators M which commute with the action of the group DM, the diffeomorphisms over M. This approach will lead to a natural class of non-linear intrinsic operators. Indeed, any group G of symmetries is, by definition, a subgroup of Diff(MM), and thus commutes with such M [4 Consequently, obtaining a non-linear invariant to a symmetry group G could be done by using a cascade of interlacing non- operators which commute with MM and linear operators which commute with G.
A notable example of linear operators that are covariant to the Lie group of translations is a given by the convolutions along the orbit of the group. These can be constructed thanks to the canonical Haar measure 132. However, such an approach fails for infinite dimensional groups, like our object of interest: contrary to Lie groups, (MM is not locally compact and it is thus not possible to define a Haar measure on this group.
Our first contribution is demonstrate that the non-linear operators which act on vector fields (elements of (M, TM)) and which commute with the group of diffeomorphisms, are actually just scalar multiplications. This implies that (Mf is too rich obtain non-trivial operators. Our second contribution is to demonstrate that non-linear operators acting on signals in LE (M,, R) are pointwise non- coneaiieasssies. This fills a gap in the results of 77, and a fortiori justifies the use of point-wise non- con-lineesss in geometric Deep Learning [4]
Let us remark that the study of equivariant operators that take input vector fields is motivated by the use of Neural Networks in physics, in particular for dynamical systems such as fluid dynamics [8] For example, one subject of interest in hydrodynamics how vector field of velocities evolves; the time evolution of such field is described by a partial differential equation (PDE), the Navier-Stokes equations, in which Neural Networks found recent applications and is more generally the case of other PDE [31]
Our paper is structured follows: Sec. introduces the necessary formalism, that use through this paper: in particular, we formally define the action of diffeomorphism. Then, we state and discuss our theorems in Sec. and sketch their proofs in Sec. 3.2 Rigorous proofs of each statement can be found in the Appendix.
2 Problem Setup
2.1 Related work and motivation
In this section, we discuss the notion of intrinsic operators, invariant and covariant non-linear operators and linear representation over standard symetry groups. Then, we formally state our objective.
Intrinsic Operators As discussed above, in this work we are interested in intrinsic operators M. E), + LP(M, E), where M is a Riemannian manifold, and E' = IR or E = TM, capturing respectively the setting of scalar signals and vector fields over M. L (MM,, RR) is the space of scalar function f MM- + IR which p-th power is integrable, similarly (MMM, TM) is the space of sections of the tangent bundle of M (denoted TM). f : M - TM, which norm |[ : M + R is in L (M, R). Here the notion of 'intrinsic' means that M is consistent with an equivalence class induced by symmetry group G in (MM, if f. e LPM, E') are related by transformation
2

---

g E G (in which case we write f= g.f), then M(f) = Naturally, a stronger equivalence class imposes a stronger requirement towards M, and consequently restrains the complexity of M. We now describe the plausible techniques used to design such operators M.
CM-Convolutions The notion of G.M-conolutiioss 34 is example of linear covariant operators which commute with the reparametrization of manifold. In practice, this implies that the weights of -o-ooooolutioon are shared and the action of GM-convolutions is local two properties that facilitate implementation and point out the similarity with Lie groups. Another example of symmetry group corresponds the isometry group of a Riemaniann manifold, whose pushforward preserves the tensor metric. In this case, it is well known that isometries [33] are the only diffeomoomorhismm which commute with manifold Laplacian. Thus, any linear operators which commute with isometries is stabilized by Laplacian's 's eigenspaces. However, little is known on the non- non-linear counterpart of the symmetry-com covariant operators. In this work, we characterize non-linear operators which commute with (MM. We will see that such operators are intrinsically defined by Diff(MM and could be combined with any linear operators covariant with a symmetry group G.
Non-linear operators It has been shown that Convolutional Neural Networks are dense in the set of non-linear covariant operators (35). The recipe of the corresponding proof is extension of the proof of the universal approximation theorem [1]. The Scattering Transform [6] [3] is also
an example of well-understood non-linear operator which corresponds to cascade of complex wavelet transforms followed by point-wise modulus non-linearity. This representation provably linearizes small deformations.
Compact Lie Groups In the context of geometric Machine Learning 5 there are several relevant notions of equivalence. For instance, we consider a compact Lie Group G acting on M, and an associated representation in F = [ MM + R]: Given g E G and f E F, then fx)
(ta) for x E M. We then consider f f~ f. related by this group action: f = g-f for some g E G. The operators M which are compatible with such group action are referred as being G-equivariant (or covariant to the action of G) in the ML literature B 4 Such groups are typically of finite and small dimension, e.g. the Euclidean transformations of M = R2, with d = 2 for computer vision applications, d for computational biology/chemistry applications. In this case, it is possible to characterize all linear intrinsic operators M as group convolutions (0)] leading to a rich family of non-ling non-linear intrinsic operators by composing such group convolutions with element-wise non-linear operators, as implemented in modern Neural Networks. We highlight that stability to symetries via non-linear operators finds useful application, in particular for flat manifolds 77)
Isometries Riemanian manifolds M come with a default equivalence class, which is given by isometries. T denotes the tangent vector space of M at point u E M. If m., : T.M TTM R denotes the Riemannian metric tensor at point u E M, a diffeomorphism &b -M M is an isometry if (1, w) 9wtu) (dva,,)) (v), (d)(()) for any E . and u, w E T.M. In words, isometries changes of variables that preserve the local distances in the domain. The ensemble of all isometries forms a Lie Group which is locally compact [27). In this case, one can also build a rich class of intrinsic operators by following the previously explained 'blueprint, namely composing linear intrinsic operators with element-wiss non-linearities. As representative example, the Laplace- Beltrami operator of M only depends on intrinsic metric properties (33 as said above, isometries preserve the invariant subspaces of a Laplacian.
Beyond Isometries While isometries are the 'natural' transformations of the geometric domain, they cannot express -igh--imennional sources of variability; indeed, if M is d-dimensional complete connected Riemannian manifold, its isometry group has dimension at most d(d + 1)/2 [0]. This raises the question whether one can characterize intrinsic operators relative to broader class of transformations. Another class of important symmetries corresponds to the ones which are gauge invariant, i.e. which leads to transformations which preserve the change of parametrization and which are used in 34 through the notion of G-structure.
In this work, we consider the class of transformations given by Difi(M), the diffeomorphisms over M. As shown the Appendix, compactly supported deformations di M + M define bounded linear operators Ly acting on LPMM, E) + (M,, and constitute far broader class
3

---

of transformations than isometries. Our proof is mainly based on the use of compactly supported diffeomorphisms.
Our objective is to characterize the (non-linear) operators M such that
VΦ DM, L ML
In other words, aim to understand continuous operators M that commute with deformations. We will show that such operators are act locally and that they can be descriped explicitly, with simple formula. The commutation condition is visualized in the following diagram:
Le
M
M
Tip > Mg
Mf
2.2 Notations
We will now formally introduce the mathematical objects of interest in this document. Let (M. g) be an orientable, connected, Riemannian manifold, finite dimension d E N'. Let TM denote the tangent bundle of M, i.e the union of tangent spaces at points u E .M. M is the cotangent bundle of M. g E T(T M & T* M) a section symmetric definite positive bilinear forms on the tangent bundle of M. It is common denote PB the collection of sections of a bundle B; A T M for n S d is the bundle n-linear alternated forms of M, and A(A" " T M) is the space of section of this vector bundle over M.
For A C M, we denote A its closure; LA is the indicator function of A, i.c. which takes value 1 if æ E A and 0 otherwise. B(u,r) r) denotes the ball of radius r around u E M. Any two vectors v, U1 V in a pre-Hiilbert space (with scalar product (.)) are orthogonal, denoted U1, when (u, (1) = 0.
Fix E [1, tol. Any volume form w E E(A (TAATT* T defines a (positive) measure on the orientable Riemannian manifold M; the total volume of M is ((M) == [M 1dw. Let us define L MM, the space of L vector fields, defined the subspace of measurable functions f .M + TM such that f() T.M almost everywhere and
I A / J.EM f(u)) dw(x) < t00.
(1)
We will also consider L2(, R) the space of measurable scalar functions (fields) f M - that fulfill
[[()] dw) < 400.
JUE
We may write || instead of || Ilp when there is no ambiguity. For a t diffcomorphism Mi, will consider the action of Ls Lg (M,TM) + LM,, which we define for for any f E Lg (M, R) as
Laf(u) a
Note that this action is contravariant:
f = d(v Φ
x) -f ((u)) = I.p
For scalar function e (M,,)) we define the action of via
Lof(u) A f(b(u)).
Let A be measurable set of M and f IPM, E), flx A is the product of with 1 i.e. flA is equal to f on A and elsewhere. what follows we introduce "constant" fields over an open set, they are denoted clp with U open subset of M. For scalar fields. 'constant' scalar field f(u)

---

is equal to the same constant E R for any u E U. On the other hand, 'constant' vector fields flo are vector fields over for which there is a chart from U an open subset of R which for any E U [()) is equal to constant vector e E IR;; in the vector case we say that the vector field f'l can be straightened.
This latter operator is also contravariant. If there is no ambiguity, we will use the same notation Lgs whether we apply it LE (M, or L (M,TMM TM). We might sometimes refer L (M, R) or (M,, as LP(M, R) or LP(M,TM)) Throughout the article we restrict ourselves to such that Lo is bounded operator. Write supp(Ø) =u, ((u) # u for the support of & and say that & has compact support if supp(Ø) is compact. We denote by (MM C Diff(M) the set of compactly supported diffeomorphisms. Recall that since M is (P IR and (M, (M,TM TM) is dense in LE (M,TM) TM). Finally, denote by O& the set of unitary operators on Rd. Throughout the article, we might not write explicitly that equalities hold almost everywhere, since this is the default in L spaces.
C. (M) is dense in
As mentioned earlier, compactly supported diffeomorphisms lead to continuous operators, which is made rigorous by the following lemma whose proof is in the appendix.
Lemma 1. If supp(Ø) is compact, then Ly bounded.
3 Main theorems
In this section we present our main results. We first show that any (non-linear) deformation- equivariant operator acting scalar fields must be point-wise (Theorem [ and then establish that any deforrmtionneeuivaaiine operator acting on vector fields corresponds to a multiplication by scalar (Theorem 2)
3.1 Theorem statements
Now, we are ready to state our two main theorems:
Theorem (Scalar case). Let M connected and orientable manifold of dimension d 2 1. We consider a Lipschitz continuous operator M (M, + (EM,, where 1 S co. Then,
VÓ Dif((M:: MLs = L.M
is equivalent to the existence of Lipschitzz continuous function p R - R that fulfills ([f(mm)= = p(((m))
a.e.
In that case, we have p(0) if oo.
Theorem 2 (Vector case). Let M be connected and orientable manifold of dimension d > 1. We consider a continuous operator M (MM,TM + (M,, TM), where 1 S < oo. Then,
V DiffMM: MLs L
is equivalent to the existence of scalar X E IR such that Vf E LE (M, TM): M[f](m)= Xf a.e.
We highlight that our theorems are quite generic in the sense that they apply to the manifolds usually used in applications or theory, Fd in particular.
The scalar case allows to recover standard operators which exploited for Deep Neural Networks architectures. However, Theorem indicates that the group of diffeomorphism is too rich obtain non-trivial non-linear operators.
Remark 1.
Remark 2. The case = oo leads to different results. For instance, in the scalar case we may consider the operator M (x) = sup, ff() which fulfills L LM but not pointwise.
Remark 3. The condition (w(M) = DO = p(0) = 0" in Theorem is necessary, since in the case M = R. the operator M 2 ff() is not in (MM
Remark The Lipschitz condition in Theorem 7 crucial, otherwise, Mf(x)= p(f(x)) might not be an operator of L2 (M, R). For instance, i = 2, M [0. and M Mf(x) = Vf(x), we see that in this case, let f(x) = then f (M,, and M f & LR(
5

---

Remark 5. If M is not Lipschitz. we can fand an example which is not even continuous. The following example holds in both cases, the scalar case and the vector case. In both cases f E LP(M, the only thing that changes is the action of T f. M IR, let for all IR),
Mf(x) = g-m f(v)= (())=(()))))(((
It is a measurable function. Let us show that this M is a counterexample to the vector case: for any e M) and x E R, one has
M M = =1 (x,lim ((oo)))= (2) dx(x) ((}(z))
(3)
(x) do(x))
(4)
(2,limm 12 ==
(5)
-g)ff(z)
(6)
However, M is not continuous as changing any function to on Q does not change its norm but changes the set where the limits exists. More precisely let > 0 be a strictly positive scalar, M[d]
let = clfx & Q]. M[f]= = 0 as {z, (2,3lim,-> f(o(y)) 0. However c f almost everywhere but M(e) + M[} therefore M is not continuous.
3.2 Proof Sketch
We now deseribe the main ideas for proving the Theorems and The appendix contains complete formal arguments and technical lemmata which we omit here due to lack of space. The two proofs share quite some similarities despite substantially different final results. Three ideas guide our proofs: First, we prove that it is possible to localize M on a certain class of open sets which behaves nicely with the manifold structure, the strongly convex sets which we denote as O,, This is closely related to the notion of pre-sheaf [55.. Secondly, we characterize M on small open-sets. In the sealar case, we will study the representation of locally constant functions. In the vector case, we will show that locally, the image M (Lve) of a vector field e is co-linear to provided that U is small enough. We will also show that those local properties are independent of the position on the manifold M via connectedness argument. Thirdly and finally, we combine compacity and a density argument to extend this characterization to M, which is developed in Sec. 3.3 Throughout the presentation, we will use the following definitions and theorems obtained from other works:
Definition (Strong convexity, from [8]. Let O₂ be the collection of open sets which are bounded and strongly corvex, i.e. such that any points p, q in such a set can be joined by geodesic contained in the set. Furthermore let Ö = {A O₁ 3B E O C B and w (A) 0}.
The intuition behind the definition of Ô, is that all of its elements are contained in a 'security' open weich avoids degenerated effects the manifold. In particular, this allows control the boundary of a given open set.
Theorem 3 (theorem adapted from 17
(1) Ón is a system of neighborhoods. (2) Any element Or is diffeomorph to Rd. (3) Both O2 and O₂ are stable by intersection.
18
Theorem 4 (Flowbox theorem, stated in (1). Let f.g (M,, For any E -M with f(m) + 0 and g(m) # 0, there exists an open set U C C and & E Dif( such that p(m) = m and (5(1 f) 10
T We will now present some lemmata that are necessary for the proofs of theorems and 2 As a first step, we argue that one may assume M = O where O denotes the constant 0-function. This is because in the appendix we show that (0) a constant function C, with C if ((M) = co. Therefore, we may substract C' from p and X, leaving with having to show the theorems only for (0) = O.
Next, a key idea of the proof is to exploit the flexibility of the deformation equivariance to localise the input, i.e. to show that the image of compactly supported functions is also compactly supported. To do so. the following lemma provides way of collapsing open ball into singleton while maintaining a good control on the support of the diffeomorphism..
Lemma 2 (Key lemma). Let 0. There exists sequence of diffeomorphisms on :d Rd. compactly supported in B(0, 1 + e) such that:
1 on d.. 1)) 13(0 n

---

and
Lemma (Image of localized vector field), For M in Theorem there U E Ón, and X(U) such [ldo( -
sup wEB(0.1)
Proof. Set (n( fn (llll)),,, where
(x(r) = :
ifrr22+e
and fn is smoothly interpolated for |rl [1, + c] in a way that it remains nondecreasing. It is then clear that on fulfills the desired properties.
We will often that if the support of ái E Dif((M) is such that supp(ø) n 0, then for any E E (M, R) one has ly = (1v.). This implies the following important lemma, for which rigorous proof can be found in the appendix:
Lemma Let U E O₂ and M as in Theorem or Theorem 2 Then, for any f E E, where
E (M,,) or E (MM,) respectively, we have:
M[flo]= luM]]
Furthermore, U is any closed set, the same conclusion applies.
Equipped with this result, our proof will characterize the image of functions of the type cly where either E E. or is vector field which can be straightened (isomorphic constant vector), via the following Lemma. In the Vector case:
that for any f E (M,,MM
M[flv] - LUU)
Here the scalar case:
Lemma (Image constant functions, scalar case). Let M as in Theorem 1 For any E Ó and C R, then: M (clu) = h(c, U)lv. Furthermore, c - h(c, U) is Lipschitz for any E Ó₁
At this stage, we note that both representations are pisnt-wise, and the next steps of the proofs will be identical both for the scalar and vector cases. The extension to LL (M, or LE, (M, TM) will be done thanks to:
Lemma 6 (Image of disjoint union of opensets). Let U1,... Un E O₂ and M as in Theorem 2 or Theorem vi + j. U. N 0. Then for any (M,
M
10.
Ml.vff.
This lemma states that we can completely characterize M disjoint union of simple sets. We will then need an argument similar to Vitali covering Lemma in order to "glue" those open sets together, which shows that simple functions with disjoint support can approximate any elements of LE (M. R) or LE (M,TM) TM) (we only state the lemma for LE (M, R) as our proof on (M, does not necessarily need this result):
Lemma 7 (Local Vitali). For f E C (M) and m E M, there exists U E Ó with m E U, such that for any e > 0, there exist subsets U1,.., Un E Ö₂ with U, C U and numbers ci,..,, Cn E R such that:
Σ Tu.cn Cn Tulll <e.
Note that this type of covering is not possible any open set without further assumptions on the manifold, such as bounds on Ricci curvature (22). Fortunately, we will only need local version which is true because charts are locally bi-Lipschitz. Both Lemma 6 and Lemma imply that:

---

Proposition 1. Consider M from either Theorem 1 or Assume that there exists U E O₁ such that M(clv) = h(c, Ly for any V C. U, with V E O2 where e. is either a vector field in the case E (M, or a constant scalar in the case E (M, 1.M. we further assume that c + h(c, U) is L-Lipschitz, then
Yf E Em E (,M([off(m) = Luh(f(m),U)
Furthermore, it does not depend on U. meaning that for any other such Ü, we have:
of L. E U n M,M(Lef lrh(f(m)),))
We briefly discuss the intuition behind Theorem is linked to the idea that the operators at hand have to commute with local rotations, and this even for locally constant vector fields. We reduce cettrmmationqqqivaareee
the characterisation
vector operators using an invariance to symmetry multiples of scalar. The intuition is
argument: functions which are invariant to rotations contained in the following lemma, which is commonly used physics:
Lemma (Invariance to rotation). f Rdd - IRd such that for any W E (IR( and Rdd, one has f(Wx) W f(x). Then, there is X gd I2 f(x) A([x=|]ax.
Proof. We write ((x ((x))++t with x- (m) # 0 and x- 1 Then, introduce W E Oa(R) such that Wx (m) - (m) and Wx(m) x (m). From f(x) f(Wx) W fz) we deduce that x = 0. Next, A(Wx) A(x) thus X(x) = A(x') for any ||>|| ||='||-
Distinction between scalar and vector case case: there are several more steps for the proof of Theorem 2 one needs to show that the point-wise non-linearity actually a scalar multiplication. We also highlight that the non-linearity is fully defined by its image on locally constant functions.
The scalar case is simpler to handle than the vector
Finally, we conclude the proof of the theorem by appealing common density argument of the functions smooth with compact support, combing all the lemmata we have just presented in Sec. 3.3
3.3 Proofs conclusions (common to the scalar and vector case)
In this section, prove that the local properties of M can be extended globally on M. The main idea is to exploit the well-known Poincaré's formula, which states that:
lu Σ
Σ
ntg T...nU,,
and to localize the action of M on each U, n NUg n....nu U. E O1 thanks to Lemma
3
Proof of Theorem Fan Theorem 2
Let f be smooth and compactly supported function. Further
consider Ujsn Ug finite covering of its support with U, E Ó, Using an inclusion-exclusion formula we obtain
together with Lemma
1u,0 M = Σ
1-
Σ
10, nig n... M[]]
k=1
1-
Σ
M[flU, nug n...
where we used that U, nU, n... Un E Ö. Now, the support of f is closed and included in U.U,
Thus using Lemma
MI = Σ 1-1*
Σ
Mffle,
k=
Note that if p is a pointwise operator with p(0) 0, then ((1 lup(f) and

---

n
Σ Si.
= Σ (-1) Σ k=1
p()
(8)
in ci.cci
=l,,u,p(f) = p(f).
(9)
Thus, f = p(f) where p obtained from Lemma or combined with Prop density in L (M,R) or (M,TM respectively. This ends the proof.
| We conclude by
4 Remarks and conclusion
In this work, we have fully characterized non-linear operators which commute under the action of smooth deformations. In some sense, settles the intuitive fact that commutation with the whole diffeomorphism group is too strong property, leading to a small, nearly trivial family of non-linear intrinsic operators. While on their own they have limited interest for geometric deep representation learning, they can 'upgrade" any family of linear operators associated with any group C (M) into a powerful non-linear class the so-called GDL Blueprint in [4] Also, this result a first step towards characterizing the non-linear operators which commute with Gauge transformations and could give useful insights for specifying novel Gauge invariant architectures. We now state couple of unsolved questions and future work.
On the commutativity assumption: Several examples and approximation results 21 35 exist for operators that commute with Lie groups and discrete groups [19]. In this case, it is possible to define measure the group that invariant by the group action (called the Haar measure), which makes it possible to define convolutions. Roughly, non-linear operators covariant with some actions of those groups can be thought of as an approximation by Group Convolution Neural Networks. It is important to note that the inputs of the operators described in these articles are functions that take real values; the much more general class of inputs that take values in vector bundles is, to our knowledge, not covered in the literature. To our knowledge, we are the first work to study the design of equivariant Neural Networks that process vector fields defined over manifold. In this setting even for M = Rd, it is unclear which type of non-linear operators commute with smaller groups of symmetry such as the Euclidean group. In fact, a generic question holds for manifolds: for a given symmetry group G, what is elementary non-linear building block of a Neural Network? This could be, for instance, useful to design Neural Networks which are Gauge invariant. It an open question for future work which would be relevant many applications in physics [16]. Furthermore, the fact that the characterization of diffcomorphism invariant operators we exhibited this paper is very restrictive opens the way for the study of other non- non-locally 'smaller' compact groups; we believe that any results in that direction are completely novel.
Example of vector operators for L It is slightly unclear how the vector case p can be handled in our framework, yet (I] seems to have interesting insights toward this direction.
Lincarization of (M) In this work, we considered exact commutation between operators and a symmetries: however, it is unclear which operators approximatively commute with given symmetry group. Such operators would be better lincarize high- diggesinnnii symmetry group like MM). An important instance of non-linear operators that are non-local and that 'nearly' commute with diffeomorphisms is the Wavelet Scattering representation
28
Acknowledgments and Disclosure of Funding
EO was supported by the Project ANR-21-CE2340 ADONIS and EMERG-ADONIS from Al- liance SU. GSP was also supported by France Relance and Median Technologies; he would like to thank very much NeurIPS Foundation for their financial support (NeurIPS 2022 Scholar Award).

---

References
[1] John C Baez. Diffeomorphism invariant generalized measures on the space of connections modulo gauge transformations. In Proceedings of the Quantum Topology Conference. World Scientific, Singapore, 1994.
[2] Christoph Bandt. Metric invariance of haar measure. Proceedings of the American Mathematical Society, pages 65-69, 1983.
[3] Alberto Bietti, Luca Venturi, and Joan Bruna. On the sample complexity of learning under geometric stability. Advances in Neural Information Processing Systems, 34, 2021.
[4] Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13478, 2021.
[5] Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geometric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine, 34(4):18-42, 2017.
[6] Joan Bruna. Scattering representations for recognition. PhD thesis, Ecole Polytechnique X. 2013.
[7] Joan Bruna and Stéphane Mallat. Invariant scattering convolution networks. IEEE transactions on pattern analysis and machine intelligence, 35(3):1872-1886, 2013.
[8] Steven Brunton, Bernd R. Noack, and Petros Koumoutsakos. Machine learning for fluid mechanics. Annual Review of Fluid Mechanics, 52(1):477-508, 2020.
[9] Craig Calcaterra and Axel Boldt. Lipschitz flow-box theorem. Journal of mathematical analysis and applications, 338(2): 11001111.11111111111111111111111 2008
[10] Zhi Chen, Yigian Shi, and Bin Xu. The riemannian manifolds with boundary and large symmetry. Chinese Annals of Mathematics, Series B, 3133)344--300 2010.
[11] Taco Cohen, Maurice Weiler, Berkay Kicanaoglu, and Max Welling. Gauge equivariant convolutional networks and the icosahedral cnn. In International conference on Machine learning, pages 1321-1330. PMLR, 2019.
[12] Taco Cohen and Max Welling. Learning the irreducible representations of commutative lie groups. International Conference on Machine Learning, pages 1755-1763. PMLR, 2014.
[13] Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference machine learning, pages 2990-2999. PMLR, 2016.
[14] George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems, 2(4):303-314, 1989.
[15] Pedro Boavida de Brito and Michael Weiss. Manifold calculus and homotopy sheaves. arXiv preprint arXiv: /202. 1305, 2012.
[16] Michael Eickenberg, Erwan Allys, Azadeh Moradinezhad Dizgah, Pablo Lemos, Elena Mas- Muntazir Abidi, ChangHoon Hahn, Sultan Hassan, Bruno Regaldo-Saine Saint Blancard, Shirley Ho, et al. Wavelet moments for cosmological parameter estimation. arXiv preprint arXiv:2204.0766 076466 2022.
[17] Stéphane Gonnord and Nicolas Tosel. Calcul différentiel: thèmes d' a'anllyse pour l'agrégation. Ellipses, 1998.
[18] Sigmundur Gudmundsson. An introduction to riemannian geometry. Lecture Notes version, pages 1-235, 2004.
[19] Nicolas Keriven and Gabriel Peyré. Universal invariant and equivariant graph neural networks. Advances in Neural Information Processing Systems, 32, 2019.
10

---

[20] Risi Kondor and Shubhendu Trivedi. On the generalization of equivariance and convolution in neural networks to the action of compact groups. In International Conference on Machine Learning, pages 2747-2755. PMLR, 2018.
[21] Wataru Kumagai and Akiyoshi Sannai. Universal approximation theorem for equivariant maps by group cnns. arXiv preprint urXiv:2012. 38822, 2020.
[22] John Lott and Cédric Villani. Ricci curvature for metric-measure spaces via optimal transport. Annals of Mathematics, pages 903-991, 2009.
[23] Stéphane Mallat. Group invariant scattering. Communications on Pure and Applied Mathematics, 65(10):1331-3343 2012.
[24] Stéphane Mallat. Understanding deep convolutional networks. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 374(2065):0050020, 2016.
[25] Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Learning with invariances in random features and kernel models. In Conference on Learning Theory, pages 3351-3418. PMLR, 2021.
[26] Peter W Michor and Cornelaa Vizman. n--rannitivvtyy of certain diffeomorphism groups. arXiv preprint dg- (i/ggu94400005, 1994.
[27] Sumner B Myers and Norman Earl Steenrod. The group of isometries of a riemannian manifold. Annals of Mathematics, pages 400-416, 1939.
[28] Edouard Oyallon. Analyzing and introducing structures in deep convolutional neural networks. PhD thesis, Paris Sciences et Lettres, 2017.
[29] Richard S Palais. Extending diffeomorphisms. Proceedings of the American Mathematical Society, 11(2):274-277, 1960.
[30] Jialun Ping, Fan Wang, and Jin-Quan Chen. Group representation theory for physicists. World Scientific Publishing Company, 2002.
[31] M. Raissi, P. Perdikaris, and G.E. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019.
[32] Mitsuo Sugiura. Unitary representations and harmonic analysis: an introduction. Elsevier, 1990.
[33] Bill Watson, Manifold maps commuting with the laplacian. Journal of Differential Geometry, 8(1):85-94, 1973.
[34] Maurice Weiler, Patrick Forré, Erik Verlinde, and Max Welling. Coordinate independent convolutional networks-isometry and gauge equivariant convolutions on riemannian manifolds. arXiv preprint arXiv:2106..6620, 2021.
[35] Dmitry Yarotsky. Universal approximations of invariant maps by neural networks. Constructive Approximation, 55(1):407-474, 2022.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
11

---

2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [N/A]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [N/A]
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [N/A]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
12


Reviews:
[{"review_id": "rWsbTmKpxG", "reviewer": "Reviewer_3uBu", "summary": "This paper studies operators mapping vector and scalar fields defined over a manifold. The author demonstrates that the non-linear operators that act on vector fields and commute with the group of diffeomorphisms are scalar multiplications, which implies that $Diff(\\mathcal{M})$ is too rich to obtain non-trivial operators. In the case of vector fields $L^{p}_{w}(\\mathcal{M}, T, \\mathcal{M})$, the author demonstrates that these operators are the scalar multiplication.", "questions": "Since I am not an expert in geometric deep learning, I have no more questions about this paper.", "limitations": "The limitations are not explicitly discussed in this paper.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "As for the strengths of the manuscript, it is well-written and well-motivated. The statements in this paper are well-supported by proof. As for the weaknesses, I prefer to some experiments to verify the proposed the theorems, or provide some examples to show the usage of the proposed methods in the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "TjPeasSmT2", "reviewer": "Reviewer_TxKA", "summary": "This paper contributes two main proofs that support the practice of using non linear activation functions in conjunction with linear operations in geometric deep learning. They also prove that Diff(M) is too rich and that there are no universal class of non-linear operators to motivate the design of Neural Networks over symmetries of M.", "questions": "One thing I suggest to the authors to help with the clarity is to define more of their notation. Much of the notation is not defined in the main paper despite having an entire subsection dedicated to it. This could be improved as there is plenty of space in the paper.\n\n", "limitations": "The authors discussed some of the unsolved questions that remain.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The paper addresses important issues of symmetries in machine learning. It also proves that no universal class of nonlinear operators exist that can handle all possible symmetries in a manifold. This is important to know when designing neural networks. Thus the significance, quality, and originality of the paper are good.\n\nWeakness: The paper is very dense mathematically and difficult to read for someone who is not embedded in the literature on this topic. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "51LnYE7Phe", "reviewer": "Reviewer_YVG6", "summary": "This paper theoretically studies the operators on a manifold $\\mathcal{M}$ that commute with the group of diffeomorphisms $\\text{Diff}(\\mathcal{M})$. Specifically, the author proves that in the case of a scalar fields, only point-wise non-linearities are the corresponding operators; and in the case of vector fields, those operators corresponds to the scalar multiplication. These result indicates that there is no universal class of non-linear operators (e.g., neural networks) that commute with any group of symmetries of $\\mathcal{M}$", "questions": "All the suggestions and questions are presented in the strength and weaknesses section.", "limitations": "Yes", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- This paper, especially the theory and its corresponding proofs, is well-written and easy to follow.\n- The proved theory appears to be novel and solid, especially the result in the context of neural networks. I think this is a very nice contribution to the geometric deep learning theory.\n\nWeaknesses and comments:\n- Overall, I don't have any criticism on the theories, my only question is that since one of the result is that there is no universal class of non-linear operators that commute with 'any' group of symmetries, I wonder what can we say about the operators if we reduce the symmetries to some specific symmetry group? In this case, would that be possible to find some particular non-linear operators that commute with such symmetries? I think it would be great if there is a discussion on this.  \n\nMinor comments:\n- The operator $M$ (not the manifold $\\mathcal{M}$) is mentioned in line 16 but it is defined later in line 64. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept