Clifford Neural Layers for PDE Modeling
Authors: Johannes Brandstetter
Rianne van den Berg
Max Welling
Jayesh K Gupta
Venue: ICLR 2023 poster (2023)
Keywords: Geometric Deep Learning
PDE modeling
multivector fields
Clifford algebra
Clifford convolution
Clifford Fourier transform

Abstract: Partial differential equations (PDEs) see widespread use in sciences and engineering to describe simulation of physical processes as scalar and vector fields interacting and coevolving over time. Due to the computationally expensive nature of their standard solution methods, neural PDE surrogates have become an active research topic to accelerate these simulations. However, current methods do not explicitly take into account the relationship between different fields and their internal components, which are often correlated. Viewing the time evolution of such correlated fields through the lens of multivector fields allows us to overcome these limitations. Multivector fields consist of scalar, vector, as well as higher-order components, such as bivectors and trivectors. Their algebraic properties, such as multiplication, addition and other arithmetic operations can be described by Clifford algebras. To our knowledge, this paper presents the first usage of such multivector representations together with Clifford convolutions and Clifford Fourier transforms in the context of deep learning. The resulting Clifford neural layers are universally applicable and will find direct use in the areas of fluid dynamics, weather forecasting, and the modeling of physical systems in general. We empirically evaluate the benefit of Clifford neural layers by replacing convolution and Fourier operations in common neural PDE surrogates by their Clifford counterparts on 2D Navier-Stokes and weather modeling tasks, as well as 3D Maxwell equations. For similar parameter count, Clifford neural layers consistently improve generalization capabilities of the tested neural PDE surrogates.

Published as a conference paper at ICLR 2023
CLIFFORD NEURAL LAYERS FOR PDE MODELING
Johannes Brandstetter
Rianne van den Berg
Microsoft Research A14Science johannesbemiccoooff... com
Microsoft Research A14Science
rvandenbeerjemlccooofff com
Max Welling
Jayesh K. Gupta
Microsoft Research AI4Science ma xwe nngiminmiimmmm cro.oft..oo...ooom.ooooomooooo.. com
Microsoft Autonomous Systems and Robotics Research jayesh. gupta ft com
ABSTRACT
Partial differential equations (PDEs) see widespread use in sciences and engi- neering to describe simulation of physical processes scalar and vector fields interacting and coevolving over time. Due to the computationally expensive na- ture of their standard solution methods, neural PDE surrogates have become active research topic to accelerate these simulations. However, current methods do not explicitly take into account the relationship between different fields and their internal components, which are often correlated. Viewing the time evolu- tion of such correlated fields through the lens of multivector fields allows overcome these limitations. Multivector fields consist of scalar, vector, as well higher-order components, such bivectors and trivectors. Their algebraic properties, such as multiplication, addition and other arithmetic operations can be described by Clifford algebras. To our knowledge, this paper presents the first usage of such multivector representations together with Clifford convolutions and Clifford Fourier transforms in the context of deep learning. The resulting Clifford neural layers are universally applicable and will find direct use in the areas of fluid dynamics, weather forecasting, and the modeling physical sys- tems in general. We empirically evaluate the benefit of Clifford neural layers by replacing convolution and Fourier operations in common neural PDE surro- gates by their Clifford counterparts 2D Navier-Stokes and weather modeling tasks, as well as 3D Maxwell equations. For similar parameter count, Clifford neural layers consistently improve generalization capabilities of the tested neu- ral PDE surrogates. Source code for our PyTorch implementation is available at https: http://mmcooooftoooooooo www.ssof...ggttb..oooo wtiithub.i//cll.. hww.bb/odaafff.o.yrrr.....
INTRODUCTION
Most scientific phenomena are described by the evolution and interaction of physical quantities over space and time. The concept of fields is one widely used construct to continuously parameterize these quantities
chosen coordinates (McMullin, 2002). Prominent examples include (i) fluid mechanics, which has applications in domains ranging from mechanical and civil engineering, to geophysics and meteorology, and (ii) electromagnetism, which provides mathematical models for electric, optical, or radio technologies. The underlying equations of these examples are famously de- scribed in various forms of the Navier-Stokes equations and Maxwell's equations. For the majority of these equations, solutions are analytically intractable, and obtaining accurate predictions neces- sitates falling back on numerical approximation schemes often with prohibitive computation costs. Deep learning's success in various fields has led to a surge of interest in scientific applications, especially at augmenting and replacing numerical solving schemes in fluid dynamics with neural networks (Li et al., 2020; Kochkov et al., 2021; 1; Lu et al., 2021; Rasp & Thuerey, 2021; Keisler, 2022: Weyn et al., 2020; Sønderby et al., 2020; Pathak et al., 2022).
motivating example to ground our discussion, two different kinds of fields emerge: scalar fields such as temperature or humidity, and vector fields such as wind velocity or pressure gradients. Current deep learning based approaches treat different vector field
Taking weather simulations

---

Published as a conference paper at ICLR 2023
(a) Scalar pressure field
(b) Vector wind velocity field
Figure Fields of the Earth's shallow water model. Vector components of the wind velocities (right) are strongly related, i.e. they form vector field. Additionally, the wind vector field and the scalar pressure field (left) are related since the gradient of the pressure field causes air movement and subsequently influences the wind components. We therefore aim to describe scalar and vector field as one multivector field, which models the dependencies correctly.
components the same as scalar fields, and stack all scalar fields along the channel dimension, thereby omitting the geometric relations between different components, both within vector fields as well as between individual vector and scalar fields. This practice leaves out important inductive bias infor- mation present in the input data. For example, wind velocities in the x- and y- directions are strongly related, i.e. they form a vector field. Additionally, the wind vector field and the scalar pressure field are related since the gradient of the pressure field causes air movement and subsequently influences the wind components. In this work, we therefore build neural PDE surrogates which model the relation between different fields (e-g. wind and pressure field) and field components (e.g. x- and y- component of the wind velocities). Figure I shows example of wind vector field as per the Earth's ' shallow water model in two dimensions, and the related scalar pressure field.
Clifford algebras (Suter, 2003; Hestenes, 2003; 2012; Dorst et al., 2010; Renaud, 2020) are the core intersection of geometry and algebra, introduced to simplify spatial and geometrical relations between many mathematical concepts. For example, Clifford algebras naturally unify real numbers, vectors, complex numbers, quaternions, exterior algebras, and many more. Most notably, in contrast to standard vector analysis where primitives scalars and vectors, Clifford algebras have additional spatial primitives for representing plane and volume segments. An expository example is the cross- product of two vectors in 3 dimensions, which naturally translates these two vectors. The cross product is often represented as a vector due to its 3 independent components, but the cross product has a sign flip under reflection that a true vector does not. In Clifford algebras, different spatial primitives can be summarized into objects called multivectors, as illustrated in Figure 2. this work, we replace operations over feature fields deep learning architectures by their Clifford algebra counterparts, which operate multivector feature fields. Operations on, and mappings between multivectors are defined by Clifford algebras. For example, will endow convolutional kernel with multivector components, such that it can convolve over multivector feature maps.
plane segment spanned by
Scalar (1)
Vectors fea. es. ta)
Bivectors
Trivector [eseves)
Multivector
(eies, euta, (sta)
6901
610y9g
Figure 2: Multivector components of Clifford algebras.
BACKGROUND: CLIFFORD ALGEBRAS
We introduce important mathematical concepts and discuss three Clifford algebras, Cl2.0(RR), (R), C1a,0 (R). which we later use for the layers introduced in Section A more detailed introduction well as connections to complex numbers and quaternions is given in Appendix A.
2

---

Published as a conference paper at ICLR 2023
Clifford algebras. Consider the vector space R" with standard Euclidean product (.,-), where n p 9, and and are non-negative integers. A real Clifford algebra (l)) is an associative algebra' generated by p p+g orthonormal basis elements ei...., Ep+y of the generating vector space IR, such that the following quadratic relations hold:
e, +1 for isish ej= - for < <jsptop e -ye for i $
(1)
The pair (p.q) q) is called the signature and defines a Clifford algebra (I)), together with the basis elements that span the vector space GP+o of Clp.q Vector spaces of Clifford algebras have scalar elements and vector elements, but can also have elements consisting of multiple basis clements of the generating vector space I", which can be interpreted plane and volume segments. Exemplary oow-dimensional Clifford algebras are: (i) (IR.)(IR)) which a one-dimensional algebra that is spanned by the basis element {1} and is therefore isomorphic to R, the field of real numbers; (ii) C (R) which is a two-dimensional algebra with vector space G spanned by {1, ei where the basis vector ej squares to -1, and is therefore isomorphic to C, the field of complex numbers; (iii) C (R) which is a 4-dimensional algebra with vector space G2 spanned by {1, é1, e2, el e1e2), where e1, e2, e2 all square -1 and anti-commute. Thus, 2 (R) is isomorphic the quaternions IH.
Grade, dual, geometric product. The grade of a Clifford algebra basis element is the dimension of the subspace represents. For example, the basis elements { e1, E2; eje2] of the vector space G2 of the Clifford algebra Cl2,,((R) have the grades {0, 1, Using the concept of grades, we can divide Clifford algebras into linear subspaces made up of elements of each grade. The grade subspace of smallest dimension is Mg. the subspace of all scalars (elements with 0 basis vectors of the generating vector space). Elements of M1 are called vectors, elements of M2 are bivectors, and so on. In general, a vector space G of a Clifford algebra Clp., (RR) can be written as the direct sum of all of these subspaces: GP-4 Mg ( M1 @ ... @ Mptq The elements of Clifford algebra are called multivectors, containing elements of subspaces, i.c. scalars, vectors, bivectors, vevectss. The basis element with the highest grade is called the pseudoscalar". which in g² corresponds to the bivector C2, and in R3 to the trivector 616233-
The dual a" of a multivector a is defined as a = aiptg where ipto represents the respective pseudoscalar of the Clifford algebra. This definition allows us to relate different multivectors to each other, which is a useful property when defining Clifford Fourier transforms. For example, for Clifford algebras in I the dual of the scalar is the bivector, and in R3, the dual of the sealar is the trivector. Finally, the geometric product is a bilinear operation on multivectors. For arbitrary multivectors a,
GPT9, and scalar X, the geometric product has the following properties: (i) closure, i.e. ab E GP+9, (ii) associativity, i.e. (ab)c a(be); (iii) commutative scalar multiplica- tion, i.e. Xa = ai; (iv) distributive over addition, i.e. ab + e) = ab + ac. The geometric product is in general non-commutative, i.c. ab of ba. Note that Equation I describe the geometric product specifically between basis elements of the generating vector space.
Clifford algebras (R2 and (la) The 4-dimensional vector spaces of these Clifford algebras have the basis vectors (1, 62, e1e2] where 61, e2 square to +1 for Cl2, and to - for Clo,2 (R). For (B2)) the geometric product of two multivectors and b- bo + ble1 + b2é2 + bige1e2 is given by:
oo taee + a2ez 012e102
ab = =aabo + aιdı + 02b2 - a12b12)1 11 + (aobi + asby - 0zb12 + a12b2)e1
+ (anb2 arb12 azby De2d))e2 + (a0b12 agbı azhı + a12b0)e1e2,
which can be derived by collecting terms that multiply the same basis clements, Appendix A. A vector x = (x1, E R² with standard Eu- clidean product (..-) can be related to æ1 + 2262 R² c G Clifford multiplication of two vectors
I, y 2 G² yields the geometric product ay:
tt A Aa
A
ry ( + x222)(11e1 + y222)
es A
2 2ivie + 22 + x1326162 + 22y1e2e1
Figure 3: Antisymmetry of bivector exterior (3)
(z. v\ + FrAg.
(wedge) product.
Operations of addition and multiplication are associative.
In contrast scalars, pseudoscalars change sign under reflections.
3

---

Published as a conference paper at ICLR 2023
where A is the exterior wedge product. The asymmetric quantity A y = is associated with the bivector, which can be interpreted as an oriented plane segment as shown in Figure 3. A unit bivector 22, spanned by the (orthonormal) basis vectors e and E2 is determined by the product:
i2 = 6162 = (61, 62) + te A2 - 62 e1 = e261
which if squared yields 03 1. Thus, in represents geometric V-I. From Equation 4, it follows that eg eid2 -2ge1 and e₁ igeg = -E222 Using the pseudoscalar ig, the dual of scalar is a bivector and the dual of vector is again a vector. The dual pairs of the base vectors are 1 + 6182 and e1 + eg. For Cl2,0(1R) (RR), these dual pairs allow us to write an arbitrary multivector as
oo + a1e1 + a2e2 + 012e1e2 1 (ao + (1212) te1 (a1 + agi2)
(5)
spinor
vector part
which can be regarded two complex-valued parts: the spinor' part, which commutes with the base element 1, i.e. li2 = 12 and the vector part, which anti-commutes with the respective base element e1. i.c. egio = eιe102 -eiege1 -1 For C1(O, ())), the vector part changes to e1 (01 - -2222). This decomposition will be the basis for Clifford Fourier transforms.
The Clifford algebra 2 isomorphic the quaternions Hi, which are an extension of com- plex numbers and are commonly written in the literature as a + bi + cj + dk. Quaternions also form a 4-dimensional algebra spanned by {1, f, k). where i, J. k all square to -1. The algebra isomor- phism 2 is easy to verify since e 6162 all square to -1 and anti-commute. The basis element is called the scalar and the basis elements i, J. & are called the vector part of a quaternion. Quaternions have practical uses in applied mathematics, particularly for expressing rotations, which we will use to define the rotational Clifford convolution layer in Section 3.
Clifford algebra (R)) The 8-dimensional vector space G3 of the Clifford algebra Cls. (R) has the basis vectors {1, el, e2, 63, é1e2, e3é1, 6263, e16263). i.e. it consists of one scalar, three vectors (er, e2, e3), three bivectors {erez, (erez,ege,, egen) and one trivector elegeg. The trivector is the pseudoscalar is of the algebra. The geometric product of two multivectors is defined anal- ogously to the geometric product of Cl2, see Appendix A. The dual pairs of Cl3.o are: 1 + Eie2e3 = i3. é1 + E2C3. E2 + EgE1, and e3 ++ e1 An intriguing example of the duality of the multivectors of Cl3,o( emerges when writing the expression of the electromagnetic field F in terms of an electric vector field E and magnetic vector field B, such that F E Bis where = + E, + Ege3 and = BBE +B +Be3. this way the electromagnetic field de- composes into electric vector and magnetic bivector parts via the pseudoscalar is (Hestenes, 2003). For example, for the base component B_e1 of B it holds that B, = Bgeie1e2c; = Bgezes which is bivector and the dual to the base component Ege1 of E. Consequently, the multivec- tor representing F consists of three vectors (the electric field components) and three bivectors (the magnetic field components multiplied by iz). This viewpoint gives Clifford neural layers natural advantage over their default counterparts as we will see in Section
3 CLIFFORD NEURAL LAYERS
Here, we introduce 2D Clifford convolution and 2D Clifford Fourier transform layers. Appendix B contains extensions to 3 dimensions. In Appendices B, D, related literature is discussed, most no- tably complex (Bassey et al., 2021) and quaternion neural networks (Parcollet et al., 2020).
Clifford CNN layers. Regular convolutional neural network (CNN) layers take input feature maps/ f 2² + Re and convolve" them with a set of Cout filters {w" Yeu with w" 2 + R
Gn Σ ((()), =)) = Σ VERZ
x))
(6)
DEZZ
"Spinors are elements of a complex vector space that can be associated with Euclidean space. vectors. spinors transform their negative when rotated 360
Unlike
"The bivector eses has negative orientation. 'In deep learning, convolution operation in the forward pass is implemented cross-correlation.
4

---

Published as a conference paper at ICLR 2023
inner product of input feature maps with the corresponding filters at every point y E z². By applying Cout filters, the output feature maps can be interpreted as Cour dimensional feature vectors at every point y Z². We now extend CNN layers such that the element-wise product of scalars f (1())o (y- x) is replaced by the geometric product of multivector inputs and multivector filters f (y)) wurh (y - x), where the chosen signature of CI is reflected in the geometric product. We replace the feature maps f 2²² + R by multivector feature maps 22 (G and convolve them with set of Cout multivector filters {w" Yuur 22 (G
which can be interpreted
[ * w0
(x)
(w (y-
WE
pwii GP
Note that each geometric product, indexed by [I,.., Cour and j {l,..., Gn now results in a new multivector rather than scalar. Hence, the output of layer is grid of Cout multivectors. We can e.g. implement C2(2, (R) Clifford CNN layer using Equation 2 where (dg, by, be, b12) + fuo w/ wh² w I corre- spond to 4 different kernels representing one 2D multivector kernel, i.e. 4 differ- ent convolution layers, and {ao. a1. 02. } - fgg f.. f2. fiz) correspond to the scalar, vector and bivector parts of the input multivector field. The channels of the different layers represent different stacks of scalars, vectors, and bivectors. Analogously, we can implement a C1(3, 0) (R) CNN layer using Equation 42 in Appendix B. A schematic sketch of a Clifford convolution layer is shown in Figure 4.
Multivector input fields
Multivector kernels
Multivector output fields
f(x
Geometric
product
Figure 4: Sketch of Clifford convolution. Multivector
input fields are convolved with multivector kernels.
Rotational Clifford CNN layers. Here we introduce an alternative parameterization to the Clifford CNN layer introduced in Equation 7 by using the isomorphism of the Clifford algebra 2 (R) to quaternions. We take advantage of the fact that a quaternion rotation can be realized by a matrix multiplication (Jia, 2008; Kuipers, 1999; Schwichtenberg. 2015). Using the isomorphism, we can represent the feature maps f and filters w as quaternions: f fo f f2j fák and w w0 + w + whi J + w.. R.S. We can now devise an alternative parameterization of the product between the feature map f and w.. To be more precise, we introduce a composite operation that results in a scalar quantity and a quaternion rotation, where the latter acts on the vector part of the quaternion f and only produces nonzero expansion coefficients for the vector part of the quaternion output. A quaternion rotation f (w">)) acts on the vector part (i, k) of få, and can be algebraically manipulated into a vector-matrix operation Ri f1, where R HH + Hi is built up from the elements of w (Kuipers, 1999). In other words, one can transform the vector part (i, J. k) of f E via a rotation matrix R that built from the scalar and vector part (1, J. k) of w E H. Altogether, a rotational multivector filter { 1:a _ + (G2) acts the feature map f through rotational transformation Rid (w w Whe, Wtot.. (20.11) acting on vector and bivector parts of the multivector feature map f z² + (G2)an, and an additional scalar response of the multivector filters:
R( [ Pi(
Gn
[f * wood (x)= - Σ
(1) x)) + +RAS x).
(8)
MEZZ
1=1
scalar oulpuu
where (y) (y m))] = 56 wor,o Wi B wo fl i.e., the scalar output of the geometric product of (lo,2(RR) as in Equation 34. A detailed description of the rotational multivector filters Ri I²) outlined in Appendix B. While in principle the Clifford CNN layer in Equation 7 and the rotational Clifford CNN layer in Equation 8 are equally flexible, our experiments in Section show that rotational Clifford CNN layers lead to better performance.
"Noee that the expansion coefficients for the feature map f and filters th 3 in terms of the basis elements of G² and in terms of quaternion elements i, j and is are the same.
5

---

Published as a conference paper at ICLR 2023
Clifford convolutions satisfy the property of equivariance under translation of the multivector inputs, as shown in theorem in Appendix B. Analogous to Theorem 1, translation equivariance can be derived for rotational Clifford CNN layers.
The discrete Fourier transform of an n-dimensional complex signal f(x) (f1,.-.;;0, :.n RR + C M1 x... x M, grid points is defined as:
Clifford Fourier layers.
Mi
Mn
(m)= =
Σ
Σ 17a
f(m1,..., m,)) )
2mi ma +... mahn
mi
M. In Fourier Neural Operators (FNO) (Li et al., 2020), dis- crete Fourier transforms on real-valued input fields and respective tock-transfomms implemented as Fast Fourier Transforms on real-valued inputs (RFFTs) are interleaved with a weight multiplica- tion by complex weight matrix of shape Cou for each mode, which results in complex-van -valued weight tensor of the form W E Co.x ((f x...x""")), where Fourier modes above cut-off frequen- zero. Additionally, residual connection is usually implemented as convolution layer with kernel size 1. In Figure 5a, sketch of an FNO layer is shown. For C1(2, (R). the Clifford Fourier transform (Ebling Scheuermann, 2005; Ebling, 2006; Hitzer, 2012) for multivector valued functions f ((x) RR2 G2 and vectors x, E E IR2 is defined as:
where (EI E 2., ...
cies (Si :.... Sn are
[ f(z)e-2mizane f(g) = F(F) = 2x 33
2mia( dr, Ag E TR?
(10)
provided that the integral exists. In contrast to standard Fourier transforms. f(x) and f({) represent multivector fields in the spatial and the frequency domain, respectively. Furthermore, i2 = 6162 is used in the exponent. Inserting the definition of multivector fields, rewrite Equation 10 as:
F(F((E)
1 / 2n
(f6le)+
fi2(()i))) + ei (1(=)+f(()))))
(m.2( dr
spinor part
vector part
=1 -1 (1ooo)) fra(t)ia) (E)
el (((i(o)) fawwyi) (5)
(11)
We obtain Clifford Fourier transform by applying two standard Fourier transforms the dual pairs fo fo(z) + (12(2) and fi fi(z) + fe(x)iz, which both can be treated as complex- valued signals fu. fi R²2 C. Consequently, f(x) can be understood as an element of C?. The 2D Clifford Fourier transform is the linear combination of two classical Fourier transforms. Discrete versions of Equation 11 are obtained analogously to Equation 9, see Appendix Similar FNO, multivector weight tensors W E (G2) (G x (8) x emo)) are applied, where again Fourier modes above cut-off frequencies (ETRX, :2 <max) are set to zero. In doing we point-wise modify the Clifford Fourier modes f(E) = (E) fo(E)+ + product. The Clifford Fourier modes follow naturally when combining spinor and vector parts of Equation 11. Finally, the residual connection is replaced by Clifford convolution with multivector kernel k. A schematic sketch is shown in Figure 5b. For C1(3, (B), Clifford Fourier transforms follow similar elegant construction, where we apply four separate Fourier transforms
via the geometric
fo(x) fo(2) + f123 fia())
f1 - (z) f23(2)
(12)
f2(x) = fa
fa(x) fa(a) + fi2(r)is
i.c scalar/trivector and vector/bivector components are combined into complex fields and then sub-
jected to a Fourier transform.
EXPERIMENTS
We assess Clifford neural layers for different architectures in three experimental settings: the incompressible Navier-Stokes equations, shallow water equations for weather modeling, and 3- dimensional Maxwell's equations. We replace carefully designed baseline architectures by their
The FFT of real-valued signal is Hermitianssymmetric, so the output contains only the positive frequen- cies below the Nyquist frequency for the last spatial dimension.

---

Published as a conference paper at ICLR 2023
FUG
Firee
MM SUOM IWMNIIOANNM
W
www
RYYT
(a) FNO layer
(b) Clifford FNO layer
Figure 5: Sketch of Fourier Neural Operator (FNO) and Clifford Fourier Operator (CFNO) layers.
The real valued Fast Fourier transform (RFFT) over real valued scalar input fields far is replaced by the complex Fast Fourier transform (FFT) over the complex valued dual parts v(x) and a(x) of multivector fields (z)). Pointwise multiplication in the Fourier space via complex weight tensor W is replaced by the geometrie product in the Clifford Fourier space via multivector weight tensor W. Additionally, the convolution path is replaced by Clifford convolutions with multivector kernels w.
Clifford counterparts. Baseline ResNet architectures comprise 8 residual blocks, each consisting of two convolution layers with 3 x 3 kernels, shortcut connections, group normalization (Wu & He, 2018), and GeLU activation functions (Hendrycks & Gimpel, 2016). Baseline 2-dimensional Fourier Neural Operators (FNOs) consist of 8 (4) FNO blocks, GeLU activations and no normal- ization scheme, using 16 (8) Fourier modes for the 2- and 3-dimensional equations, respectively. For Clifford networks, we change convolutions and Fourier transforms to their respective Clifford operation, and substitute normalization techniques and activation functions with Clifford counter- parts, keeping the number of parameters similar. We evaluate different training set sizes, and report losses for scalar and vector fields. All datasets share the common trait of containing multiple in- put and output fields. More precisely, one scalar and one 2-dimensional vector field in case of the Navier-Stokes and the shallow water equations, and a 3-dimensional (electric) vector field and its dual (magnetic) bivector field in case of the Maxwell's equations.
Example inputs and targets the neural PDE surrogates are shown in Figure 6. The number of input timesteps vary for different experiments. The one-step loss is the mean-squared eтror at the next timestep summed over fields. The rollout loss is the mean-squared error after applying the neural PDE surrogate 5 times, summing over fields and time dimension. More information on the implementation details of the tested architectures, loss functions, and more detailed results can be found in Appendix C.
Scalar
Vector 2
Vector g
Input
Navier-Stokes in 2D. The incompressible Navier-Stokes equations (Temam, 2001) conserve the velocity flow fields v: X + 2 where X IR2 via:
Target
du
Figure 6: Example input and target fields for the Navier-Stokes experi- ments. Input fields comprise a timestep history.
Øt V. VU VU+pVE Vp+f, V- (13)
where V Ve is the convection, i.c. the rate of change of v along v. a 2 the viscosity, i.c. the diffusion or net move- ment of v. Vp the internal pressure and f an external force, which in our case is a buoyancy force. yields mass conservation of the Navier-Stokes
An additional incompressibility constraint V equations. In addition to the velocity field, we introduce scalar field representing scalar quantity, i.e smoke, that is being transported via the velocity field. The scalar field advected by the vector field, i.e. as the vector field changes, the scalar field is transported along with it, whereas the scalar field influences the vector field only via an external force term. We call this weak coupling between vector and scalar fields. We implement the 2D Navier-Stoke equation using oF @Flow"(Ho (Holl et al., 2020), obtaining data on a grid with spatial resolution of 128 x 128 (Ar = 0.25, Ay = 0.25),
hhtpps: :/gi thub. com/ coom/uum-te ttwttttuupsssse pbs/ Phi Fl

---

Published as a conference paper at ICLR 2023
Rollout
Rollout
Rollout
Rollout
0.2
ResNet
0.2
FNO
ResNet
FNO
CResNet
CENO
CResNet
CENO
0.15
CResNeloo
0.15
CResNefet
0.1
0.1
0.5
0.05
0.05
MSE
0
MSE
One-step
One-step
One-step
One-step
0.02
0.02
-10
0.02
0.02
0.01
0.01
2
2
15600
10400
5200
10400
5200
2080
2048
1792
8996
890
448
192
Num. Train Trajectories
Num. Train Trajectories
(a) Navier-Stokes equations
(b) Shallow water equations
Figure 7: Results for ResNet based (left) and Fourier based (right) architectures on the 2-dimensional
Navier-Stokes and Shallow water experiments. One-step and rollout loss are shown.
and temporal resolution of At 1.5s. Results for one-step loss and rollout loss the test set shown in Figure 7a. For ResNet-like architectures, we observe that both CResNet and CResNetau improve upon the ResNet baseline. Additionally, the two Clifford based architectures, which we attribute to better and more stable models that do not overfit to one-step predictions so easily. Lastly, while in principle CResNet and CResNetrn based architectures are equally flexible, CResNet,o ones in general perform better than CResNet ones. For FNO and respective Clifford Fourier based (CFNO) architectures, the loss in general much lower than for ResNet based architectures. CFNO architectures improve upon FNO architectures for all dataset sizes, and for one-step as well as rollout losses.
observe that rollout losses also lower for
Shallow water equations. This set of coupled equations (Vreugdenhil, 1994) can be derived from integrating the incompressible Wavier-Stoker equations, cases where the horizontal length scale is much larger than the vertical length scale. As such, the equations model a thin layer of fluid of constant density in hydrostatic balance, bounded from below by the bottom topography and from above by free surface via 3 coupled PDEs, describing the velocity in direction, the velocity in the y- direction, and the scalar pressure field. The shallow water equations can be therefore be used simplified weather model, done in this work and exemplified in Figure 1. The relation between vector and scalar components is relatively strong (strong coupling due to the 3-coupled PDEs). We obtain data for the 2D shallow water equations on grid with spatial resolution of 192 x 96 (Ar = 1.875", Ay = 3.75%), and temporal resolution of At 6h. We observe similar results than for the Navier-Stokes experiments. For low number of trajectories, ResNet architectures seem to lack expressiveness, where arguably some data smoothing is learned first. Thus, ResNets need significantly more trajectories compared to (C)FNO architectures obtain reasonable loss values, which seems to go hand in hand with Clifford layers gaining advantage. In general, performance dif- ferences between baseline and Clifford architectures are even more pronounced, which we attribute to the stronger coupling of the scalar and the vector fields.
Maxwell's equations in matter in 3D. In isotropic media, Maxwell's equations (Griffiths, 2005) propagate solutions of the displacement field D, which is related to the electrical field via D E, where 6y is the permittivity of free space and 6, is the permittivity of the medium, and the magnetization field H, which is related to the magnetic field B via H Molt where bi is the permeability of free space and μ, is the permeability of the medium. The electromagnetic field F has the intriguing property that the electric field F. and the magnetic field B are dual pairs, thus
8

---

Published as a conference paper at ICLR 2023
F= E+ Big, Le. strong coupling between the electric field and its dual (bivector) magnetic field. This duality also holds for and H. Concretely, the fields interest are the vector- valued D-field (Da, Dy. D2) and the vector- vector-valued H-field (H y: H2). We obtain data for the 3D Maxwell's equations grid with spatial resolution of 32 x 32 32 (Az Ay A 5 10 m)), and temporal resolution of At s0 We randomly place 18 different light sources outside cube which emit light with different amplitude and different phase shifts, causing the resulting D and H fields to interfere. The wavelength of the emitted light is 10 "m.
Rollout
We test FNO based architectures and respective Clifford counterparts (CFNO). Due to the vector-bivector character of electric and magnetic field components, Maxwell's equations stress-test the inductive bias advantages of Clifford base architectures.
-10-2
FNO
ideal playground to
1.5
CPNO
Results for one-step loss and rollout loss on the test set are shown in Figure 8. CFNO architectures improve upon FNO architectures, espe- cially for low numbers of trajectories. Results demonstrate the much stronger inductive bias of Clifford based 3-dimensional Fourier lay- ers, and their general applicability to 3-dimensional problems, which are structurally even more interesting than 2-dimensional ones.
One-step
MSC
CONCLUSION
We introduced Clifford neural layers that handle the various scalar (e.g. charge density), vector (e.g. electric field), bivector (magnetic field) and higher order fields proper geometric objects organized as multivectors. This geometric algebra perspective allowed us to naturally generalize convolution and Fourier transformations their Clifford counterparts, providing an elegant rule to design new neural network layers. The multivector viewpoint denotes an inductive bias advantage, leading to better representation of the relationship be- tween fields and their individual components, which prominently demonstrated by the fact that our Clifford layers significantly outper- formed equivalent standard neural PDE surrogates.
6400
3200
1280
640
Num. Train Trajectories
Figure 8: Results for Fourier based architectures on Maxwell equation's s.
Limitations. One limitation is the current speed of Fast Fourier Transform (FFT) operations on ma- chine learning accelerators like GPUs. While an active area of research, current available versions of CuFFT" kernels wrapped in PyTorch (Paszke et al., 2019) are not yet as heavily optimized! especially for the gradient pass. In contrast to FNO layers, which operate on real-valued signals, complex- voluuex-valued FFT operations where the backward pass is approxi-
Clifford Fourier layers mately twice as slow. For similar parameter counts, inference times of FNO and CFNO networks similar. Similar to Grassucci al. (2021) who investigated the speed of geometric convolution layers, we found that Clifford convolutions are more parameter efficient since they share parameters among filters, with the downside that the net number of operations larger, resulting in increased training times by a factor of about 2. Finally, from a PDE point of view, the presented approaches obtain PDE surrogates are limited since the neural networks have be retrained for different equation parameters or e.g. different At.
Future work. Besides modeling of PDEs, weather, and fluid dynamics, we see potential applica- tions of Clifford layers for e.g. MRI or radar data, and for neural implicit representations (Xie et al., 2022; Rella et al., 2022). Extensions towards graph networks and attention based models will be useful to explore. Furthermore, custom multivector GPU kernels can overcome many of the speed issues as the compute density of Clifford operations is much higher which is better for hardware accelerators (Hoffmann et al., 2020). The use of just-in-time compiled language with better ar- ray abstractions like Julia (Bezanson et al., 2017) could significantly simplify the interface. Finally, combining the ideas of multivector modeling together with various approaches (Raissi et al., 2019; Lutter et al., 2018; Gupta et al., 2019; Cranmer et al., 2020; Zubov al., 2021) is an attractive next step.
informed neural network
Pttpps: :ttpps::/deeee devel oper pererrvvvie ivviii. ccom/cuffe Foro alternative efficient GPU-aceelerated multidimensional FFT libraries see e.g. https://githhs. coo/ m/ VKFFT

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY AND ETHICAL STATEMENT
Reproducibility statement. We have included error bars, and ablation studies wherever we found necessary and appropriate. We have described our architectures in Section and provided fur- ther implementation details in Appendix Section C. We have further include pseudocode for the newly proposed layers in Appendix Section B.6. We open-sourced our PyTorch implementation at https: / mi crosoft. gi thub. htwcliffoordlaeeeeeee..oo for others to use. We aim to de- velop this codebase further in the future.
Ethical statement. Neural PDE surrogates will play an important role in modeling many natural phenomena, and thus developing them further might enable us to achieve shortcuts or alternatives for computationally expensive simulations. For example, if used as such, PDE surrogates will po- tentially help to advance different fields of research, especially in the natural sciences. Examples related this paper are fluid dynamics or weather modeling. Therefore, PDE surrogates might po- tentially be directly or indirectly related to reducing the carbon footprint. On the downside, relying on simulations always requires rigorous cross-checks and monitoring, especially when we "learn to simulate".
REFERENCES
Azzam Alfarraj and Guo-Wei Wei. Geometric algebra generation of molecular surfaces. Journal of the Royal Society Interface, 19(189):0220111, 17, 2022.
Troy Arcomano, Istvan Szunyogh, Jaideep Pathak, Alexander Wikner, Brian R Hunt, and Edward Ott. A machine learning- based global atmospheric forecast model. Geophysical Research Letters, 47(9):2000GL088777, 2020.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint 1rX1vv160..06450, 2016.
Yohai Bar-Sinai, Stephan Hoyer, Jason Hickey, and Michael P Brenner. Learning data-driven dis- cretizations for partial differential equations. Proceedings of the National Academy of Sciences, 116(31):15344- 1534--5344444999 15349, 2019.
Joshua Bassey, Lijun Qian, and Xianfang Li. survey of complex-valued neural networks. arXiv preprint arXiv:2 2021.
Jeff Bezanson, Alan Edelman, Stefan Karpinski, and Viral Shah. Julia: fresh approach to nu- merical computing. SIAM review, 59(1):65-98, 2017. URL 141000671.
Saakaar Bhatnagar, Yaser Afshar, Shaowu Pan, Karthik Duraisamy, and Shailendra Kaushik. Predic- tion of aerodynamic flow fields using convolutional neural networks. Computaaiooal Mechanics, 64(2):525 -0:555555555100 2019.
Fred Brackx, Eckhard Hitzer, and Stephen Sangwine. History of quaternion and clifford-fourier transforms and wavelets. Quaternion and Clifford Fourier transforms and wavelets, 27:XI- XXVII, 2013.
Johannes Brandstetter, Rob Hesselink, Elise van der Pol, Erik Bekkers, and Max Welling, Ge- ometric and physical quantities improve (3) equivariant message passing. arXiv preprint 10.00:21 2021.
Johannes Brandstetter, Max Welling, and Daniel E Worrall. Lie point symmetry data augmentation for neural pde solvers. arXiv preprint arXiv:2202.07643, 2022a.
Johannes Brandstetter, Daniel Worrall, and Max Welling. Message passing neural PDE solvers. arXiv preprint 0rX3v:2202.03376, 2022b.
Susanne Brenner, L Ridgway Scott, and L Ridgway Scott. The mathematical theory of finite element methods, volume 3. Springer, 2008.
10

---

Published as a conference paper at ICLR 2023
Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geomet- ric deep learning: Going beyond euclidean data. IEEE Signal Processing Magazine, 34(4):18 42, 2017.
Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.1347, 2021.
Sven Buchholz. A theory of neural computation with clifford algebras. 2005.
Sven Buchholz and Gerald Sommer. Introduction neural computation in clifford algebra. In Geometric computing with Clifford algebras, pp. 291-314. Springer, 2001.
Shuhao Cao. Choose transformer: Fourier galerkin. Advances in neural information processing
systems, 21849244-24400 2021.
Gengxiang Chen, Yingguang Li, Qinglu Meng, Jing Zhou, Xiaozhong Hao, et al. Residual fourier neural operator for thermochemical curing of composites. arXiv preprint arXiv:21 10266, 2021.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International Confer- ence on Machine Learning (ICML), pp. 2990-2999. PMLR, 2016a.
Taco S Cohen and Max Welling. Steerable cnns. arXiv preprint arXiv:
2016b.
Taco S Cohen, Mario Geiger, and Maurice Weiler. A general theory of equivariant CNNs homo- geneous spaces. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
James W Cooley and John W Tukey, An algorithm for the machine calculation of complex fourier series. Mathematics of computation, 19990))297-301, 1965.
Miles Cranmer, Sam Greydanus, Stephan Hoyer, Peter Battaglia, David Spergel, and Shirley Ho. Lagrangian neural networks, arXiv preprint arXiv:2003. 10Xx:20003.04630, 2020.
Lco Dorst, Daniel Fontijne, and Stephen Mann. Geometric algebra computer science: an object- oriented approach geometry. Elsevier, 2010.
Ebling and G. Scheuermann. Clifford Fourier transform on vector fields. IEEE Transactions on Visualization and Computer Graphics, 11(4):469 4799, 2005. doi: 10. 10.1109/TVCG.2055.50
Julia Ebling. Visualization and analysis of flow fields based clifford convolution. 2006.
Julia Ebling and Gerik Scheuermann. Clifford convolution and pattern matching on vector fields. In
IEEE Visualization, 2003. VIS 2003., pp. 193 200. IEEE, 2003.
Todd A Ell. Quaternion-fourier transforms for analysis of two-dimensional linear time-invariant partial differential systems. In Proceedings of 32nd IEEE Conference on Decision and Control, pp. 1830-1841. IEBE, 1993.
Todd Ell and Stephen Sangwine. Hypercomplex fourier transforms of color images. IEEE Transactions image processing. 16(1):22-35, 2006.
Todd A Ell, Nicolas Le Bihan, and Stephen Sangwine. Quaternion Fourier transforms for signal and image processing. John Wiley & Sons, 2014.
Todd Anthony Ell. Hypercomplex spectral transformations. PhD thesis, University of Minnesota, 1992.
Thomas Frerix, Dmitrii Kochkov, Jamie Smith, Daniel Cremers. Michael Brenner, and Stephan Hoyer. Variational data assimilation with a learned inverse observation operator. In International Conference on Machine Learning (ICML), pp. 3449-3458. PMLR, 2021.
Kunihiko Fukushima and Sei Miyake. Neocognitron: A self-organizing neural network model for a mechanism of visual pattern recognition. In Competition and cooperation in neural nets, pp- 267-285. Springer, 1982.

---

Journal of Published as a conference paper at ICLR 2023
Victor Garcia Satorras, Zeynep Akata, and Max Welling. Combining generative and discriminative models for hybrid inference. In H. Wallach, H. Larochelle, A. Beygelzimer, d' A'Alcéé Buc, E. Fox, and R. Garnett Advances in Neural Information Processing Systems (NeurIPS), pp- 13802-13812. Curran Associates, Inc., 2019.
Chase Gaudet and Anthony S Maida. Deep quaternion networks. In International Joint Conference on Neural Networks (IJCNN), pp. 1-8, IEEE, 2018.
Mario Geiger and Tess Smidt. e3nn: Euclidean neural networks. arXiv preprint arXiv:2207.09453 2022.
Nicholas Geneva and Nicholas Zabaras. Modeling the dynamics of pde systems with physics- constrained deep auto-regressive networks, Journal of Computational Physics, 403:109056, 2020.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In International Conference on Artificial Intelligence and Statistics (AISTATS), pp- 249-256. JMLR Workshop and Conference Proceedings, 2010.
Elconora Grassucci, Aston Zhang, and Danilo Comminiello. Lightweight convolutional neural net- works by hypercomplex parameterization. arXiv preprint arXiv:21 10.04/76, 2021.
Daniel Greenfeld, Meirav Galun, Ronen Basri, Irad Yavneh, and Ron Kimmel. Learning to optimize multigrid PDE solvers. In International Conference on Machine Learning (ICML), pp. 2415- 2423, 2019.
David J Griffiths.
Introduction to electrodynamics, 2005.
Steven Guan, Ko-Tsung Hsu, and Parag V Chitnis. Fourier neural operator networks: A fast and general solver for the photoacoustic wave equation. arXiv preprint arXiv:2108.09374, 2021.
John Guibas, Morteza Mardani, Zongyi Li, Andrew Tao, Anima Anandkumar, and Bryan Catan- zaro. Adaptive fourier neural operators: Efficient token mixers for transformers. arXiv preprint arXiv:2111.13581 2021.
Xiaoxiao Guo. Wei Li, and Francesco Iorio. Convolutional neural networks for steady flow ap- proximation. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 481 490, 2016.
Jayesh K Gupta, Kunal Menda, Zachary Manchester, and Mykel J Kochenderfer. general frame- work for structured learning of mechanical systems. arXiv preprint arXiv:1902.08705. 2019.
Jiequn Han, Arnulf Jentzen, and Weinan E. Solving high-dimensional partial differential equations using deep learning. Proceedings of the National Academy of Sciences, 115(34):8505-8510, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpass- ing human-level performance on imagenet classification. In IEEE International Conference on Computer Vision (ICCV), pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Yacov Hel-Or and Patrick Teo. Canonical decomposition of Mathematical Imaging and Vision, 9(1):83-95 1998.
steerable functions.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint 18040455 2016.
Jan Hermann, Zeno Schätzle, and Frank Noé. Deep-neural-networkk solution of the electronic schrödinger equation. Nature Chemistry, 12(10)8891887,, 2020.
David Hestenes. Oersted medal lecture 2002: Reforming the mathematical language physics, 2003.
12

---

Published as a conference paper at ICLR 2023
David Hestenes. New foundationss for classical mechanics, volume 15. Springer Science & Business Media, 2012.
David Hestenes and Garret Sobezyk. Clifford algebra to geometric calculus: a unified language for mathematics and physies, volume 5. Springer Science Business Media, 2012.
Eckhard Hitzer. The clifford fourier transform in real clifford algebras. 2012.
Eckhard Hitzer. Quaternion and Clifford Fourier Transforms. Chapman and Hall/CRC, 2021.
Eckhard Hitzer and Stephen J Sangwine. Quaternion and Clifford Fourier transforms and wavelets. Springer, 2013.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):
1735-1780, 1997.
Jordan Hoffmann, Simon Schmitt, Simon Osindero, Karen Simonyan, and Erich Elsen. Algebranets. arXiv preprint arXiv:2006.07360) 2020.
Philipp Holl, Vladlen Koltun, Kiwon Um, and Nils Thuerey. phiflow: A differentiable pde solving framework for deep learning via physical simulations. In NeurlPS Workshop. volume 2, 2020.
Jun-Ting Hsieh, Shengjia Zhao, Stephan Eismann, Lucia Mirabella, and Stefano Ermon. Learning neural PDE solvers with convergence guarantees. arXiv preprint arXiv: 01200, 2019.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7132-7141,2015
Rakhoon Hwang, Jae Yong Lee, Jin Young Shin, and Hyung Ju Hwang. Solving pde-constrained control problems using operator learning. In AAAI Conference on Artificial Intelligence, vol- ume 36, pp. 4504-4512, 2022.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning (ICML), pp- 448-456 PMLR, 2015.
Erik Jenner and Maurice Weiler. Steerable partial differential operators for equivariant neural net- works. arXiv preprint arXiv:2 10163, 2021.
Yan-Bin Jia. Quaternions and rotations. Com S, 477(577):15, 2008
Xiaowei Jin, Shengze Cai, Hui Li, and George Em Karniadakis. Nsfnets (navier-stokes flow nets): Physics-informed neural networks for the incompressible navier-stokes equations. Journal of Computational Physics, 426:109951 2021.
Ryan Keisler. Forecasting global weather with graph neural networks. arXiv preprint 0:Xi5:2202.07575. 2022.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Georgios Kissas, Jacob H Seidman, Leonardo Ferreira Guilhoto, Victor M Preciado, George J Pap- pas, and Paris Perdikaris, Learning operators with coupled attention. Journal of Machine Learning Research, 23(215):1-63, 2022.
Clement Kleinstreuer. Engineering fluid dynamics: an interdisciplinary systems approach. Cam- bridge University Press, 1997.
Milan Klöwer, Tom Kimpson, Alistair White, and Mosè Giordano. milnn//speedyweahherjj.. v0.2.1, July 2022. URL https:/ //tp:///iii.////oo org/10.../...ooooooooo.oo...11 /10.0000000 5281/zenoda 6788067.
Dmitrii Kochkov, Jamie A Smith, Ayya Alieva, Qing Wang, Michael P Brenner, and Stephan beaming--acceleratee
Hoyer. Machine learning accelerated computational fluid dynamics. Proceedings of the National Academy of Sciences, 118(11):210178411 18, 2021.
13

---

Published as a conference paper at ICLR 2023
Risi Kondor and Shubhendu Trivedi. On the generalization of equivariance and convolution in neural networks to the action of compact groups. In International Conference on Machine Learning (ICML). pp. 2747-2755, PMLR, 2018.
Nikola Kovachki, Samuel Lanthaler, and Siddhartha Mishra. On universal approximation and error bounds for fourier neural operators. Journal of Machine Learning Research (JMLR), 22:Art-No, 2021.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. Advances in Neural Information Processing Systems (NeurIPS), 25, 2012.
Fred Kucharski, Franco Molteni, Martin P. King, Riccardo Farneti, In-Sik Kang, and Laura Feu- dale. On the need of intermediate complexity general circulation models: A "SPEEDY" exam- ple. Bulletin of the American Meteorological Sociery, 94(1):25-30 January 2013, doi: 10.1175/ 1.
bams-d- 1-00388.... URL https: /ttppp::://i.... /ttp:/oi.oo///////ooo org/ 1-t.//1..1////ammm-1--1-0001
Jack B Kuipers. Quaternions and rotation sequences: a primer with applications orbits, aerospace, and virtual reality. Princeton university press, 1999.
Yasuaki Kuroe. Models of clifford recurrent neural networks and their dynamics. In The 2011 international joint conference on neural networks, pp. 1035-1041. IEEE, 2011.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):227--2224 1998.
Zijie Li, Kazem Meidani, and Amir Barati Farimani. Transformer for partial differential equations' operator learning. arXiv preprint arXiv:2205.1306 2022a
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, An- drew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08899 2020.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, An- drew Stuart, and Anima Anandkumar. Markov neural operators for learning chaotic systems. arXiv preprint arXiv: 2ee:2220600000 06898, 2021a.
Zongyi Li. Hongkai Zheng, Nikola Kovachki, David Jin, Haoxuan Chen, Burigede Liu, Kamyar Azizzadenesheli, and Anima Anandkumar. Physics-informed neural operator for learning partial differential equations. arXiv preprint arXiv:2111.0377 1.03994, 2021b.
Zongyi Li, Daniel Zhengyu Huang, Burigede Liu, and Anima Anandkumar. Fourier neural operator with learned deformations for pdes on general geometries. arXiv preprint wrXiv:220.005209 2022b.
Marten Lienen and Stephan Günnemann. Learning the dynamics of physical systems from sparse observations with finite element networks. arXiv preprint arXiv:2203.08883 2022.
Joowon Lim and Demetri Psaltis. Maxwellnet: Physics-driven deep neural network training based on maxwell's equations. Apl Photonics, 7(1):01130,, 2022.
Burigede Liu, Nikola Kovachki, Zongyi Li, Kamyar Azizzadenesheli, Anima Anandkumar, An- drew M Stuart, and Kaushik Bhattacharya. A learning-based multiscale method and its applica- tion to inelastic impact problems. Journal of the Mechanics and Physics of Solids, 158:104668, 2022.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.0399 2016.
Winfried Lötzsch, Simon Ohler, and Johannes S Otterbach. Learning the solution operator of bound- ary value problems using graph neural networks. arXiv preprint vrXiv:2206.1409) 2022.
Pertti Lounesto. Clifford algebras and spinors. In Clifford Algebras and Their Applications in Mathematical Physics, pp. 25-37. Springer, 1986.
14

---

Published as a conference paper at ICLR 2023
Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature Machine Intelligence. 3(3):218- 222-229, 2021.
Lu Lu, Xuhui Meng, Shengze Cai, Zhiping Mao, Somdatta Goswami, Zhongqiang Zhang, and George Em Karniadakis. A comprehensive and fair comparison of two neural operators (with practical extensions) based fair data. Computer Methods Applied Mechanics and Engineer- ing. 393:114778, 2022.
Michael Lutter, Christian Ritter, and Jan Peters, Deep lagrangian networks: Using physics as model prior for deep learning. In International Conference on Learning Representations (ICLR), 2018
Hao Ma, Yuxuan Zhang, Nils Thuerey, Xiangyu Hu, and Oskar Haidn. Physics-driven learning of the steady navier-stokes equations using deep convolutional neural networks. arXiv preprint arXiv:2/06.09331, 2021a.
Wei Ma, Zhaocheng Liu, Zhaxylyk Kudyshev, Alexandra Boltasseva, Wenshan Cai, and Yongmin Liu. Deep learning for the design of photonic structures. Nature Photonics, 15(2):77-90, 2021b.
Romit Maulik, Vishwas Rao, Jiali Wang, Gianmarco Mengaldo, Emil Constantinescu, Bethany Lusch, Prasanna Balaprakash, Ian Foster, and Rao Kotamarthi. Efficient bigh-dimensional varia- tional data assimilation with machine-learne reduced-order models. Geoscientific Model Devel- opment, 15(8): 3433334333 2022.
Andreas Mayr, Sebastian Lehner, Arno Mayrhofer, Christoph Kloss, Sepp Hochreiter, and Jo- hannes Brandstetter. Boundary graph neural networks for 3d simulations. arXiv preprint arXiv:2106. 1299, 2021.
Ernan McMullin. The origins of the field concept in physics. Physics in Perspective, 4(1):13-39, 2002.
Pavlo Melnyk, Michael Felsberg, and Märten Wadenbäck. Embed me if you can: A geometric perceptron. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 1276-1284, 2021.
Philip Mirowski. More heat than light: Economics as social physics, physics as nature's economics. Cambridge University Press, 1991.
Franco Molteni. Atmospheric simulations using gem with simplified physical parametrizations. Model climatology and variability in multi-decadal experiments. Climate Dynamics, 20(2): 175-191, 2003.
C Eddie Moxey, Stephen J Sangwine, and Todd A Ell. Hypercomplex correlation techniques for vector images. IEEE Transactions on Signal Processing, 51(7):1941-1953 2003.
Ulises Moya-Sánchan Sebastià Xambó-Descamps, Abraham Sánchez Pérez, Sebastián Salazar- Colores, and Ulises Cortés. A trainable monogenic convnet layer robust in front of large contrast changes in image classification. IEEE access, 9: 93163335-663746, 2021.
Tu Dinh Nguyen, Dinh Phung, et al. Quaternion graph neural networks. In Asian Conference on Machine Learning, pp. 236-251, PMLR, 2021.
Titouan Parcollet, Mirco Ravanelli, Mohamed Morchid, Georges Linarès, Chiheb Trabelsi. Re- nato De Mori, and Yoshua Bengio. Quaternion recurrent neural networks. arXiv preprint arXiv:/806. 1804.0441...8... 2018a.
Titouan Parcollet, Ying Zhang, Mohamed Morchid, Chiheb Trabelsi, Georges Linarès, Renato De Mori, and Yoshua Bengio. Quaternion convolutional neural networks for end-to-end auto- matic speech recognition. arXiv preprint arXiv:1806.07789, 2018b.
Titouan Parcollet, Mohamed Morchid, and Georges Linarès. Quaternion convolutional neural net- works for heterogencous image processing. In ICASSP 2019 2019 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8514-8518, 2019. doi: 10. D0.109//CASSSP221988888
15

---

Published as a conference paper at ICLR 2023
Titouan Parcollet, Mohamed Morchid, and Georges Linarès. A survey of quaternion neural net- works. Artificial Intelligence Review, 53(4):295772288) 2982, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Veti,o Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, nigh-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buee E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems (NeurIPS), pp. 8024-8035. 2019.
Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizzadenesheli, Pedram Hassanzadeh, Karthik Kashinath, and Animashree Anandkumar. Fourcastnet: A global data- driven high-resolution weather model using adaptive fourier neural operators. arXiv preprint arXiv:2202.1 1 14, 2022.
JK Pearson and DL Bisset. Neural networks in the clifford domain. In Proceedings of 1994 IEEE International Conference Neural Networks (ICNN 94), volume 3. pp. 1465 1469. IEEE, 1994.
Justin Pearson. Clifford networks. In Complex- Valued Neural Networks: Theories and Applications, pp. 81-106. World Scientific, 2003.
Raphaël Pestourie, Youssef Mroueh, Chris Rackauckas, Payel Das, and Steven G Johnson. Physics- enhanced deep surrogates for pdes. arXiv preprint arXiv:2111.088 1.008841,, 2021.
Tobias Pfaff, Meire Fortunato, Alvaro Sanzhez- and Peter W Battaglia. Learning mesh- based simulation with graph networks. arXiv preprint arXiv:2010.03409 2020.
David Pfau, James S Spencer, Alexander GDG Matthews, and W Matthew C Foulkes, Ab initio solution of the many-electron schrödinger equation with deep neural networks. Physical Review Research, 2(3):033429, 2020.
Timothy Praditia, Matthias Karlbauer, Sebastian Otte, Sergey Oladyshkin, Martin V Butz, and Wolf- gang Nowak. Finite volume neural network: Modeling subsurface contaminant transport. arXiv preprint arXiv:2104.06010, 2021.
Md Ashiqur Rahman, Manuel A Florez, Anima Anandkumar, Zachary E Ross, and Kamyar Aziz- zadenesheli. Generative adversarial neural operators. arXiv preprint vrrXi:2205.03011 2022a.
Md Ashiqur Rahman, Zachary E Ross, and Kamyar Avizzadenesheli. U-no: U-shaped neural oper- ators. arXiv preprint arXiv:2204. 127, 20226.
Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-707, 2019.
Maziar Raissi, Alireza Yazdani, and George Em Karniadakis. Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations. Science, (6481))2 1030, 2020.
Yongming Rao, Wenliang Zhao, Zheng Zhu, Jiwen Lu, and Jie Zhou. Global filter networks for image classification. Advances in Neural Information Processing Systems (NeurIPS), 34, 2021.
Stephan Rasp and Nils Thuerey. Data-driven medium-range weather prediction with a resnet pre- trained on climate simulations: new model for weatherbench. Journal of Advances in Modeling Earth Systems, 13())=20200S0002400, 2021.
Edoardo Mello Rella, Ajad Chhatkuli, Ender Konukoglu, and Luc Van Gool. Neural vector fields for surface representation and inference. arXiv preprint arXiv:2204.06533 06552, 2022.
Pierre Renaud. Clifford algebras lecture notes applications in physics, 2020.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed-
ical image segmentation. In International Conference on Medical Image Computing and Computer-assisted Intervention, pp. 234 241. Springer, 2015.
16

---

Published as a conference paper at ICLR 2023
Alvaro
Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Pe- ter W. Battaglia. Learning to simulate complex physics with graph networks. arXiv preprint arXiv:2002.09405. 2020.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chinh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In IEEE Conference on Computer Vision and Patlern Recognition (CVPR), pp. 4510-4520, 2018.
Stephen J Sangwine and Todd A Ell. Colour image filters based on hypercomplex convolution. IEE Proceedings- Vision, Image and Signal Processing, 147(2):89-93, 2000.
Jakob Schwichtenberg. Physics from synmetrr.. Springer. 2015.
Wenlei Shi, Xinquan Huang, Xiaotian Gao, Xinran Wei, Jia Zhang, Jiang Bian, Mao Yang, and Tie-
Yan Liu. Lordnet: Learning to solve parametric partial differential equations without simulated data. arXiv preprint arXiv:2206.0 000.:2266.09418. 2022.
Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Ya Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. Convolutional LSTM network: machine learning approach for precipitation nowcasting. Ad- vances in neural information processing systems, 28, 2015.
Justin Sirignano and Konstantinos Spiliopoulos. Dgm: A deep learning algorithm for solving partial differential equations. Journal of computational physics, 375:1339-1364, 2018.
Casper Kaae Sønderby, Lasse Espeholt, Jonathan Heek, Mostafa Dehghani, Avital Oliver, Tim Sal- imans, Shreya Agrawal, Jason Hickey, and Nal Kalchbrenner. Metnet: A neural weather model for precipitation forecasting. arXiv preprint arXiv:2003.12140 2020.
Matthew Spellings. Geometric algebra attention networks for small point clouds. arXiv preprint arXiv:2110.02391. 2021.
Kimberly Stachenfeld, Drummond B Fielding, Dmitrii Kochkov, Miles Cranmer, Tobias Pfaff, Jonathan Godwin, Can Cui, Shirley Ho, Peter Battaglia, and Alvaro Sanchez Gonzalez. Learned coarse models for efficient turbulence simulation. arXiv preprint arXiv:2112.152775 2021.
Andrew M Steane. An introduction spinors. arXiv preprint arXiv: 2013.
Jaap Suter.
Geometric algebra primer. geometri petooettric-allgbbeeeee pdd.
http:/ http:://wwww... www.wwapsuter.c.ommm
pdf, 2003.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural net-
works. In International Conference on Machine Learning (ICML), pp. 6105-6114. PMLR, 2019.
Roger Temam. Navier-Stokes equations: theory and numerical analysis, volume 343. American
Mathematical Soc., 2001.
Nils Thuerey, Philipp Holl, Maximilian Mueller, Patrick Schnell, Felix Trost, and Kiwon Um. Physics-based deep learning. arXiv preprint arXiv:2109.05237. 2021.
Chiheb Trabelsi, Olexa Bilaniuk, Dmitriy Serdyuk, Sandeep Subramanian, João Felipe Santos San- tos, Soroush Mehri, Negar Rostamzadeh, Yoshua Bengio, and Christopher Joseph Pal. Deep complex networks. International Conference on Learning Representations (ICLR), 2017.
Marco AS Trindade, Vinicius NL Rocha, and Floquet. Clifford algebras, quantum neural networks and generalized quantum fourier transform. arXiv preprini arXiv:2206.01888 2022.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6924 6932, 2017.
Kiwon Um. Robert Brand, Yun Raymond Fei, Philipp Holl, and Nils Thuerey. Solver-in-the-loop Learning from differentiable physics to interact with iterative pde-solvers. Advances in Neural Information Processing Systems (NeurIPS), 33:6111-6122, 2020.
Charles Van Loan. Computational frameworks for the fast Fourier transform. SIAM, 1992.
17

---

Published as a conference paper at ICLR 2023
Comelis Boudewijn Vreugdenhil. Numerical methods for shallow-water volume 13. Springer Science & Business Media, 1994.
Nils Wandel, Michael Weinmann, and Reinhard Klein. Learning incompressible fluid dynam- ics from ccratch-towards fast, differentiable fluid models that generalize. arXiv preprint wwXiv:2006.08762, 2020.
Nils Wandel, Michael Weinmann, Michael Neidlin, and Reinhard Klein. ppline-pinn, Approaching pdes without data using fast, physics-informed hermite-spline cnns. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8529-8538, 2022.
Rui Wang, Karthik Kashinath, Mustafa Mustafa, Adrian Albert, and Rose Yu. Towards physics- informed deep learning for turbulent flow prediction. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1457-1466, 2020a.
Rui Wang, Robin Walters, and Rose Yu. Incorporating symmetry into deep dynamics models for improved generalization. arXiv preprint arXiv:2002.03001, 2020b.
Sifan Wang, Hanwen Wang, and Paris Perdikaris. Learning the solution operator of parametric par- tial differential equations with physics- informed DeepONets. Science advances, 7(40):eabi8605, 2021.
Maurice Weiler and Gabriele Cesa. General e(2) equiivarrat steerable CNNs. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco S Cohen. 3d steerable CNNs: Learning rotationally equivariant features in volumetric data. Advances in Neural Infor- mation Processing Systems (NeurlPS), 31, 2018.
Gege Wen, Zongyi Li, Kamyar Azizzadenesheli, Anima Anandkumar, and Sally M Benson. U- fno--a an enhanced fourier neural operator- deep-learning model for multiphase flow. Ad- vances in Water Resources, 163:104180. 2022.
Jonathan A Weyn, Dale R Durran, and Rich Caruana. Improving data-driven global weather pre- diction using deep convolutional neural networks on cubed sphere. Journal of Advances in Modeling Earth Systems, 22220:0020000000 2020.
Jonathan A Weyn, Dale Durran, Rich Caruana, and Nathaniel Cresswell-Clay. Sub-seasonal fore- casting with a large ensemble of deep-learning weather prediction models. Journal of Advances in Modeling Earth Systems, 13(7):c202 MSS00500, 2021.
Daniel E Worrall, Stephan J Garbin, Daniyar Turmukhambetov, and Gabriel J Brostow. Harmonic networks: Deep translation and rotation equivariance. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5028-5037, 2017.
Tailin Wu, Takashi Maruyama, and Jure Leskovec. Learning to accelerate partial differential equa- tions via latent global evolution. arXiv preprint arXiv:2206.07682 2022.
Yuxin Wu and Kaiming He. Group normalization. In European Conference Computer Vision (ECCV), pp. 3-19, 2018.
Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. Neural fields in visual comput- ing and beyond. In Computer Graphics Forum, volume 41, pp. 641-676. Wiley Online Library, 2022.
Yan Yang, Angela F Gao, Jorge C Castellanos, Zachary E Ross, Kamyar Azizzadenesheli, and Robert W Clayton. Seismic wave propagation and inversion with neural operators. The Seismic Record, 1(3):126-144 2021.
Di Zang, Xihao Chen, Juntao Lei, Zengqiang Wang, Junqi Zhang, Jiujun Cheng, and Keshuang Tang. A multi-channel geometric algebra residual network for traffic data prediction. IET Intelli- gent Transport Systems, 2022.
18

---

Published as a conference paper at ICLR 2023
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6848 8848-6856, 2018.
Xuanyu Zhu, Yi Xu, Hongteng Xu, and Changjian Chen. Quaternion convolutional neural networks. In European Conference on Computer Vision (ECCV), 2018.
Yinhao Zhu and Nicholas Zabaras. Bayesian deep convolutional encoder-decodee networks for surrogate modeling and uncertainty quantification. Journal of Computational Physics, 366:415- 447, 2018.
Kirill Zubov, Zoe McCarthy, Yingbo Ma, Francesco Calisto, Valerio Pagliarino, Simone Azeglio, Luca Bottero, Emmanuel Luján, Valentin Sulzer, Ashutosh Bharambe, et al. Neuralpde: Au- tomating physics-informed neural networks (pinns) with error approximations. arXiv preprint arXiv:2107.09443, 2021.
19

---

Published as a conference paper at ICLR 2023
Appendices
CONTENTS
Introduction
Background: Clifford algebras
3 Clifford neural layers
Experiments
Conclusion
Mathematical background
22
A.I
Clifford algebras
22
A.2
Examples low-dimensional Clifford algebras
24
A.2.1
Clifford algebra Clo.1
24
A.2.2
Clifford algebra Cl2,0
24
A.2.3
Clifford algebra Cla (R)
................................
25
A.2.4
Clifford algebra Cl3,0(
26
A.3
The electromagnetic field dimensions
27
Clifford neural layers
28
B.1
Clifford convolution layers
28
B.1.1
Translation equivariance of Clifford convolutions
................................
28
B.1.2
Rotational Clifford CNN layers
29
B.1.3
3D Clifford convolution layers
................................
30
B.2
Clifford normalization
30
B.3
Clifford initialization
B.4
Equivariance under rotations and reflections
31
B.5
Clifford Fourier layers
................................
32
B.5.1
2D Clifford Fourier transform
33
B.5.2
2D Clifford convolution theorem
34
B.5.3
3D Clifford Fourier transform
35
B.5.4
3D Clifford convolution theorem
36
B.5.5
Implementation of Clifford Fourier layers
37
B.6
Pseudocode
37
Experiments
39
C.1
Loss function and metrics
39
20

---

Published as a conference paper at ICLR 2023
C.2
Models
39
C,3 Training and model selection.
40
C,4
Navier-Stokes in 2D
41
C.5 Shallow water equations.
45
C.6 Maxwell's equations in matter in 3D
51
Related work
53
Glossary
56
21

---

Published as a conference paper at ICLR 2023
A MATHEMATICAL BACKGROUND
This appendix supports Section of the main paper. We give a more detailed explanation of real Clifford algebras and have closer look (TR), Cl),2(R)), and (R). For detailed in- troduction into Clifford algebras we recommend Suter (2003); Hestenes (2003; 2012); Dorst et al. (2010); Renaud (2020)
A.1 CLIFFORD ALGEBRAS
Vector spaces and algebras over field. A vector space over a field F is set V together with two binary operations that satisfy the axioms for vector addition and scalar multiplication. The axioms of addition ensure that if two elements of V get added together, we end up with another element of V. The elements of F are called scalars. Examples of a field F are the real numbers R and the complex numbers C. Although it is common practice to refer to the elements of general vector space V as vectors, to avoid confusion we will reserve the usage of this term to the more specific case of elements of IR.. As we will see below, general vector spaces can consist of more complicated, higher-order objects than scalars, vectors or matrices.
An algebra over field consists of vector space V over a field F' 'ogether with an additional bilinear law of composition elements of the vector space, V x V + V, that is, if a and are any two elements of V, then ab: V V > V is an element of V, satisfying pair of distribution laws: a(Apb + Azc) = ab + Azac and (A1a + dzb)c = aiac + X2be for λ1, X2 E F and a, b, c E V. Note that general vector spaces don't have bilinear operations defined their elements.
Clifford algebras over TR. In this manuscript we will focus on Clifford algebras over IR. For a more general exposition on Clifford algebras over different fields the reader is referred to Lounesto (1986).
A real Clifford algebra is generated by the re-dimensional vector space R" through a of relations that hold for the basis elements of the vector space RR. Let us denote the basis elements of II* with e....., en. and without loss of generality choose these basis elements to be mutually orthonormal. Taking two nonnegative integers p and g, such that p + n, then a real Clifford algebra Clp.q (IR)
with the "signature" (p. q). is generated through the following relations that define how the bilinear product of the algebra operates on the basis elements of R":
of +1
for 1 1sisp,
(14)
9 -
for p < S 4q
(15)
c4e Eje;
for i f
(16)
Through these relations we can generate basis for the vector space of the Clifford algebra, which we will denote with G. Equations 14 and 15 show that the product between two vectors yields a scalar. According to the aforementioned definition of an algebra over field, a Clifford algebra with a vector space G is equipped with a bilinear product G x G - G. that combines two elements from the vector space G and yields another element of the same space G. Therefore, both scalars and vectors must be elements of the vector space G. Equation 16 shows that besides scalar and vector elements, higher order elements consisting of a combination of two basis elements, such as eje, and eye, are also part of the vector space G. Finally, by combining Equations 14, 15, 16 we can create even higher order elements such as C,eyCk for + j F k. or C1C2...Cpqg which all must be part of the vector space G.
In order determine what the basis elements are that span the vector space of Clp.g (I), we note that elements (2)))) and é eg....e are related through simple scalar multiplicative factor of plus or minus one, depending on the sign of the permutation a. Therefore, it suffices to consider the unordered combinations of basis elements of R: the basis of the vector space G is given by {1, E1, E2, Ep+gg e162, Cptg-1ppttt-p-a-ppep Eptg).
In summary, we have introduced two different vector spaces. First, the vector space R" which generates the Clifford algebra, and second the vector space G, which is the vector space spanned by the basis elements of the Clifford algebra Clp.q(RR (R). Convention is to denote the vector space of real Clifford algebra with a superscript n of the dimension of the generating vector space, yielding Gm for generating vector space IR... Note that the dimension of the vector space GD is 2n 2
22

---

Published as a conference paper at ICLR 2023
Exemplary low-dimensional Clifford algebras are: (i) (R) which is one-dimensional algebra that is spanned by the vector {1} and is therefore isomorphic to R, the field of real numbers; (ii) C (R) which is a wwo-dimensional algebra with vector space G' spanned by {1, where the basis vector en squares to -1, 1, and is therefore isomorphic to C, the field of complex numbers; (iii) 2 which is 4-dimensional algebra with vector space G2 spanned by {1, 61, e2, (16)). where e2 square to -1 and anti-commute. Thus, (lo,2( is isomorphic to the quaternions H..
Definition 1: Grade of Clifford algebra element
The grade Clifford algebra basis element the dimension of the subspace it represents.
For example, the basis elements {1, éi, 62; e2) of the Clifford algebras Cla (R) and Cl2 have the grades {0, 1, 1, Using the concept of grades, we can divide the vector spaces of Clifford algebras into linear subspaces made
of clements of each grade. The grade subspace of smallest dimension is Mo. the subspace of all scalars (elements with basis vectors). Elements of M1 are called vectors, elements of M2 are bivectors, and so on. In general, the vector space GP+9 of Clifford algebra Clp.g can written as the direct sum of all of these subspaces:
G = Mo M1 .... 2 Mp+g
(17)
The elements of Clifford algebra are called nultivectors, containing elements of subspaces, i.c. scalars, vectors, bivectors, trivectors etc. The basis element with the highest grade is called the pseudoscalar" which in R² corresponds the bivector 61 and in R the trivector 61e263- The pseudoscalar is often denoted with the symbol iptq- From hereon, only multivectors will be denoted with boldface symbols.
Geometric product. Using Equations 14, 15, 16, we have seen how basis elements of the vector space GP+g of the Clifford algebra are formed using basis elements of the generating vector space V. We now, look at how elements of GT++ are combined, i.e. how multivectors are bilinearly operated on. The geometric product is the bilinear operation on multivectors Clifford algebras. For arbitrary multivectors b, E GT+,, and scalar À the geometric product has the following properties:
ab E GP+4 (ab)e a(be)
closure.
(18)
associativity,
(19)
Xa = aA
commutative scalar multiplication,
(20)
a(b + ab +
distributive over addition.
(21)
The geometric product in general commmmmttaaeee,, i.e. ab * ba. As we describe later, the geometric product is made up of two things: an inner product (that captures similarity) and exterior (wedge) product that captures difference.
Definition 2: Dual of a multivector
The dual a' of a multivector a is defined as:
a' aiptg
(22)
where ipto represents the respective pseudoscalar of the Clifford algebra.
This definition allows relate different multivectors each other, which is useful property when defining Clifford Fourier transforms. For example, for Clifford algebras in RR² the dual of scalar is bivector, and for the Clifford algebra R the dual of scalar is trivector.
In contrast scalars, pseudoscalars change sign under reflection.
23

---

Published as a conference paper at ICLR 2023
A.2 EXAMPLES OF LOW-DIMENSIONAL CLIFFORD ALGEBRAS
A.2.1 CLIFFORD ALGEBRA 1lo,1
The Clifford algebra (R) a two-dimensional algebra with vector space G spanned by (1,c1), J. and where the basis vector e1 squares to -1. Cla.1 (R) is thus algebra-isomorphic to C, the field of complex numbers. This becomes more obvious if we identify the basis element with the highest grade, i.e. 61, as the pseudoscalar i which is the imaginary part of the complex numbers. The geometric product between two multivectors a = On + are1 and b = bo + by 61 is therefore also isomorphic to the product of two complex numbers:
ab agbo + abbi el + aabb ei aabbeie
= (aobo anbi) + (agb) + a1bb))
(23)
A.2.2 CLIFFORD ALGEBRA C12,0
The Clifford algebra C12,o [R) is 4-dimensional algebra with vector space G2 spanned by the basis vectors (1,61, f 6162) where 61, 62 square +1. The geometric product of two multivectors a = ag + 0161 + age2 + 0126162 and bo + biet + bee2 + b126162 is defined via:
ab agbo agbi
agb2 +
e2 + abb12 6162
a1 +
a1b1261e1 e102 a1be + abbbeier +
62
+ agbo
+ azhı 62
agbze2e2 e2e1
azb1262 e1 62
+ a12bo e162
+ e
62 el + a12b2 e 6262 + a1bbi2cie2eiee
(24)
Using the relations f = 1, 6262 = 1, and e,e, = -eje for i + E (21, from which follows that ejege162 1, we obtain:
ab
agdn + ajhı + a2b2 - 012h12
(agb1 aıdu azb12 a12b2) C1
+ (aob2 + 01b12 + azho a12h1) 62
+ (agb12 + azb2 azbı a12bo) e162
(25)
vector R C G2 is identified with 2 + x262 E R G Clifford multiplication of two
vectors I, R² C G2 yields the geometric product zy:
ry = (x) eq
x2
e2
+ +
=
sc enyne? e + x222262
T1y2e1 + 322/1 6261
(x. y)
Agy
(26)
Inner product
Outer/ product
The asymmetric quantity wAy = - A is associated with the now often mentioned bivector, which can be interpreted an oriented plane segment.
Equation 26 can be rewritten to express the (symmetric) inner product and the (anti-symmetric) outer product in terms of the geometric product:
37 y
(xy yr)
(27)
(x,, v)
(zy + yx)
(28)
24

---

Published as a conference paper at ICLR 2023
From the basis vectors of the vector space G of the Clifford algebra (l,,, (1, probably the most interesting is e1 We therefore have a closer look the unit bivector i2 = 6162 which is the plane spanned by é and e2 and determined by the geometric product:
12 61 (21 + -2 Ae
6261
(29)
where the inner product (e1, e2) is zero due to the orthogonality of the base vectors. The bivector 2g if squared yields 2 11, and thus in represents true geometric v-1. From Equation 29, it follows that
C2 = c1 = i261
e1 i262 6212
(30)
Using definition 2, the dual of a multivector a G2 is defined via the bivector as iga. Thus, the dual of scalar bivector and the dual of a vector is again vector. The dual pairs of the base vectors are 1 ++ eje2 and + Ez. These dual pairs allow us to write an arbitrary multivector a as
ug + djei 0262 012612
(ag + 01212) C1 (a1 + 02l2)
(31)
spinor part
vector part
which can be regarded as two complex-valued parts: the spinor part, which commutes with ig and the vector part, which anti-commutes with ig.
A.2.3 CLIFFORD ALGEBRA Clo. (RR)
The Clifford algebra Cla,2 is 4-dimensional algebra with vector space G2 spanned by the basis vectors {1, e1,e1,
622 E162] where é,, 62 square to -1. The Clifford algebra 2 (R) is algebra- isomorphic to the quaternions H., which commonly written in literature (Schwichtenberg, 2015)
as bi + cÿ dk, where the (imaginary) base elements J. and fulfill the relations:
1 j
ij
ji -k
82 ijtj 2jf0 =11
(32)
Quaternions also form 4-dimensional algebra spanned by {1, k, where i, j. k all square to -1. The basis element 1 is often called the scalar part, and the basis elements J. k are called the vector part a quaternion.
The geometric product of two multivectors 0o + 0161 + 0262 + 01261e2 and b by + bie1 + b2e2 + b126162 is defined as:
ab = agbo + agbi en agb2
C2
aghi2 +
e1e2
+ bo e1 + agbieier
a3b2 +
e102
+ a1b12e1es
+ agbo 62 agbi
agbzegez ege1
02b1s
e1 62
+ +0120n e162
+ #izblei
el
a12b2 +
+ 62e2
212b1ee1e2e1e..
(33)
Using the relations 1ιlι = 1, E2e2 = 1,, and eie; = ejes for i i E {er, e2), from which
follows that e1eg6162 11, we obtain:
ab agdn - aibı - u2b2 - a12b12
(aobi aıdo azbi2 a12b²)
e1
+ (anb2 ard12 + azbo + a12b1)
62
+ (agb12 + 0rd2 azdı a12bb)
e162
(34)
25

---

Published as a conference paper at ICLR 2023
A.2.4 CLIFFORD ALGEBRA (R)
The Clifford algebra is a 8-dimensional algebra with vector space Go spanned by the basis vectors (1, 62, Cg. e1 6163; CgCs, cieges), i.e. one scalar, three vectors {e1.e2, e2, three bivectors {e162, C3 C. and one trivector eye2 The trivector is the pseudoscalar ig of the algebra. The geometric product of two multivectors defined analogously the geometric product of C12.0 (IR), following the associative and bilinear multiplication multivectors follows:
for 1,2,3
(35)
e,eY = e
for i.j = 1, +
(36)
Using Definition 2, the dual pairs of Cl3,0 are:
1 + G10263 is
(37)
61 + 6263
(38)
e2 + e391
(39)
63 O 6162
(40)
The geometric product for Cl3,o(R) is defined analogously to the geometric product of Cl2,0( via:
ab agbo + agbi 61
+ agb2 e2
+ agb3 63
+ agb12 e1e2
+ anbis e1e3 + agb23 6263
+ agbi22 e1e263
aibo
ei
aibbeiei + b2 e1e2
+ aabb e1e3
+ a1b120111
02 + tabbue1c1 Cs
+ 01b23
616263
+ 01b1236111 6203
azbo
e2
d261 e261
+ agbr
agbs e2e3
a2b12e2
61
62
+
02b13
616362
+ 02b23e2e2 e3
a2b1230222 6163
azbo
azhi e3e1
+ azb2
e362
+ 03b3e3e3
agb12
616362
0zb13
ei e3e3
0zb23
62 e3e3
azb123
6162 6303
+
e102 a1zb0
dq2b1
02 c1e1 + 012b2
e1 6262 + a1zbs 61e263
a12b12e1eze1e2
- ai1bbaeee 6263 + 012b23e2e2 e163
41221230 63
a1zhg e103
an3hi 63 G161
013d2
610362 013b3 61 6303
0136120161 6362 + #rgbbsere3e1es - 413b23 e162 E363
413/123616341
02ghg 6203 + 023h1 e1e362 + 023b2e2 es e2 + 023b3 62 6363
a2gb1se2 egc1 e2 agsbis e2e1 e3c3 + azsbygezeezzes
6j anab12sezes
E2e3
a12360 e1e263 + dlzabrer 6g63 ei - 012362 e163 e1e2 + a123b3 e1e2 6363
0123b1261e e2 61 + 0123b13e1 62 636103 a12zbzs é 62036263
(41)
where minus signs appear do reordering of basis elements. Equation 41 simplifies
26

---

Published as a conference paper at ICLR 2023
ab = agbo + arb; + 02b2 + az03 - 012012 - 013b13 025b23 - 01230123'
+ (aod1 + anbp 02b12
asb1s + 012by + d1zb3 - 423b123 - a12zb23) e1
+ (aab2 a1b12 azbo
43b23
a12bl + a13b123 a2363 + d123b13) e2
+ (aobs + a1b13 a2b2s
+ azbo a12b123 013h1
023h2
4123b12)
63
+ (00d12 01b2 izhı + 03b123 + 012 013b23
a23b13 + 4123b3)
61
+ (agb13 a103 a2b123 az6ı + 012b23 01zb0
423b12
d123b2)
e163
+ (aob23 + 01b123 + azh3 - agh2 - 41bb1s + 013012 + 023bu + d123/1)
6253
+ (a0b123 + an b23 aabis azb12 012b3 01zb2 + a23bi + a123bn) e16263
(42)
A.3 THE ELECTROMAGNETES FIELD IN 3 DIMENSIONS
Through the lense of C3 (R), an intriguing example of the duality of multivectors found when writing the expression of the electromagnetic field F in terms of an electric vector field F and a magnetic vector field B (Hestenes & Sobezyk, 2012; Hestenes, 2003), such that
= Big.
(43)
Both the electric field E' and the magnetic field B are described by Maxwell's equations (Griffiths, 2005). The two fields are strongly coupled, e.g. temporal changes of electric fields induce mag- netic fields and vice versa. Probably the most illustrative co-occurence of electric and magnetic fields is when describing the propagation of light. In standard vector algebra, E is vector while B is pseudovector, i.e. the two kinds of fields are distinguished by a difference in sign under space inversion. Equation 43 naturally decomposes the electromagnetic field into vector and bivec- tor parts via the pseudoscalar iz. For example, for the base component Br of B holds that B,Er Bgeiéieges Bze2e3, which is bivector and the dual to the base component e1 of E. Geometric algebra reveals that a pseudovector is nothing else than a bivector represented by its dual, the magnetic field B in Equation 43 is fully represented by the complete bivector Bíg, rather than alone. Consequently, the multivector representing F consists of three vectors (the electric field components) and three bivectors egág e2iz E3E1, ezi3 eje2 (the magnetic field components multiplied by iz).
27

---

Published as a conference paper at ICLR 2023
B CLIFFORD NEURAL LAYERS
This appendix supports Section of the main paper.
Clifford convolutions are related the work on complex networks by Trabelsi et (2017), and closely related to work quaternion neural networks (Zhu et al., 2018; Parcollet et al., 2018a; Gaudet & Maida, 2018; Parcollet et al., 2018b; 2019; 2020; Nguyen et al., 2021). Probably the most related work are (i) by Zang et al. (2022) who build geometric algebra convolution networks to process spatial and temporal data, and (ii) Spellings (2021) who build rotation- and permutation- equivariant graph network architectures based on geometric algebra products of node features. Higher order information is built from available node inputs.
B. CLIFFORD CONVOLUTION LAYERS
We derive the implementation of translation equivariant Clifford convolution layers for multivectors G², i.e. multivectors of Clifford algebras generated by the 2-dimensional vector space R2, Finally, we make the extension to Clifford algebras generated by the 3-dimensional vector space RR:.
Regular CNN layers. Regular convolutional neural network (CNN) layers take as input feature maps f z² > Re and convolve!? them with set of Cout filters {w" Ti=l 22 R
[f x w (x) (z)= Σ
(f(v), w x))
(44)
VE2
Σ yEZA
1 (y)
(45)
Equation 44 can be interpreted as inner product of the input feature maps with corresponding filters every point y E Z². By applying Cour filters, the output feature maps can be interpreted Cour dimensional features vectors at every point y E Z We now want to extend convolution layers such that the elementwise product of scalars f fly)) - x) are replaced by the geometric product of multivector inputs and multivector filters f (y)w*)
x).
Clifford CNN layers. We replace the feature maps f 2 R by multivector feature maps f 72 + (G2) co and convolve them with set of Cou multivector filters (w*) z + (G2)
[f wt] [)]()))=
(x)
ΣΣ f (y) (
(46)
yET?
fooid G2
B.1.1 TRANSLATION EQUIVARIANCE OF CLIFFORD CONVOLUTIONS
Theorem 1: Translation equivariance of Clifford convolutions
Let ft 2 (G2)au be multivector feature map and let w z² (G2)an i be multivector kernel, then for CI(2, (R) [L. + wo (x) = [L, [f wil (x)
In deep learning, convolution operation in the forward pass is implemented cross-correlation.
28

---

Published as a conference paper at ICLR 2023
Proof.
[[,,]] xw] (x)
Σ Σ fy - 1)w((y - x)
WE22 1=1 j=1
Gr
Σ Σ
foly- +)wool =2) + fu - t) - x2) + fa(v t)w2(U x) f12(v +))11( x)
VER2 1=1
(fo(y -)) (y 1) + fi (y =) - f2(y - ()wi2(y 2) + fi2 - ()wz(y -m))en
(Joly () =) fi ())zz( 2) [y L)w fiz 1)wi (y) - xx)
(alyy 1) - 2) fa 12 2) fa(ʃ l)wi(y x) + fi2(y Dwwol 2)) e102
(using - y w-11
Σ fo(w)wo(y (z 1)) + f1(w)w1(H (e ()) + f2(y)w2(y - (2 ()) f12(w)012(y - (x 1))
yERR
(Jo())oo(y - (x ()) + (v) (x ( 12(1) (x - 1)) + fiz(y) (2 - 1))
(fo())uz(v - (x - ()) (2) (x ()) + f2())oo (x 1)) fiz(w) (y (2 - 1))
(o(()))1(W () (1))waz( (a ()) f2(2) (v (a 1)) fi2()wolly (o 1))
e102
[Le [f x wx)l
(47)
Implementation of Cl2.0 (.) and (lR,2(R)) layers. We can implement a CI((2, ()R) Clifford
CNN layer using Equation 25 where {bo. b1 b2, b12} + {wo w. w w12 correspond to 4 different kernels representing one 2D multivector kernel, i.e. 4 different convolution layers, and {ao.
41, 022 )12) > (fá- H.. f2 }i2} correspond to the scalar, vector and bivector parts of the input multivector field. The channels of the different layers represent different stacks of scalars, vectors, and bivectors. All kernels have the same number of input and output channels (number of input and output multivectors), and thus the channels mixing occurs for the different terms of Equations 25, 42 individually. Lastly, usually not all parts of the multivectors are present in the input vector fields. This can easily be accounted for by just omitting the respective parts of Equations 25, 42. A similar reasoning applies to the output vector fields. For Cl(O, (RR) the signs within the geometric product change slightly.
B.1.2 ROTATIONAL CLIFFORD CNN LAYERS
Here we introduce an alternative parameterization the Clifford CNN layer introduced in Equa- tion 7 by using the isomorphism of the Clifford algebra (lR,((IR) to quaternions 13, We take ad- vantage of the fact that a quaternion rotation can be realized by matrix multiplication (Jia, 2008; Kuipers, 1999; Schwichtenberg, 2015). Using the isomorphism, we can represent the feature maps f' and filters w as quaternions: f = + + + f3 and w 1-2 +w i+ j+ w 7,14 Leveraging this quaternion representation, we can devise an alternative parameterization of the prod- uet between the feature map f and - To be more precise, we introduce composite operation that results in a scalar quantity and a quaternion rotation, where the latter acts on the vector part of the quaternion f and only produces nonzero expansion coefficients for the vector part of the quaternion output. A quaternion rotation wi f (wni>)- acts on the vector part Gi,j, k) of f and
13 We could not find neural rotational quaternion convolutions in existing literature, we however used the codebase of https://github.co ctattuub..o//Orkiiss wwwm///iis-eseearr///eaaa Research/Pytorch werrtoorc-uuattmionn.... Quatermion-Neura--ee Neural-Networkss inspiration.
NNot that the expansion coefficients for the feature map f and filters i in terms of the basis elements of G² and in terms of quatemnion elements i, and i are the same.
29

---

on the feature Published as a conference paper at ICLR 2023
be algebraically manipulated into vector netor-matrix operation R where R H Hi is built up from the elements of w (Kuipers, 1999). In other words, one can transform the vector part (i, J. k) of f E via a rotation matrix R that built from the scalar and vector part (1, j. k) of w H. Altogether, rotational multivector filter [w) 16m 2² (G2)
map f through rotational transformation R (wrn. ma,01 Wind Wra, 20 (2,,12) acting on vector and bivector parts of the multivector feature map f =² - (G2)au, and an additional scalar response of the multivector filters:
Gn
[f worl (x)
Σ yE22
f w x)
j=1
R(v) f
(y) x))] + +R
(48)
yE22
(2(
scalar output
where (f (y) (y .))] fg w
fiz 1or, which is the scalar
1o1.2
output of Equation 34. The rotational matrix R (y - x)
written out form reads:
1 - ((aat)) (wiil 12)
2 pi fij fn
no 1iii2 W oi
2 Tia ond
1 Wio 12 + noi 20
R
2 f
1or.1 wi + wit 20 20
1 2 (10 2² + (10 1u
2 o mayy t Wi
rot, ror.1
2 2o0 1i1 3 20122
2 th 12 + i 20..22
(w
(49)
where widd (y - x) = non,o (y - x) + w (y øJei + 2ar,2 x)e2 + w 12 (y x)e12 12
normalized filter with || 1. The dependency (y x) is omitted inside the rotation matrix Rid for clarity.
B.1.3 3D CLIFFORD CONVOLUTION LAYERS
Implementation of (R) layers. Analogously to the 2-dimensional case, we can implement 3D Clifford CNN layer using Equation 42, where {bv, b1, b2, b12. b13, b23. b12s} correspond to 8 different kernels representing one 3D multivector kernel, i.e. different convolution layers, and (ao, 01, 02 012, 013, (2g3, d123) correspond to the scalar, vector, bivector, and trivector parts of the input multivector field. Convolution layers for different 3-dimensional Clifford algebras change the signs in the geometric product.
B.2 CLIFFORD NORMALIZATION
Different normalization schemes have been proposed to stabilize and accelerate training deep neural networks (loffe & Szegedy, 2015; Ba et al., 2016; Wu He, 2018; Ulyanov et al., 2017). Their standard formulation applies only real values. Simply translating and scaling multivectors such that their mean is 0 and their variance is insufficient because it does not ensure equal variance across all components.
Batch normalization Trabelsi et al. (2017) extended the batch normalization formulation to apply complex values. We build the same principles first propose an appropriate batch normaliza- tion scheme for multivectors, similar to the work of Gaudet & Maida (2018) for quaternions. For 2D multivectors of the form a ag 0101 + 0ge2 + 0126162, we can formulate the problem of batch normalization as that of whitening 4D vectors:
(V) $ E(al)
(50)
where the covariance matrix V is
V Yooa
Vage:
V
Vagaz
Va0a12
V Vayag
Vaja
Vajag
Vaja1a
(51)
V
Vagan
Vagaz
V
Vazag
Vaza12
41200
Vazaa
Varzas
Vaiaa12
30

---

Published as a conference paper at ICLR 2023
The shift parameter 3 is multivector with 4 learnable components and the scaling parameter yy is x positive matrix. The multivector batch normalization is defined as:
(N a
(52)
When the batch sizes are small, it can be more appropriate to use Group Normalization or Layer Normalization. These can be derived with appropriate application of Eq. 50 along appropriate tensor dimensions. As such, batch, layer, and group normalization can be easily extended to 3-dimensional Clifford algebras.
B.3 CLIFFORD INITIAIIZATION
Parcollet et al. (2018a); Gaudet & Maida (2018) introduced initialization schemes for quaternions which expands upon deep network initialization schemes proposed by Glorot & Bengio (2010); He et al. (2015). Similar to Clifford normalization, quaternion initialization schemes can be adapted to Clifford layers in a straight forward way. Effectively, tighter bounds are required for the uniform distribution form which Clifford weights sampled. However, despite intensive studies did not observe any performance gains over default Py Torch initialization schemes" 15 for 2-dimensional experiments. Similar findings reported in Hoffmann et al. (2020), However, 3-dimensional implementations necessitate much smaller initialization values (factor 1/8).
B.4 EQUIVARIANCE UNDER ROTATIONS AND REFLECTIONS
Clifford convolutions satisfy the property of cquivariance under translation the multivector in- puts, as shown in this Appendix B. However, the current definition of Clifford convolutions is not equivariant under multivector rotations or reflections. Here, we derive a general kernel constraint which allows us to build generalized Clifford convolutions which equivariant w.r. rotations or reflections of the multivectors. That is, we like to prove equivariance of a Clifford layer under ro- tations and reflections (i.e. orthogonal transformations) if the multivector kernel multivector filters (w*) Cue Z + (G) satisfies the constraint:
w Tw 4 (x)
for S < Gin- We first define an orthogonal transformation on multivector by,
Tf fuful, u'u=1
(53)
where and f are multivectors which are multiplied using the geometric product. The minus sign is picked by reflections but not by rotations, i.c. depends on the parity of the transformation. This construction is called "vveroor" product. The construction vectors and its extension to arbitrary multivectors. The above construction makes immediately clear that T(fg) (T) When write Tr, Euclidean vector (which can principle also be defined using versors). To show equivariance, we wish prove for multivectors f z² (G) and of Cour multivector filters {w ) Z²
be found in c.g. Suter (2003) for
orthogonal transformation of an
(G) that:
f((T)) Tff(),
(54)
and
w (Tx)) Two(
(55)
Equations 54, 55 yield:
[ * w (Tx) = T [f [ww] (x)
(56)
That is: if the input multivector field transforms multivector, and the kernel satisfics the stated equivariance constraint, then the output multivector field also transforms properly as a multivec- tor. Note that T might act differently the various components (scalars, vectors, pseudoscalars, pseudovectors) under rotations and/or reflections.
The default PyTorch initialization of linear and convolution layers is He Uniform initialization (He et al., 2015) for 2-dimensional problems. The gain is calculated using LeakyRelu activation functions with negative part of 5, which effectively results in Glorot Uniform initialization.
31

---

Published as a conference paper at ICLR 2023
Now,
[f * wo]' (Tx)
Σ
(y)
yET? j=1
E
(I(T-'y ( x)))
WEE? j=1
= Σ
Σ
(T x))), yf = T
Ty E ==1
Σ
(T z)))
j=1
Σ
Tf Tf
y'EZ2 j=1
T(f (f) -x)))
E2
T Σ E'E j=1
( x)))
T [ w. (x)
(57)
where in the fourth line we transform variables y + y' in the fifth line we use the invariance of the summation "measure" under T, in the sixth line we use the transformation property of f and equivariance for w³, in the seventh line we use the property of multivectors, and the eighth line use linearity of T.
B.5 CLIFFORD FOURIER LAYERS
We derive the implementation of Clifford Fourier layers for multivectors in G2 and G i.e. multi- vectors of Clifford algebras generated by the 2-dimensional vector space R2 and the 3-dimensional vector space R
Classical Fourier transform. In arbitrary dimension n, the Fourier transform f(E) (()
for continuous n-dimensional complex-valedd signal f(x) f(x In) R" - C is defindd as:
-()==U(( (2x)n/2 [ f(x)e f(z)e- 2mi((,)) dr R" R"
(58)
provided that the integral exists, where and are n-dimensional vectors and (a, {) is the contrac- tion of x and ę. Usually, (x, {) is the inner product, and ş is an element of the dual vector space R"* The inversion theorem states the back-transform from the frequency domain into the spatial domain:
f(x) (F)=))))))))) = (2m))/2 J f(E) de, Vx E R
(59)
We can rewrite the Fourier transform of Equation 58 in coordinates:
f(Ei....,EE ........)))- =(fY(E En. = (2 / f(21..... on 200((1 taxEu) dr dxn
(60)
32

---

Published as a conference paper at ICLR 2023
Discrete/Fast Fourier transform. The discrete counterpart of Equation 58 transforms
dimensional complex signal f(z) = f(r....., Rn)
R + at M1 x .... x Mn grid points
into its complex Fourier modes via:
M Σ my
Mn
-2ri
mafn
f(E.......EN F(f) ..)
f mn)
178
(61)
where (EI,...,E.) S.) E ΣM, ... x x.. BM. X.. Fast Fourier transforms (FFTs) (Cooley & Tukey, 1965; Van Loan, 1992) immensely accelerate the computation of the transformations of Equation 61 by factorizing the discrete Fourier transform matrix into a product sparse (mostly zero) factors.
B.5.1 2D CLIFFORD FOURIER TRANSFORM
Analogous Equation 58, for CI(2, (R) the Clifford Fourier transform (Ebling & Scheuermann, 2005; Hitzer, 2012) and the respective inverse transform for multivector valued functions f f(x R2a G and vectors x, TR² are defined as:
f(() = F(f((E) = 2T [ f(x)e ((a))-2mislatt dz, VE E ?
(62)
JR2
- [ f(E)e de, Vr E IR?
(63)
2n
provided that the integrals exist. The differences to Equations 58 and 59 are that fxx) and f(E) represent multivector fields in the spatial and the frequency domain, respectively, and that the pseu- doscalar 122 = 61 is used in the exponent. Inserting the definition of multivector fields, we can rewrite Equation 62
1
F(f(()) = 2m Ra
f(x)e 2mia(anE) dx,
1
/
( fo(x)) + 512 +
[1())) + f2(m)iz) )
22222(2 dx vector part
2T JRZ
spinor part
1
fo(x) + f12(1)(2)
(- dr
2m
Ra
+
1
(f((z) + fe(x))z)) e1
(0ri dx
2x
BB2
F (allo)) (12())aa) (5)
e1 +(16))+
(64)
We obtain Clifford Fourier transform by applying two standard Fourier transforms for the dual pairs fo = [z) + (12(2)12 and ft fi + f2(1r)2, which both can be treated as complex- valued signal for fi R C. Consequently, f(x) can be understood as an element of C2. The 2D Clifford Fourier transform is the linear combination of two classical Fourier transforms. The discretized versions of the spinor/vector part (fs/v) reads analogously to Equation 61:
(E) F(fa = Σ
Mi
Σ
M2
(m/ m2)-
22r
mid
28
mafa
(65)
m==0 m2
where again (51,E2 E RM, x 7MM_- Similar to Fourier Neural Operators (FNOs) where weight
tensors are applied pointwise in the Fourier space, we apply multivector weight tensors W (G2) ( x point-wise. Fourier modes above cut-off frequencies (ETTa <m=x) are set to
zero. In doing so, we modify the Clifford Fourier modes
f(C) = F()(() = fo(E) + f(()er + fe(E)ez + f12(E)e12
(66)
33

---

Published as a conference paper at ICLR 2023
via the geometric product. The Clifford Fourier modes follow naturally when combining spinor and vector parts of Equation 64. Analogously to FNOs, higher order modes are cut off. Finally, the residual connections used in FNO layers is replaced by multivector weight matrix realized as Clifford convolution, ideally a (R) convolution layer. A schematic sketch of Clifford Fourier layer is shown in Figure 5b in the main paper. For (R)), the the vector part changes to
(1(()) - (((())_)())))))
B.5.2 2D CLIFFORD CONVOLUTION THEOREM
In contrast to Ebling & Scheuermann (2005), we proof the 2D Clifford convolution theorem for multivector valued filters applied from the right, such that filter operations are consistent with Clif- ford convolution layers. We first need to show that the Clifford kernel commutes with the spinor and anti-commutes with the vector part multivectors. We can write the product ae'* for every scalar E R and multivector E G as
a = (a(cos(s) + i2 (in(()))
ae 22
(67)
For the basis of the spinor part, we obtain lig i21, and for the basis of the vector part 61 e1e102 =1fge -2 Thus, the Fourier kernel e e-2riz(zE)) commutes with the spinor part, and anti-commutes with the vector part of a, both for Cl(2, (RR and C1(O,, (IR). We therefore proof the convolution theorem for the commuting spinor and the anti-commuting vector part a.
Theorem 2: 2D Clifford convolution theorem.
Let the field f R G² be multivector valued, the filter kg IR2 G2 be spinor valued, exist, then
and the filter k, IR2 - G2 be vector valued, and let F(k), F(r,,
Fif * k. (E) = FF() F (E) F(f (E) = -((,)())
where (i(k)(()) = F(k, | {- and Fi Firko = F(ke)(--5
Proof.
See fly)ka(y z)dy 22iia(1 dx -2)dy
Flf x ke)(E) (2
1
/ Na x) 2xbalais) dx dy
fwy)
Jz0
(2
Jg2
f(y)
ks ( 201 (2- dx 201
dy
(2
JT FT H())
f(y)e 200ia dy Fl(ke)(E)
2T
J82
=F(((E) (t()))(E).
(68)
34

---

Published as a conference paper at ICLR 2023
Fif +cF()) = (2x)² /
|
f(y)ko(y - x)dy -2 da
fw)
kin - ade -2 dx dy
)2x)²
J22
JR2
f(y)
Ki (x)e 2miz(u-a.8) dx
dy
(2x)²
R22
FF (Ro (E) 2220000 2712(011) [(k.)) (E), where ---
Jec S(U)E 2 dy
F(K))(EE
2
= (E) F(k
(69)
B.5.3 3D CLIFFORD FOURIER TRANSFORM
For C3(3, (R), analogous to Equation 58, the Clifford Fourier transform (Ebling & Scheuermann, 2005) and the respective inverse transform for multivector valued functions f R G3 and vectors x, 3 E TR3 are defined as:
=()))) (2T)3/2 Ja
f(z)e-2nistee -2xis dæ, Vg E R3,
(70)
[(0) -=)= = (2m)3//2
f(8)e2nis(a de, Vr E R
(71)
JB3
provided that the integrals exist. A multivector valued function f IR3 G3
= fo fiei + fee + faes + f12512 + fi3e13 f23023 + f1230123
(72)
can be expressed via the pseudoscalar i3 616203 as:
f (fo f12si5]
+ (fi + f2sia)e1
+ (f2 + faria)ez
+ (fa + fi2la)es,
(73)
We obtain a 3-dimensional Clifford Fourier transform by applying four standard Fourier transforms for the four dual pairs fo fu(x)) + f122s(z) fi = fx( + fza(z) fz = fz and which all can be treated
fs = f12 Consequently. f fxx can be understood as an element of C+. The 3D Clifford Fourier transform is the linear combination of four classical Fourier transforms:
complex-values signal for fi, fs R C.
35

---

Published as a conference paper at ICLR 2023
F(S(()) = (2
( 20ig(ar)) dæ,
1 (fo(t) + l12a(1)(s) e (ni(e) + (zz(z))ss)
(2x)3/2
(fa(z) fsa())(s) cs [f((z) + fia(z)is) )
2mial( do
+
e2
(folo) + f12s(1))s) 25is(m.E) dr
2//2
(((=) faa(z)is)
2mig(m)) de
(2T) (2/)
e1
(fa(z) + fai(z)
2nia(m de
(2m))//2
e2
(1o(x)) + f12(1))s)
2mig(ane) do
(2m)
JRS
1 F (fo(o)) + (aa(a))s)) (5) + ((i(() - (2s(r))ss) (5)
F (FF(x)) + (z) (5)
+
F [s(x)) + fre(a)is)
(E) (6)
(74)
Analogous to the 2-dimensional Clifford Fourier transform, we apply multivector weight ten- x x x point-wise. Fourier modes above cut-off frequencies
sors W E (G3)cxxomn
(STTNN 22
cma)) are set to zero. In doing we modify the Clifford Fourier modes 53
f(L) = F(F)()
= fo(E) + f((5)e + fa(E)e2 + fa(E)s + fia(5)e12 + fai (E)ea + fza(E)e23 + fiza(E)e123
(75)
via the geometric product. The Clifford Fourier modes follow naturally when combining the four parts of Equation 74 Finally, the residual connections used in FNO layers replaced by mul- tivector weight matrix realized as Clifford convolution, ideally Cla.o convolution layer. For other 3-dimensional Clifford algebras, the signs the dual pairs Equation 73 change accordingly.
B.5.4 3D CLIFFORD CONVOLUTION THEOREM
This theorem adapted from Ebling Scheuermann (2005)). First, again let's check the Clifford kernel commutes with the different parts of multivectors. We can write the product ae for every scalar E and multivector a E 3 as
ae a( cos(s) + bg sin()))
(76)
First, check again if the different basis vectors of the Fourier transforms of Equation 74 commute with the pseudoscalar iz:
lig ig1 V
é 61618ge3 -16 = 6262611 = i361 V
621g = 62616263 = -E1626263 = E1626362 = i362
e313 с391е9E3 -E1E362E3 c1626363 i3e3 V
(77)
In contrast to the 2-dimensional Clifford Fourier transform, now all four parts of the multivector of Equation 73 commute with ig. This holds for all 3-dimensional Clifford algebras.
36

---

Published as a conference paper at ICLR 2023
Theorem 3: 3D Clifford convolution theorem.
Let the field f IR' + GD be multivector valued, the filter ka R2 G be multivector valued, and F(S): [(f).F(Ke) [lka exist, then
F(f = = (E)
where Fi (K(a)((E) = F(ka)(-E)
Proof.
F(f x(f*ka)(E) (2T)3 Jma S fly)kalv z)dy 2 de (x)dy
J f(y)
JR3 ka z)ee (m. dx dy
(2
J f(y)
ka (a(D) 2010(1-2288 2.8) dr
dy
(2
1
f(y)e (1y)e-2ristu.))) dy Fl(kal(5)
(2Tr3/2
F(f)(E)) (k(ka)())
(78) L
B.5.5 IMPLEMENTATION OF CLIFFORD FOURIER LAYERS
implement a 2D Clifford Fourier layer by applying two standard Fourier transforms on the dual pairs of Equation 11. These dual pairs can be treated as complex valued inputs. Similarly, implement a 3D Clifford Fourier layer by applying four standard Fourier transforms on the dual pairs of c.g. Cl3.0 (Equation 37 Equation 40). Since Clifford convolution theorems hold both for the vector and the spinor parts and for the four dual pairs for C12,0 and Cla,0- respectively. we multiply the modes in the Fourier space using the geometric product. Finally, apply inverse Fourier transformation and resemble the multivectors in the spatial domain.
B.6 PSEUDOCODE
Algorithm 1 sketches the implementation of Clifford convolution, Algorithm of rotational Clifford convolution, and Algorithm 3 of Clifford Fourier layer.
1: function CLIFFORD 2DW))
[[
W[1
W[2]
-W3
W
W
[3[3]
W
2: kernel +
W
W
[
W[1
W[3
[
W[1]
W[0]
3: return kernel
4: function CLLFFORDCONV2IDWWW
5: kernel + CLITFORDEERNEL22(V))
input <- VIEW LAS. REALVECTOR(Z)
7: output +- CoNv2D(kemel, input)
return VIEW AS (ULTIVECTOR(Ou(put)
Algorithm I: Pseudocode for 2D Clifford convolution using Cl2.0-
37

---

Published as a conference paper at ICLR 2023
1: function CLJFFORDKERNEEI22eoor
8Q12 < W[1] W
3: 8Q13 - + W
4: 5423 + +
5:
sumsq WT0] +
W[1
W + W +c
6:
rot12 < /100/sss/umss sumsq
7:
rot WOO 2 sumsq
8:
ro <- W W 3
sumsq
9:
rol23 W W 2
sumsq
10:
rot21 - [ W
3
sumss
ro131 W 2 W
sumsq
W 0
W[1
WW[2] W[4](rot23 rot14)
-W[3] W[4](rot24 + rot13)
5
(1. - 8q23)
12:
kernel
5
(1o](r + ro)
W[4](1.0 2013) W[4] + rot12)
W[4] 1ol
W
[rot - rot1s)
W[4] sQ12)
13:
return kernel
14:
function CLJFFORDDCONV2Dee (W x)
kernel CLIFFORDEERNEL2D (W)
16:
input <- VIEW (.w._AS_.EEALVECTOR())
17:
output -- CoNv2D(kermel, input)
18:
return VIEW LS MULTIVECTOR
Algorithm 2: Pseudocode for 2D rotational Clifford convolution using Clo,2-
1: function CLIFFORSSPECTRA.COONDDDD x, m1, m2,
Fys Fy +- VIEW AS QUALSPARTS(e)
3: [(x)) < FFT2 x.)
Complex 2D FFT of vector part
f(x.)) + FFT2 (x.)
b Complex 2D FFT of scalar part
(x,) f(x,) ... m1,: m2]
f(z,) 1[ mi, -m2 :]
5:
f((x))
m1 m2
f(z,) m1 m72 :
Vector modes
((x)) + [(xa ..... mi, m2
[xx
m.. - :]
6:
((xa) ...., mh ;; m2
[xx)) ......... -m - :]_
Scalar modes
(()) (x.))++++++++++ (x,) (xx))
f (*xx).. b. Multivector Fourier modes
8:
f*(x) + (z)
D Geometric product in the Fourier space D Inverse 2D FFT of vector part
9:
f. + (=)
10:
22 (
Inverse 2D FFT of scalar part
11:
+ VIEW AS MULTIVECTOR( 22)
12:
return à
13:
function CLAFFORRDFFOURRRRRLLLYYYRRRDDDDD W. .)
14:
Ø1 CLIFOROSPPICTRALCCOVVWVYEEE m,, m2)
15:
<- VIEW REA.EEETRALLVECO(()))
16:
8/ CLIFFORDCONYWWW..c
17:
< VIEW AS MULTIVECTOR
18:
out <- ACTIVATION()) + y/2
19:
return out
20:
Algorithm 3: Pseudocode for 2D Clifford Fourier layer using Cla
38

---

Published as a conference paper at ICLR 2023
C
EXPERIMENTS
This appendix supports Section the main paper.
C.I Loss FUNCTION AND METRICS
We report the summed MSE (SMSE) loss defined as:
N. Nust
SSMSE
Σ
Σ
[u,(v.ty) - 0z fy)
(79)
No
yE (orZ2)
where u is the target, the model output, Niclds comprises scalar fields well individual vector field components, and N2, is the total number of spatial points. Equation 79 is used for training with N, 1, and further allows us define four metrics:
One-step loss where N₂- and Nields comprises all scalar and vector components.
Vector loss where N, and Nields comprises only vector components.
Scalar loss where N₂ and Nuelas comprises only the scalar field.
Rollout loss where N₂ = 5 and Niad, comprises all scalar and vector components.
For Maxwell's equation, electric and magnetic loss are defined analogously to the vector and the scalar loss for Navier-Stokes and shallow water experiments.
C.2 MODELS
We experiment with two architecture families: ResNet models (He et al., 2016) and Fourier Neural Operators (FNOs) (Li al., 2020). All baseline models are fine-tuned for all individual experiments with respect to number of blocks, number of channels, number of modes (FNO), learning rates, normalization and initialization procedures, and activation functions. The best models are reported, and for reported Clifford results each convolution layer is substituted with Clifford convolution, each Fourier layer with a Clifford Fourier layer, each normalization with Clifford normalization and each non-linearity with Clifford non- non-linearity. A Clifford non-linearity in this context is a the application of the corresponding default linearity to the different multivector components.
ResNet architectures. For Navier-Stokes and shallow water experiments, tectures with 8 residual blocks, each consisting of two convolution layers with 3 x kernels, shortcut connections, group normalization (Wu & He, 2018), and GeLU activation functions (Hendrycks & Gimpel, 2016). We further use two embedding and two output layers, i.c. the overall architectures could be classified Res-20 networks. In contrast to standard residual networks for image classifi- cation, we don't 't use any nown-projection techniques, e.g. convolution layers with strides larger than or via pooling layers. In contrast, the spatial resolution stays constant throughout the network. We therefore also use the same number of hidden channels throughout the network, that is 128 channels per layer. Overall this results roughly 2.4 million parameters. Increasing the number of residual blocks or the number of channels did not increase the performance significantly.
ResNet archi-
Clifford ResNet architectures. For every ResNet-based experiment, we replaced the fine-tuned ResNet architectures with two Clifford counterparts: each CNN layer is replaced with (i) Clifford CNN layer, and (ii) with a rotational Clifford CNN layer. To keep the number of weights similar, instead of 128 channels the resulting architectures have 64 multivector channels, resulting again in roughly 1.6 million floating point parameters. Additionally for both architectures, GeLU activation functions are replaced with Clifford GeLU activation functions, group normalization is replaced with Clifford group normalization, Using Clifford initialization techniques did not improve results.
Fourier Neural Operator architectures. For Havier-Stoker and shallow water experiments, we used 2-dimensional Fourier Neural Operators (FNOs) consisting of 8 FNO blocks, two embedding and two output layers. Each FNO block comprised a convolution path with a x 1 kernel and an FFT path. We used 16 Fourier modes (for x and y components) for wisn weight multiplication,
39

---

Published as a conference paper at ICLR 2023
and overall use 128 hidden channels. We used GeLU activation functions (Hendrycks & Gimpel, 2016). Additional shorteut connections or normalization techniques, such as batchnorm or group, norm did not improve performance, neither did larger numbers of hidden chamnels, more FNO blocks. Overall this resulted in roughly 140 million parameters for FNO based architectures.
For 3-dimensional Maxwell experiments, used 3-dimensional Fourier Neural Operators (FNOs) consisting of 4 FNO blocks, two embedding and two output layers. Each FNO block comprised kernel and an FFT path. We used Fourier modes (for x,
3D convolution path with a and components) for point-wise weight multiplication, and overall used 96 hidden channels. Interestingly, using more layers or more Fourier modes degraded performances. Similar the 2D experiments, we applied GeLU activation functions, and neither apply shortcut connections nor normalization techniques, such as batchnorm or groupnorms. Overall this resulted in roughly 65 million floating point parameters for FNO based architectures.
Clifford Fourier Neural Operator architectures. every FNO-based experiment, we replaced the fine-tuned FNO architectures with respective Clifford counterparts: each FNO layer is replaced by its Clifford counterpart. To keep the number of weights similar, instead of 128 channels the re- sulting architectures have 48 multivector channels, resulting in roughly the same number of parame- ters. Additionally, GeLU activation functions are replaced with Clifford GeLU activation functions. Using Clifford initialization techniques did not improve results.
For 3-dimensional Maxwell experiments, we replaced each 3D Fourier transform layer with a 3D Clifford Fourier layer and each 3D convolution with respective Clifford convolution. We also use 6 Fourier modes (for y. and components) for point-wise weight multiplication, and overall used 32 hidden multivector channels, which results in roughly the same number of parameters (55 millions). In contrast to 2-dimensional implementations, Clifford initialization techniques proved important for 3-dimensional architectures. Most notably, too large initial values of the weights of Clifford convolution layers hindered gradient flows through the Clifford Fourier operations.
C.3 TRAINING AND MODEL SELECTION.
We optimized models using the Adam optimizer (Kingma & Ba, 2014) with learning rates [10-1, 2. 10-4,5 10-4 for 50 epochs and minimized the summed mean squared error (SMSE) which is outlined in Equation 79. We used cosine annealing as learning rate scheduler (Loshchilov & Hutter, 2016) with a linear warmup. For baseline ResNet models, optimized number of layers, number of channels, and normalization procedures. We further tested different activation functions. For baseline FNO models, we optimized number of layers, number of channels, and number of Fourier modes. Larger numbers of layers or channels did not improve the performances for both ResNet and FNO models. For the respective Clifford counterparts, we exchanged convolution and Fourier layers by Clifford convolution and Clifford Fourier layers. We further used Clifford normalization schemes. We decreased the number of layers to obtain similar numbers of parameters. We could have optimized Clifford architectures slightly more by e.g. using different numbers of hidden layers than the baseline models did. However, this would (i) slightly be against the argument of having "plug- and play" replace layers, and (ii) would have added quite some computational overhead. Finally, we are quite confident that the used architectures are very close to the optimum for the current tasks.
Computational resources. All FNO and CFNO experiments used 4x16 GB NVIDIA V100 ma- chines for training. All ResNet and Clifford ResNet experiments used 8x32 GB NVIDIA V100 machines. Average training times varied between 3 and 48 h, depending task and number of trajectories. Clifford runs on average took twice as long to train for equivalent architectures and epochs.
40

---

Published as a conference paper at ICLR 2023
C.4 MAVIER-STOKER IN 2D
The incompressible Navier-Stokes equations are built upon momentum and mass conservation of fluids. Momentum conservation yields for the velocity flow field
dv ôt
U. Vu + 2V Vp f
(80)
where Ve the convection, v the viscosity, Vp the internal pressure and an external force. Convection is the rate of change of vector field along vector field (in this case along itself), viscosity the diffusion of vector field, i.e. the net movement form higher valued regions to lower concentration regions, μ is the viscosity coefficient. The incompressibility constrained yields mass conservation via
(81)
Additional to the velocity field v(z), we introduce scalar field s(x) representing scalar quantity that is being transported through the velocity field. For example, e might represent velocity of air inside room, and might represent concentration of smoke. Ás the vector field changes, the scalar field is transported along it, i.e. the scalar field is advected by the vector field. Similar to convection, advection is the transport of scalar field along a vector field:
ds dt
Vs.
(82)
We implement the 2D Navier-Stokes equation using &Flow (Holl al., 2020). Solutions are propagated where we solve for the pressure field and subtract its spatial gradients afterwards. Semi- Lagrangian advection (convection) is used for u, and MacCormack advection for 8. Additionally, we express the external buoyancy force in Equation 80 as force acting the scalar field. Solutions obtained using Boussinesq approximation (Kleinstreuer, 1997). which ignores density differences except where they appear in terms multiplied by the acceleration due gravity. The essence of the Boussinesq approximation is that the difference in inertia is negligible but gravity is sufficiently strong make the specific weight appreciably different between the two fluids.
Equation details. We obtain data for the 2D Navier-Stokes equations on grid with spatial reso- lution of 128 x 128 (Ar 0.25, Ay 0.25), and temporal resolution of At 1.5s. The equation is solved on closed domain with Dirichlet boundary conditions (v 0) for the velocity, and Neu- mann boundaries Øa de 0 for the scalar smoke field. The viscosity parameter is set to 0.01, and a buoyancy factor of (0, 0.5)" is used. The scalar field is initialized with random Gaussian noise fluctuations, and the velocity field is initialized to O. We the simulation for 21 and sample every 1.5s. Trajectories contain scalar and vector fields at different time points.
Results. Results are summarized in Figures 10, 9, and detailed in Table Figure 11 displays ex- amples of Savier-Stoker rollouts of scalar and vector fields obtained by Clifford Fourier surrogates, and contrasts them with ground truth trajectories. For ResNet-like architectures, we observe that both CResNet and CResNet... improve upon the ResNet baseline. Additionally, we observe that rollout losses are also lower for the two Clifford based architectures, which we attribute to better and more stable models that do not overfit to one-step predictions so easily. Lastly, while in prin- ciple CResNet and CResNetot based architectures are equally flexible, CResNetio ones in general perform better than CResNet ones. For FNO and respective Clifford Fourier based (CFNO) archi- tectures, the loss is in general much lower than for ResNet based architectures. CFNO architectures improve upon FNO architectures for all dataset sizes, and for one-step as well rollout losses.
"nttps:/ //gi wtt..tcoottummmts/////oo
41

---

Published as a conference paper at ICLR 2023
Rollout
One-step
-10-2
ResNet
CResNet
0.15
CResNelor
0.1
5-10-2
MSE
Scalar
Vector
-10-a
-10-2
15600
10400
5200
2080
15600
Num. Train Trajectories
Figure 9: Results on Navier-Stokes equations obtained by ResNet based architectures. Unrolled loss. one-step loss, scalar loss and vector loss reported for ResNet, CResNet, and CResNet architectures. Models are trained on training sets with increasing number of trajectories. ResNet based architectures have much higher loss than FNO based architectures in the low data regime, where possibly smearing and averaging operations are learned first.
42

---

Published as a conference paper at ICLR 2023
Rollout
One-step
-10-2
-10
FNO
CFNO
MSE
0
Scalar
Vector
-10m3
-10
10400
$200
2080
10400
5200
2080
Num. Train Trajectories
Figure 10: Results on Navier-Stokes equations obtained by Fourier based architectures. Rollout loss, one-step loss, scalar loss and vector loss are reported for FNO and CFNO architectures. Models are trained on three training sets with increasing number of trajectories.
Table 1: Model comparison on four different metrics for neural PDE surrogates which are trained on Navier-Stokes training datasets of varying size. Error bars are obtained by running experiments with three different initial seeds.
SMSE
METHOD
Trajs.
scalar
vector
onestep
rollout
ResNet
0. t 0.000 03
0.01255
0.000 08
0.01553 + 0.000
0.11362 0.000 48
CResNet
2080
0.005 66 + 0.000 62
0.022 52 0.002
84
0.02806 : 0.003 46
0.158 0.026 77
CResNet..
0.003 76 1: 0.000 28
0.014 : 0.001 16
0.017 0.001 43
0.106 0.004 76
ResNet
0.003 + 0.000 28
0.01431 0.001 02
0.017 67 + 0.001 38
0.13234
0.000
CResNet
5200
0.002 65 + 0.000
0.009 88 0.000 24
0.01250 + 0.000 22
0.099
0.000
CResNet...
0.002 34 +: 0.000 14
0.008 57 0.000 66
0.01087 + 0.000 74
0.09427 0.000
ResNet
0.003 21 + 0.000 04
0.01337 0.000 44
0.01653 + 0.000 48
0.138 02 0.002 23
CResNet
10400
0.003 15 + 0.000 06
0.011 62 0.000 19
0.014 73 + 0.000 18
0.106 71 0.002 46
CResNetau
0.002 01 + 0.000 20
0.007 19 0.000 74
0.009 17 + 0.000
0.100 0.002 29
ResNet
0.003 42 + 0.000
0.013
0.000 79
0.017 16 0.000
0.130 0.003
CResNet
15600
0.002 85 + 0.000 19
0.010 0.00051
0.01357 + 0.000 63
0.103
0.002
CResNetau
0.002 + 0.000
0.007 36 0.000 69
0.009 38 + 0.000 87
0.097
0.001 39
FNO
2080
0.003 18 + 0.000 21
0.006 13 + 0.000 44
0.009 31 + 0.000 64
0.04281 + 0.003 00
CFNO
0.002 66 + 0.000 02
0.004 84 + 0.000 06
0.007 49 + 0.000 08
0.03461 t 0.00033
FNO
$200
0.002 + 0.000 04
0.003
0.000
0.005 36 0.000
0.026 0.000
CFNO
0.001 89 + 0.000
0.00293 0.000
0.004 82 + 0.000 03
0.024 0.000
FNO
10400
0.001 56 + 0.000 03
0.002 20 + 0.000 07
0.003 75 + 0.000 10
0.020
0.000-12 42
CENO
0.001 48 + 0.000 01
0.002 05 + 0.000 01
0.003 53 + 0.
0.018 86 0.00006
43

---

Published as a conference paper at ICLR 2023
Prediction
Target
Prediction
Target
Scalar field
Prediction
Target
Prediction
Target
(b) Vector field
Figure 11: Example rollouts of the scalar and vector field of the Navier- Stokes experiments, obtained by a Clifford Fourier PDE surrogate and the ground truth.
44

---

0 Published as a conference paper at ICLR 2023
C.5 SHALLOW WATER EQUATIONS.
The shallow water equations (Vreugdenhil, 1994) describe a thin layer of fluid of constant density in hydrostatic balance, bounded from below by the bottom topography and from above by free surface. For example, the deep water propagation of a tsunami can be described by the shallow water equations, and so can simple weather model. The shallow water equations read:
Ve'
On
01
U
dr
Uy
dy
0, + 9 dx
0vg
Bvy
On
+ U.
0g
9 Oy
dL
dr
Oy
On θ
0
+
(17 + h)) b
(17 h)uy Dy
0
(83)
dx
where Er is the velocity in the x-direction, or zonal velocity, By is the velocity in the y-direction, or meridional velocity, 9 is the acceleration due to gravity, 17 y) is the vertical displacement of free surface, which subsequently is used derive pressure fields; h(,, y) is the topography of the earth's surface. We modify the implementation in SpeedyWeather. j1" " (Klöwer et al., 2022) to further randomize initial conditions to generate our dataset. SpeedyWeathez ji combines the shallow water equations with spherical harmonics for the linear terms and Gaussian grid for the non-linear terms with the appropriate spectral transforms. scheme with a Robert and William's 's filter to dampen the computational modes and achieve 3rd oder accuracy. SpeedyWeat her. j1 is based on the atmospheric general circulation model SPEEDY in Fortran (Molteni, 2003; Kucharski et al., 2013).
internally uses a leapfrog time
Equation details. We obtain data for the 2D shallow water equations on grid with spatial res- olution of 192 96 (Ar 1.875%, Ay 3.75)), and temporal resolution of At = 6.. The equation is solved on a closed domain with periodic boundary conditions. We rollout the simulation for 20 days and sample every h. Here 20 days is of course the actual simulation time but rather the simulated time. Trajectories contain scalar pressure and wind vector fields at 84 different time points.
Results. Results are summarized in Figures 12, 13, 14, and detailed in Tables 2, 3. Figure 15 displays examples of shallow water equations rollouts of scalar pressure and vector wind fields obtained by Clifford Fourier surrogate models, and contrasts them with ground truth trajectories. The predictions are fairly indistinguishable from ground truth trajectories. We observe similar results than for the Navier-Stokes experiments. However, performance differences between baseline and Clifford architectures are even more pronounced, which we attribute to the stronger coupling of the scalar and the vector fields. For ResNet-like architectures, CResNet and CResNNet,or improve upon the ResNet baseline, rollout losses are much lower for the two Clifford based architectures, and CResNet,a based architectures general perform better than CResNet based ones. For Fourier based architectures, the loss is in general much lower than for ResNet based architectures (a training set size of 56 trajectories yields similar (C)FNO test set performance than training set size of 896 trajectories for ResNet based architectures). CFNO architectures improve upon FNO architectures for all dataset sizes, and for one-step as well as rollout losses, which especially pronounced for low number of training trajectories.
Phttps:/ /i http://mmmmmmnmmpppepeeeeeatterr
45

---

Published as a conference paper at ICLR 2023
Rollout
One-step
-10-2
ENO
0.8
CENO
0.4
0.2
MSE
Scalar
Vector
-10-2
-10
1
896
448
192
56
896
448
192
56
Num. Train Trajectories
Figure 12: Results on the shallow water equations obtained by Fourier based architectures using two timestep history input. Unrolled loss, one-step loss, scalar loss and vector loss are reported for FNO and CFNO architectures. Models are trained on three training sets with increasing number of trajectories.
46

---

Published as a conference paper at ICLR 2023
Rollout
One-step
-10-2
FNO
0.6
CFNO
0.4
0.2
MSE
Scalar
Vector
-10-2
-10-2
896
448
192
56
896
448
192
56
Num. Train Trajectories
Figure 13: Results on the shallow water equations obtained by Fourier based architectures using four timestep history input. Rollout loss, one-step loss, scalar loss and vector loss are reported for FNO and CFNO architectures. Models are trained on three training sets with increasing number of trajectories.
47

---

Published as a conference paper at ICLR 2023
Rollout
One-step
2
ResNet CResNet
CResNelor
0.1
1
MSE
Scalar
Vector
-10-2
-10
6
0
2
0
2048
1792
896
448
192
Num. Train Trajectories
Figure 14: Results on the shallow water equations obtained by ResNet based architectures using two timestep history input. Rollout loss, one-step loss, scalar loss and vector loss reported for ResNet, CResNet, and CResNet,.t architectures. Models are trained on training sets with increasing number of trajectories. ResNet based architectures have much higher loss than FNO based archi- tectures in the low data regime, where possibly smearing and averaging operations are learned first.
48

---

Published as a conference paper at ICLR 2023
Prediction
Target
Prediction
Target
(a) Scalar field
Prediction
Target
Prediction
Target
(b) Vector field (z-component)
Prediction
Target
Prediction
Target
(c) Vector field (g-component)
Figure 15: Example rollouts of the scalar and vector field of the shallow water experiments, obtained by a Clifford Fourier PDE surrogate (top) and the ground truth (bottom).
49

---

Published as a conference paper at ICLR 2023
Table 2: Model comparison on four different metries for neural PDE surrogates which are trained on the shallow water equations training datasets of varying size. Results are obtained by using a two timestep history input. Error bars are obtained by running experiments with three different initial seeds.
SMSE
METHOD
Trajs.
scalar
vector
onestep
rollout
ResNet
0.0240 + 0.0002
0.0421 = 0.0010
0.0661 + 0.0011
1.1195 + 0.0197
CResNet
192
0.0617 + 0.0016
0.0823 + 0.0027
0.1440 = 0.0042
2.0423 + 0.0494
CR
0.0319 + 0.0003
0.0576 = 0.0005
0.0894 = 0.0007
1.4756 + 0.0044
ResNet
0.0140 + 0.0003
0.0245 + 0.0007
0.0385 + 0.0010
0.7083 + 0.0119
CResNet
448
0.0238 + 0.0007
0.0448 = 0.0023
0.0685 = 0.0030
1.1727 + 0.0483
CResNetao
0.0114 0.0001
0.0221 = 0.0001
0.0335 = 0.0002
0.6127 = 0.0018
ResNet
0.0086
0.0156 = 0.0003
0.0242 = 0.0003
0.4904 = 0.0080
CResNet
896
0.0095 = 0.0002
0.0183 = 0.0004
0,0278 = 0.0006
0.5247 + 0.0101
CResNetaa
0.0055
0.0106 = 0.0001
0.0161 + 0.0001
0.3096 + 0.0010
ResNet
0.0061 + 0.0002
0.0123 + 0.0009
0.0184 + 0.0010
0.4780 + 0.0062
CResNet
1792
0.0039
0.0071
0.0111 + 0.0001
0.2842 + 0.0067
CResNet,o
0.0025
0.0044
0.0069
0.2370
ResNet
0.0060 + 0.0002
0.0121 + 0.0003
0.0181 + 0.0005
0.4480 + 0.0058
CResNet
2048
0.0039 + 0.0001
0.0072 + 0.0002
0.0111 + 0.0003
0.2816 + 0.0065
CResNet,ot
0.0028 + 0.0005
0.0480 = 0.0006
0.0075 = 0.0011
0.2164 + 0.0070
FNO
56
0.0271 + 0.0016
0.0345 + 0.0007
0.0616 + 0.0022
0.8032 + 0.0043
CFNO
0.0071 + 0.0003
0.0177 + 0.0004
0.0250 + 0.0007
0.4323 + 0.0046
FNO
192
0.0021 + 0.0002
0.0057 + 0.0001
0.0077 + 0.0003
0.1444 + 0.0026
CENO
0.0012
0.0040
0.0001
0.0053 + 0.0001
0.0941 0.0021
FNO
448
0.0007 + 0.0001
0.0026
0.0034 + 0.0001
0.0651 + 0.0014
CFNO
0.0005
0.0020
0.0026 = 0.0001
0.0455 + 0.0009
FNO
896
0.0004
0.0016
0.0020 = 0.0001
0.0404 + 0.0005
CFNO
0.0003
0.0013
0.0017 = 0.0001
0.0315 + 0.0004
Table 3: Model comparison on four different metrics for neural PDE surrogates which are trained on the shallow water equations training datasets of varying size. Results are obtained by using a four timestep history input. Error bars are obtained by running experiments with three different initial seeds.
SMSE
METHOD
Trajs.
scalar
vector
onestep
rollout
FNO
56
0.0276 + 0.0017
0.0388 + 0.0023
0.0663 + 0.0038
0.6821 + 0.0379
CFNO
0.0093 + 0.0003
0.0252 0.0005
0.0345 + 0.0009
0.4357 + 0.0056
FNO
192
0.0033 + 0.0007
0.0069 0.0009
0.0102 + 0.0015
0.1612 + 0.0057
CENO
0.0015 + 0.0001
0.0050 + 0.0003
0.0065 + 0.0003
0.1023 + 0.0026
FNO
448
0.0009 = 0.0001
0.0023 = 0.0002
0.0032 + 0.0003
0.0687 + 0.0023
CFNO
0.0010 + 0.0006
0.0039 + 0.0027
0.0050 + 0.0033
0.1156 + 0.0913
FNO
896
0.0004 + 0.0001
0.0012 + 0.0001
0.0016 + 0.0001
0.0436 + 0.0011
CENO
0.0003
0.0012 0.0001
0.0015 = 0.0001
0.0353 + 0.0010
50

---

Published as a conference paper at ICLR 2023
C.6 MAXWELL'S EQUATIONS IN MATTER IN 3D.
Electromagnetic simulations play a critical role in understanding light-matter interaction and de- signing optical elements. Neural networks have been already successful applied in inverse-designing photonic structures (Ma et al., 202 lb; Lim & Psaltis, 2022).
Maxwell's equations in matter read:
V D
Gauss's law
(84)
V B =
Gauss's law for magnetism
(85)
E' 0B
Faraday's s law of induction
(86)
0D
V
H
Öt +
Ampère's circuital law
(87)
In isotropic media, the displacement field D is related the electrical field via D cuc, E, where 6o is the permittivity of free space and e,, is the permittivity of the media. Similarly, the magnetization field H in isotropic media is related to the magnetic field B via H Pojt, where μi is the permeability of free space and is the permeability of the media. Lastly, j is the electric current density and p the total electric charge density.
We propagate the solution of Maxwell's equation in matter using finiteddifference time-domain ' where the discretized Maxwell's equations are solved leapfrog manner. First, the electric field vector components in volume of space are solved at given instant in time. Second, the magnetic field vector components in the same spatial volume are solved the next instant in time.
Equation details. of 32 x x32 x 32 (Ax Ay A: = 5-) and temporal resolution of At 50s. We randomly place 18 (6 in the x-y plane, in the x- plane, 6 in the y- plane) different light sources outside cube which emit light with different amplitude and different phase shifts, causing the resulting D and H fields to interfere with each other. The wavelength of the emitted light is 10-$m. The equation is solved on a closed domain with periodic boundary conditions. We run the simulation for 400 and sample data every s0s. Trajectories contain displacement D and the magnetization field H components. Exemplary trajectories are shown Figure 16.
We ohtain data for the 3D Maxwell's equations on grid with spatial resolution
Dg reslame
De Seofame
Figure 16:
An example propagation of the displacement field / and the magnetization field H. Shown are the field components for arbitrary slice of the x- plane.
Results. Results summarized in Figure 17 and detailed in Table 4.
"https::/ www.bi/oo/o//aoo////////o//o
51

---

Published as a conference paper at ICLR 2023
Rollout
One-step
-10
-10
1.5
CENO
2
MSE
D
H
-10-3
-10-3
3
2
2
0
6400
3200
1280
640
6400
3200
1280
6440
Num. Train Trajectorics
Figure 17: Results on the Maxwell equations obtained by Fourier based architectures using two timestep history input. Rollout loss, one-step loss, displacement field D loss, and magnetization field loss are reported for FNO and CFNO architectures. Models are trained on four training sets with increasing number of trajectories.
Table 4: Model comparison on four different metrics for neural PDE surrogates which are trained on the Maxwell equations training datasets of varying size. Results obtained by using two timestep history input. Error bars are obtained by running experiments with three different initial seeds.
SMSE
МETHOD
Trajs.
D
H
onestep
rollout
FNO
640
0.0030 + 0.0006
0.002
0.000 50
0.0054 0.0011
0.0186 + 0.0083
CFNO
0.0006 + 0.0001
0.000 72 + 0.000 10
0.0013 + 0.0002
0.0054 + 0.0023
FNO
1280
0.0010 + 0.0002
0.000 85 + 0.000 20
0.0019 + 0.0004
0.0068 + 0.0036
CENO
0.0003 + 0.0001
0.000 41 + 0.000 10
0.0007 0.0002
0.0029 + 0.0016
FNO
3200
0.0003 + 0.0001
0.000 25 + 0.000 10
0.0005 + 0.0001
0.0020 + 0.0011
CENO
0.0002
0.000 20 0.000 10
0.0004
0.0001
0.0015 + 0.0009
FNO
6400
0.0001
0.000 09
0.0002
0.0008 0.0004
CENO
0.0001
0.000 09
0.0002
0.0007 0.0004
52

---

Published as a conference paper at ICLR 2023
D RELATED WORK
This appendix supports detailed discussions of how our work relates to complex and quaternion neural networks, to work on Clifford algebras and Clifford Fourier transforms computer vision, to Fourier Neural Operators, equivariant neural networks and geometric deep learning approaches, to neural operator learning and neural PDE surrogates.
Clifford (geometric) algebra and Clifford Fourier transform. (Real) Clifford algebras (also known as geometric algebras), as an extension of elementary algebra to work with geometrical ob- jects such as vectors are extensively discussed in Suter (2003); Hestenes (2003); Dorst et al. (2010); Hestenes (2012); Renaud (2020). Compared to other formalisms for manipulating geometric ob- jects, Clifford algebras are tailored towards vector manipulation of objects of different dimensions. Hypercomplex and quaternion Fourier transforms are extensively discussed in Ell (1992; 1993); Ell & Sangwine (2006); Ell et al. (2014). This work heavily builds on the concepts introduced in Ebling & &Shheuermann (2003; 2005); Ebling (2006). Comprehensive summaries of Clifford and quaternion Fourier transforms can be found in Hitzer Sangwine (2013); Brackx et al. (2013); Hitzer (2021). Clifford algebras and Clifford Fourier transforms already deployed to solve PDEs numerically in Alfarraj & Wei (2022). More precisely, the Hifford-FFurrce transform is used to solve the mode decomposition process in PDE transforms.
Clifford neural networks. Neural networks in the Clifford domain were proposed already in 1994 by Pearson& & Bisset (1994), and later by Pearson (2003). These works put the emphasis on the geo- metric perceptron (Melnyk et al., 2021), i.e. how to recast vanilla multilayer perceptrons (MLPs) as Clifford MLPs. Similarly, (Hoffmaan nions to set of alternative algebras. Besides Clifford MLPs, Clifford algebras have been used in recurrent neural networks (RNNs) (Kuroe, 2011), and have been used to formulate quantum neu- ral networks (Trindade et al., 2022). Their applicability to neural computing has been studied in general (Buchholz & Sommer, 2001; Buchholz, 2005), exploring global exponential stabilitics of Clifford MLPs with varying delays and impulsive effects. Probably the most related wors are: (i) Zang et al. (2022) who build geometric algebra convolution networks to process spatial and tem- poral data of 3D traffic data. Multidimensional traffic parameters
al., 2020) generalized from complex numbers and quater-
encoded as multivectors which allows to model correlation between traffic data in both spatial and temporal domains. (ii) Spellings (2021) who build rotation- and sermutation-equivariaa graph network architectures based on ge- ometric algebra products of node features. Higher order information is built from available node inputs.
In contrast to previous works, we are the first to introduce the multivector viewpoint of field compo- nents which allows us effectively connect Clifford neural layers with the geometric structure of the input data. We further connect neural Clifford convolutions on multivectors with various works on complex numbers and quaternions. We are further the first to introduce neural Clifford Fourier transforms.
Complex and quaternion neural networks. Trabelsi et al. (2017) introduced the key compo- nents for complex-valued deep neural networks. More precisely, they introduced convolutional (Le- Cun et al., 1998) feed-forward and convolutional LSTM (Shi et al., 2015; Hochreiter & Schmid- huber, 1997) networks, together with complex batch-normalization, and complex weight initial- ization strategies. Quaternions are a natural extension of complex neural networks. Already in classical computer vision, quaternions as hypercomplex convolution (Sangwine & Ell, 2000) and hypercomplex correlation (Moxey et al., 2003) techniques were introduced for color image pro- cessing. Quaternion based deep learning architectures are a natural extension of complex neural networks. In quaternion neural networks (Zhu et al., 2018; Parcollet et al., 2018a; Gaudet & Maida, 2018; Parcollet et al., 2018b; 2019; 2020; Nguyen et al., 2021; Moya-Sånchez et al., 2021), con- cepts such as complex convolution, complex batchnorm, and complex initialization are transfered from the complex numbers C, which are algebra-isomorph to CI(O, 1) to CT(O, (RR) which is gegebra-isomorph the quaternions H. Although Hoffmann et al. (2020) generalized these from complex numbers and quaternions to a set of alternative algebras, their tasks did not really leverage any multivector structure in data.
53

---

Published as a conference paper at ICLR 2023
Fourier Neural Operators. Fourier Neural Operators (FNOs) (Li et al., 2020) have had tremen- dous impact towards improving neural PDE solver surrogates. Efficient implementations of FNO layers come
neural networks (PINO) (Li et al., 2021b), as U-shaped network architectures (UNO) (Rahman et al., 2022b), as spectral surrogate for vision transformer archi- tectures (Rao et al., 2021; Guibas et al., 2021), Markov neural operators (MNO) for chaotic systems (MNO) (Li et al., 2021a), and as generative adversarial neural operators (GANOs) (Rah- man et al., 2022a). Applications range from weather forecasting (Pathak et al., 2022), CO2-water multiphase problems (Wen et al., 2022), multiscale method for crystal plasticity (Liu et al., 2022), seismic wave propagation (Yang et al., 2021). photoaccustic wave propagation (Guan et al., 2021), PDE-constrained control problems (Hwang et al., 2022), and for thermochemical curing of com- posites (Chen et al., 2021). Recently, FNOs have been successfully applied to PDEs on general geometries (Li et al., 2022b). Furthermore, universal approximation and error bounds have been studied for FNOs (Kovachki et al., 2021).
Neural PDE solvers/surrogates.
The intersection of PDE solving, deep learning. fluid dynamics, and weather forecasting has developed into a very active hub of research lately (Thuerey et al., 2021 1). We roughly group recent approaches to learn neural PDE surrogates and neural PDE solvers into three categories:
hybrid approaches, where neural networks augment numerical solvers or replace parts of numerical solvers;
(ii) direct approackes.
(a) where the mapping from an initial state to a solution learned, i.e. the solution function of the underlying PDE is approximated:
(b) where the mapping from an initial state final state of an underlying PDE, i.c. the solution operator is learned, idcally as mapping between function spaces to be able to generalize across c.g. parameters.
Ad (i): Neural networks augment numerical solvers by learning data-driven discretizations for PDEs (Bar-Sinai et al., 2019) or by controlling learned approximations inside the calculation of standard numerical solver used for computational fluid dynamics (Kochkov et al., 2021). In Green- feld et al. (2019), a prolongation is learned which maps from discretized PDE solutions to multigrid solutions, Hsich et al. (2019) learn to modify the updates of an existing solver, Praditia et al. (2021) adopt the numerical structure of Finite Volume Methods (FVMs), and Um et al. (2020) learn cor- rection function of conventional PDE solvers to improve accuracy. All these approaches are hybrid approaches (Garcia Satorras et al., 2019), where the computational graph of the solver is preserved and heuristically-chosen parameters are predicted with a neural network. A different flavor of hy- brid approaches can be assigned to the works of Sanchez-Gonzalez et al. (2020); Pfaff et al. (2020); Mayr et al. (2021) who predict accelerations of particles/meshes to numerical update the respective positions. Finally, PauliNet (Hermann et al., 2020) and FermiNet (Pfau et al., 2020) approximate wave-functions of many-electron systems, and thus replace the hand-crafted ansatz which is con- ventionally used in variational quantum Monte Carlo methods.
Ad (ii.a): Sirignano & Spiliopoulos (2018); Han et al. (2018) approximate the solution of high-dimensional Black-Scholes and Hamilton-Jcoobi-elllne
equations, respectively. Physics- informed neural networks (PINNs) (Raissi et al., 2019) embed the underlying physics in the training process, and can be used to solve both forward (Jin et al., 2021) as well as backward (Raissi al., 2020) dynamics. Zubov et al. (2021) allow automating many of these aspects under single coherent interface.
Ad (ii.b): Guo et al. (2016) learned a surrogate CNN-based model to approximate steady-state flow field predictions, similarly Bhatnagar et al. (2019) trained a surrogate CNN-based model to predict solutions for unseen flow conditions and geometries, and Zhu & Zabaras (2018) used Baysian CNNs for surrogate PDE modeling and uncertainty quantification. Fourier Neural Operators (FNOs) (Li et al., 2020) proposed the mapping from parameter space to solution spaces, and had tremendous impact towards improving neural PDE solver surrogates. In parallel, Lu et al. (2021) introduced DeepONet, which learns mappings between function spaces, and was successfully applied to many parametric ODEs and PDEs. Both, FNOs and DeepONets have been combined with PINNs and trained in physics-informed style (Li et al., 2022b; Wang et al., 2021). A comprehensive compar-
54

---

Published as a conference paper at ICLR 2023
ison of these two neural operator approaches is done by Lu et al. (2022). Other directions include the modeling of PDE solution operators via latent space models, transformers, and graph neural networks (GNNs). Wu et al. (2022) present the modeling of the systems dynamics in latent space with fixed dimension where the latent modeling is done via MLPs, and the encoding and decod- ing via CNNs, which can also be replaced by graph neural networks (GNNs). Cao (2021) propose the Galerkin transformer, a simple attention based operator learning method without softmax nor- malization, LOCA (Learning Operators with Coupled Attention) (Kissas et al., 2022) maps the input functions to a finite set of features and attends to them by output query locations, and Li et al. (2022a) propose a transformer which provides a flexible way to implicitly exploit the patterns within inputs. Brandstetter et al. (2022b) formulated a message passing neural network approach that rep- resentationally contains several conventional numerical PDE solving schemes. Further GNN based approaches are Lötzsch et al. (2022) who learn the operator for boundary value problems on finite element method (FEM) (Brenner et al., 2008) ground truth data, and Lienen& & Günnemann (2022) who derive their GNN models from FEM in principled way.
A practical use case for neural PDE surrogates is replacing expensive classical PDE solvers. There is however major chicken-and-ege problem here (Brandstetter et al., 2022a; Shi et al., 2022): obtaining high quality ground truth training data for neural PDE surrogates often requires using these expensive solvers. Minimizing this data requirement is beginning to be approached in recent works. Geneva & Zabaras (2020); Wandel et al. (2020; 2022) achieve "data-free" training in various settings. "Data-free" refers to the self-supervised training steps, which are done without ground truth data. The current state-of-art generic approach is introduced in Shi et al. (2022) as the mean squared residual (MSR) loss constructed by the discretized PDE itself. However, for e.g. generating realistic initial conditions numerical solvers are still needed. Pestourie et al. (2021) identify how incorporating limited physical knowledge in the form of a low-fidelity "coarse" solver can allow training PDE surrogate models with order of magnitude less data. Another direction to improve data efficiency is by exploiting the Lie point symmetries of the underlying PDEs, either via data augmentation (Brandstetter et al., 2022a) by building equivariant PDE surrogates (Wang et al., 2020b). Our current work in a way also improves data efficiency by capturing the inductive bias appropriate for multivector fields. Overall we believe hybrids of such approaches going to be necessary for making neural PDE surrogates of practical use in many domains.
Neural PDE surrogates for fluid flow and weather forecasting applications gaining momentum. In weather forecasting, Pathak al. (2022) introduced FourCastNet as high-resolution weather mod- eling built on Adaptive Fourier Neural Operators (Guibas et al., 2021), Keisler (2022) successfully applied a graph neural network based approach to weather forecasting, Rasp & Thuerey (2021) achieved data-driven medium-rang -aang weather prediction with a ResNet which was pretrained on cli- mate simulations, Weyn et al. (2020) use CNNs on cubed sphere for global weather prediction, Weyn et al. (2021) forecast weather sub-seasonally with a large ensemble of deep-learning weather prediction models, Arcomano et al. (2020) build a reservoir computing- baaeed, low-resolution, global prediction model, and MetNet (Sønderby et al., 2020) takes as input radar and satellite data to fore- cast probabilistic precipitation maps. Finally, data assimilation is improved by deep learning tech- niques in Frerix et al. (2021) and Maulik et al. (2022). Similarly, in fluid dynamics, Ma et al. (2021a) applied U-Nets (Ronneberger et al., 2015) to achieve physics-driven learning of steady Navier-Stokes equations, Stachenfeld et al. (2021) learned coarse models for turbulence simulations, TF-Net (Wang al., 2020a) introduced domain-specific variations of U-Nets along with trainable spectral filters in a coupled model of deynolds-averagee Navier-Stokes and Large Eddy Simulation.
This exhaustive list of neural PDE solver surrogates shows that many of the architectures are based on convolutional or Fourier layers. For these two, Clifford layers are applicable as drop-in replace- ment in almost all cases. For graph neural network and attention based architectures, we leave the implementation of respective Clifford counterparts to future work.
Geometric deep learning. The core idea of geometric deep learning (Bronstein et al., 2017; 2021) is to exploit underlying low-dimensionality and structure of the physical world, in order to design deep learning models which can better learn in high dimensional spaces. Incorporating underlying symmetries would be one way to achieve this. If done correctly, it can drastically shrink the search space, which has proven to be quite successful in multiple scenarios. The most obvious examples are CNNs (Fukushima & Miyake, 1982; LeCun et al., 1998), where the convolution operation com- mutes with the shift operator, and thus provides way to equip layers and subsequently networks
55

---

Published as a conference paper at ICLR 2023
with translation equivariant operations. Group convolution networks (Cohen & Welling, 2016a; Kondor & Trivedi, 2018; Cohen et al., 2019) generalize equivariant layers beyond translations, i.c provide concept of how to build general layers that are equivariant to broader range of groups, such as rotation groups. An appealing way of how to build such group equivariant layers is via so-called steerable basis functions (Hel-Or & Teo, 1998), which allow write transformation by specific groups as a linear combination of fixed, fimite set of basis functions. This concept leads to steerable group convolution approaches (Cohen & Welling, 2016b; Worrall et al., 2017). Two con- crete examples are: (i) circular harmonics, which are respective basis functions for building layers that are equivariant to the group SO(2), the rotation group in 2 dimensions (Worrall et al., 2017; Weiler & Cesa, 2019); (ii) spherical harmonics, which are respective basis functions for building layers that are equivariant to the group SO(3,, the rotation group in dimensions (Weiler et al., 2018; Geiger & Smidt, 2022; Brandstetter et al., 2021). I). The similarity to multivector fields becomes more obvious we have closer look at spherical harmonics, which are defined as homogeneous polynomials of degree I, where the / case corresponds to scalars, the / case to vectors, and I 2 2 to higher order objects. Finally, Jenner & Weiler (2021) built steerable PDE operators such as curl divergence equivariant neural network components.
Grouped convolution. In their seminal work, Krizhevsky al. (2012) introduced filter grouping. which allowed them to reduce the parameters in CNNs. The respective grouped convolutions (not be confused with group convolutions) divide the filter maps channel dimension, the channel dimension most of the time increases strongly for deeper layers, and thus dominates the parameter count. Subsequent work showed that is beneficial additionally shuffle the channels for each filter group (Zhang al., 2018), and to adaptively recalibrate channel-wise feature responses (Hu et al., 2018). All these approaches can seen the wider spectrum of effective model scaling (Tan & Le, 2019; Sandler et al., 2018).
Clifford convolutions in contrast do not have groupings in the channel dimensions, but instead group together elements multivectors. In Clifford convolution, the Clifford kernel is therefore a constrained object where weight blocks appear multiple times (due to the nature the geometric product). Thus, Clifford convolutions all tricks of effective model scaling could in principle be applied on top of Clifford convolutions. Findings from Hoffmann al. (2020) with respect to higher compute density of alternative algebras are applicable to our work as well.
more parameter efficient than standard convolutions, and
E GLOSSARY
This short appendix summarizes notations used throughout the paper (Table 5), and contrasts the most fundamental concepts which arise when using Clifford algebras.
Table 5: Notations used throughout the paper.
Notation
Meaning
E1, e2, E3
Basis vectors of the generating vector space of the Clifford algebra. Wedge (outer) product of basis vectors e; and ej. Inner product of basis vectors e; and Ej. Basis bivectors of the vector space of the Clifford algebra. Basis trivector of the vector space of the Clifford algebra. Pseudoscalar for Clifford algebras of grade 2.
6; Aej
6 ej ee),e))
6162, Cyc1, 6263
e1e2
i2 e162
ig e1e263
Pseudoscalar for Clifford algebras of grade 3. Euclidean vector E I. wedge (outer) product of Euclidean vectors x and y. Inner product of vectors and y. Multivector.
x
x
x v.y = (ar, v)
a
ab
Geometric product of multivectors a and b.
i.j.k
Base elements of quaternions.
56

---

Published as a conference paper at ICLR 2023
Geometric, Exterior, and Clifford algebras. A geometric algebra is a Clifford algebra of the real numbers. Since we are only using (R), (I), and Cl3,o(R), we are effectively working with geometric algebras. The exterior or Grassmann algebra is built up from the same concepts of scalars, vectors, bivectors, exterior algebra has zero quadratic form (all base vectors square zero). Clifford algebras are generalization thereof with nonzero quadratic forms.
k-vectors, but only exterior (wedge) products exist. Therefore, the
Complex numbers, quaternions, hypercomplex numbers. Hypercomplex numbers are ele- ments of finite-dimensional algebras over the real numbers that are unital, i.e. contain multi- plicative identity element, but not necessarily associative or commutative. Elements are generated for a basis . such that e are all hypercomplex numbers which can be characterized by different Clifford algebras. The bivec- tor, trivector (and higher objects) of the Clifford algebras directly translate into basis elements of the respective algebras. For example, quaternions (which are of the form + bi ej dk, where 12 f k22 -) are isomorphic to the Clifford algebra 2 where the basis element e1, e2, and e2 directly translate to i, J. k.
0, Complex numbers, quaternions, octonions
Spinor. Spinors arise naturally in discussions of the Lorentz group, the group to describe trans- formations in special relativity. One could say that spinor is the most basic sort of mathematical object that can be sorentz-transfonne like quantity in which rotations and Lorentz boosts (relativistic translations) are built into the overall formalism. More generally, spinors are elements of complex vector spaces that can be associated with Euclidean vector spaces. However, unlike vectors, spinors transform to their negative when the space rotated by 360". In this work, the subalgebra (2, 0) (R). spanned by even-graded basis elements of Cl2,0 (IR), i.e. 1 and e1e2, determines the space of spinors via linear combinations of and C1 is thus isomorphic to the field of complex numbers C. Most notably, spinors of Cl2,0 commute with the Fourier kernel, whereas vectors do not. For a detailed introduction spinors recommend Steane (2013), and the comprehensive physics book of Schwichtenberg (2015).
In its essence, spinor complex two-component vector-
Pseudoscalar. A pseudoscalar- unlike a scalar changes sign when you invert the coordinate axis. The easiest example of pseudoscalar is the scalar triplet product of three arbitrary vectors of an Euclidean vector space x, y, z E IR" with inner product (-.-). The scalar triplet product becomes negative for any parity inversion, i.e. swapping any two of the three operands: (y) x z) -x (z x y) = y (xr x z) - x x))
Scalar field, vector field. A field is any (physical) quantity which takes different values different points in space (space-time).
scalar field map ID - R, where D C R". A vector field map D IR", where ID C R" For example, a vector field in space. For an interesting history of the evolution of the concept of fields in physics we recommend Mirowski (1991); McMullin (2002). Table list various important vector and scalar fields for comparison.
results vector field in plane, and n 3 in
57

---

Published as a conference paper at ICLR 2023
Table 6: Examples of various vector and scalar fields. Vector fields ascribe a vector each point in space, c.g. force, electric current (stream of charged particles), or velocity. Scalar fields on the other hand collate each field point with scalar value such temperature.
Example
Field quantity
Type
Coordinates
Gravitational field (strength)
Force per unit mass (N/kg)
Vector
R + R
Electric field (strength)
Force per unit electric charge (N/C) Electric current per meter (A/m)
Vector
R R3
Magnetic field (strength)
Vector
R + R
Pressure field
Force per unit square (N/m²)
energy per unit volume (J/m²)
Scalar
IR3 IR
Mean sea level pressure Flow velocity field Flow speed field
Pressure field at mean sea level
Scalar
IR + IR
Change of point along its streamline' (v)
Vector
IR + IR22 RR - RR3
Length of flow velocity vector ((u))
Scalar
IR IR, 1R
Wind velocity field
Air flow velocity field (v)
Vector
R² + IH R - R
Temperature field
Temperature at space point (K)
Scalar
I + IR
Signed distance field (SDF)
Signed distance
Scalar R + IR
Occupancy field
Occupancy
Scalar R3 + IR
Streamlines family curves whose tangent vectors constitute the velocity vector field the flow. Streamlines
differ over time when the flow fluid changes. flow velocity vector field itself shows direction in which massless fluid element will travel at any spatial coordinate time, and therefore describes and characterizes a fluid.
58


Reviews:
[{"review_id": "l4LiBF9GFV", "reviewer": "Reviewer_S1uq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper addresses the problem of modelling a PDE in the means of neural networks. The main motivation stems from the fact that Clifford algebras is a very popular tool in modelling physical relationships. As a consequence, the paper proposes to use the product of the algebra (instead of the products of a scalar field) to define both, convolutions and FT (Fourier transforms).\n\nWith these basic operations at hand, solutions to both, Navier-Stokes and Maxwell equations are explored. To this end, standard neural networks like ResNet and FNOs (Fourier Neural Operator Networks) are used and convolutions resp. FT are replaced with the Clifford counterpart.", "summary_of_the_review": "Overall, I consider the idea behind the paper very nice and I think the concept of Clifford neural networks should be presented to the machine learning community. ~Due to the missing datasets, I consider the paper below the acceptance threshold. If the authors can provide the datasets for the final version, I would raise my vote to an acceptance vote.~\n\nEdit: Since the data set will be provided, I am happy to raise my vote to an acceptance vote.", "strength_and_weaknesses": "Strength:\n+ Every details of the Clifford algebra as well as the two new operations (C-convolutions and C-FT) is explained extensively\n+ The conducted experiments are based on well-known architecture where only the basic operations are replaced\n+ The experiments focus on physical problems, which is the motivation for these new operations\n\nWeakness:\n- For the experiments it is not clear how the data is generated. Thus, results are difficult to reproduce.\n- Section 2 takes 2 pages to explain well-known facts about Clifford algebras. This could be explained in a more compact manner.\n- The paper including appendices is too excessive with more than 55 pages. It appears a bit too long for a conference paper", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "hDMfF6N7Xe", "reviewer": "Reviewer_TyjS", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a new architecture for Neural PDEs based on Clifford algebra. By incorporating fields with different dimensions together into one algebra system, the method yields better results when used in neural networks due to better coupling among the data.", "summary_of_the_review": "It is a very comprehensive and self-supportive paper. I would recommend acceptance.", "strength_and_weaknesses": "Strengths:\n1. The theory is comprehensive and self-contained. The results are promising and supportive.\n\nWeakness:\n1. In Fig. 7, it is not discussed why CResNet works worse than ResNet in some of the experiments. Please add some analysis here.\n2. Clifford algebra seems to be general and can potentially be applied to more than Neural PDE, as long as there are inputs of multiple data types. It would be great if the author can provide more insight into other applicable scenarios and future directions to achieve this.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "f4Zc0W0u9vd", "reviewer": "Reviewer_ugoG", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces Clifford neural layers for both convolution and Fourier operations in the context of deep learning. The main motivation is that Clifford algebras can describe the algebraic property (e.g. multiplication, addition) of multivector fields consisting scalar, vector, as well as higher-order components. The resulting Clifford neural layers are universally applicable, and the authors explore their usage in neural PDE surrogate models (for solving 2D Navier-Stokes and 3D Maxwell's equations). By replacing convolution and Fourier operations in common neural PDE surrogates by their Cliffold couterparts, the authors show that Clifford neural layers consistently improves their performance. ", "summary_of_the_review": "The proposed Clifford neural layers is a novel and solid technical contribution. The results look promising for the first attempt of this kind. Though there are some obvious limitations (e.g., slow runtime), they are potentially solvable. Therefore, I recommend for acceptance.", "strength_and_weaknesses": "Strengths:\n\n- The proposed Clifford neural layers are novel and technically solid. Based on the observation that different fields in PDE are often related, the authors propose to leverage Clifford algebras to naturally incorporate this inductive bias into neural PDE surrogate models. As far as I know, this is the first time that people introduces Clifford algebras to neural PDE modeling.\n- The experiment results look promising.  Through experiments in three different settings, Cliffod neural layers bring consistent improvement compared to their standard counterparts. This is promising for an first attempt of this type.\n\nWeaknesses:\n\n- Some experimental results need further explanations or investigations. For example, I noticed that in Figure 7 ResNet based results, CResNet gives larger error than ResNet for low numbers of trajectories. This is different from 2D Navier-Stokes (Figure 6). It also contradicts with the claim in Sec. 5 (\"the performance gap increased with less data availability across all settings\"). Any explanation for this particular result.\n- For 3D Maxwell's equations, the paper only shows the results for FNO based architectures. There is no explanation why ResNet based architectures are note tested here. If possible, I think it's better to include them for completeness.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DMa-p7OjR2n", "reviewer": "Reviewer_EJCG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper the authors present Clifford neural layers that can be useful in the areas of fluid dynamics, weather forecasting, and the modeling of physical systems in general.  The author emipirically validate the use of Cifford layer by incorporating them in the common neural PDE surrogates for 2D Navier-Stokes, weather modeling, and 3D Maxwell equations. ", "summary_of_the_review": "The paper present clifford layer which can be used for application in Physical sciences. The method is tested on three datasets and the results look promising. The experiment section is missing some details which makes it a bit difficult for non-experts to read.", "strength_and_weaknesses": "The main strenghts of the paper are:\n\n1) The paper is well-written and the main ideas are effectively presented.\n2) The background section covers the pre-requisite knowledge required to understand the clifford neural layers\n3) A novel method is proposed in this paper which can be useful in the Physical science domain\n\n\nThe main weaknesses of the paper in my opinion are:\n1) As someone who does not work in the Physical sciences I could not get much information about the datasets used from the paper. I am not sure what the training sets are for these dataset and hence can't make an informed judgement on the quality of the method. [I did not go through the supplementary material in any detail ]\n2) The baseline information is not very clear form the paper.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster