GA-ReLU: an activation function for Geometric Algebra Networks applied to 2D Navier-Stokes PDEs

Authors: alberto  pepe
joan  lasenby
sven  buchholz
Venue: ICLR_cc_2024_Workshop_AI4DiffEqtnsInSci
Published: 2024-02-10 16:28:19

Abstract: Many differential equations describing physical phenomena are intrinsically geometric in nature. It has been demonstrated how this geometric structure of data can be captured effectively through networks sitting in Geometric Algebra (GA) that work with multivectors, making them suitable candidates to solve differential equations. GA networks however, are still mostly uncharted territory. In this paper we focus on non-linearities, since applying them to multivectors is not a trivial task: they are generally applied in a point-wise fashion over each real-valued component of a multivector. This approach discards interactions between different elements of the multivector input and compromises the geometric nature of GA networks. To bridge this gap, we propose GA-ReLU, a GA approach to the rectified linear unit (ReLU), and show how it can improve the solution of Navier-Stokes PDEs.

Main paper: 
 introduction: 
 Geometric (or Clifford) Algebra (GA) has been recently rediscovered as a suitable mathematical space in which to build neural networks that are truly geometrical (Brandstetter et al., 2022; Ruhe et al., 2023) . While not new (Pearson & Bisset, 1994) , GA networks have only recently been employed successfully in computational biology (Pepe et al., 2024a) , physics (Ruhe et al., 2023) , vision (Pepe et al., 2024b) and partial differential equations (PDEs) (Brandstetter et al., 2022) . Data in all these fields presents an intrinsic geometric structure that can be captured by operating on multivectors. In higher dimensional Clifford Algebras, however, it is not easy to define the concept of differentiability and hence construct an expressive function theory. Ever since the early proposals of Clifford neurons in Pearson & Bisset (1994) ; Buchholz & Sommer (2001) ; Arena et al. (1994) until today, activations have always been applied onto multivectors element-wise. When doing so, we are losing part of the geometric coupling of data that GA networks strive to achieve: is there a better way to apply non-linearities to multivectors? We try to address this issue in this paper. 
problem definition: 
 We wish to define a non-linear function ψ(x) : G n → G n for networks operating in GA in order to extend the geometric flavour of the approach also to the activation function. In particular, we refer to the Navier-Stokes PDE problem as formulated in Brandstetter et al. (2022) and train two networks, namely the Clifford ResNet and Clifford Fourier Neural Operator (FNO) with ReLU and GA-ReLU activation functions, respectively.The incompressible Navier-Stokes equations in 2D are given by: in which ∂u ∂t is the time derivative of the fluid velocity vector u = (u x , u y ), (u•∇)u is the convective term, -1 ρ ∇p is the pressure gradient, ν∇ 2 u is the viscous diffusion and f is the external force term. The incompressibility of the fluid is ensured by ∇ • u = 0.EQUATIONSince there exists a coupling between vector quantities (the velocity u) and scalar ones (the pressure field p, or the smoke s advected by u), it makes sense to express this coupling by "wrapping" together the vector and scalar information as a single multivector of the type:EQUATIONwhere u e1 and u e2 are the components of u along the e 1 and e 2 direction, respectively. Given two multivectors {x ti , x ti+1 } at two different time instants t i , t i+1 , we want to use a machine learning pipeline that estimates x ti+2 . This multivector approach has been demonstrated to be more successful than estimating (s, u x , u y ) t+2 independently through a network not in GA, since the coupling between different geometric quantities has to be inferred by the network rather being explicitly expressed through a multivector structure. We wish to keep the same coupling also when applying non-linearities. 
ga-relu: a relu in geometric algebra: 
 We want to design ψ(x) in such a way that (i) it preserves the behaviour of the equivalent activation function defined over R, and (ii) it is able to differentiate between grades and model interactions between components of the same grade. An activation function ϕ (that we will assume from now on to be the ReLU function) has commonly been applied to a multivector x element-wise, i.e.:EQUATIONwhere x i is the i-th blade component of x and n is the space dimensionality. This approach satisfies (i), but not (ii) (see Fig. 1a ). To fill in this gap, we introduce GA-ReLU. GA-ReLU is the composition of (1) a coefficient-wise ReLU, that depends on the magnitude of each multivector coefficient, and(2) a phase-dependent ReLU, that grasps the interaction between vector coefficients and depends on their phase difference. We will derive GA-ReLU below.We look at the complex domain and express our 2D multivector in terms of complex numbers. The phase-dependent ReLU has been inspired from the complex cardioid activation function firstly introduced in Virtue et al. (2017) , defined asEQUATIONin which z = a + bi ∈ C, i 2 = -1 and K an attenuation function dependent on the argument of z.The complex cardioid is an extension of the ReLU function over C and it is dependent only on the phase of the input rather than on its magnitude. It is known that the 2D Geometric Algebra G 2,0 is isomorphic to C by simply taking I ≜ e 12 as our imaginary unit, since e 2 12 = (e 1 e 2 ) 2 = e 1 e 2 e 1 e 2 = -e 1 e 1 e 2 e 2 = -1 = i 2 . By keeping this in mind, it is easy to see that a generic multivector x = x 0 + x 1 e 1 + x 2 e 2 + x 12 e 12 ∈ G 2,0 can be decomposed into "a sum of two complex numbers" as follows:x = (x 0 + Ix 12 ) + e 1 (x 1 + Ix 2 ) = z S + e 1 z V .(5)Following Brandstetter et al. (2022) , we will refer to z S as the spinor part and to z V as the vector part. We can then evaluate Eq.4 on x as follows:EQUATIONIn 2D Navier-Stokes there is no bivector component, so Eq. 6 reduces to:f (x) = K(arg(z S ))z S + e 1 K(arg(z V ))z V = ϕ(x 0 ) + e 1 K(arg(z V ))z V (7)since K(arg(z S )) = K(arg(0)) = 1 for x 0 > 0 and K(arg(0)) = 0 for x 0 ≤ 0. On the other hand, the second term can be computed to be:EQUATIONEventually, we obtain thatEQUATIONmeaning that the complex ReLU acts like a real ReLU over the scalar part x 0 and attenuates the vector components x 1 , x 2 by an amount proportional to the phase between them. Eq.9, however, is unbounded for negative vector components (see Fig. 1b ), which could cause numerical instability and negatively impact convergence. Hence, we would still want to keep the element-wise ReLU on the vector components to guarantee also a dependence on their magnitude. The final expression of GA-ReLU will then be:EQUATION)In Eq.10 we have the advantage of having a magnitude scaling similar to a ReLU (i.e. 0 output for negative input), but also a phase dependency deriving from the complex ReLU (see Fig. 1c ).We generated our own dataset of fluid in motion over a regular grid through PhiFlow (Holl et al., 2020) following the specifications of Brandstetter et al. (2022) (see Appendix A.1). It is composed of 15600, 4680 and 3120 training, validation and testing sequences, respectively. We call "sequence" a pair of inputs and targets (x ti , x ti+1 ; x ti+2 ). We trained a Clifford ResNet and a Clifford FNO with standard ReLU and GA-ReLU activation functions (see Appendix A.2). We report the summed mean squared error over the three multivector coefficients (One Step Loss) and over the scalar (Scalar Loss) and vector (Vector Loss) coefficients.  
results: 
 The 3 metrics measured for datasets of different sizes are reported in Fig. 2 and in Fig. 3 for the Clifford ResNet and Clifford FNO, respectively. Note how, albeit small, the improvement from GA-ReLU is consistent for different dataset sizes. We plot the difference between ground truth and predicted scalar fields s, ŝ for 5 different sequences in the test set in Fig. 4 and the difference between ground truth and predicted vector fields u, û in Fig. 5 . Note how, despite having minimally modified the activation function, it is possible to identify for each frame regions that deviate more from ground truth (i.e. areas in which the PDE solution is less exact) when employing an activation function that treats multivector components independently. 
conclusions: 
 We introduced GA-ReLU, a modified version of ReLU for multivector-valued networks that attempts to take into account the coupling between multivector coefficients, and showed how it can improve over the baseline error for a 2D Navier-Stokes PDEs problem. GA-ReLU has the limitations of being designed as an adaptation real-valued activation (ReLU) and for a specific mathematical space (G 2,0 ). Nevertheless, we hope that GA-ReLU can highlight the importance of non-linearities that take into account the structure of multivectors. 
 
 Appendix:  


Decision: The reviewer marks this paper as a clear accept and I agree. There are some questions of novelty that I encourage the authors to address in the camera-ready version.

Reviews: review: title: title: Good Paper, Accept
main_review: main_review: Congratulations to the authors in getting this research to this stage. I enjoyed reading this nicely drafted paper. Draft starts with a short, concise and excellent Introduction and defines the problem very well. 

The title and abstract focus on the proposal of a new activation function for geometric algebra however I fail to grasp the novelty in the paper. The approach of combining two existing components to formulate the GA-RELU activation function and its subsequent application to a new domain is both commendable and indicative of the paper’s creative efforts to advance the field. The application is definitely new and hence makes the case for this paper. However, upon careful review,  the novelty of the concept of GA-RELU prompts further questions. The description and promotion of complex cardioid  activation under a new name of GA-RELU may overstate its uniqueness. 

- Quality 
  - The written quality of paper is good. Draft nicely explains the problem and solution. The paper use the modified complex cardioid  activation for the GA and demonstrate the improvement with this approach and 
  - Draft clearly associates the physical meaning associated with mathematical equations used and derived. 

- Clarity 
  - The text is clear and describe the problem and solution with all necessary mathematical details. 

- Originality
  - The concept of modifying the complex cardioid  activation is original and has not be proposed in the filed earlier. 
  - Moreover, the application of GA-RELU for Geometric Algebra Networks is new and its application to the 2D NS PDE. 

- Significance 
  - The proposed research mark an significant improvement for Geometric Algebra Networks, however the experimentation uses on one example. 
  - Due to limited experimental results it is hard to judge its usability and applications in different problems in GA.