SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The above implies that  $\mathrm{O}(V, \mathfrak{q})$  acts on whole  $\operatorname{Cl}(V, \mathfrak{q})$  in a well-defined way. Concretely, if  $x \in \operatorname{Cl}(V, \mathfrak{q})$  is of the form  $x = \sum_{i \in I} c_i \cdot v_{i,1} \cdots v_{i,k_i}$  with  $v_{i,j} \in V$ ,  $c_i \in \mathbb{F}$  and  $\Phi \in \mathrm{O}(V, \mathfrak{q})$  is given by  $\bar{\rho}^{(1)}([w]) = \Phi$  with  $w \in \Gamma(V, \mathfrak{q})$ , then we have:
$$
\rho (w) (x) = \sum_ {i \in I} c _ {i} \cdot \rho (w) \left(v _ {i, 1}\right) \dots \rho (w) \left(v _ {i, k _ {i}}\right) = \sum_ {i \in I} c _ {i} \cdot \Phi \left(v _ {i, 1}\right) \dots \Phi \left(v _ {i, k _ {i}}\right). \tag {12}
$$
This means that  $\Phi$  acts on a multivector  $x$  by applying an orthogonal transformation to all its vector components  $v_{i,j}$ , as illustrated in Figure 1. Theorem 3.5 implies that that a map  $f:\mathrm{Cl}(V,\mathfrak{q})^{\ell_{\mathrm{in}}}\to \mathrm{Cl}(V,\mathfrak{q})^{\ell_{\mathrm{out}}}$  is equivariant to the Clifford group  $\Gamma (V,\mathfrak{q})$  if and only if it is equivariant to the orthogonal group  $\mathrm{O}(V,\mathfrak{q})$  when both use the actions on  $\operatorname {Cl}(V,\mathfrak{q})$  described above (for each component). To leverage these results for constructing  $\mathrm{O}(V,\mathfrak{q})$ -equivariant neural networks acting on vector features, i.e.,  $x\in V^{\ell_{\mathrm{in}}}$ , we refer to Section 4.
To prove Theorem 3.5, we invoke Cartan-Dieudonné (Theorem C.13), stating that every orthogonal transformation decomposes into compositions of reflections, and recall that  $\rho$  reduces to a reflection when restricted to  $V$ , see Theorem E.25. Theorem 3.5 also allows one to provide explicit descriptions of the element of the Clifford group, see Corollary E.27.
Finally, it is worth noting that our method is also Pin and Spin group-equivariant<sup>6</sup>. These groups, sharing properties with the Clifford group, are also often studied in relation to the orthogonal group.
We restrict our layers to use  $\mathbb{F} \coloneqq \mathbb{R}$ . Our method is most similar to steerable methods such as  $[\mathrm{BHP}^+22]$ . However, unlike these works, we do not require an alternative basis representation based on spherical harmonics, nor do we need to worry about Clebsch-Gordan coefficients. Instead, we consider simply a steerable vector basis for  $V$ , which then automatically induces a steerable multivector basis for  $\operatorname{Cl}(V,\mathfrak{q})$  and its transformation kernels. By steerability, we mean that this basis can be transformed in a predictable way under an action from the Clifford group, which acts orthogonally on both  $V$  and  $\operatorname{Cl}(V,\mathfrak{q})$  (see Figure 1).
We present layers yielding Clifford group-equivariant estimable transformations. All the main ideas are based on Corollary 3.3 and Corollary 3.4. It is worth mentioning that the methods presented here form a first exploration of applying our theoretical results, making future optimizations rather likely.
Linear Layers Let  $x_{1},\ldots ,x_{\ell}\in \mathrm{Cl}(V,\mathfrak{q})$  be a tuple of multivectors expressed in a steerable basis, where  $\ell$  represents the number of input channels. Using the fact that a polynomial restricted to the first order constitutes a linear map, we can construct a linear layer by setting
$$
y _ {c _ {\text {o u t}}} ^ {(k)} := T _ {\phi_ {c _ {\text {o u t}}}} ^ {\operatorname {l i n}} \left(x _ {1}, \dots , x _ {\ell}\right) ^ {(k)} := \sum_ {c _ {\text {i n}} = 1} ^ {\ell} \phi_ {c _ {\text {o u t}} c _ {\text {i n}} k} x _ {c _ {\text {i n}}} ^ {(k)}, \tag {13}
$$
where  $\phi_{c_{\mathrm{out}}c_{\mathrm{in}}k}\in \mathbb{R}$  are estimable coefficients and  $c_{\mathrm{in}},c_{\mathrm{out}}$  denote the input and output channel, respectively. As such,  $T_{\phi}:\operatorname {Cl}(V,\mathfrak{q})^{\ell}\to \operatorname {Cl}(V,\mathfrak{q})$  is a linear transformation in each algebra subspace
Here we restrict  $(V,\mathfrak{q})$  to be non-degenerate as we never consider degenerate quadratic forms in our experiments. However, in the supplementary material we generalize this also to the degenerate case by carefully taking the radical subspace  $\mathcal{R}$  of  $(V,\mathfrak{q})$  into account on both sides of the isomorphism.
<sup>6</sup>We discuss in Appendix E.6 a general definition of these groups.
$k$ . Recall that this is possible due to the result that  $\rho(w)$  respects the multivector subspaces. This computes a transformation for the output channel  $c_{\mathrm{out}}$ ; the map can be repeated (using different sets of parameters) for other output channels, similar to classical neural network linear layers. For  $y_{c_{\mathrm{out}}}^{(0)} \in \mathbb{R}$  (the scalar part of the Clifford algebra), we can additionally learn an invariant bias parameter.
Geometric Product Layers A core strength of our method comes from the fact that we can also parameterize interaction terms. In this work, we only consider layers up to second order. Higher-order interactions are indirectly modeled via multiple successive layers. As an example, we take the pair  $x_{1}, x_{2}$ . Their interaction terms take the form  $\left(x_{1}^{(i)}x_{2}^{(j)}\right)^{(k)}, i,j,k = 0,\ldots ,n$ ; where we again make use of the fact that  $\rho (w)$  respects grade projections. As such, all the grade-  $k$  terms resulting from the interaction of  $x_{1}$  and  $x_{2}$  are parameterized with
$$
P _ {\phi} \left(x _ {1}, x _ {2}\right) ^ {(k)} := \sum_ {i = 0} ^ {n} \sum_ {j = 0} ^ {n} \phi_ {i j k} \left(x _ {1} ^ {(i)} x _ {2} ^ {(j)}\right) ^ {(k)}, \tag {14}
$$
where  $P_{\phi}:\mathrm{Cl}(V,\mathfrak{q})\times \mathrm{Cl}(V,\mathfrak{q})\to \mathrm{Cl}(V,\mathfrak{q})$ . This means that we get  $(n + 1)^3$  parameters for every geometric product between a pair of multivectors<sup>7</sup>. Parameterizing and computing all second-order terms amounts to  $\ell^2$  such operations, which can be computationally expensive given a reasonable number of channels  $\ell$ . Instead, we first apply a linear map to obtain  $y_{1},\ldots ,y_{\ell}\in \operatorname {Cl}(V,\mathfrak{q})$ . Through this map, the mixing (i.e., the terms that will get multiplied) gets learned. That is, we only get  $\ell$  pairs  $(x_{1},y_{1}),\ldots ,(x_{\ell},y_{\ell})$  from which we then compute  $z_{c_{\mathrm{out}}}^{(k)}\coloneqq P_{\phi_{c_{\mathrm{out}}}}(x_{c_{\mathrm{in}}},y_{c_{\mathrm{in}}})^{(k)}$ . Note that here we have  $c_{\mathrm{in}} = c_{\mathrm{out}}$ , i.e., the number of channels does not change. Hence, we refer to this layer as the element-wise geometric product layer. We can obtain a more expressive (yet more expensive) parameterization by linearly combining such products by computing
$$
z _ {c _ {\text {o u t}}} ^ {(k)} := T _ {\phi_ {c _ {\text {o u t}}}} ^ {\text {p r o d}} \left(x _ {1}, \dots , x _ {\ell}, y _ {1}, \dots , y _ {\ell}\right) ^ {(k)} := \sum_ {c _ {\text {i n}} = 1} ^ {\ell} P _ {\phi_ {c _ {\text {o u t}} c _ {\text {i n}}}} \left(x _ {c _ {\text {i n}}}, y _ {c _ {\text {i n}}}\right) ^ {(k)}, \tag {15}
$$
which we call the fully-connected geometric product layer. Computational feasibility and experimental verification should determine which parameterization is preferred.
Normalization and Nonlinearities Since our layers involve quadratic and potentially higher-order interaction terms, we need to ensure numerical stability. In order to do so, we use a normalization operating on each multivector subspace before computing geometric products by putting
$$
x ^ {(m)} \mapsto \frac {x ^ {(m)}}{\sigma \left(a _ {m}\right) (\bar {\mathfrak {q}} \left(x ^ {(m)}\right) - 1) + 1}, \tag {16}
$$
where  $x^{(m)} \in \mathrm{Cl}^{(m)}(V, \mathfrak{q})$ . Here,  $\sigma$  denotes the logistic sigmoid function, and  $a_{m} \in \mathbb{R}$  is a learned scalar. The denominator interpolates between 1 and the quadratic form  $\bar{\mathfrak{q}}\left(x^{(m)}\right)$ , normalizing the magnitude of  $x^{(m)}$ . This ensures that the geometric products do not cause numerical instabilities without losing information about the magnitude of  $x^{(m)}$ , where a learned scalar interpolates between both regimes. Note that by Theorem 3.2,  $\bar{\mathfrak{q}}(x^{(m)})$  is invariant under the action of the Clifford group, rendering Equation (16) an equivariant map.
Next, we use the layer-wide normalization scheme proposed by  $\mathrm{[RGdK^{+}23]}$ , which, since it is also based on the extended quadratic form, is also equivariant with respect to the twisted conjugation.
Regarding nonlinearities, we use a slightly adjusted version of the units proposed by  $\mathrm{RGdK}^{+}23$ . Since the scalar subspace  $\mathrm{Cl}^{(0)}(V,\mathfrak{q})$  is always invariant with respect to the twisted conjugation, we can apply  $x^{(m)}\mapsto \mathrm{ReLU}\left(x^{(m)}\right)$  when  $m = 0$  and  $x^{(m)}\mapsto \sigma_{\phi}\left(\bar{\mathfrak{q}}\left(x^{(m)}\right)\right)x^{(m)}$  otherwise. We can replace ReLU with any common scalar activation function. Here,  $\sigma_{\phi}$  represents a potentially parameterized nonlinear function. Usually, however, we restrict it to be the sigmoid function. Since we modify  $x^{(m)}$  with an invariant scalar quantity, we retain equivariance. Such gating activations are commonly used in the equivariance literature [WGW+18, GS22].
In practice, we use fewer parameters due to the sparsity of the geometric product, implying that many interactions will invariably be zero, thereby making their parameterization redundant.
O(3) Signed Volumes
O(5) Convex Hulls
Figure 3: Left: Test mean-squared errors on the O(3) signed volume task as functions of the number of training data. Note that due to identical performance, some baselines are not clearly visible. Right: same, but for the O(5) convex hull task.
O(5) Regression
Figure 4: Test mean-squared-errorson the O(5) regression task.
Embedding Data in the Clifford Algebra In this work, we consider data only from the vector space  $V$  or the scalars  $\mathbb{R}$ , although generally one might also encounter, e.g., bivector data. That is, we have some scalar features  $h_1, \ldots, h_k \in \mathbb{R}$  and some vector features  $h_{k+1}, \ldots, h_\ell \in V$ . Typical examples of scalar features include properties like mass, charge, temperature, and so on. Additionally, one-hot encoded categorical features are also included because  $\{0, 1\} \subset \mathbb{R}$  and they also transform trivially. Vector features include positions, velocities, and the like. Then, using the identifications  $\mathrm{Cl}^{(0)}(V, \mathfrak{q}) \cong \mathbb{R}$  and  $\mathrm{Cl}^{(1)}(V, \mathfrak{q}) \cong V$ , we can embed the data into the scalar and vector subspaces of the Clifford algebra to obtain Clifford features  $x_1, \ldots, x_\ell \in \mathrm{Cl}(V, \mathfrak{q})$ .
Similarly, we can predict scalar- or vector-valued data as output of our model by grade-projecting onto the scalar or vector parts, respectively. We can then directly compare these quantities with ground-truth vector- or scalar-valued data through a loss function and use standard automatic differentiation techniques to optimize the model. Note that invariant predictions are obtained by predicting scalar quantities.
Here, we show that CGENNs excel across tasks, attaining top performance in several unique contexts. Parameter budgets as well as training setups are kept as similar as possible to the baseline references. All further experimental details can be found in the public code release.
5.1 Estimating Volumetric Quantities
O(3) Experiment: Signed Volumes This task highlights the fact that equivariant architectures based on scalarization are not able to extract some essential geometric properties from input data. In a synthetic setting, we simulate a dataset consisting of random three-dimensional tetrahedra. A main advantage of our method is that it can extract covariant quantities including (among others) signed volumes, which we demonstrate in this task. Signed volumes are geometrically significant because they capture the orientation of geometric objects in multidimensional spaces. For instance, in computer graphics, they can determine whether a 3D object is facing towards or away from the camera, enabling proper rendering. The input to the network is the point cloud and the loss function is the mean-squared error between the signed volume and its true value. Note that we are predicting a covariant (as opposed to invariant) scalar quantity (also known as a pseudoscalar) under O(3) transformations using a positive-definite (Euclidean) metric. The results are displayed in the left part of Figure 3. We compare against a standard multilayer perceptron (MLP), an MLP version of the  $\mathrm{E}(n)$ -GNN [SHW21] which uses neural networks to update positions with scalar multiplication, Vector Neurons (VN) [DLD $^{+}$ 21], and Geometric Vector Perceptrons (GVP) [JES $^{+}$ 21]. We see that the scalarization methods fail to access the features necessary for this task, as evidenced by their test loss not improving even with more available data. The multilayer perceptron, although a universal approximator, lacks the correct inductive biases. Our model, however, has the correct inductive biases (e.g., the equivariance property) and can also access the signed volume. Note that we do not take
the permutation invariance of this task into account, as we are interested in comparing our standard feed-forward architectures against similar baselines.
O(5) Experiment: Convex Hulls We go a step further and consider a five-dimensional Euclidean space, showcasing our model's ability to generalize to high dimensions. We also make the experiment more challenging by including more points and estimating the volume of the convex hull generated by these points - a task that requires sophisticated algorithms in the classical case. Note that some points may live inside the hull and do not contribute to the volume. We use the same network architectures as before (but now embedded in a five-dimensional space) and present the results in Figure 3. We report the error bars for CGENNs, representing three times the standard deviation of the results of eight runs with varying seeds. Volume (unsigned) is an invariant quantity, enabling the baseline methods to approximate its value. However, we still see that CGENNs outperform the other methods, the only exception being the low-data regime of only 256 available data points. We attribute this to our method being slightly more flexible, making it slightly more prone to overfitting. To mitigate this issue, future work could explore regularization techniques or other methods to reduce overfitting in low-data scenarios.
5.2 O(5) Experiment: Regression
We compare against the methods presented by [FWW21] who propose an  $\mathrm{O}(5)$ -invariant regression problem. The task is to estimate the function  $f(x_{1},x_{2})\coloneqq \sin (\| x_{1}\|) - \| x_{2}\|^{3} / 2 + \frac{x_{1}^{\top}x_{2}}{\|x_{1}\|\|x_{2}\|}$ , where the five-dimensional vectors  $x_{1},x_{2}$  are sampled from a standard Gaussian distribution in order to simulate train, test, and validation datasets. The results are shown in Figure 4. We used baselines from [FWW21] including an MLP, MLP with augmentation (MLP+Aug), and the O(5)- & SO(5)-MLP architectures. We maintain the same number of parameters for all data regimes. For extremely small datasets (30 and 100 samples), we observe some overfitting tendencies that can be countered with regularization (e.g., weight decay) or using a smaller model. For higher data regimes (300 samples and onward), CGENNs start to significantly outperform the baselines.
5.3 E(3) Experiment:  $n$ -Body System
The  $n$ -body experiment  $\mathrm{[KFW^{+}18]}$  serves as a benchmark for assessing the performance of equivariant (graph) neural networks in simulating physical systems [HRXH22]. In this experiment, the dynamics of  $n = 5$  charged particles in a three-dimensional space are simulated. Given the initial positions and velocities of these particles, the task is to accurately estimate their positions after 1000 timesteps. To address this challenge, we construct a graph neural network (GNN) using the Clifford equivariant layers introduced in the previous section. We use a standard message-passing algorithm  $\mathrm{[GSR^{+}17]}$  where the message and update networks are CGENNs. So long as the message aggregator is equivariant, the end-to-end model also maintains equivariance. The input to the network
Method	MSE (↓)
SE(3)-Tr.	0.0244
TFN	0.0155
NMP	0.0107
Radial Field	0.0104
EGNN	0.0070
SEGNN	0.0043
CGENN	0.0039 ± 0.0001
Table 1: Mean-squared error (MSE) on the  $n$ -body system experiment.
consists of the mean-subtracted positions of the particles (to achieve translation invariance) and their velocities. The model's output is the estimated displacement, which is to the input to achieve translation-equivariant estimated target positions. We include the invariant charges as part of the input and their products as edge attributes. We compare against the steerable SE(3)-Transformers [FWFW20], Tensor Field Networks [TSK $^{+}$ 18], and SEGNN [BHP $^{+}$ 22]. Scalarization baselines include Radial Field [KKN20] and EGNN [SHW21]. Finally, NMP [GSR $^{+}$ 17] is not an E(3)-equivariant method. The number of parameters in our model is maintained similar to the EGNN and SEGNN baselines to ensure a fair comparison.
Results of our experiment are presented in Table 1, where we also present for CGENN three times the standard deviation of three identical runs with different seeds. Our approach clearly outperforms earlier methods and is significantly better than  $\left[\mathrm{BHP}^{+}22\right]$ , thereby surpassing the baselines. This experiment again demonstrates the advantage of leveraging covariant information in addition to scalar quantities, as it allows for a more accurate representation of the underlying physics and leads to better predictions.
Model	Accuracy (↑)	AUC (↑)	1/εB(↑)(εS=0.5)	1/εB(↑)(εS=0.3)
ResNeXt [XGD+17]	0.936	0.9837	302	1147
P-CNN [CMS17]	0.930	0.9803	201	759
PFN [KMT19]	0.932	0.9819	247	888
ParticleNet [QG20]	0.940	0.9858	397	1615
EGNN [SHW21]	0.922	0.9760	148	540
LGN [BAO+20]	0.929	0.9640	124	435
LorentzNet [GMZ+22]	0.942	0.9868	498	2195
CGENN	0.942	0.9869	500	2172
Table 2: Performance comparison between our proposed method and alternative algorithms on the top tagging experiment. We present the accuracy, Area Under the Receiver Operating Characteristic Curve (AUC), and background rejection  $1 / \epsilon_{B}$  and at signal efficiencies of  $\epsilon_{S} = 0.3$  and  $\epsilon_{S} = 0.5$ .
5.4 O(1,3) Experiment: Top Tagging
Jet tagging in collider physics is a technique used to identify and categorize high-energy jets produced in particle collisions, as measured by, e.g., CERN's ATLAS detector  $\mathrm{[BTB^{+}08]}$ . By combining information from various parts of the detector, it is possible to trace back these jets' origins  $\mathrm{[KNS^{+}19,ATL17]}$ . The current experiment seeks to tag jets arising from the heaviest particles of the standard model: the 'top quarks' [IQWW09]. A jet tag should be invariant with respect to the global reference frame, which can transform under Lorentz boosts due to the relativistic nature of the particles. A Lorentz boost is a transformation that relates the space and time coordinates of an event as seen from two inertial reference frames. The defining characteristic of these transformations is that they preserve the Minkowski metric, which is given by  $\gamma (ct,x,y,z)\coloneqq (ct)^2 -x^2 -y^2 -z^2$ . Note the difference with the standard positive definite Euclidean metric, as used in the previous experiments. The set of all such transformations is captured by the orthogonal group O(1,3); therefore, our method is fully compatible with modeling this problem.
We evaluate our model on a top tagging benchmark published by [KPTR19]. It contains 1.2M training entries, 400k validation entries, and 400k testing entries. For each jet, the energy-momentum 4-vectors are available for up to 200 constituent particles, making this a much larger-scale experiment than the ones presented earlier. Again, we employ a standard message passing graph neural network  $\mathrm{[GSR^{+}17]}$  using CGENNs as message and update networks. The baselines include ResNeXt  $\mathrm{[XGD^{+}17]}$ , PCNN [CMS17], PFN [KMT19], ParticleNet [QG20], LGN  $\mathrm{[BAO^{+}20]}$ , EGNN [SHW21], and the more recent LorentzNet  $\mathrm{[GMZ^{+}22]}$ . Among these, LGN is a steerable method, whereas EGNN and LorentzNet are scalarization methods. The other methods are not Lorentz-equivariant. Among the performance metrics, there are classification accuracy, Area Under the Receiver Operating Characteristic Curve (AUC), and the background rejection rate  $1 / \epsilon_{B}$  at signal efficiencies of  $\epsilon_{S} = 0.3$  and  $\epsilon_{S} = 0.5$ , where  $\epsilon_{B}$  and  $\epsilon_{S}$  are the false positive and true positive rates, respectively. We observe that LorentzNet, a method that uses invariant quantities, is an extremely competitive baseline that was optimized for this task. Despite this, CGENNs are able to match its performance while maintaining the same core implementation.
We presented a novel approach for constructing  $\mathrm{O}(n)$ - and  $\mathrm{E}(n)$ -equivariant neural networks based on Clifford algebras. After establishing the required theoretical results, we proposed parameterizations of nonlinear multivector-valued maps that exhibit versatility and applicability across scenarios varying in dimension. This was achieved by the core insight that polynomials in multivectors are  $\mathrm{O}(n)$ -equivariant functions. Theoretical results were empirically substantiated in three distinct experiments, outperforming or matching baselines that were sometimes specifically designed for these tasks.
CGENNs induce a (non-prohibitive) degree of computational overhead similar to other steerable methods. On the plus side, we believe that improved code implementations such as custom GPU kernels or alternative parameterizations to the current ones can significantly alleviate this issue, potentially also resulting in improved performances on benchmark datasets. This work provides solid theoretical and experimental foundations for such developments.
[WSK+15] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaou Tang, and Jianxiong Xiao, 3d ShapeNets: A deep representation for volumetric shapes., Computer Vision and Pattern Recognition (CVPR), 2015, pp. 1912-1920.
$\left[\mathrm{XGD}^{+}17\right]$  Saining Xie, Ross B. Girshick, Piotr Dólár, Zhuowen Tu, and Kaiming He, Aggregated Residual Transformations for Deep Neural Networks., Computer Vision and Pattern Recognition (CVPR), 2017, pp. 5987-5995.
[YCo20] YCor, Analogue of the special orthogonal group for singular quadratic forms, Math-Overflow, 2020, url:https://mathoverflow.net/q/378828 (version: 2020-12-13).
$\left[\mathrm{ZCD}^{+}20\right]$  C Lawrence Zitnick, Lowik Chanussot, Abhishek Das, Siddharth Goyal, Javier Heras-Domingo, Caleb Ho, Weihua Hu, Thibaut Lavril, Aini Palizhati, Morgane Riviere, and others, An introduction to electrocatalyst design using machine learning for renewable energy storage, arXiv, 2020.
$\left[\mathrm{ZKR}^{+}17\right]$  Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola, Deep sets, Advances in neural information processing systems, vol. 30, 2017.
[ZXXC18] Xuanyu Zhu, Yi Xu, Hongteng Xu, and Changjian Chen, *Quaternion convolutional neural networks*, Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 631-647.
We thank Leo Dorst and Steven de Keninck for insightful discussions about Clifford (geometric) algebras and their applications as well as pointing us to the relevant literature. Further, we thank Robert-Jan Schlimbach and Jayesh Gupta for their help with computational infrastructure and scaling up the experiments. This work used the Dutch national e-infrastructure with the support of the SURF Cooperative using grant no. EINF-5757 as well as the DAS-5 and DAS-6 clusters  $\mathrm{[BEdL^{+}16]}$ .
Broader Impact Statement