SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. Let  $x$  be a  $k$ -blade and  $u$  a vector of unit norm. We can decompose  $u$  into  $u = t + v$  with  $t \wedge x = 0$  (the part tangential to the subspace of  $x$ ) and  $v \rfloor x = 0$  (the normal part). This decomposition is unique unless  $x$  is ideal in the projective GA, in which case the  $e_0$  component of  $u$  is both normal and tangential, and we choose  $t$  Euclidean.
In either case, note the following equalities:  $xt = (-1)^{k - 1}tx; xv = (-1)^{k}vx; vt = -tv$  and note  $\exists \lambda \neq 0$  such that  $vtx = \lambda x$ , which can be shown e.g. by picking a basis. Then:
$$
u [ x ] = (- 1) ^ {k} (t + v) x (t + v) = (t + v) (- t + v) x = (- \| t \| ^ {2} + \| v \| ^ {2}) x - 2 v t x
$$
We have  $u[x] \propto x \iff vtx = 0$ . If  $x$  is not ideal, this implies that either  $v = 0$  (thus  $u \wedge x = 0$  and  $u[x] = -x$ ) or  $t = 0$  (thus  $u \rfloor x = 0$  and  $u[x] = x$ ). If  $x$  is ideal, this implies that either  $v \propto e_0$  (thus  $u \wedge x = 0$  and  $u[x] = -x$ ) or  $t = 0$  (thus  $u \rfloor x = 0$  and  $u[x] = x$ ).
Lemma 3. Let  $r \in \{0,1\}$ . Any linear  $\operatorname{Pin}(n,0,r)$ -equivariant map  $\phi : \mathbb{G}_{n,0,r} \to \mathbb{G}_{n,0,r}$  can be decomposed into a sum of equivariant maps  $\phi = \sum_{lkm} \phi_{lkm}$ , with  $\phi_{lkm}$  equivariantly mapping  $k$ -blades to  $l$ -blades. If  $r = 0$  (Euclidean algebra) or  $k < n + 1$ , such a map  $\phi_{lkm}$  is defined by the image of any one non-ideal  $k$ -blade, like  $e_{12\dots k}$ . Instead, if  $r = 1$  (projective algebra) and  $k = n + 1$ , then such a map is defined by the image of a pseudoscalar, like  $e_{01\dots n}$ .
Proof. The  $\operatorname{Pin}(n,0,r)$  group action maps  $k$ -vectors to  $k$ -vectors. Therefore,  $\phi$  can be decomposed into equivariant maps from grade  $k$  to grade  $l$ :  $\phi(x) = \sum_{lk} \phi_{lk}(\langle x \rangle_k)$ , with  $\phi_{lk}$  having  $l$ -vectors as image, and all  $k'$ -vectors in the kernel, for  $k' \neq k$ . Let  $x$  be an non-ideal  $k$ -blade (or pseudoscalar if  $k = n + 1$ ). By lemmas 1 and 4, in both Euclidean and projective GA, the span of the  $k$ -vectors in the orbit of  $x$  contains any  $k$ -vector. So  $\phi_{lk}$  is defined by the  $l$ -vector  $y = \phi_{lk}(x)$ . Any  $l$ -vector can be decomposed as a finite sum of  $l$ -blades:  $y = y_1 + \ldots + y_M$ . We can define  $\phi_{lk_m}(x) = y_m$ , extended to all  $l$ -vectors by equivariance, and note that  $\phi_{lk} = \sum_m \phi_{lk_m}$ .
Proposition 4. For an  $n$ -dimensional Euclidean geometric algebra  $\mathbb{G}_{n,0,0}$ , any linear endomorphism  $\phi : \mathbb{G}_{n,0,0} \to \mathbb{G}_{n,0,0}$  that is equivariant to the  $\operatorname{Pin}(n,0,0)$  group (equivalently to  $O(n)$ ) is of the type  $\phi(x) = \sum_{k=0}^{n} w_k \langle x \rangle_k$ , for parameters  $w \in \mathbb{R}^{n+1}$ .
Proof. By decomposition of Lemma 3, let  $\phi$  map from  $k$ -blades to  $l$ -blades. Let  $x$  be a  $k$ -blade. Let  $u$  be a 1- versor. By Lemma 2, if  $u$  is orthogonal to  $x$ ,  $u[\phi(x)] = \phi(u[x]) = \phi(x)$  and  $u$  is also orthogonal to  $\phi(x)$ . If  $u \wedge x = 0$ , then  $u[\phi(x)] = \phi(u[x]) = \phi(-x) = -\phi(x)$  and  $u \wedge \phi(x) = 0$ . Thus any vector in  $x$  is in  $\phi(x)$  and any vector orthogonal to  $x$  is orthogonal to  $\phi(x)$ , this implies  $\phi(x) = w_k x$ , for some  $w_k \in \mathbb{R}$ . By Lemma 3, we can extend  $\phi$  to  $\phi(y) = w_k y$  for any  $k$ -vector  $y$ .
Projective geometric algebra How about equivariant linear maps in projective geometric algebra? The degenerate metric makes the derivation more involved, but in the end we will arrive at a result that is only slightly more general.
Lemma 4. The Pin group of the projective geometric algebra,  $\operatorname{Pin}(n,0,1)$ , acts transitively on the space of  $k$ -blades with positive norm  $\|x\| = \lambda > 0$ . Additionally, the group acts transitively on the space of zero-norm  $k$ -blades of the form  $x = e_0 \wedge y$  (called ideal blades), with  $\|y\| = \kappa$ .
Proof. Let  $x = x_{1} \wedge \ldots \wedge x_{k}$  be a  $k$ -blade with positive norm  $\lambda$ . All vectors  $x_{i}$  can be written as  $x_{i} = v_{i} + \delta_{i}e_{0}$ , for a nonzero Euclidean vector  $v_{i}$  (meaning with no  $e_0$  component) and  $\delta_i\in \mathbb{R}$ , because if  $v_{i} = 0$ , the norm of  $x$  would have been 0. Orthogonalize them as  $x_2^{\prime} = x_2 - \langle x_1,x_2\rangle x_1$ , etc., resulting in  $x = x_1^\prime \wedge \dots \wedge x_k^\prime$  with  $x_{i}^{\prime} = v_{i}^{\prime} + \delta_{i}^{\prime}e_{0}$  with orthogonal  $v_{i}^{\prime}$ .
Define the translation  $t = 1 + \frac{1}{2}\sum_{i}\delta_{i}^{\prime}e_{0}\wedge v_{i}^{\prime}$ , which makes  $x^{\prime}$  Euclidean:  $t[x^{\prime}] = v_1^{\prime}\wedge \ldots \wedge v_k^{\prime}$ . By Lemma 1, the Euclidean Pin group  $\operatorname {Pin}(n,0,0)$ , which is a subgroup of  $\operatorname {Pin}(n,0,1)$ , acts transitively on Euclidean  $k$ -blades of a given norm. Thus, in the projective geometric algebra  $\operatorname {Pin}(n,0,1)$ , any two  $k$ -blades of equal positive norm  $\lambda$  are related by a translation to the origin and then a  $\operatorname {Pin}(n,0,0)$  transformation.
For the ideal blades, let  $x = e_0 \wedge y$ , with  $\|y\| = \kappa$ . We take  $y$  to be Euclidean without loss of generality. For any  $g \in \operatorname{Pin}(n, 0, 1)$ ,  $g[e_0] = e_0$ , so  $g[x] = e_0 \wedge g[y]$ . Consider another  $x' = e_0 \wedge y'$  with  $\|y'\| = \kappa$  and taking  $y'$  Euclidean. As  $\operatorname{Pin}(n, 0, 0)$  acts transitively on Euclidean  $(k - 1)$ -blades with norm  $\kappa$ , let  $g \in \operatorname{Pin}(n, 0, 0)$  such that  $g[y] = y'$ . Then  $g[x] = x'$ .
We can now construct the most general equivariant linear map between projective geometric algebras, a key ingredient for GATr:
Proposition 5. For the projective geometric algebra  $\mathbb{G}_{n,0,1}$ , any linear endomorphism  $\phi : \mathbb{G}_{n,0,1} \to \mathbb{G}_{n,0,1}$  that is equivariant to the group  $\operatorname{Pin}(n,0,r)$  (equivalently to  $E(n)$ ) is of the type  $\phi(x) = \sum_{k=0}^{n+1} w_k \langle x \rangle_k + \sum_{k=0}^n v_k e_0 \langle x \rangle_k$ , for parameters  $w \in \mathbb{R}^{n+2}$ ,  $v \in \mathbb{R}^{n+1}$ .
Proof. Following Lemma 3, decompose  $\phi$  into a linear equivariant map from  $k$ -blades to  $l$ -blades. For  $k < n + 1$ , let  $x = e_{12\dots k}$ . Then following Lemma 2, for any  $1 \leq i \leq k$ ,  $e_i \wedge x = 0$ ,  $e_i[x] = -x$ , and  $e_i[\phi(x)] = \phi(e_i[x]) = \phi(-x) = -\phi(x)$  and thus  $e_i \wedge \phi(x) = 0$ . Therefore, we can write  $\phi(x) = x \wedge y_1 \wedge \ldots \wedge y_{l-k}$ , for  $l - k$  vectors  $y_j$  orthogonal to  $x$ .
Also, again using Lemma 2, for  $k < i \leq n$ ,  $e_i \rfloor x = 0 \implies e_i[\phi(x)] = \phi(x) \implies e_i \rfloor \phi(x) = 0 \implies \forall i, \langle e_i, y_j \rangle = 0$ . Thus,  $y_j$  is orthogonal to all  $e_i$  with  $1 \leq i \leq n$ . Hence,  $l = k$  or  $l = k + 1$  and  $y_1 \propto e_0$ .
For  $k = n + 1$ , let  $x = e_{012\dots k}$ . By a similar argument, all invertible vectors  $u$  tangent to  $x$  must be tangent to  $\phi(x)$ , thus we find that  $\phi(x) = x \wedge y$  for some blade  $y$ . For any non-zero  $\phi(x)$ ,  $y \propto 1$ , and thus  $\phi(x) \propto x$ . By Lemma 3, by equivariance and linearity, this fully defines  $\phi$ .
A.2 Bilinear maps
Next, we turn towards bilinear operations. In particular, we show that the geometric product and the join are equivariant.
For the geometric product, equivariance is straightforward: Any transformation  $u \in \operatorname{Pin}(n,0,r)$ , gives a homomorphism of the geometric algebra, as for any multivectors  $x,y$ ,  $u[xy] = u\widehat{xy}u^{-1} = u\widehat{x}yu^{-1} = u\widehat{x}u^{-1}u\widehat{y}u^{-1} = u[x]u[y]$ . The geometric product is thus equivariant.
Dual and join in Euclidean algebra For the join and the closely related dual, we again begin with the Euclidean geometric algebra, before turning to the projective case later.
The role of the dual is to have a bijection  $\cdot^{*}:\mathbb{G}_{n,0,0}\to \mathbb{G}_{n,0,0}$  that maps  $k$ -vectors to  $(n - k)$ -vectors. For the Euclidean algebra, with a choice of pseudoscalar  $\mathcal{I}$ , we can define a dual as:
$$
x ^ {*} = x \mathcal {I} ^ {- 1} = x \tilde {\mathcal {I}} \tag {6}
$$
This dual is bijective, and involutive up to a sign:  $(y^{*})^{*} = y\tilde{\mathcal{I}}\tilde{\mathcal{I}} = \pm y$ , with  $+y = 1$  for  $n \in \{1,4,5,8,9,\ldots\}$  and  $-y$  for  $n \in \{2,3,6,7,\ldots\}$ . We choose  $\tilde{\mathcal{I}}$  instead of  $\mathcal{I}$  in the definition of the dual so that given  $n$  vectors  $x_{1},\ldots,x_{n}$ , the dual of the multivector  $x = x_{1} \wedge \ldots \wedge x_{n}$ , is given by the scalar of the oriented volume spanned by the vector. We denote the inverse of the dual as  $x^{-*} = x\mathcal{I}$ . Expressed in a basis, the dual yields the complementary indices and a sign. For example, for  $n = 3$  and  $\mathcal{I} = e_{123}$ , we have  $(e_1)^* = -e_{23}$ ,  $(e_{12})^* = e_3$ .
Via the dual, we can define the bilinear join operation, for multivectors  $x,y$ :
$$
x \vee y := (x ^ {*} \wedge y ^ {*}) ^ {- \star} = ((x \tilde {\mathcal {I}}) \wedge (y \tilde {\mathcal {I}})) \mathcal {I}.
$$
Lemma 5. In Euclidean algebra  $\mathbb{G}_{n,0,0}$ , the join is  $\mathrm{Spin}(n,0,0)$  equivariant. Furthermore, it is  $\mathrm{Pin}(n,0,0)$  equivariant if and only if  $n$  is even.
Proof. The join is equivariant to the transformations from the group  $\mathrm{Spin}(n,0,0)$ , which consists of the product of an even amount of unit vectors, because such transformations leave the pseudoscalar  $\mathcal{I}$  invariant, and the operation consists otherwise of equivariant geometric and wedge products.
However, let  $e_{12\dots n} = \mathcal{I}\in \operatorname {Pin}(n,0,0)$  be the point reflection, which negates vectors of odd grades by the grade involution:  $\mathcal{I}[x] = \hat{x}$ . Let  $x$  be a  $k$ -vector and  $y$  an  $l$ -vector. Then  $x\vee y$  is a vector of grade  $n - ((n - k) + (n - l)) = k + l - n$  (and zero if  $k + l < n$ ). Given that the join is bilinear, the inputs transform as  $(-1)^{k + l}$  under the point reflection, while the transformed output gets a sign  $(-1)^{k + l - n}$ . Thus for odd  $n$ , the join is not  $\operatorname {Pin}(n,0,0)$  equivariant.
To address this, given a pseudoscalar  $z = \lambda \mathcal{I}$ , we can create an equivariant Euclidean join via:
$$
\operatorname {E q u i J o i n} (x, y, z = \lambda \mathcal {I}) := \lambda (x \vee y) = \lambda \left(x ^ {*} \wedge y ^ {*}\right) ^ {- *}. \tag {7}
$$
Proposition 6. In Euclidean algebra  $\mathbb{G}_{n,0,0}$ , the equivariant join EquiJoin is  $\mathrm{Pin}(n,0,0)$  equivariant.
Proof. The EquiJoin is a multilinear operation, so for  $k$ -vector  $x$  and  $l$ -vector  $y$ , under a point reflection, the input gets a sign  $(-1)^{k + l + n}$  while the output is still a  $k + l - n$  vector and gets sign  $(-1)^{k + l - n}$ . These signs differ by even  $(-1)^{2n} = 1$  and thus EquiJoin is  $\operatorname{Pin}(n,0,1)$ -equivariant.
We prove two equalities of the Euclidean join which we use later.
Lemma 6. In the algebra  $\mathbb{G}_{n,0,0}$ , let  $v$  be a vector and  $x, y$  be multivectors. Then
$$
v \rfloor (x \vee y) = (v \rfloor x) \vee y \tag {8}
$$
$$
x \vee (v \rfloor y) = - (- 1) ^ {n} \widehat {v \rfloor x} \vee y. \tag {9}
$$
Proof. For the first statement, let  $a$  be a  $k$ -vector and  $b$  an  $l$ -vector. Then note the following two identities:
$$
a \vee b = \left\langle a ^ {*} b \tilde {\mathcal {L}} \right\rangle_ {2 n - k - l} \mathcal {I} = \left\langle a ^ {*} b \right\rangle_ {n - (2 n - k - l)} \tilde {\mathcal {L}} \mathcal {I} = \left\langle a ^ {*} b \right\rangle_ {k + l - n} = a ^ {*} \rfloor b,
$$
$$
(v \rfloor a) ^ {*} = \langle v a \rangle_ {k - 1} \tilde {\mathcal {I}} = \langle v a \tilde {\mathcal {I}} \rangle_ {n - k + 1} = \langle v a ^ {*} \rangle_ {n - k + 1} = v \rfloor (a ^ {*}).
$$
Combining these and the associativity of  $\rfloor$  gives:
$$
(v \rfloor a) \vee b = (v \rfloor a) ^ {*} \rfloor b = v \rfloor (a ^ {*}) \rfloor b = v \rfloor (a \vee b)
$$
For the second statement, swapping  $k$ -vector  $a$  and  $l$ -vector  $b$  incurs  $a \vee b = (a^{*} \wedge b^{*})^{-*} = (-1)^{(n - k)(n - l)}(b^{*} \wedge a^{*})^{-*} = (-1)^{(n - k)(n - l)}(b \vee a)$ . Then we get:
$$
\begin{array}{l} a \vee (v \rfloor b) = (- 1) ^ {(n - k) (n - l - 1)} (v \rfloor b) \vee a \\ = (- 1) ^ {(n - k) (n - l - 1)} v \rfloor (b \vee a) \\ = (- 1) ^ {(n - k) (n - l - 1) + (n - k) (n - l)} v \rfloor (a \vee b) \\ = (- 1) ^ {(n - k) (n - l - 1) + (n - k) (n - l)} \left(v \rfloor a\right) \vee b \\ = (- 1) ^ {(n - k) (2 n - 2 l - 1)} (v \rfloor a) \vee b \\ = (- 1) ^ {k - n} (v \rfloor a) \vee b \\ = - (- 1) ^ {k - 1 - n} (v \rfloor a) \vee b \\ = - (- 1) ^ {n} \widehat {(v \rfloor a)} \vee b. \\ \end{array}
$$
This generalizes to multivectors  $x, y$  by linearity.
Dual and join in projective algebra For the projective algebra  $\mathbb{G}_{n,0,1}$  with its degenerate inner product, the dual definition of Eq. 6 unfortunately does not yield a bijective dual. For example,  $e_0 e_{012\dots n} = 0$ . For a bijective dual that yields the complementary indices on basis elements, a different definition is needed. Following Dorst [17], we use the right complement. This involves choosing an orthogonal basis and then for a basis  $k$ -vector  $x$  to define the dual  $x^*$  to be the basis  $n + 1 - k$ -vector such that  $x \wedge x^* = \mathcal{I}$ , for pseudoscalar  $\mathcal{I} = e_{012\dots n}$ . For example, this gives dual  $e_{01}^* = e_{23}$ , so that  $e_{01} \wedge e_{23} = e_{0123}$ .
This dual is still easy to compute numerically, but it can no longer be constructed solely from operations available to us in the geometric algebra. This makes it more difficult to reason about equivariance.
Proposition 7. In the algebra  $\mathbb{G}_{n,0,1}$ , the join  $a \lor b = (a^* \land b^*)^{-*}$  is equivariant to  $\operatorname{Spin}(n,0,1)$ .
Proof. Even though the dual is not a  $\mathbb{G}_{n,0,1}$  operation, we can express the join in the algebra as follows. We decompose a  $k$ -vector  $x$  as  $x = t_x + e_0p_x$  into a Euclidean  $k$ -vector  $t_x$  and a Euclidean  $(k - 1)$ -vector  $p_x$ . Then Dorst [17, Eq (35)] computes the following expression
$$
\begin{array}{l} \left(t _ {x} + e _ {0} p _ {x}\right) \vee \left(t _ {y} + e _ {0} p _ {y}\right) = \left(\left(t _ {x} + e _ {0} p _ {x}\right) ^ {*} \wedge \left(t _ {y} + e _ {0} p _ {y}\right) ^ {*}\right) ^ {- *} \\ = t _ {x} \vee_ {\mathrm {E u c}} p _ {y} + (- 1) ^ {n} \widehat {p _ {x}} \vee_ {\mathrm {E u c}} t _ {y} + e _ {0} \left(p _ {x} \vee_ {\mathrm {E u c}} p _ {y}\right), \tag {10} \\ \end{array}
$$
where the Euclidean join of vectors  $a, b$  in the projective algebra is defined to equal the join of the corresponding vectors in the Euclidean algebra:
$$
a \vee_ {\mathrm {E u c}} b := ((\widetilde {a e _ {1 2 \dots n}}) \wedge (\widetilde {b e _ {1 2 \dots n}})) e _ {1 2 \dots n}
$$
The operation  $a \vee_{\mathrm{Euc}} b$  is  $\operatorname{Spin}(n,0,0)$  equivariant, as discussed in Lemma 5. For any rotation  $r \in \operatorname{Spin}(n,0,1)$  (which is Euclidean), we thus have  $r[a \vee_{\mathrm{Euc}} b] = r[a] \vee_{\mathrm{Euc}} r[b]$ . This makes the PGA dual in Eq. (10) equivariant to the rotational subgroup  $\operatorname{Spin}(n,0,0) \subset \operatorname{Spin}(n,0,1)$ .
We also need to show equivariance by translations. Let  $v$  be a Euclidean vector and  $\tau = 1 - e_0v / 2$  a translation. Translations act by shifting with  $e_0$  times a contraction:  $\tau[x] = x - e_0(v\rfloor x)$ . This acts on the decomposed  $x$  in the following way:  $\tau[t_x + e_0p_x] = \tau[t_x] + e_0p_x = t_x + e_0(p_x - v\rfloor t_x)$ .
We thus get:
$$
\begin{array}{l} \tau [ x ] \vee \tau [ y ] = (\tau [ t _ {x} ] + e _ {0} p _ {x}) \vee (\tau [ t _ {y} ] + e _ {0} p _ {y}) \\ = \left(t _ {x} + e _ {0} \left(p _ {x} - v \rfloor t\right)\right) \vee \left(t _ {y} + e _ {0} \left(p _ {y} - v \rfloor t _ {y}\right)\right) \\ = x \vee y - t _ {x} \vee_ {\mathrm {E u c}} (v \bigupharpoonright t _ {y}) - (- 1) ^ {n} \widehat {v \bigupharpoonright t _ {x}} \vee_ {\mathrm {E u c}} t _ {y} \\ - e _ {0} \left(p _ {x} \vee_ {\mathrm {E u c}} (v \rfloor t _ {y}) + (v \rfloor t _ {x}) \vee_ {\mathrm {E u c}} p _ {y}\right) \quad \text {U s e d (1 0)} \\ = x \vee y - e _ {0} \left(p _ {x} \vee_ {\mathrm {E u c}} (v \rfloor t _ {y}) + (v \rfloor t _ {x}) \vee_ {\mathrm {E u c}} p _ {y}\right) \quad \text {U s e d} (9) \\ = x \vee y - e _ {0} \left(- (- 1) ^ {n} \widehat {v \perp p _ {x}} \vee_ {\mathrm {E u c}} t _ {y} + (v \perp t _ {x}) \vee_ {\mathrm {E u c}} p _ {y}\right) (9) \\ = x \vee y - e _ {0} \left((- 1) ^ {n} (v \rfloor \widehat {p _ {x}}) \vee_ {\mathrm {E u c}} t _ {y} + (v \rfloor t _ {x}) \vee_ {\mathrm {E u c}} p _ {y}\right) \\ = x \vee y - e _ {0} (v \rfloor \left\{\left(- 1\right) ^ {n} \widehat {p _ {x}} \vee_ {\mathrm {E u c}} t _ {y} + t _ {x} \vee_ {\mathrm {E u c}} p _ {y} \right\}) \quad \text {U s e d} (8) \\ = \tau [ x \vee y ] \\ \end{array}
$$
The join is thus equivariant  $^{12}$  to translations and rotations and is therefore  $\mathrm{Spin}(n,0,1)$  equivariant.
Similar to the Euclidean case, we obtain full  $\mathrm{Pin}(n,0,1)$  equivariance via multiplication with a pseudoscalar. We thus also use the EquiJoin from Eq. (7) in the projective case.
A.3 Expressivity
As also noted in Ref. [17], in the projective algebra, the geometric product itself is unable to compute many quantities. It is thus insufficient to build expressive networks. This follows from the fact that the geometric product preserves norms.
12The authors agree with the reader that there must be an easier way to prove this.
Lemma 7. For the algebra  $\mathbb{G}_{n,0,r}$ , for multivectors  $x,y$ , we have  $\|xy\| = \|x\| \|y\|$ .
$$
\text {P r o o f .} \| x y \| ^ {2} = x y \widetilde {x y} = x y \tilde {y} \tilde {x} = x \| y \| ^ {2} \tilde {x} = x \tilde {x} \| y \| ^ {2} = \| x \| ^ {2} \| y \| ^ {2}
$$
Hence, any null vector in the algebra can never be mapped to a non-null vector, including scalars. The projective algebra can have substantial information encoded as null vector, such as the position of points. This information can never influence scalars or null vectors. For example, there is no way to compute the distance (a scalar) between points just using the projective algebra. In the GATr architecture, the input to the MLPs that operate on the scalars, or the attention weights, thus could not be affected by the null information, had we only used the geometric product on multivectors.
To address this limitation, we use besides the geometric product also the join. The join is able to compute such quantities. For example, given the Euclidean vector  $e_{12\dots n}$ , we can map a null vector  $x = e_{012\dots k}$  to a non-null vector  $x \vee e_{12\dots n} \propto e_{12\dots k}$ .
Equivariant join One of the primitives in GATr is the equivariant join EquiJoin  $(x,y;z)$ , which we define in Eq. (7). For  $x$  and  $y$ , we use hidden states of the neural network after the previous layer. The nature of  $z$  is different: it is a reference multivector and only necessary to ensure that the function correctly changes sign under mirrorings of the inputs. We find it beneficial to choose this reference multivector  $z$  based on the input data rather than the hidden representations, and choose it as the mean of all inputs to the network.
Auxiliary scalars In addition to multivector representations, GATr supports auxiliary scalar representations, for instance to describe non-geometric side information such as positional encodings or diffusion time embeddings. In most layers, these scalar variables are processed like in a standard Transformer, with two exceptions. In linear layers, we allow for the scalar components of multivectors and the auxiliary scalars to freely mix. In the attention operation, we compute attention weights as
$$
\operatorname {S o f t m a x} _ {i} \left(\frac {\sum_ {c} \left\langle q _ {i ^ {\prime} c} ^ {M V} , k _ {i c} ^ {M V} \right\rangle + \sum_ {c} q _ {i ^ {\prime} c} ^ {s} k _ {i c} ^ {s}}{\sqrt {8 n _ {M V} + n _ {s}}}\right), \tag {11}
$$
where  $q^{MV}$  and  $k^{MV}$  are query and key multivector representations,  $q^s$  and  $k^s$  are query and key scalar representations,  $n_{MV}$  is the number of multivector channels, and  $n_s$  is the number of scalar channels.
Distance-aware dot-product attention As we argue in Sec. 3.3, it can be beneficial to extend queries and keys with nonlinear features. We use the following choice:
$$
\phi (q) = \omega (q _ {\backslash 0}) \left( \begin{array}{c} q _ {\backslash 0} ^ {2} \\ \sum_ {i} q _ {\backslash i} ^ {2} \\ q _ {\backslash 0} q _ {\backslash 1} \\ q _ {\backslash 0} q _ {\backslash 2} \\ q _ {\backslash 0} q _ {\backslash 3} \end{array} \right) \quad \text {a n d} \quad \psi (k) = \omega (k _ {\backslash 0}) \left( \begin{array}{c} - \sum_ {i} k _ {\backslash i} ^ {2} \\ - k _ {\backslash 0} ^ {2} \\ 2 k _ {\backslash 0} k _ {\backslash 1} \\ 2 k _ {\backslash 0} k _ {\backslash 2} \\ 2 k _ {\backslash 0} k _ {\backslash 3} \end{array} \right) \quad \text {w i t h} \quad \omega (x) = \frac {x}{x ^ {2} + \epsilon}.
$$
Here the index  $\backslash i$  denotes the trivector component with all indices but  $i$ . Then
$$
\phi (q) \cdot \psi (k) = - \omega (q _ {\backslash 0}) \omega (k _ {\backslash 0}) \| k _ {\backslash 0} \vec {q} - q _ {\backslash 0} \vec {k} \| _ {\mathbb {R} ^ {3}} ^ {2},
$$
where we use the shorthand  $\vec{x} = (x_{\backslash 1}, x_{\backslash 2}, x_{\backslash 3})^T$ . When the trivector components of queries and keys represent 3D points, with  $q_{\backslash 0} = k_{\backslash 0} = 1$ , it becomes proportional to the pairwise negative squared Euclidean distance between the points.
In this notation, a trivector  $q = (q_{0}, \vec{q})$  transforms under rotation  $R \in \mathrm{O}(3)$  and translation  $\vec{t} \in \mathbb{R}^3$
as  $(q_{\backslash 0},\vec{q})\mapsto (q_{\backslash 0},R\vec{q} +q_{\backslash 0}\vec{t})$  . It is easy to see that the inner product is invariant:
$$
\begin{array}{l} \phi (q) \cdot \psi (k) \mapsto - \omega (q _ {\backslash 0}) \omega (k _ {\backslash 0}) \| k _ {\backslash 0} (R \vec {q} + q _ {\backslash 0} \vec {t}) - q _ {\backslash 0} (R \vec {k} + k _ {\backslash 0} \vec {t}) \| _ {\mathbb {R} ^ {3}} ^ {2} \\ = - \omega (q _ {\backslash 0}) \omega (k _ {\backslash 0}) \| R (k _ {\backslash 0} \vec {q} - q _ {\backslash 0} \vec {k}) + q _ {\backslash 0} k _ {\backslash 0} \vec {t} - q _ {\backslash 0} k _ {\backslash 0} \vec {t} \| _ {\mathbb {R} ^ {3}} ^ {2} \\ = - \omega (q _ {\backslash 0}) \omega (k _ {\backslash 0}) \| k _ {\backslash 0} \vec {q} - q _ {\backslash 0} \vec {k} \| _ {\mathbb {R} ^ {3}} ^ {2} \\ = \phi (q) \cdot \psi (k). \\ \end{array}
$$
The normalization function  $\omega$  could have been chosen as  $\omega(x) = 1$ , but this would make the attention logit a fourth-order polynomial of the keys and queries, which tends to lead to instabilities when taking the exponential in the softmax. The choice above makes the attention logit a quadratic function for larger values of the keys and queries, just like regular dot-product attention.
For the interested reader, we'd like to note that the above inner product is closely related to the inner product in another geometric algebra, the conformal geometric algebra. We plan on exploring this connection in future work.
With this final piece, GATr's attention mechanism computes attention weights from three sources: the  $\mathbb{G}_{3,0,1}$  inner product of the multivector queries and keys  $\langle q,k\rangle$ , the distance-aware inner product of the nonlinear features  $\phi (q)\cdot \psi (k)$ , and the Euclidean inner product of the auxiliary scalars  $q_{s}\cdot k_{s}$ .