SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Maksim Zhdanov*
AMLab, University of Amsterdam m.zhdanov@uva.nl
Nico Hoffmann
Helmholtz-Zentrum Dresden-Rossendorf
Gabriele Cesa
AMLab, University of Amsterdam
Steerable convolutional neural networks (CNNs) provide a general framework for building neural networks equivariant to translations and transformations of an origin-preserving group  $G$ , such as reflections and rotations. They rely on standard convolutions with  $G$ -steerable kernels obtained by analytically solving the group-specific equivariance constraint imposed onto the kernel space. As the solution is tailored to a particular group  $G$ , implementing a kernel basis does not generalize to other symmetry transformations, complicating the development of general group equivariant models. We propose using implicit neural representation via multi-layer perceptrons (MLPs) to parameterize  $G$ -steerable kernels. The resulting framework offers a simple and flexible way to implement Steerable CNNs and generalizes to any group  $G$  for which a  $G$ -equivariant MLP can be built. We prove the effectiveness of our method on multiple tasks, including N-body simulations, point cloud classification and molecular property prediction.
Equivariant deep learning is a powerful tool for high-dimensional problems with known data domain symmetry. By incorporating this knowledge as inductive biases into neural networks, the hypothesis class of functions can be significantly restricted, leading to improved data efficiency and generalization performance [6]. Convolutional neural networks [28] (CNNs) are a prominent example as they are equivariant to translations. Group-equivariant CNNs [9] (G-CNNs) generalize CNNs to exploit a larger number of symmetries via group convolutions, making them equivariant to the desired symmetry group, such as the Euclidean group  $E(n)$  that encompasses translations, rotations, and reflections in  $n$ -dimensional Euclidean space. In physics and chemistry, many important problems, such as molecular modelling or point clouds, rely on the Euclidean group. Objects defined in physical space have properties that are invariant or equivariant to Euclidean transformations, and respecting this underlying symmetry is often desired for the model to perform as expected.
Neural networks can be parameterized in various ways to incorporate equivariance to the Euclidean group. One option is to use a message-passing neural network as a backbone and compute/update messages equivariantly via convolutions. This approach generalizes well to point clouds and graphs and offers the high expressivity of graph neural networks. Equivariant convolutional operators can be further categorized as regular [3, 9, 14, 26] or steerable group convolutions [10, 11, 52]. The latter recently proved to be especially suitable for incorporating physical and geometric quantities into a model [5]. The key idea behind Steerable CNNs is using standard convolution - which guarantees translation equivariance - with  $G$ -steerable kernels that ensure commutativity with the transformations of another group  $G$ , such as rotations. The commutation requirement imposes a constraint onto the kernel space that must be solved analytically for each group  $G$ . This, in turn, does not allow generalizing a convolution operator tailored to a specific group to other symmetry transformations. In
*Work done while at Helmholtz-Zentrum Dresden-Rossendorf.
the case of the Euclidean group, Cesa et al.[7] proposed a generally applicable way of parameterizing steerable convolutions for sub-groups of  $E(n)$ . The method relies on adapting a pre-defined kernel basis explicitly developed for the group  $E(n)$  to an arbitrary sub-group by using group restriction.
However, because only a finite basis can be chosen, a basis tailored for  $E(n)$  can be sub-optimal in terms of expressiveness for its sub-groups; see Section 3.6 more details. Hence, we propose an alternative way of building steerable convolutions based on implicit neural kernels, i.e. convolutional kernels implemented as continuous functions parameterized by MLPs [38, 39]. We demonstrate how  $G$ -steerable convolutions with implicit neural kernels can be implemented from scratch for any sub-group  $G$  of the orthogonal group  $O(n)$ . The method allows us to ultimately minimize requirements to implement equivariance to new groups; see Section 3.3. The flexibility of neural functions also permits the injection of geometric and physical quantities in point convolutions, increasing their expressiveness [5]; see Section 3.2. We validate our framework on synthetic N-body simulation problem, point-cloud data (ModelNet-40 [56]) and molecular data (QM9 [55]) and demonstrate key bene
fits of our approach such as flexibility and generalizability. Besides, we demonstrate that implicit kernels allow Steerable CNNs to achieve performance competitive with state-of-the-art models and surpass them with the correct choice of the task-specific symmetry group.
Figure 1: Illustration of the proposed approach: computing the response of an implicit kernel  $k$  (background) of a steerable point convolution for the node  $i$  (upper right corner) of a graph with steerable features (visualized as spherical functions).
2 Background: Steerable Convolutions
In this work, we propose a general solution to easily build Steerable CNNs equivariant to translations and any compact group  $G^3$ . In Section 2.1, we provide some necessary prerequisites from group theory and group representation theory [43]. Then, we review the framework of Steerable CNNs and discuss the constraint it induces on the convolutional kernels in Section 2.2.
2.1 Groups, Representations and Equivariance
Definition 1 (Group action). An action of a group  $G$  on a set  $\mathcal{X}$  is a mapping  $(g, x) \mapsto g.x$  associating a group element  $g \in G$  and a point  $x \in \mathcal{X}$  with some other point on  $\mathcal{X}$  such that the following holds:
$$
g. (h. x) = (g h). x \quad \forall g, h \in G, x \in \mathcal {X}
$$
Definition 2 (Group representation). A linear representation  $\rho$  of a group  $G$  is a map  $\rho : G \to \mathbb{R}^{d \times d}$  that assigns an invertible matrix  $\rho(g) \forall g \in G$  and satisfies the following condition
$$
\rho (g h) = \rho (g) \rho (h) \forall g, h \in G.
$$
A group representation  $\rho(g): V \to V$  furthermore can be seen as a linear action of  $G$  on a vector space  $V$ . Additionally, if two (or more) vectors  $v_1 \in \mathbb{R}^{d_1}$  and  $v_2 \in \mathbb{R}^{d_2}$  belong to vector spaces transforming under representations  $\rho_1$  and  $\rho_2$ , their concatenation  $v_1 \oplus v_2 \in \mathbb{R}^{d_1 + d_2}$  transforms under the direct sum representation  $\rho_1 \oplus \rho_2$ .  $(\rho_1 \oplus \rho_2)(g)$  is a  $d_1 + d_2$  dimensional block-diagonal matrix containing  $\rho_1(g)$  and  $\rho_2(g)$  in its diagonal blocks.
There are three types of group representations that are important for the definition of our method:
- Trivial representation: all elements of  $G$  act on  $V$  as the identity mapping of  $V$ , i.e.  $\rho(g) = I$ .
- Standard representation: the group  $O(n)$  of all orthogonal  $n \times n$  matrices has a natural action on  $V = \mathbb{R}^n$ ; similarly, if  $G$  is a subgroup of  $O(n)$ , elements of  $G$  can act on  $V = \mathbb{R}^n$  via the inclusion mapping, i.e.  $\rho_{st}(g) = g \in \mathbb{R}^{n \times n}$ .
<sup>3</sup>We provide the definition for a group in Appendix A.1.
- Irreducible representations is a collection of generally known representations that can be used as a building block for larger representations via direct sum. As argued in [49], Steerable CNNs can be parameterized without loss of generality solely in terms of irreducible representations.
Definition 3 (Equivalence). Let us have two spaces  $\mathcal{X},\mathcal{Y}$  endowed with a symmetry group  $G$  , i.e. with an action defined on them. A function  $\phi :\mathcal{X}\to \mathcal{Y}$  is called  $G$  -equivariant, if it commutes with the action of  $G$  on the two spaces, i.e.  $\phi (g.x) = g.\phi (x)$  for all  $g\in G,x\in X$
As we have discussed, the layers of conventional CNNs are translation equivariant by design; however, they do not commute with transformations of other groups, such as rotations and reflections.
2.2 Steerable CNNs
Steerable CNNs provide a more general framework that allows building convolutions that are equivariant to a group of isometries of  $\mathbb{R}^n$ , i.e.  $(\mathbb{R}^n, +) \rtimes G \leq E(n)$ . Those groups are decomposable as a semi-direct product of the translations group  $(\mathbb{R}^n, +)$  and an origin-preserving compact group  $G \leq O(n)$ , where  $O(n)$  is the group of  $n$ -dimensional rotations and reflections. As translation equivariance is guaranteed [52] by the convolution operator itself, one only has to ensure equivariance to  $G$ . See [51] for a more in-depth description of Steerable CNNs.
The feature spaces of Steerable CNNs are described as collections of feature fields. A feature field of type  $\rho$  is a feature map  $f: \mathbb{R}^n \to \mathbb{R}^d$  endowed with a group representation  $\rho: G \to \mathbb{R}^{d \times d}$  that defines how an element  $g \in G$  transforms the feature:
$$
[ g. f ] (x) := \rho (g) f \left(g ^ {- 1}. x\right) \tag {1}
$$
Furthermore, each convolutional layer is a map between feature fields. For the map to be equivariant, it must preserve the transformation laws of its input and output feature fields. In practice, it means that the following constraint onto the space of convolution kernels must be applied [52]:
$$
k (g. x) = \rho_ {o u t} (g) k (x) \rho_ {i n} (g) ^ {- 1} \quad \forall g \in G, x \in \mathbb {R} ^ {n} \tag {2}
$$
where  $k: \mathbb{R}^n \to \mathbb{R}^{d_{out} \times d_{in}}$ , and  $\rho_{in}: G \to \mathbb{R}^{d_{in} \times d_{in}}$ ,  $\rho_{out}: G \to \mathbb{R}^{d_{out} \times d_{out}}$  are respective representations of input and output feature fields.
To parameterize the kernel  $k$ , the constraint in equation 2 needs to be solved analytically for each specific group  $G$  of interest. This renders a general solution challenging to obtain and limits the applicability of steerable convolutions.
3 Implicit neural kernels
Instead of deriving a steerable kernel basis for each particular group  $G$ , we propose parameterizing the kernel  $k: \mathbb{R}^n \to \mathbb{R}^{d_{out} \times d_{in}}$  with an MLP satisfying the constraint in Eq. 2. The approach only requires the  $G$ -equivariance of the MLP and suggests a flexible framework of implicit steerable convolutions that generalizes to arbitrary groups  $G \leq O(n)$ . We argue about the minimal requirements of this approach in Section 3.3.
We first define the kernel as an equivariant map between vector spaces that we model with an MLP (see Section 3.1). Then, we demonstrate that  $G$ -equivariance of an MLP is a sufficient condition for building the implicit representation of steerable kernels for compact groups. We indicate that the flexibility of neural representation allows expanding the input of a steerable kernel in Section 3.2. Next, we describe how a  $G$ -equivariant MLP can be implemented in section 3.3. Later, we describe how one can implement  $G$ -steerable point convolution in the form of equivariant message passing [5, 40] in Section 3.4 and its generalization to dense convolution in Section 3.5. Finally, we compare our method with the solution strategy proposed in [7] in Section 3.6.
4See Appendix A.1 for the definition of a semi-direct product.
To remain in the scope of the manuscript, we abstain from the mathematical definition of compact groups, which requires introducing topological groups. One can find more information about compact groups in [26].
3.1 Kernel vectorization and equivariance
Our goal is to implement the kernel  $k: \mathbb{R}^n \to \mathbb{R}^{d_{out} \times d_{in}}$  of a  $G$ -steerable convolution that maps between spaces of feature fields with representations  $\rho_{in}$  and  $\rho_{out}$ . The kernel itself is a function whose input in  $\mathbb{R}^n$  transforms under the standard representation  $\rho_{st}$  (as defined in Section 2.1) and which we will model with an MLP. Since MLPs typically output vectors, it is convenient to vectorize the  $d_{out} \times d_{in}$  output of the kernel. We denote the column-wise vectorization of a matrix  $M \in \mathbb{R}^{d_1 \times d_2}$  as  $vec(M) \in \mathbb{R}^{d_1 d_2}$ . Henceforth, we will consider kernel's vector form  $vec(k(\cdot)): \mathbb{R}^n \to \mathbb{R}^{d_{out} d_{in}}$ .
Let  $\otimes$  denote the Kronecker product between two matrices. Then,  $\rho_{\otimes}(g) \coloneqq \rho_{in}(g) \otimes \rho_{out}(g)$  is also a representation of  $G$ . We suggest an implicit representation of the vectorized kernel  $k$  using an  $G$ -equivariant MLP  $\phi: \mathbb{R}^n \to \mathbb{R}^{d_{out}d_{in}}$  based on the following lemma (see A.2 for the proof):
Lemma 1. If a kernel  $k$  is parameterized by a  $G$ -equivariant  $MLP\phi$  with input representation  $\rho_{st}$  and output representation  $\rho_{\otimes} \coloneqq \rho_{in}\otimes \rho_{out}$ , i.e.  $vec(k)(x)\coloneqq \phi (x)$ , then the kernel satisfies the equivariance constraint in Equation 2 for a compact group  $G$ .
In other words,  $G$ -equivariance of MLP is a sufficient condition for  $G$ -equivariance of the convolutional layer whose kernel it parameterizes. Using implicit kernels also has a very favourable property - it allows arbitrary steerable features as its input, which we discuss in the following section.
3.2 Expanding the input
Note that in the case of standard steerable convolutions, the input  $x \in \mathbb{R}^n$  of a kernel is usually only the difference between the spatial position of two points. However, there is no requirement that would disallow the expansion of the input space except for practical reasons. Hence, here we augment steerable kernels with an additional feature vector  $z \in \mathbb{R}^{d_z}$ . This formulation allows us to incorporate relevant information in convolutional layers, such as physical and geometric features. For example, when performing convolutions on molecular graphs,  $z$  can encode the input and output atoms' types and yield different responses for different atoms. If introducing additional arguments into a kernel, the steerability constraint in equation 2 should be adapted to account for transformations of  $z$ :
$$
k (g. x, \rho_ {z} (g) z) = \rho_ {\text {o u t}} (g) k (x, z) \rho_ {\text {i n}} (g) ^ {- 1} \tag {3}
$$
which must hold for all  $g \in G, x \in \mathbb{R}^n, z \in \mathbb{R}^{d_z}$ , where  $\rho_z: G \to \mathbb{R}^{d_z \times d_z}$  is the representation of  $G$  acting on the additional features.
Again, analytically solving the constraint 3 to find a kernel basis for arbitrary  $\rho_z$  is generally unfeasible. Note also that the solution strategy proposed in [7] requires a basis for functions over  $\mathbb{R}^{n + d_z}$ , whose size tends to grow exponentially with  $d_{z}$  and, therefore, is not suitable. Alternatively, we can now use the flexibility of neural representation and introduce additional features into a kernel at no cost.
3.3 Implementing a  $G$ -equivariant MLP
We are now interested in how to build an MLP that is equivariant to the transformations of the group  $G$ , i.e. a sequence of equivariant linear layers alternated with equivariant non-linearities [15, 37, 44]. It is important to say that our approach does not rely on a specific implementation of  $G$ -MLPs, and any algorithm of preference might be used (e.g. [15] or enforcing via an additional loss term).
The approach we employed in our experiments is described below. Since the irreducible representations of  $G$  are typically known<sup>7</sup>, one can always rely on the following properties: 1) any representation  $\rho$  of a compact group  $G$  can be decomposed as a direct sum of irreducible representations  $\rho(g) = Q^T\left(\bigoplus_{i \in I} \psi_i(g)\right)Q$  with a change of basis  $Q^8$  and 2) Schur's Lemma, which states that there exist equivariant linear maps only between the irreducible representation of the same kind<sup>9</sup>. Hence, one can apply the right change of basis to the input and output of a linear layer and then learn only maps between input and output channels associated with the same irreducible representations. In the context of implicit kernels, the tensor product representation  $\rho_{in} \otimes \rho_{out}$  in the last layer is
This representation is formally known as the tensor product of the two representations.
This is a reasonable assumption since irreducible representations are often used to construct other representations used in Steerable CNNs. If not, there exist numerical methods to discover them from the group algebra.
[7] describes and implements a numerical method for this.
${}^{9}$  Typically,these maps only include scalar multiples of the identity matrix.
decomposed by a matrix containing the Clebsch-Gordan coefficients, which often appears in the analytical solutions of the kernel constraint in the related works. Note that the non-linearity  $\sigma$  used in the Steerable CNNs are  $G$ -equivariant and, therefore, can be used for the MLP as well.
3.4  $G$ -steerable point convolutions
Point convolutions operate on point clouds - sets of  $N$  points endowed with spatial information  $X = \{x_{i}\}_{i = 0}^{N - 1}\in \mathbb{R}^{n\times N}$ . A point cloud thus provides a natural discretization of the data domain, which renders a convolution operator as follows:
$$
f _ {o u t} \left(x _ {i}\right) = \left(k * f _ {i n}\right) \left(x _ {i}\right) = \sum_ {0 \leq j \leq N - 1} k \left(x _ {i} - x _ {j}\right) f _ {i n} \left(x _ {j}\right) \tag {4}
$$
To reduce the computational cost for large objects, one can induce connectivity onto  $x \in X$  and represent a point cloud as a graph  $\mathcal{G} = (\mathcal{V}, \mathcal{E})$  with nodes  $v_i \in \mathcal{V}$  and edges  $e_{ij} \in \mathcal{E} \subseteq \mathcal{V} \times \mathcal{V}$ , where each node  $v_i$  has a spatial location  $x_i$ , node features  $z_i$  and a corresponding feature map  $f_{in}(x_i)$  (see Figure 1). Additionally, each edge  $e_{ij}$  can have an attribute vector  $z_{ij}$  assigned to it (as in Section 3.2). This allows for a learnable message-passing point convolution whose computational cost scales linearly with the number of edges:
$$
f _ {o u t} \left(x _ {i}\right) = \left(k * f _ {i n}\right) \left(x _ {i}\right) = \sum_ {j \in \mathcal {N} (i)} k \left(x _ {i} - x _ {j}, z _ {i}, z _ {j}, z _ {i j}\right) f _ {i n} \left(x _ {j}\right) \tag {5}
$$
where  $\mathcal{N}(i) = \{j:(v_i,v_j)\in \mathcal{E}\}$  and the kernel  $k(\cdot)$  is parameterized by a  $G$ -equivariant MLP.
3.5 Extension to  $G$ -steerable CNNs
Straightforwardly, the proposed method can be extended to dense convolutions. In such case, the kernel is defined as  $k: \mathbb{R}^n \to \mathbb{R}^{c_{out} \times c_{in} \times K^n}$  - that is, a continuous function that returns a collection of  $K \times K \times \ldots$  kernels given a relative position (and, optionally, arbitrary steerable features). In this case, the vectorized kernel is parameterized in exactly the same way as described above but is estimated at the center of each pixel.
3.6 Comparison with the analytical solution
A general basis for any  $G$ -steerable kernel is described in [7]. Essentially, it relies on two ingredients: i) a (pre-defined) finite  $G$ -steerable basis [17] for scalar filters and ii) a learnable equivariant linear map. Let us look at those in detail. Firstly, a finite  $G$ -steerable basis is essentially a collection of  $B$  orthogonal functions, i.e.  $Y:\mathbb{R}^n\to \mathbb{R}^B$ , with the following equivariant property:  $Y(g.x) = \rho_{Y}(g)Y(x)$ , for some representation  $\rho_{Y}$  of  $G$ . The linear map in ii), then, is a general equivariant linear layer, whose input and output transform as  $\rho_{Y}$  and  $\rho_{in}\otimes \rho_{out}$ .
In practice, it means that one has to provide a pre-defined basis  $Y$  for a group of interest  $G$ . Since the design might not be straightforward, [7] suggest a way to reuse an already derived  $O(n)$  steerable basis for any subgroup  $G \subset O(n)$ . While general, such a solution can be sub-optimal. For example, if  $n = 3$ , an  $O(3)$ -steerable basis has local support inside a sphere, which is suitable for the group of 3D rotations  $SO(3)$  but not ideal for cylindrical symmetries, i.e. when  $G$  is the group  $SO(2)$  of planar rotations around the  $Z$  axis.
In comparison, the solution proposed in Section 3.1 replaces the pre-defined basis  $Y$  with a learnable  $G$ -MLP. It allows us to learn  $G$ -specific kernels without relying on a pre-derived basis for a larger group, which in turn means that we can theoretically obtain  $G$ -optimal kernel basis via learning (see A.3 for further details). Furthermore, the kernels defined in Section 3.1 can now be conditioned on arbitrary steerable features, which makes them more task-specific and expressive. In the context of a general basis, one can interpret the last linear layer of an implicit kernel as the map in  $ii$ , and the activations before this layer as a learnable version of the basis  $Y$  in  $i$ .
Group convolutions. Multiple architectures were proposed to achieve equivariance to a certain symmetry group. It has been proven [26] that the convolutional structure is a sufficient condition for
building a model that is equivariant to translations and actions of a compact group. One can separate group convolutions into two classes depending on the space on which the convolution operates: regular [3, 4, 9, 14, 26] and steerable [7, 10, 11, 46, 49, 52, 54] group convolutions. In the first case, the input signal is represented in terms of scalar fields on a group  $G$ , and the convolution relies on a discretization of the group space. Steerable convolutions are a class of  $G$ -equivariant convolutions that operate on feature fields over homogeneous spaces and achieve equivariance via constraining the kernel space. They further avoid the discretization of the group space and can reduce the equivariance error in the case of continuous groups.
$G$ -steerable kernels. In the case of rigid body motions  $G = SO(3)$ , the solution of the equivariance constraint is given by spherical harmonics modulated by an arbitrary continuous radial function, which was analytically obtained by Weiler et al.[52]. Lang and Weiler [27] then applied the Wigner-Eckart theorem to parametrize  $G$ -steerable kernel spaces over orbits of a compact  $G$ . The approach was later generalized by Cesa et al.[7], who proposed a solution for any compact sub-group of  $O(3)$  based on group restriction. Using this approach, one can obtain a kernel basis for a group  $G \leq H$  if the basis for  $H$  is known. Despite its generalizability, the method still requires a pre-defined basis for the group  $H$  that is further adapted to  $G$ . The resulting solution is not guaranteed to be optimal for  $G$ ; see Section 3.6. We note that the practical value of steerable kernels is also high as they can be used for convolution over arbitrary manifolds in the framework of Gauge CNNs [8, 21, 50].