SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Implicit kernels. Using the implicit representation of convolution kernels for regular CNNs is not novel. It was used, for example, to model long-range dependencies in sequential data [39] or for signal representation [45]. Romero et al.[39] demonstrated that such parametrization allows building shallower networks, thus requiring less computational resources to capture global information about the system. Continuous kernels were recently used to build an architecture [25] for processing data of arbitrary resolution, dimensionality and length, yet equivariant solutions are scarce [47]. Finzi et al.[14] proposed parametrizing convolutions on Lie groups as continuous scalar functions in the group space. The method relies on discretizing a continuous group, which might lead to undesirable stochasticity of the model's output. Instead, we use Steerable CNNs that define the kernel as a function on a homogeneous space. While the discretization of this space is still required, in most cases, it is naturally given by the data itself, e.g. for point clouds; hence, no sampling error arises. It is also important to mention the key difference between implicit kernels and applying  $G$ -MLPs [15] directly - the latter is incapable of processing image/volumetric data as convolutions do. Henceforth, we focus on CNNs with consideration for potential extensions to various data modalities.
Equivariant point convolutions. A particular momentum has been gained by point convolutions in the form of equivariant message-passing [5, 40, 41, 46] specifically for problems where symmetry provides a strong inductive bias such as molecular modelling [1] or physical simulations [18]. Thomas et al.[46] pioneered  $SE(3)$ -equivariant steerable convolutions whose kernels are based on spherical harmonics modulated by a radial function. The approach was further generalized by Batzner et al.[2], who uses an MLP conditioned on the relative location to parameterize the radial function, although the basis of spherical harmonics is preserved. Brandstetter et al.[5] demonstrated that introducing geometric and physical information into an equivariant message-passing model improves the expressivity on various tasks, which we also observe in this work. Note that, for  $G = SO(3)$  or  $O(3)$  and without additional edge features  $z$ , our MLP can only learn a function of the radius and, therefore, is equivalent to the models proposed in [2].
In this section, we implement Steerable CNNs with implicit kernels and apply them to various tasks<sup>10</sup>. First, we indicate the importance of correctly choosing the symmetry group on a synthetic N-body simulation problem where an external axial force breaks the rotational symmetry (see Section 5.2). Then, we prove the generalizability of the proposed approach as well as the gain in performance compared to the method proposed in [7] on ModelNet-40 (see Section 5.3). Afterwards, we show that one can introduce additional physical information into a kernel and significantly improve the performance of steerable convolutions on molecular data (see Section 5.4). Code and data to reproduce all experiments are available on GitHub.
10All datasets were downloaded and evaluated by Maksim Zhdanov (University of Amsterdam).
Figure 2: Final position estimation in the N-body system experiment, where particles are connected to one another and to the XY-plane by springs (see left upper corner). Our model with correct axial symmetry  $SO(2)$ , significantly outperforms the state-of-the-art model SEGNN [5], which is  $O(3)$ -equivariant, as the relative contribution of the plane string increases.
Figure 3: Performance comparison of Steerable CNNs on the rotated ModelNet-40 dataset for different  $G$ . Bars show mean average accuracy with error bars indicating standard deviation, both computed from 5 runs. The numbers above bars denote the performance gain of implicit kernels (orange) over [7] (blue). Statistically significant ( $p < 0.05$ ) positive differences are green, negative ones are red, and insignificant ones are yellow.  $SO(2)$  contains planar rotations around the  $Z$  axis,  $M$  and  $Inv$  contain mirroring along the  $X$  axis and origin respectively, while  $F$  contains rotations by  $\pi$  around the  $X$  axis.  $O(2) \cong SO(2) \rtimes M$  achieves the best accuracy as it best represents the real symmetries of the data.
Implicit kernels. To parameterize implicit kernels, we employ linear layers (see Section 3.3) followed by quotient ELU non-linearities [7]. The last layer generates a steerable vector, which we reshape to yield a final convolutional kernel. The  $G$ -MLP takes as input a steerable vector obtained via direct sum (concatenation) of batch-normalized harmonic polynomial representation of the relative location  $x$ , edge features  $z_{ij}$  and input node features  $z_i$ . See details on pre-processing and task-specific inputs in Appendix B.1.
Steerable convolutions. We employ steerable point convolutional layers as described in Equation 5. For each task, we tune hyperparameters of all models on validation data: the number of layers, the number of channels in each layer, the depth and the width of implicit kernels, and the number of training epochs. For all the experiments, except the one reported in Table 2, we yield a number of parameters similar to the baselines with a deviation of less than  $10\%$ . For ModelNet-40 experiments, our architecture is partially inspired by the model introduced in [34], which uses gradual downsampling of a point cloud<sup>11</sup>. For N-body and QM9 experiments, we add residual connections to each steerable layer to learn higher-frequency features [20]. For QM9 and ModelNet-40 experiments, the last steerable layer returns a vector with invariant features for each node, to which we apply global pooling to obtain a global representation of an object, which is further passed to a classification MLP. In the case of N-body experiments, we output the standard representation corresponding to the particles' coordinates. Details about optimization and model implementation for each task can be found in Appendix B.2.
5.2 The relevance of smaller  $G < O(n)$ : N-body simulation
Dataset. We conduct experiments using the N-body system [23], where particles are connected through springs, and their interaction is governed by Hooke's law. Similar to previous studies [5, 40], we modify the original trajectory prediction task to calculate the position of each particle in a 3-dimensional space, given its initial position and velocity. We attach each particle to the XY plane using strings with random equilibrium lengths and pre-defined stiffness, which can slide freely over the plane (see Figure 2, upper left corner). This additional force term breaks the rotational symmetry of the system into only azimuthal symmetry; this system resembles a simplified version
<sup>11</sup>Poulenard et al.[34] use kd-tree pooling to compute coarser point clouds, while we only use random sampling of points. Note also that the spherical quotient ELU we used is similar in spirit to the proposed functional non-linearity described there, yet it does not employ a deep MLP.
of the problem of molecule binding to the surface of a larger molecule. We choose the model's hyperparameters to have a similar parameter budget to the SEGNN model [5], which has shown state-of-the-art performance on a similar task (we also compare against a non-equivariant baseline; see Table A2). For all models, we use the highest frequency of hidden representations in both  $G$ -MLP and Steerable CNNs equal to 1. We use the velocity of a particle and the equilibrium length of the attached XY spring as input; the model's output transforms under the standard representation. We train a separate model for each value of plane strings' stiffness (3000 training points) to measure the impact of symmetry breaks on performance.
Results. As can be seen in Fig. 2, a Steerable CNN with azimuthal symmetry  $SO(2)$  significantly outperforms SEGNN, which is equivariant to a larger group  $O(3)$ . Since we introduced a force term that now breaks the rotational symmetry, SEGNN struggles to learn it. Furthermore, while in the default setting (plane strings' stiffness is 0), models achieve roughly the same performance, the absolute difference grows exponentially once the plane strings are introduced.
The synthetic problem is meant to show the importance of choosing the correct symmetry when designing a model. While the Euclidean group  $\mathrm{E}(3)$  is often enough for N body systems in a vacuum, it is important to be careful when an external influence or underlying structure can disrupt global symmetries and negatively impact a model with a larger symmetry group. This can be relevant in scenarios like molecular docking simulations [12], where the system can align with the larger molecule, or material science [22], where the arrangement of atoms and crystal lattice structures yield a discrete group of symmetries smaller than  $O(n)$ .
5.3 Generalizability of implicit kernels: ModelNet-40
Dataset. The ModelNet-40 [56] dataset contains 12311 CAD models from the 40 categories of furniture with the orientation of each object aligned. The task is to predict the category of an object based on its point cloud model. 2468 models are reserved for the test partition. From the remaining objects, we take  $80\%$  for training and  $20\%$  for validation. We augment each partition with random rotations around the Z-axis. We induce connectivity on point clouds with a k-nearest neighbour search with  $k = 10$  at each model layer and use normals as input node features.
Results. In this section, we demonstrate how one can build a Steerable CNN that is equivariant to an arbitrary subgroup of the Euclidean group  $G \rtimes (\mathbb{R}^3, +) \leq E(3)$ . We compare the performance of implicit kernels with the standard steerable kernels obtained by group restriction [7] and keep the number of parameters similar. The results are shown in Figure 3. Implicit kernels achieve significant improvement in accuracy on test data for the majority of groups. The only statistically significant negative difference is presented for  $G = SO(3)$ , for which a tailored and hence optimal kernel basis is already available. When a custom solution is unknown, implicit kernels often significantly outperform the previously proposed method. There-
fore, they pose an efficient toolkit for building a kernel basis for an arbitrary subgroup of  $E(3)$ . We report the result of the best-performing model in Table 1. Although we do not outperform more task-specific and involved approaches such as PointMLP [31], our model is on par with other group equivariant models. We emphasize that our model is a simple stack of convolutional layers and hypothesize that a smarter downsampling strategy, residual connections and hierarchical representations would significantly improve the overall performance of our model, which we leave for further research.
Table 1: Overall accuracy (OA) on ModelNet-40. Group equivariant methods are denoted by \*.
Method	OA, %
Spherical-CNN [13] *	88.9
SE(3)-ESN [16] *	89.1
TFN[mlp] P [34] *	89.4
PointNet++ [35]	91.8
SFCNN [36]	92.3
PointMLP [31]	94.5
Ours	89.3 ± 0.06
5.4 Flexibility of implicit kernels: QM9
Dataset. The QM9 dataset [55] is a public dataset consisting of about 130k molecules with up to 29 atoms per molecule. Each molecule is represented by a graph with nodes denoting atoms and edges indicating covalent bonds. Each node is assigned a feature vector consisting of one-hot encoding of the type of the respective atom (H, C, N, O, F) and its spatial information corresponding to a low energy conformation. Additionally, each molecule is described by 19 properties from which we select
Table 2: Mean Absolute Error (MAE) between model predictions and ground truth for the molecular property prediction on the QM9 dataset. Linear steerable convolutions are denoted by  $*$ . L stands for the number of layers, and W stands for the number of channels in each layer.
Task
Units	α
bohr3	Δε
meV	εHOMO
meV	εLUMO
meV	μ
D	Cν
cal/mol K	G
meV	H
meV	R2
bohr3	U
meV	U0
meV	ZPVE
meV
NMP[19]	.092	69	43	38	.030	.040	19	17	0.180	20	20	1.50
SchNet[41]	.235	63	41	34	.033	.033	14	14	0.073	19	14	1.70
SE(3)-Tr.[18]	.142	53	35	33	.051	.054	-	-	-	-	-	-
DimeNet++[24]	.043	32	24	19	.029	.023	7	6	0.331	6	6	1.21
SphereNet[29]	.046	32	23	18	.026	.021	8	6	0.292	7	6	1.12
PaiNN[42]	.045	45	27	20	.012	.024	7	6	0.066	5	5	1.28
EGNN[40]	.071	48	29	25	.029	.031	12	12	0.106	12	12	1.55
SEGNN[5]	.060	42	24	21	.023	.031	15	16	0.660	13	15	1.62
TFN[46]*	.223	58	40	38	.064	.101	-	-	-	-	-	-
Cormorant[1]*	.085	61	34	38	.038	.026	20	21	0.961	21	22	2.02
L1Net[32]*	.088	68	46	35	.043	.031	14	14	0.354	14	13	1.56
LieConv [14]*	.084	49	30	25	.032	.038	22	24	0.800	19	19	2.28
Ours (W=24, L=15)	.078	45.3	24.1	22.3	.033	.032	21.1	19.6	0.809	19.7	19.5	2.08
Ours (W=16, L=30)	.077	43.5	22.8	22.7	.029	.032	19.9	21.5	0.851	22.5	22.3	1.99
12 commonly taken in literature [5] for regression tasks. Different from common practice [5], we perform convolution on molecular graphs, i.e. with connectivity pre-defined by molecular structure instead of inducing it. The design choice is motivated by the presence of edge features, which we include in an implicit kernel to display the flexibility of neural representation.
Results. We first demonstrate how one can use the flexibility of neural representation to introduce additional features of choice into a steerable convolutional layer. As each pair of atoms connected by a covalent bond is assigned a one-hot encoding of the bond type  $z_{ij}$ , we use it as a condition for  $O(3)$ -equivariant implicit kernels. Additionally, we follow Musaelian et al.[33] and embed one-hot encoding of the center and neighbour atom types  $z_{i}$  and  $z_{j}$  into the MLP. We include each property one by one and indicate corresponding performance gain in Figure 4. First, we observed a sharp improvement by switching from standard steerable kernels to implicit ones. We attribute it to the higher expressivity of implicit kernels and their ability to learn more complex interactions. Furthermore, injecting edge attributes into the kernel computation reduced MAE even further. Introducing both atom types and an edge type significantly influenced the per
Figure 4: Using implicit kernels  $k_{MLP}^{G}$  and injecting it with bond and atoms properties significantly improves the performance of Steerable CNNs on the QM9 dataset (Mean Absolute Error on the  $\varepsilon_{HOMO}$  regression problem). Bars denote mean average accuracy on the test dataset with error bars corresponding to standard deviation; both computed on 5 runs. The kernel is  $O(3)$ -equivariant rendering the final architecture  $E(3)$ -equivariant.
formance, corresponding to the model learning how to process each specific combination differently, thus adding to its expressivity. It is not surprising since a similar result was obtained by Brandstetter et al.[5], who used non-linear message aggregation conditioned on physical information, yielding state-of-the-art performance.
Scaling the model up. Table 2 shows the results of an  $E(3)$ -equivariant Steerable CNN with implicit kernels on the QM9 dataset. Both models we reported here have approx.  $2 \cdot 10^{6}$  parameters and only differ in length and width of the convolutional layers. For non-energy regression tasks ( $\alpha, \Delta \varepsilon, \varepsilon_{HOMO}, \varepsilon_{LUMO}, \mu$  and  $C_\nu$ ), we obtain results that are on par with the best-performing message-passing based approaches (we do not compare against transformers). We also indicate that Steerable CNNs with implicit kernels significantly outperform steerable linear convolutions (TFN, LieConv, L1Net) on most tasks. This is consistent with the observation of Brandstetter et al.[5], who pointed out that non-linear convolutions generally perform better than linear ones. For the remaining energy variables (G, H, U,  $U_0$ , ZPVE) and  $R^2$ , our model significantly falls behind the task-specific benchmark approaches. We theorize that it can be attributed to two factors. First, steerable convolutions generally do not perform well on these tasks compared to problem-tailored
frameworks (PaiNN [42], DimeNet++ [24], SphereNet [29]), also in the non-linear case (e.g. SEGNN [5]). Second, we hypothesize that molecular connectivity does not produce a sufficient number of atom-atom interactions, which is crucial for the performance of a message-passing-based model [5]. However, as the goal of the section was to demonstrate the flexibility of implicit kernels that can be conditioned on features of graph edges, we leave developing more involved architectures (e.g. with induced connectivity) for further work.
We propose a novel approach for implementing convolutional kernels of Steerable CNNs, allowing for the use of smaller groups and easy integration of additional features under the same framework with minor changes. To avoid analytically solving the group  $G$ -specific equivariance constraint, we use a  $G$ -equivariant MLP to parameterize a  $G$ -steerable kernel basis. We theoretically prove that MLP equivariance is sufficient for building an equivariant steerable convolutional layer. Our implicit representation outperforms a previous general method, offering a way to implement equivariance to various groups for which a custom kernel basis has not been developed. The N-body experiment suggests that this method will be particularly applicable in scenarios where rotational symmetry is disturbed, such as in material science or computational chemistry. The critical force term, which violated rotational symmetry, was effectively captured by our model, while the state-of-the-art model struggled with it. Additionally, our flexible neural representation enables the introduction of arbitrary features into a convolutional kernel, enhancing the expressivity of Steerable CNNs. We validate this advantage by applying Steerable CNNs to point cloud and molecular graph data and achieving competitive performance with state-of-the-art approaches. In conclusion, we present a simple yet efficient solution for constructing a general kernel basis equivariant to an arbitrary compact group.
Steerable CNNs generally have high computational complexity and higher memory requirements compared to traditional CNNs. Implicit neural kernels do not help to mitigate the issue, yet they provide additional control over the kernel complexity. We found that when the kernels are parameterized by a single linear layer, the run time slightly decreases compared to the analytical solution $^{12}$ , while a relative performance gain remains. We suggest that more efficient ways to implement  $G$ -MLPs would significantly contribute to the acceleration of the method and leave it for further research. From the implementation point of view, the most troublesome was achieving initialization similar to the non-implicit convolutions. We expect the problem to manifest even stronger when dealing with dense convolutions. This, combined with the discretization error of matrix filters, might negatively affect the performance. We are, however, convinced that it is a matter of time before a robust way of initializing  $G$ -equivariant implicit kernels will be obtained.
All the experiments were performed using the Hemera compute cluster of Helmholtz-Zentrum Dresden-Rossendorf and the IvI cluster of the University of Amsterdam. This research results from a collaboration initiated at the London Geometry and Machine Learning Summer School 2022 (LOGML). The authors thank Anna Mészáros, Chen Cai and Ahmad Hammoudeh for their help at the initial stage of the project. They also thank Rob Hesselink for his assistance with visualizations.
12as implemented in escnn [7]
[55] Zhenqin Wu, Bharath Ramsundar, Evan N. Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S. Pappu, Karl Leswing, and Vijay S. Pande. Molecularnet: A benchmark for molecular machine learning. ArXiv, 2017. 2, 8
[56] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. CVPR, pages 1912-1920, 2015. 2, 8