Rotation-Invariant Random Features Provide a Strong Baseline for Machine Learning on 3D Point Clouds
Authors: Owen Melia
Eric M Jonas
Rebecca Willett
Venue: Accepted by TMLR ()

Abstract: Rotational invariance is a popular inductive bias used by many fields in machine learning, such as computer vision and machine learning for quantum chemistry. Rotation-invariant machine learning methods set the state of the art for many tasks, including molecular property prediction and 3D shape classification. These methods generally either rely on task-specific rotation-invariant features, or they use general-purpose deep neural networks which are complicated to design and train. However, it is unclear whether the success of these methods is primarily due to the rotation invariance or the deep neural networks. To address this question, we suggest a simple and general-purpose method for learning rotation-invariant functions of three-dimensional point cloud data using a random features approach. Specifically, we extend the random features method of Rahimi & Recht (2007) by deriving a version that is invariant to three-dimensional rotations and showing that it is fast to evaluate on point cloud data. We show through experiments that our method matches or outperforms the performance of general-purpose rotation-invariant neural networks on standard molecular property prediction benchmark datasets QM7 and QM9. We also show that our method is general-purpose and provides a rotation-invariant baseline on the ModelNet40 shape classification task. Finally, we show that our method has an order of magnitude smaller prediction latency than competing kernel methods.


Published in Transactions on Machine Learning Research (07/2023)
Rotation-Invarian Random Features Provide a Strong Base- line for Machine Learning on 3D Point Clouds
Owen Melia
melioo meliao@uchioago.ede
Deportment of Computer Science University of Chicago
Eric Jonas
ericjOhuchicago.co edu
Department of Computer Science University of Chicago
Rebecca Willett
willett Sullitt@uc edu
Department of Computer Science and Statistics University of Chicago
Reviewed on OpenReview: /tpp::://penreettte net/ forum? id= id-nYzhlFyjjd
Abstract
Rotational invariance is a popular inductive bias used by many fields in machine learning, such as computer vision and machine learning for quantum chemistry. Rotation-invariant machine learning methods set the state of the art for many tasks, including molecular property prediction and 3D shape classification. These methods generally either rely on task-specific invariann features, or they use general purpose deep neural networks which are complicated to design and train. However, it is unclear whether the success of these methods is primarily due lo the rolation invariance or the deep neural networks. To address this question, we suggest a simple and general-purpose method for learning rotation- invariant functions of three-dimensional point cloud data using a random features approach. Specifically, we extend the random features method Rahimi & Recht (007) by deriving a version that is invariant three-dimensional rotations and showing that fast to evaluate on point cloud data. We show through experiments that our method matches or outperforms the performance of general-purpose property prediction benchmark datasets QM7 and QM9. We also show that our method is perpral-purpose and provides a rotation-invariant baseline on the ModelNet40 shape classification task. Finally, we show that our method has an order of magnitude smaller prediction latency than competing kernel methods.
neural networks standard molecular
1 Introduction
Many common prediction tasks where the inputs are diree-dimensional physical objects are known to be iotationinnvarian;; the ground-truth label does not change when the object is rotated. Building rotation invariance into machine learning models is an important inductive bias for such problems. The common intuition is that restricting the learning process to rotation-invariant models will remove any possibility of poor generalization performance due to rotations of test samples and may improve sample efficiency by reducing the effective complexity of Icarned models. These idcas have inspired linc of rescarch bogun by Kondor et al. (2018); Cohen et al. (2018a); [steves et al. (2018) into building general-purpose deep neural network architectures that are invariant to rotations of their input. However, in these studies, it not clear whether the reported high accuracies are due the expressive power of the neural networks or primarily attributable rotation invariance. We introduce rotation-invarian random feature model which helps us explore the impact of rotation invariance alone, outside of the neural network framework. Our method is general-purpose

---

Published in Transactions on Machine Learning Research (07 (07/202) 2023)
Random spherical harmonic function 8j
Dij = sin (e..* d)
B = y
50(3)
i.
Data sample Pi
Figure 1: An overview of the rotation-invariant random features method. First we construct feature matrix @ defined in Equation 7 The rows of 4 correspond to Pt, different samples in our training set, and the columns of & correspond to Ij, different random sums of spherical harmonic functions, which are defined in Equation 10 The entries of the feature matrix are iaation-invariat random features. To compute the random fealures, we analytically evaluate the integral over all possible rotations of the data sample ps. We describe our method for evaluating the random features in Section 4.1 After constructing the feature matrix, we fit set of lincar weights B and make predictions by evaluating the lincar model $ = 9.
and does not require expert knowledge for feature architecture design. For certain prediction tasks, our proposed method can be computed with very low prediction latency with only a small reduction in accuracy compared to neural network methods, making it a viable method in a range of applications.
In this paper, we consider prediction problems that are rolation-inwarian, that is, the ground-truth respouse fx()) y does not change when an arbitrary rotation is applied the input x. We represent the input data as a 3D point cloud, unordered set of points in IR3 possibly with accompanying labels. Common examples of rotation-invarians prediction problems the point cloud consists of the positions of a molecule's constiluent atoms, and 3D shape classification, where the points are sampled from the surface of an object. Section 2 provides a detailed overview of methods used for rotation-invarian prediction, their underlying insights, and their various advantages and disadvantagos.
3D point clouds include molccular property prediction, where
We propose a new method for learning rolation-invariant functions of point cloud data, which we call rotation random features. We extend the random features method of Rahimmi & Recht (2007) by deriving version that is invariant to three-dimensional rotations and showing that it is fast to evaluate point cloud data. The rotation invariance property of our features is clear from their definition, and computing the features only requires two simple results from the representation theory of SO(3). Despite its simplicity, our method achieves performance near state-of-the-art on small-molecule energy regression and 3D shape classification. The strong performance on both tasks gives evidence that our method is general-purpose. An overview of our method can be found in Figure] [
Our experiments in Section 5.1.1 show that our model is an order of magnitude faster than state-of-the kernel methods when predicting new samples for energy regression tasks. Thus conclude our model is promising candidate for tasks which require real-time
predictions. Minimizing prediction latency is a growing concern for many machine learning methods; there are multiple applications requiring low-latency predictions on point clouds. Using machine learning predictions for real-time system control places hard requirements on prediction latencies. For example, trigger algorithms in the ATLAS experiment at the CERN Large Hadron Collider require prediction latencics ranging from 2us 40ms at different stages of the event filtering hicrarchy (Collaboration 2008). The input these trigger algorithms particle jet, which can be interpreted a point cloud in four-dimensional spacetime. The majority of data-driven trigger
2

---

Published in Transactions on Machine Learning Research (07/2023)
algorithms either impose rotational invariance in phase space (Komiske et al. 2018 Thaler & Van Tilburg 2011) or enforce invariance to the Torentz group, which includes three-dimensionall rotations and relativistic boosts (Bogatskiy et al. 2022 Gong et al. 2022). These definitions of invariance are compatible with our framework, with slight adjustments. Another use-case for low-latency models is as replacements for force fields in molecular dynamics simulations |Gilmer el al. 2017). These calculations make serial subroutine calls to force field models, meaning latency improvements of these force field models are necessary to speed up the outer simulations. Other rotation-invariant machine learning models either rely on deep computational graphs, which provide fundamental limit to speedup via parallelization, or they rely kernel methods Christensen et al. 2020) which have prediction-time complexity linear in the number of training samples.
1.1 Contributions
We derive a rotation-invariant version of the standard random features approach presented in Rahimi Recht (007). By using simple ideas from the representation theory of SO(3), our ootation-invariaal random ura method is easy to describe and implement, and requires minimal expert knowledge in its design (Section 4
We show with experiments that our rolation-invariane random feature method outperforms or matches the performance of general-purpose equivariant architectures optimized for small-molecule energy regression tasks. In particular, we outperform or match the performance of Spherical CNNs (Cohon et al. Sa) on the QM7 dataset and Cormorant (Anderson al., 2019) on the QM9 dataset. We compare our test errors with state-of-the-art methods on the QM9 dataset quantify the benefit of designing task-specific rotation invariant architectures (Section 5.1
We show that our method makes predictions an order of magnitude faster than kernel methods at Lest time (Section 5.1.1).
We show the votation-invarian random feature method is general-purpost by achieving low test errors using the same method different. task, shape classification on the ModelNet440 benchmark dataset (Section 5.2h.
We make our code publicly available at httpp:////iitaab..co//eelll///// http://eeet.ttttttoco.aaaaaaaaar
invarianii
2
Related Work
In this section, we describe two major types of methods extracts rotation-invariant features using expert chemistry knowledge. The second class of methods uses deep neural networks to build gencral-purposs invariant architectures. Finally, we briefly survey the random features work that inspire our method.
prediction methods. The first class of
Descriptors of Atomic Environments In the computational chemistry community, siguificant effort has been invested in using physical knowledge to generate "descriptors of atomic environments." " or rotation- invariant feature embeddings of molecular configurations. One such feature embedding is the Coulomb matrix, used by Montavon et al. (2012). In Rupp et al. (2012), the sorted eigenvalues of the Coulomb matrix are used as imput features. The sorted eigenvalues are invariant to rotations of the molecule and give more compact representation than the full Coulomb matrix. Noticing that the Coulomb matrix can relatively faithfully represent a molecule with only pairwise atomic interactions has been an important idca in this community. Many methods have been developed to generate descriptions of atomic environments by decomposing the representation into contributions from all atomic pairs, and other higher-order approxinations consider triplets and quadruplets of atoms and beyond (Christensen et al. 2020 Bartók et al. 2013 Kovács al.
2021
Drautz 2019h
The FCHL19 (Christensen et al. 2020) method creates featurization that relies
MMan ceneral-purpose invariant methods follow this design well. Cormorant Anderson et
2019 has internal nodes spherical images from
which correspond all pains of atoms the molecule, and Spherical CNNs (Cohen al. 2018a) molecular point clouds using ideas from pairwise electrostatic (Coulombir) repulsion.
3

---

Published in Transactions on Machine Learning Research (07/2023)
chemical knowledge and is optimized for learning the energy of small organic molecules. The FCHL19 feature embedding depends on 2-body and 3-body terms. There are multiple different approaches, such as Smooth Overlap of Atomic Potentials (SOAP) (Bartók et al. 2013) and Atomic Cluster Expansion (ACE) (Drautz 2019: Kovács et al., 2021) which first compute a rotation-invariant basis expansion of a chemical system, and then learn models in this basis expansion. In Appendix [ we show that our proposed method can be interpreted as using a highly-simplified set of ACE basis functions to learn a randomized nonlinear model.
Deep learning is an effective tool in computational chemistry as well. Task-specific deep neural network architectures for learning molecular properties of small organic molecules outperform other approaches on common energy learning benchmarks. Multiple methods (Schütt et al., 2018 Unke & Meuwly 2019) design neural networks to take alomic charges and interatomic distances as input. These methods use highly-optinn optimiee neural network architectures, inchuding specially-designee atomic interaction blocks and self-attention layers. OrbNet (Qiao et al., 2020), another neural network method, takes the output of a low-cost density functional theory calculation as cotation-invarian input features and uses a messago-passingg graph neural network architecture.
Gencral-Purposs Invariant Architectures While many gencral-purpose invariant architectures operate directly on point clouds, the first examples of such architectures, Spherical CNNs (Cohen el al. 2018a Esteves et al., 2018 Kondor et al. 2018) operate "spherical images", functions defined on the unit sphere. These networks define spherical convolutions between a spherical image and a filter. These convolutions are performed in Fourier space, and the input spherical image is transformed using a fast Fourier transform on the unit sphere. Much like a traditional convolutional neural network, these networks are comprised of multiple convolutional layers in series with nonlinearities and pooling layers in between. To accommodate multiple different types of 3D data, including 3D point clouds, these mothods take a preprocessing step to render the information of 3D object into spherical image.
Other methods operate on point clouds directly. SPHNet (Poulenard et al. harmonic power spectrum of the input point cloud, which is a set of rotation iitatiio-invarian features, and passes this through a series of point convolutional layers. Cormorant (Anderson et al. 2019 and Tensor Field Networks Thhoms et al. 2018) both design neural networks that at the first layer have activations corresponding to each point in the point cloud. All activations of these networks are so-called "spherical tensors," which means they are equivariant to rotations of the input point cloud. To combine the spherical tensors at subsequent layers, Clebsch-Gordas products are used. Many other deep neural network architectures have been designed to operate on point clouds, but the majority have not been designed to respect rotational invariance. A survey of this literature can be found in Guo et al. 2021).
2019) computes the spherical
There are many ways to categorize the broad field of group-invariant deep neural network architectures, and we have chosen to form categorics based on the imput data type. Another helpful categorization is the distinction between regular group-CNNs and sleerable group-CNNs introduced by CCohen el al.] (2018h) 8b). In this distinction, regulor group- CNNs form intermediate representations that are scalar functions of the sphere or the rotation group, and examples of this category are Cohen et al. (2018a); Kondor et al. (2018) Poulenard Croup-CNNs
et al. (2019). Steerable group- CNNs form intermediate representations that are comprised of "spherical tensors" and examples of this category are Anderson et al. (2019); Weiler et al. (0188: Thomas et al. 2018))
More recently, some works have investigated methods that endow non-invariant point cloud neural network architectures with a group-invariance property, either through a modification of the architecture or a data preprocessing step. Xiao et al.] (2020) suggests using a preprocessing step that aligns all point clouds in a canonical rotation-invariant alignment; [unyy et al. (2022) expands this idea to general transformation groups. After the preprocessing step, any point cloud neural network architecture can be used to make rotation-invariant predictions of the input. Vector Neurons (Deng et al., 2021 endows neural network architectures with rotation equivariance by using three-dimensional vector activations for cach neuron in the work and cleverly modifying the equivariant nonlinearities and pooling. These methods have been successfully applied to dense point clouds that arise in computer vision problems such as shape classification and segmentation. Molccular point clouds are qualitatively different; they are quite sparse, and the Euclidean nearest neighbor graph does not always match the graph defined by molecular bond structure. Because of these differences, the alignment methods and network architectures designed for solving shape classification
4

---

Published in Transactions on Machine Learning Research (07/2023)
and segmentation tasks are not priori good models for learning functions of molecular point clouds. To the best of our knowledge, they have not been applied successfully in molecular property prediction tasks. Finally, Villar et al. (2021) show that any rotation-invariant function of a point cloud can be expressed as a function of the inner products between individual points; this insight reduces the problem of designing inlaaion-nvvariant models to one of designing permulalion-invarinn architectures.
Random Features Random Fourier features (Rahimi & Recht 2007) are an efficient, randomized method of approximating common kernels. For kernel methods, prediction time scales linearly with the size of the dataset. Random Fourier feature methods require time linear in d, approximation parameter. The approximation works by drawing a random vector and defining low-dimensional feature embedding p(æ; called random Fourier feature. The random Fourier features approximate a hhift-nnvarian kernel k(x,x") = k(x - x') in the sense that the inner product of two random Fourier feature evaluations is an unbiased estimate the kernel:
Eg [(p(1;; <), ((((;())) k(x,x')
This approximation holds when the random vectors are drawn from the Fourier transform of the kernel k and the random Fourier features have the following functional form:
p( {) (x), sin((2.E))
To reduce the variance of this approximation, one can draw d such random vectors Ea))---Ea) and concatenate the resulting random features. As practical method, Rahimi & Recht (2007) proposes building feature matrix & E Raxd by drawing d different random Fourier features şp( :55) and evaluating the random Fourier features at each of the n samples in the training dataset:
t ((rsit))
Once the feature matrix is formed, lincar model is trained to fit response vector y using regularized linear regression, such as ridge regression:
arg min 43 v112 + 1111
(1)
Experiments in Rahimi & Recht (2007) show this is an effective method for fitting data, even when d < n. When d < n, the model can be evaluated much faster than the original kernel.
Follow-up work, including Rahimi & Recht 2008), suggested that interpreting random feature methods as kernel approximators is not necessary. This work uses random nonlinear features with varying functional forms, including random decision stumps and randomlyy-nittialized sigmoid neurons o({g,x)). These random nonlinear features form effective models for diverse types of data, In Section | we introduce random features similar to those in Rahimi & Recht (2007). Our method does not approximate any explicit kernel, and it is designed to be invariant to any rotation of the input data, so call method rolation-invariant random features.
Finally, the notion of group-invariant random feature models also appears in Mei et al. (021 as a technical tool to understand the sample complexity benefit of enforcing group invariances overparameterized models. In this work, the invariances considered are transformations on onc-dimensional signals that arise from cyclic and translation groups.
3 Rotational Invariance and Spherical Harmonics
In this section, we will introduce rotational invariance and the spherical harmonics. In our method, we use the definitions presented in this section and only two simple facts about the spherical harmonics and rotations, which we list at the end of this section. We say a function mapping point cloud p to an element of the vector space ) is rotation-equivariinnt if there exists some group action the vector space y such that
S(Q p) f()) AQ E S0(3)
5

---

Published in Transactions on Machine Learning Research (07/2023)
Intuitively, this means that when an imput to f is rotated, f preserves the group structure and the output changes predictably. We say that function f is rolation-invariant if the group action on the output vector space y is the identity:
= Swt...................... (x2.1...QQN1)) AQ E SO(3)
(2)
Finally, our goal is to learn a function over point clouds that does not change when any rotation is applied to the point cloud. Formally, given some distribution D generating pairs of data (p, y) our learning goal is to find
= arg min (D [([(p): v)]
where is a classification or regression loss depending on the task and the minimization is over all functions that satisfy rotational invariance constraint Equation 2
3.1 Spherical Harmonics
When decomposing a periodic function f [0, 2x] + R a natural choice of basis functions for the decomposition is the complex exponentials eima In this basis, f(x) may be expressed f(x) Σm a e where am (f(z), eima)). When decomposing function the unit sphere S similar set of basis functions emerges, and they are called the spherical harmonics. The complex exponentials are indexed by single parameter m, and because the spherical harmonics span a more complicated space of functions, they roquire two indices, & and m. By convention, the indices are restricted to 2 0 and -f S m S l. A single spherical harmonic function y(() maps from the unit sphere S2 to the complex plane C. For function f(z) S² - E we can compute an expansion in the spherical harmonic basis:
[(x) Σ a (x)
m.t
am
(f(x),
In designing our method, we use two simple facts about spherical harmonics:
If one has evaluated the spherical harmonics of some original point x E S and wants to evaluate those spherical harmonics at a new rotated point Qe for some rotation matrix Q, the rotated evaluation is a linear combination of the un-rotated spherical harmonics the same index &:
(Qx)
Σ
y (1) mm.m
(3)
m'
Here D²( is Wigner-D matrix; it square matrix of size (2l + 1 28 1).
Evaluating the inner product between two elements the Wigner D-matrices is simple. In particular, when dQ is the uniform measure over SO(3), we have the following expression:
(Q) ((t)) (Q) dQ = (-1) 1i-mi-k -k. 8m
SSO
8 m. m 8 k, de,
(4)
28
This is a corollary of Schur's lemma from group representation theory, and we discuss this fact in Appendix C
4 Rotation-Invariant Random Features
In our setting, an individual data sample is a point cloud = (2i, æi,N with points an E 12, We model the data function: p=(x)= Σ 5(x mig) where 5(.) is delta function centered at 0. We

---

Published in Transactions on Machine Learning Research (07/2023)
then functional version of the random feature method where our data ps is a function, 9 is random function drawn from some pre-specified distribution, and (:))) is the I,2 inner product:
(p:g)) = sin ((pr.g))
(5)
We want the random feature to remain unchanged after rotating the point cloud, but Equation 5 does not satisfy this, For general functions 9, (Q o pt.g) + (pi,g). We achieve rotational invariance by defining the following rotation-invarin random feature:
p(pag) = =sin
Pi. pi. 9)
(Q
(6)
We "symmetrize" the inner product by integrating over all possible orientations of the point cloud pi eliminating any dependence of ((:g)) on the data's initial orientation. Because (p(: does not depend on the initial orientation of the data, it will not change if this initial orientation changes, i.c., if pa is rotated by Q. To build regression model, construct feature matrix
R (p:;;}))
(7)
and fit. a linear model to this feature matrix using ridge regression, as in Equation
1
4.1 Evaluating the Random Features
In this section, we describe how to efficiently evaluate the integral in Equation dimensional manifold and the integral has a nonlinear dependence on the data, so methods evaluating this integral are not immodiately clear. However, representation theory SO(3) renders this integral analytically tractable and efficiently computable. The main ideas used to evaluate this integral are exploiting the linearity of inner products and integration, and choosing particular distribution of random functions g- First, we observe that because the rotated data function Q o p, is the sum of a few delta functions, we can expand the inner product as a sum of evaluations of the random function g:
G SO(3) is a three-
(Q. = Σ ((Qui.))
(8)
Expanding the inner product and the quadratic that appear in Equation 6 and using the linearity of the integral, we are left with sum of simpler integrals:
N.
ISo (Qpr..)) 92Q
Jid2== SSo0s
Σ
g(Quisi (9(Qu.,;))Q
(9)
Next, choose particular distribution for g which allows for easy integration over SO(3). We choose to decompose as a sum of madomll-wweihteee spherical harmonics with maximum order L and K fixed radial functions:
((x)= =
wmik (e)
(10)
k= E= m=-t
2 and Rg R. + R is where (0 are random weights drawn from Gaussian distribution A/ ((000²2) Tal a radial function. Two examples of such radial functions are shown in Figure 2 This definition of random functions gives us few hyperparameters; L, the maximum frequeney of the spherical harmonics; d, the standard deviation of the randomly-drawn weights; and the choice of radial functions. In Appendix we show that the performance of our method is most sensitive to d.
By repeated application of the lincarity of the integral and the two facts about the spherical harmonics ment ioned in Section 3.1 we are able to write down a closed-form expression for the integral on the left-hand side of Equation (9. We include these algebraic details in Appendix[A] A Computing this integral has relatively low complexity; requires ovaluating a table of spherical harmonics for cach point æij up to maximum order L, and then performing a particular tensor contraction between the array of spherical harmonic evaluations and the array of random weights. This tensor contraction has complexity O K2).
7

---

Published in Transactions on Machine Learning Research (07 (07/2023)
5 Experiments
We conduct multiple experiments comparing our method with other landmark rotation-invarian machine learning methods. We find that for predicting the atomization energy of small molecules, our method outperforms neural networks in the small dataset setting, and we have competitive test errors in the large dataset setting. Our method is an order of magnitude faster than competing kernel methods at test time. We also find that our method performs competitively on a completely different task, 3D shape classification. We perform these experiments to show that our method can be used fast, simple, and flexible baseline for rotation-invariant prediction problems on 3D point cloud data.
5.1 Small-Molecule Energy Regression
A common target for machine learning in chemistry the prediction of potential energy surface, which maps from 3D atomic configurations to the atomization energy of molecule. Large standardized datasets such QM7 (Blum&& & Reymond 2009 Rupp et al. 2012) and QM9 (Ruddigkeit et al. 2012 Ramakrishnani ct al. 2014) offer a convenient way to test the performance of those machine learning models across wide chemical space of small organic molecules. Both datasets contain the 3D atomic coordinates at equilibrium and corresponding internal energies. The 3D coordinates and molecular properties are obtained by costly quantum mechanical calculations, so there may be settings which require high-throughput screening where an approximate but fast machine learning model may provide an advantageous alternative to classical methods. To adapt our method this lask, we make two small changes.
Element-Tyye Type Encoding The rotation-invariant random feature method defined above works for general, unlabeled point clouds. However, in the chemistry datasets mentioned above, we are given more information than just the 3D coordinates of particles in the molecule: the particles are individual atoms, and we know their clement type. Incorporating the element type of individual atoms in a machinc learning method is crucial for accurale prediction. Different. elements interact in quantitatively different ways, as predicted by the laws of gravitation and electrostatic repulsion, and they interact in qualitatively different ways due their electron configurations.
We use element-tyye encoding method inspired by FCHL19 (Christensen el al. 2020 and ACE (Kovács et al. 2021 To construct this element-type encoding, we look at the local view of molecule created by centering the atomic coordinates at a given atom, separate the atoms into different point clouds for each element type, and compute one random feature per element type. We then repeat this procedure for all elements of given type and sum their feature vectors.
Expressed mathematically, we are given sample p: (Ii,1,...,, with charges (G.l...., C.l....,,,,,,,, C,N, J. Let Pi ((5)) be the collection of atoms with charge Cj, and let (cz) FiAh denote the point cloud of atoms in pt with charge Cj centered at point Ti, In this notation, (()) xi, specifies a point cloud, which we can treat as unlabeled because they all have the same charge. As before, a random feature is denoted p(:ig;;) Then for all possible element-typp type pairs (2), we compute individual entries in our feature matrix
Σ
(c2) 4 Pp
Ighhig))
h: 04 =01
The column index j' depends on j, C1, and c2-
set of random functions as randomly-weighted spherical harmonics with fixed radial functions. The choice of radial functions appears as design decision in many rotation-invariant machine learning methods. For example, radial functions appear as explicit design choices in Kovács et al. (2021): | Christensen et al. (2020); Poulenard el al. (2019) and implicitly in the design of Cohen et al. (2018a). This design choice is often paramount to the empirical success of these methods. FCHL19 [Christensen et al. 2020) optimize their radial functions over multiple hyperparameters and carefully balance multiplicative terms including log-normal radial functions, polynomial decay, and soft cut-offs. The ACE models in Kovács al. (2021) use a set of orthogonal polynomials defined over carefully-chosen subset of
Radial Functions We choose to parameterize
8

---

Published in Transactions on Machine Learning Research (07 2222)
1.0
1.0
1/
Radial function
length Radial function
Radial function
Radial function
value
Radial function
Radial function value
0.6
0.6
0.4
0.4
0.0
0.0
1
5
0.0
0.2
0.4
0.6
0.8
1.0
Radius (Bohrs)
Radius
(a) Radial functions for stmall-molecule energy regression
(b) Radial functions for 3D shape classification
Our cotation-invarians random feature method requires simple user-defined radial functions. Figure 2a shows the radial functions used in our small-molecule energy regression experiments. We include and the average Carbon- Carbon-Hydrogen bond length give context to the horizontal axis. Figure 2bossw the radial functions used in our 3D shape classification experiments. The 3D shapes in the benchmark dataset. are normalized to fit inside the unit sphere.
Figure 2:
the real line, a mlysivally-motivateed spatial transformation, and use extra ad-hoc radial functions that control the behavior of extremely nearby points. For our radial functions, use two Gaussians, both centered at 1, with width parameters chosen so the full widths at half maximum are 2 and 4 respectively. Our radial functions are shown in Figure 2
5.1.1 QM7 Atomization Energy Regression
The QM7 dataset contains 7,165 small molecules with element types H, C, N, O, S and a maximum of heavy elements. We compare with two lightweight methods, FCHL19 and Spherical CNNs, that show learning results on the QM7 dataset. We reproduce their original training methods and compare test errors in Table I Notably, our rotation-invariant random feature method has average errors half of those of Spherical CNNs while being faster to train. Our best-performing model uses 2,000 random features. With the element-type encoding described above, this corresponds to 50,000 trainable parameters.
In this experiment, we use a training datasct of 5,732 samples and a test set of size 1,433. For our method and FCHL19, we use 90% of the training set to train the models and the remaining 10% as a validation set for hyperparameter optimization. To train the FCHL.19 models, we use the validation set optimize over Gaussian kernel widths and L2 regularization parameters. In our method, we search over the number of random features and 2 regularization parameters. We solve our ridge regression problem by taking singular value decompasition of the random feature matrix, and then constructing solution for each regularization level.
In addition to our ridge regression models, we also experiment using our random features as an input to deep neural networks. We are unable find deep neural network models with reasonable prediction latencies that outperformed our ridge regression model (Appendix D We also experimented with a model which combined rotation-invariant PCA alignment (Xiao et al. 2020 Puny et al. (022) with three-dimensional random features, but the alignments produced by this method were uninformative, and the model failed to meet basic basclines.
The space of small organic drug-like molecules is extremely large. The GDB-13 datasel (Blm & Reymond 2009) contains nearly one billion reference molecules, even after filtering for molecule size and synthesis prospects. Most of the machine learning methods approximating potential energy surfaces are considered fast approximate replacements for costly quantum mechanical calculations. To screen the extremely large space of small organic drug-like molecules, these methods require high prediction throughput, which
9

---

Published in Transactions on Machine Learning Research (07/2023)
Method
General- Purpose
Mean Absolute Error (eV)
Train Time (s)
Train Device
Spherical CNNs
0.1565
546.7
single GPU
Cohen et al. 2018a)
Random Features
V
0.0660 + 0.00275
203.4
24 CPU cores
(Ours) FCHL19
0.0541
260.7
CPU cores
Christensen et al. 2020
Table 1: Test error and training time the QM7 dataset. We report the mean absolute error the test. set for all methods and the standard error of the mean for method. We discuss this experiment in Section 5.1.1
be achieved by exploiting parallelism in the models' computational graphs and readily-available multicore CPU architectures. When parallelism inside the model is not available, the set of candidate molecules can be divided and data parallelism can be used to increase screening throughput. However, consider another setting where experimental samples need to be analyzed in real time to make decisions about ongoing dynamic experiments. In this setting, prediction latency is paramount.
In the low-latency setting, our method can gracefully trade off prediction latency and prediction error by tuning the number of random features used in a given model. Importantly, all training samples are used to train the model. FCHT.19 with the number of training samples used. Thus, FCHL19 can only improve test latency the cost of using fewer training samples and incurring higher test crrors.
kernel method, so the time required to predict a new data point scales linearly
To explore the tradeoff between prediction latency and prediction error, we train random feature models and FCHL19 models of different sizes. We measure their prediction latencies and plot the results in Figure 3 We note that at similar error levels, FCHL19 models show prediction latencies that are an order of magnitude slower than ours. Spherical CNNs exhibit low test latency because their method is implemented for a GPU architecture, but their test errors are high, and the neural network approach does not admit obvious ways to achieve a tradeoff between latency and test error. The prediction latencies of FCHL19 and our random feature models depend the number of atoms in the molecule, and we show this dependence in Figure
5.1.2 QM9 Atomization Energy Regression
The QM9 dataset contains 133,885 molecules with up to 9 heavy atoms C, O, N, and F, as well as Hydrogen atoms. This is larger and more complex dataset than QM7, and because there is more training data, neural network methods perform well on this benchmark. For this dataset, follow Anderson et al. (2019); Gilmer et al. (2017) by constructing atomization energy the difference between the internal energy at OK and the thermochemical energy of molecule's constituent atoms. We train a large-scale random feature model by taking 100,000 training samples from QM9 and generating 10,000 random features. We select an L regularization parameter a priori by observing optimal regularization parameters from smaller scale experiments on a proper subset the QM9 training data.
We compare the performance of our model with the reported errors of FCHL19 (Christensen al., 2020) and other neural network methods. All methods considered in this comparison enforce rotational invariance in their predictions. The results of this comparison are shown in Table 2 Our rotation-invanl feature method matches the performance of Cormorant, and it provides very strong bascline against which we can quantify the effect of expert chemistry knowledge or neural network design. OrbNet (Qiao el al. 2020) uses a graph neural network architecture, and their imputs are carefully chosen from the results of density functional theory calculations in a rotation-invarian basis. PhysNet (Unke & Meuwly 2019) usees an architecture heavily inspired by that of SCHNet (Schütt et al. 2018), and this iteration in model design resulted in almost. 50% reduction in test. error.
random
10

---

Published in Transactions on Machine Learning Research (07 2022)
0.25
FCHL19 Invariant Random Features
Model
Prodiction
Train time (e)
Error (oV)
Size
Latency (a)
Spherical CNNS
0.20
FCHL19
100
0.0151
0.4
0.2020
FCHL19
200
0.2191
FCHLL19
400
0.0604
2.3
0,2514
0.15-
Error
0.1444
FCHL19
1600
0.3170
54,0
0.1157
0.10
FCHLI19
3200
0.6835
258.5
0.0700
FCHL19
5000
0.0541
FCHL19
5732
1.2191
326.1
0.0535
0.05
Random Reaurres [Ous)
250
0.0177
0.0775
Raudom Festure (Ours
500
0.0178
107.1
0.0711
Random Features (Ours)
750
0.0189
157.8
0.0701
0.00
10
10
10a
Random Fastures (Ours]
1000
0.0263
203.4
0.0660
Prediction Latency (seconds)
Random Features (Ouzs
2000
0.0212
151.4
0.0687
Spherical
N/A
Figure Invariant Random Features are faster prediction time than FCHL19 and more accurate than Spherical CNNs. In this experiment, we compare the prediction latency and prediction error for models of different sizes evaluated the QM7 dataset. For each method, measure the latency of predicting cach molecule held-out test set 5 times. In the figure, plot the median prediction latencies with error bars spanning the 25th and 75th percentile measurements. The vertical axis of the figure is mean absolute error measured in cV on a held-out test set. The FCHL19 method is evaluated on 24 CPU cores using kernels with sample sizes 100, 200, 400, 800, 1600, 3200, 5000, and 5732. The rotation-invariant random features method is evaluated on 24 CPU cores with model sizes of 250, 500, 750, 1000, and 2000 random features. The Spherical CNNs method evaluated on single GPU, We discuss this experiment in Section 5.1.1
Method
General- Purpose
Model Type
Mean Absolute Input Features
Error (eV)
Cormorant 2019
General-purpose rotation-invariant
0.022
Charges
Anderson et al.
Atomic locations
architecture
Random Features (Ours)
Ridge Regression
0.022 + 4.45e-04
Charges
Rotation-invariant
random features
FCHL119
X
Kernel Ridge Regres- sion
0.011
Charges
Christensen et al.
Interatomic
dis-
2020
tances
3-body angles
SchNet Schütt et al.
X
Neural Network with atomic interaction blocks
0.014
Charges
2018
Interatomic dis-
tances
PhysNet
Neural Network with 0.008 attention layers
Charges dis-
Unke & Meuwly 2019
Interatomic
Lances
OrbNet
Message-Passing GNN
0.005
Mean-field DFT cal- culations
Qiao et al. 2020
Table 2: Comparing large-scale models trained QM9. FCHL19 (Christensen et al., 2020) was trained 75,000 samples from QM9 and the other methods were trained on 100,000 samples. We report the mean absolute error the test set for all methods and the standard error of the mean for our method. We discuss this experiment in Section 5.1.2
11

---

Published in Transactions on Machine Learning Research (07/2023)
Training models on large subsets of the QM9 dataset is difficult. FCHL19 requires 27 hours to construct complete kernel matrix of size (133,855 x 133,855) a compute node with 24 processors. The 27 hours does not include the time required to find the model's linear weights. The neural network methods require long training sequences on GPUs. Both Cormorant and PhysNet report training for 48 hours on a GPU.
When using a large sample size, our method is similarly difficult to train. We are able to construct a matrix of random features of size (100,000 x 250,000) in 27 minutes on machine with 24 processors, but the matrix is highly ill-conditioned, and using an iterative method to approximately solve the ridge regression problem requires 76.5 hours. Using an iterative method introduces approximation error which we did not encounter when performing experiments on the other datascts. We attribute some of the performance gap between our method and FCHL19 to this approximation error. Fortunately, solving large, dense, overdetermined ridge regression problems is an active area of research, and it is quite likely that applying methods from this area of research to our problem would result a speed-up. We outline some promising approaches in Appendix F] We believe the conditioning of our problem is caused by the choice of clement-type encoding, and we leave finding new element-type encoding that produces well-conditioned feature matrices for future work.
5.2 Shape Classification
To show that our method is general-purpose, we test our same method on a completely different task, 3D shape classification. In partieular, we consider multiclass classification on the ModelNet benchmark dataset. The Mo benchmark dataset Zhirong Wu et al. (2015) is set of computer-generater 3D models of common shapes, such as mugs, tables, and airplanes. We evaluate the performance of our model in three train/lest settings, where the train and test. sets either contain rolations about the = axis, or arbilrary rotations in 50(3). There are 9,843 training examples and 2,468 test examples spread across 40 different. classes. The shape objects are specified by 3D triangular meshes, which define a (possibly disconnected) object surface. To generate a point cloud from individual objects in this dataset, one must choose a sampling strategy and sample points from the surface of the object. We use the dataset generated by Qi et al. (a)) which samples 1,024 points on the mesh faces uniformly at random. The point clouds are then centered at the origin and scaled to fit inside the unit sphere.
We solve the multiclass classification problem with multinomial logistic regression. More specifically, we optimize the binary cross entropy loss with L2 regularization to Iearn a set of lincar weights for cach of the 40 classes in the dataset. At lest time, we evaluate the 40 different linear models and predict by choosing the class with the highest prediction score. We also slightly change the definition of our data function; for a point cloud p: (.i,1,-..,i..,, J, we use normalized data function pı(x) = N. Σÿ= 5(x - Fi.j) to climinate any dependence on the number of points sampled from the surface of the shape objects. For this setting, we use radial functions that are three Gaussian bumps with centers at 0.5, and 1, with width o = 0.75.
We compare our method with another rotation-invariant mothod, SPHNet (Poulenard et al. 2019). SPHNet is a multilayer point-convolutional neural network that enforces rotational invariance by computing the spherical harmonic power spectrum of the input point cloud. SPHNet's 's rotation-invariant method also requires a choice of radial functions, and they use two Gaussians with different centers and vary the width of the Gaussians at different layers of their network. The results of our comparison are in Table 3. Our method does not achieve near state-of-the-art results, but we conclude it provides strong baseline on this challenging multiclass problem. We discuss the prediction latency of our method on the ModelNet40 dataset in Appendix D.
6 Discussion and Future Work
Our method is a simple, flexible, and competitive baseline for rotationally- invariiat prediction problems on 3D point clouds. Rotation-invariant random features are simple to explain and implement, and using them in varied prediction lasks requires a minimal amount of design choices. Our method does not achieve state-of the--rr-tt--tttteee accuracy on any prediction task, but it provides a competitive baseline that allows us to begin to quantify the effect of neural network models and expert chemistry knowlodge in methods that enforce rotational invariance. In particular, we find that for molocular property prediction tasks, the inductive bias of rotation invariance is enough to provide very strong performance with our general-purpose random feature
12

---

Published in Transactions on Machine Learning Research (07/2023)
Method
Rotation Invariant
2/2
(O(3)/SO(3)
z 7/SO(3)
Spherical CNNs (Esteves al.
2018)
0.889
0.869
0.786
SPHNet (Poulenard et al. 2019
0.789
0.786
0.779
Vector Neurons (Deng et al.
2021
0.902
0.895
0.895
Random Features (Ours)
0,693
0.692
0,666
PointNet++ (Qi et al. 2017bp
X
0.918
0.850
0.284
Table 3:
Test accuracy on the ModelNet40 shape classification beuchmark task. Our method sets a strong bascline to compare against neural network methods, especially in the challenging z/SO(3) wrain/test regime. We discuss this experiment in Section 5.2.
model. For 3D shape classification, we find that the inductive bias of rotation invariance provides a non-trivial baseline, but the gap between our model and neural networks is larger.
Our method shows promise in settings where low prediction latency is desired. In high-throughput experiments, such as the ATLAS experiment the CERN Large Hadron Collider (Collaboration 2008). data generated at such high velocity that real-time decisions must be made whether to save discard individual samples. For this type of initial screening task, we imagine our low-latency and flexible prediction method will be an attractive candidate. Another experimental use-case for low-latency models is as replacements for force fields in molecular dynamics simulations (Gilmer et al. 2017). In these simulations, the energy and forces in a molccular system is repeatedly queried in a scrial fashion. Any improvement in the latency of energy or force prediction would translate to speedup of the overall system. Because our model has an extremely shallow computational graph, we expect it will have much lower latency than deep neural networks once implemented on a GPU.
One can also interpret our work as an initial investigation into the use of random Fourier feature methods for approximating common kernel methods in computational chemistry. Kernel methods are well-studied and highly performant in machine learning for computational chemistry; examples of such methods include Rupp et al. (2012); Montavon et al.] (2012): Bartók et al. (2013): Christensen et al. (2020). It is a natural question to ask whether these methods can benefit from low prediction latency while maintaining high test accuracy when approximated by random features. Our method does not implement an exact approximation to any of the kernel methods above, but our experiments show promise in this area. Our test errors are near those of FCHL19 parameters have rcasonable test errors. However, when training our model on the QM9 dataset, we have seen that the particular element-type encoding we have used creates ill-oonditioned feature matrices and makes optimization difficult. To fully realize the potential of random feature methods in accelerating kernel learning for computational chemistry, a new element-type encoding is needed.
the QM7 dataset, and versions of our model using only 10% of the optimal number of
7 Broader Impact Statement
The task of molecular property prediction is important to a wide range of applications, including the development of new pharmaceuticals, materials, and solvents. Some of these applications may be misused.
Acknowledgments
RMW was supported by AFSOR award 1A9550-18-1-0166 and NSF DMS-2023100 OJM was supported by NSF Research Traineeship under grant NSF 2022023.
References
Brandon Anderson, Truong Son Hy, and Risi Kondor. Cormorant: Covariant molecular neural networks. In H. Wallach, H. Larochelle, A. Beygekimer, F. d' d'AlchéBBuc, E. Fox, and R. Garnett (eds.), Advances
13

---

Published in Transactions on Machine Learning Research (07/2023)
in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https:
pdd
Albert P. Bartók, Risi Kondor, and Gábor Csányi. On representing chemical environments. Physical Reniew
B, 87118), May 2013. ISSN 1098-0121, 1550-235X. doi: 10.108//PyysRee:888.111... URL
https://linie wtwppppgg...o////....r.p.ooooooo doi/10. /tt////...../////aa////e///o/ 17.14411
Michele Benzi. Preconditioning techniques for large linear systems: A survey. Journal of Compulational Physics, 182(2):418 477, Nov 2002. ISSN 0021-9991. doi: 00.1006//cch..2022.77.
Lorenz C. Blum and Jean-Louis Reymond. 970 Million Druglike Small Molecules for Virtual Screening in the Chemical Universe Database GDB-13. Journal of the American Chemical Society, 131 July 2009. TSSN 0002-7863, 1520-5126. doi: 10.1021/ja9022 ja902302h
URL https: htttss:/ppub...eee acs ott.b///..ccrrr//bb/1000011
Alexander Bogatskiy, Timothy Hoffman, David W. Miller, and Jan T. Offermann. Pelican: Permutation equivariant and lorentz invariant or covariant aggregalor network for particle physics. (arXiv:2211.00%) Dec 2022. URL http:://axii...tt/// 0rg/0bb//2211.0045 00454 arXiv:2211.0445 [hep-ex, physics:hep-ph).
Agniva Chowdhury, Jiasen Yang, and Petros Drineas. An iterative, sketching-based framework for ridge regression. In Proceedings of the 35th International Conference on Machine Learning, pp. 989 998. PMLR, Jul 2018. URL
mlr. www.p/press////e/e (tress//80//hoondury11..... wwwpc/hrrywa.aaaaao html
Anders S Christensen, Lars A Bratholm, and Felix A Faber. FCHL revisited: Faster and more accurate quantum machine learning. The Journal of Chemical Physics, pp. 16, 2020.
Taco S. Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical CNNs. In International Conference www.://ooorridddddddaaaaa
on Learning Representations, 2018a. URL
Taco S. Cohen, Mario Geiger, and Maurice Weiler. Intertwiners between induced representations (with applications to the theory of equivariant neural networks), 2018b. URL https://arrxv..rr//ab...
10743
ATLAS Collaboration. The atlas experiment at the cern large hadron collider. Journal of Instramentation,, 3 (08):8080033, August 2008. ISSN 1748-0221. doi: 10.1088 /2444002223/00/S005550 URL /tttps///xx.oo doi org/ 0tt/:/0..066/1148822////////8119
Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, and Leonidas Guibas. Vector Neurons: A General Framework for (0(3)- Equiivriiar Networks. In 2021 IEEE/CVE International Conference on Computer Vision (ICCV), pp. 12180-12189, Montreal, QC, Canada, October 2021. IEEE. doi: 10.1109/ICCVV88922 20A.://CVVV99222222..1110110 2021.01198.
ISBN 978-1-665-42-812-5
URL https::/ieeexplore. org/
document/8711444/
Ralf Drautz. Atomic cluster expansion for accurate and transferable interatomic potentials. Physical Review B, (911:1444,, Jamuary 2019. ISSN 2469-9950, 2469-9969.
doi: 10.1005/PhyssteeB..9.11111 URL
https://link... aps
Petros Drineas and Michael W. Mahoney. Randnla: randomized numerical linear algebra. Communications of the ACM, 59(6):80 90, May 2016. ISSN 0001-0782, doi:
Carlos Esteves, Christine Allen-Blanchette Ameesh Makadia, and Kostas Daniilidis. Learning SO(3) Equivariant Representations with Spherical CNNs, September 2018. URL http:///rxiv.. wrg. aa//a/bs//1111 06721 arXiv:1711.0672) [cs).
Justin Gilmer, Samuel Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In Proceedings of the 94th International Conference on Machine Learning Volume 70, ICML/17, pp. 1263 1272. Jor,, 2017.
14

---

Published in Transactions on Machine Learning Research (07/2023)
Shiqi Gong, Qi Meng, Jue Zhang, Huilin Qu, Congqiao Li, Sitian Qian, Weitao Du, Zhi-Ming Ma, and Tie-Yan Liu. An efficient lorentz equivariant graph neural network for jet tagging. Journal of High Energy Physics, 2723(7):30, Jul 2022. ISSN 1029-8479. doi: 10.1007/JHEP0772200 physics:hep-phl.
arXiv:220.08187 [hep-ex,
Yulan Guo, Hanyun Wang, Qingyong Hu, Hao Liu, Li Liu, and Mohammed Bennamoun. Deep Learning for 3D Point Clouds: A Survey. IEEE Transactions Pottern Analysis and Machine Intelligence, 43(12): 4338-4364, 4364, December 2021. ISSN 1939-3539. doi: 10.T109/TPAMI.2000 20003M..000.0054433 Conference Name: IEEE Transactions on Pattern Analysis and Machine Intelligence.
Pranceth Kacham and David Woodruff. Sketching algorithms and lower bounds for ridge regression. In Proceedings of the 39th International Conference on Machine Learning, pp- 10539 10556. PMLR, Jun 2022.
URL https: httpp::/prroceedingg mlr wtt.smmlm..pprrr.. wtwppprsss/v/epeeee http:////.kaaaaaaaaaaaaaa html
Patrick T. Komiske, Eric M. Metodiev, and Jesse Thaler. Energy flow polynomials: complete linear basis for jet substructure. Journal of High Energy Physics, (4) Apr 2018. ISSN 1029-8479. doi: 1,.10077HHEP4420330003. arXiv:1112.07124 [hep-ex, physics:hep-ph).
Risi Kondor, Zhen Lin, and Shubhendu Trivedi. Clebsch Clebsch-gordan nets: A fully fourier space spherical convolutional neural work. In Proceedings of the 32nd International Conference on Newral Information Processing Systems, NIPS'18, pp. 10138-1014 Red Hook, NY, USA, 2018. Curran Associates Inc.
Dávid Péter Kovács, Cas der Oord, Jiri Kucera, Alice F. A. Allen, Daniel J. Cole, Christoph Ortner, and Gábor Csányi. Linear Atomic Cluster Expansion Force Fields for Organic Molecules: Beyond RMSE. Journal of Chemical Theory and Computation, 17(12):669-7771 December 2021. ISSN 1549-9618, 1549-9626. doi: /000//a...aaaa. 10.1021 /acs. jetsseeccc...... URL https://pubs acs wwt/ org./////...11///as. jctc. wtt.:jcccccc....7.
Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Learning with invariances in random features and kernel models. In Mikhail Belkin and Samory Kpotufe (eds.), Proceedings of Thirty Fourth Conference on Learning Theory, volume 134 of Proceedings of Machine Learning Research, pp. 3351 3418. PMLR, 15 19 Aug 2021. URL htttps:/pproceedii:
mlr wwwpppeesssvieeeeee http///134mme/em2 html
Xiangrui Meng, Michael A. Saunders, and Michael W. Mahoney. LSRN: A Parallel Iterative Solver for Strongly Over- or Underdetermined Systems. SIAM Journal on Scientific Computing, 36(2):C95 C118, January 2014. ISSN 1064-8275, 1095-7197. 10. 1117
doi: 10. 10.1137 (0.1137/12086661 120866580. URL http://epubs. si a.a.. www...rg//aa......oo...o..w /o1/
Grégoire Montavon, Katja Hansen, Siamac Fazli, Matthias Rupp. Franziska Biegler, Andreas Ziche, Alexandre Tkatchenko, O. Anatole von Lilienfeld, and Robert Müller. Learning invariant representations of molecules for atomization energy prediction. In Proceedings of the 25th International Conference on Neural Information Processing Systems Volume 1, NIPS'12, pp. 440 448, Red Hook, NY, USA, 2012. Curran Associates Inc
Christopher C. Paige and Michael A. Saunders. Lsqr: An algorithm for sparse linear equations and sparse least squares. ACM Transactions on Mathematical Software, 8(1):43-7B 71, Mar 1982a. ISSN 0098-3500. doi: 10.1145/355984.35550
Christopher C. Paige and Michael A. Saunders. Algorithm 583: Lsqr: Sparse linear equations and least squares problems. ACM Transactions on Mathematical Software, 8(2):195 209, Jun 1982b. ISSN 0098-3500. doi:
Adrien Poulenard, Marie-Julie Rakotosaona, Yann Ponty, and Maks Ovsjanikov. Effectivv Rotation-Invariant Point CNN with Spherical Harmonies Kernels. In 2019 International Conference 3D Vision (3DV), pp. 47-56, September 2019. doi: 10.1109/3DV, 101...00.000015. ISSN: 2475-7888.
Omri Puny, Matan Atzmon, Edward J. Smith, Ishan Misra, Aditya Grover, Heli Ben-Hamu, and Yaron Lipman. Frame Averaging for Invariant and Equivariant Network Design. In International Conference on Learning Representations, 2022. URL http::///ppaaeeriieeeeee///orrrr
15

---

Published in Transactions on Machine Learning Research (07 (203))
Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017a.
Charles Ruizhongtai Qi, Yi, Hao Su, and Lconidas J Guibas. Pointnet+++: Deep hicrarchical feature learning on point sets in a metric space. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017b. URL Inttps://procedddiiii
wwwprpps.c/pppprrftiitii. filess/paper/2017/
Zhuoran Qiao, Matthew Welborn, Animashree Anandkumar, Frederick R. Manby, and III Miller, Thomas F. Orbnet: Deep learning for quantum chemistry using symmetry-adapted atomic-orbital features. The Journal of Chemical Physics, (1)(11111,, Sep 2020. ISSN 0021-9606. doi:
Ali Rahimi and Benjamin Rocht. Random Features for Large-Scale Kernel Machines. In J. Platt, D. Koller, Y. Singer, and S. Roweis (eds.), Advances in Neural Information Processing Systems, vol- ume 20. Curran Associates, Inc., 2007. URL https://proceeeding.. neurips. cc/paperr2007/ffie/ par
1130000000b665577fff188811111111
Ali Rahimi and Benjamin Recht. Weighted Sums of Random Kitchen Sinks: Replacing minimization with randomization in learning. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou (eds.), Advances
in Neural Information Processing Systems, volume 21. Curran Associates, Inc., 2008. URL https:
Raghunathan Ramakrishnan, Pavlo Dral, Matthias Rupp, and O. Anatole von Lilienfeld. Quantuun chemistry structures and properties of 134 kilo molecules. Scientific Data, 1(1):140022, December 2014.
ISSN 2052-4463. doi: 2..108//daata.2114.2.. URL http:://www..atturr..cooaaaaaaie
Lars Ruddigkeit, Ruud van Deursen, Lorenz C. Blum, and Jcan-Louis Reymond. Enumcration of 166 Billion Organic Small Molecules in the Chemical Universe Database GDB-17. Journal of Chemical Information and Modeling, 52(11):2866 2875, November 2012. ISSN 1549-9596, 1549-960X. doi: 10.1021/ci300413 www.c..rr....eeeeeo
URL /ttpp::/pubs.. acs org org//oi/10...oo 1001//1000100111104115555
Matthias Rupp, Alexandre Tkatchenko, Klaus-Robers Müller, and O. Anatole von Lilienfeld. Fast and Accurate Modeling of Molecular Atomization Energies with Machine Learning. Physical Review Letters, 108(5)005830,, Jamuary 2012. ISSN 0031-9007, 1079-7114. doi: 10.1103 httos//haaaereeet.00...... https://link. www........................... /oo/ httg:///a......////yeeeeettttt. 1088000000 058301
URL
K. T. Schütt, H. E. Sauceda, P.-. Kindermans, A. Tkatchenko, and K.-R. Müller. SchNet A deep learning architecture for molecules and materials. The Journal of Chemical Physics, 148(24):241722, June 2018. ISSN 0021-9606, 1089-7690. doi: 10.1063 (L0663/..011979. URL http:///app.ccitaaion...rr wttp:/iiiiiii./////////o///////. 1063/1 5019779
Jesse Thaler and Ken Van Tilburg. Identifying boosted objects with n-subjettiness. Journal of High Energy Physics, 23:1( Mar 2011. ISSN 1029-8479. doi: 10.1007 (000000JHEP00(001110
Nathaniel Thomas, Tess Smidt,, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation- and translation-equivarran neural networks for 3D point clouds, May 2018. URL Number: arXiv:1802.08219 #xXiv:1802.08211 [es].
htttp://arxiv/ wwg. htrg/abs/18000 08219
William J. Thompson. Angular momentum: an illustrated guide to rotational symmetries for physical systems. Wiley, New York, 1994. ISBN 978-0-171-65264-22
Oliver T. Unke and Markus Meuwly. PhysNet: A Neural Network for Predicting Energies, Forces, Dipole Moments, and Partial Charges. Journal of Chemical Theory and Computation, 15 3693, June 2019. ISSN 1549-9618, 1549-9626. doi: 10.1021 /0.1021/acs.jctc.9beb jet...991. URL acs.:://upsss...orr.o. http:///ppp...ccc.oooroooo////10 acs. jctc 9b00181
16

---

Published in Transactions on Machine Learning Research (07/2023)
Soledad Villar, David W Hogg, Kate Storey-Fisher, Weichi Yao, and Ben Blum-Smith. Scalars are universal: Equivariant machine learning, structured like classical physics. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL
https://openreviil wtt/fooumnddddaa22 Ra2
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, J Carey, îhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, lan Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for 0686-2.
Scientific Computing in Python. Nature Methods, 17:261 272, 2020. doi:
A. J. Wathen. Preconditioning. Acta Numerica, 24:329 376, May 2015. ISSN 0962-4929, 1474-0508. doi:
10.0017/9096244995500000
Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco Cohen. 3d steerable cmns: Learning rotationally equivariant features in volumetric data. In Proceedings of the 32nd International Conference Neural Information Processing Systems, NIPS 18, pp. 10402 10413, Red Hook, NY, USA, 2018. Curran Associates Inc.
Zelin Xiao, Hongxin Lin, Renjie Li, Lishuai Geng, Hongyang Chao, and Shengyong Ding. Endowing Deep 3d Models With Rotation Invariance Based On Principal Component Analysis. In 2020 IEEE International Conference on Multimedia and Expo (ICME), pp. 1-6, London, United Kingdom, July 2020. IEEE. ISBN 378-1-72811-331-1 doi:
URL https://ieeeepplore ieee
htg/doccumen//910290
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3D ShapeNets: A deep representation for volumetric shapes. In 2015 IEEE Conference on Computer Vision and Patfern Recognition (CVPR), pp. 1912-1920, Boston, MA, USA, June 2015. TEFF. ISBN 078-1-4673--664-- doi: 10.1109/CVPR.200 7298801. URL http:://eeeeeppoooee.eeeeoooooos
A Full Integration Details
We wish to solve the following integral, which appears in the definition of rotation-invann random features Equation 6
(Q pt. 9r.
(11)
5013)
From Section 4.1 we know this integral can decompose into
I so03)
N. Σ
(Q (p)p
g(Qzs,,, (9(Qxa,2))
(12)
SO(3)
J1:j2=1
And we also have chosen particular functional form for our random function g:
g(x)
(6) w (&)
(13)
Repeated application of the linearity of integration gives us:
[ SO(3)
(Qopi,))dde
(14)
N
= Σ
(15)
h
5013)
17

---

Published in Transactions on Machine Learning Research (07 (07/2023)
K
Σ
Σ
(f1) mm
W m2..k Rx, aI Rkg (1
k==
mi
E
(Q ) ( (Q8
(16)
50(3)
We can now focus on the integral Equation [6.. apply the rotation rule for spherical harmonics introduced in Section 3.1 and again exploit the linearity of the integral.
SSors)
yru mi
(Q8i5, |r (QB:)2))
(17)
) (fr.dh ) (1) )
=/ (5(
Σ
(8
(2) mmmmm
dQ
(18)
m2
mi
mG=-12
y((ti) (Bir)) (1 (f2,52) / [sors)
(() D(ta) mL dQ
(19)
m²
m
t1 my
82
(20)
At this point, we are able to apply the integration rule for elements of Wigner-D matrices introduced in Scction 3.1
Y-(e)
(Q
(rlla) (04.)QQii,.) (Qfi.ss) )dQ
(21)
5O(3)
8T
y(hi) (fi.j) /rlle (fi.jz) mm;
8 mi ma m 66
(22)
28 +
1
mi & ml E2
8T²²
(-1)mi-mi ylt)) (fi.5 (r(()) (fir.2)
61g 8 mm
(23)
2
To put it all together, we substitute Equation the sums:
23 into Equation
16 and we see many terms drop out of
N:
K
160
(f1) (22)
wg Ra, (1l (I) Rag (1
J1.J2
1 0
E .82=0
E M2 82
8 mi m2
S
Σ (-1) - mi meixl1a (bi)
(f: (fr. (r (R1.z) (24)
28 +
mi
m
(Q o pt, g)
Σ
Σ
mmy ki -mi .ba
(61) w (Ei)
Ra, (11 RND (11=:..)21))
SO(3)
J=dz=1 k2
8T
Σ
(-1)m:-mily (101))(( (f ) (x (ĀxJz) (25)
28 +1
mi
m/=-01
Once the spherical harmonics and radial functions are evaluated, this sum has O(N2K3L3) terms.
A.1 Basis Expansion
When implementing Equation 25
compute rotationally-invarii random features, one can see that a large amount of computational effort can be re-used between different random features evaluated on the same
18

---

Published in Transactions on Machine Learning Research (07 (07/2023)
sample. A table of spherical harmonic evaluations for all the points (f....... ) can be pre-computed once and re-used. Also, by re-arranging the order of summation in Equation 225 we arrive at this expression:
K
1000 1Q (Q 5013)
Σ
(61)
w (41)
ma
k2
my
N:
Re, (1
Rkg (11 Il)
8T
Σ
(-1)m--mg
miy (la)) mi ( ( (Bisz) (26)
26
Pre-computing everything after the line break allows for the re-use of a large amount of computational work between different random feature evaluations. We can interpret this pre-computation as an expansion of our point cloud in a votationally-inartii relates this basis expansion to the Atomic Cluster Expansion method.
basis B, which depends on the choice of radial function. Appendix
8
B(e, m, k1,
ke, k2] k2]= Σ
Ray [1 (I) Re, a1 II)
Σ (-1)m-m yl1) (fi )V (Fi.e))
(27)
28 +
J=-==1
N.
Σ
Ra, (1l (I) Re, (1
SSora)
(Q ( (Q
(28)
JiJu=1
In implementation precompute this B tensor for each sample, and then we perform tensor contraction with the random weights:
K Σ
SSOTS (0(3)
(Q (Q
w mi w (81) w-m ke B[ei, mi, mi, k2]
(4)
(29)
k1 6 =1=0
B Connection Between Our Method and Randomized ACE Methods
B.1 Overview of the ACE basis
The atomic cluster expansion (ACE) method builds a rotationally--inariial basis for point clouds that are extension of our pre-computed basis expansion described in Appendix A.T. In the ACE method, preliminary
basis A&, f. is formed by projecting a point cloud function (z = D; 8(x - xi,)) onto a single-particle basis function (x))
(30)
Axe.m(p)) = (ppke.m (.mm..p.))). Pt) Σ (m
(31)
(32)
where R2 is a set of radial functions. In Kovács et al. 2021 this set of radial functions is defined by taking
nonlinear radial transformation, and then the radial functions are set of orthogonal polynomials defined on the transformed radii. This basis set is augmented with auxiliary basis functions to ensure the potential energy two atoms at extremoly nearby points diverges to infinity.
The Ag..m basis is a first step, but it only models single particles at time. To model paírwise particle interactions, or higher-order interactions, the A basis is extended by taking tensor products. First, a "correlation order" N is chosen. Then, the A basis is extended via tensor products:
Agt. (p:)
II a=l
Ak. m. (p:)
(33)
19

---

Published in Transactions on Machine Learning Research (07/2023)
where & = (ka) - (La) and m (ma) multi-indices.
Finally, to ensure invariance with respect to permutations, reflections, and rotations of the point cloud, the ACE method forms a Haar measure dg over ((3) and computes a symmetrized version of the A basis:
I A p:)dH
B (pi)
(34)
Drautz 2019 gives detailed description of how this integral is performed. Applications of representation theory give concise descriptions of which multi-indices k, L, m remain after performing this integral; many are identically zero.
B.2 Connection to Our Method
If one chooses to compute an ACE basis with correlation order N = 2, the resulting basis is very similar to our pre-computed basis expansion in Equation (28
Bx (ps) = I Ak. (H pr)HH
(35)
[ A (H o Pi) Akzksmma (H o pi)dH
(36)
-
Σ pm (Haiss)
Σ
Pkz Pkizi (Harisp) dH
(37)
Σ Rki (112i, 11) (H )
{
Σ R&z Riz (1
(Hāis)))
dH
(38)
= Σ Re, (11:2,, Rba (11=3,n0 I ((h)) WHH (alla) )d
(39)
J1.J2
The remaining difference between our basis expansion in Equation 28 and the ACE basis expansion with correlation order N is the ACE basis integrates over all of O(3), while our method only enforces invariance with respect SO(3) C 033).
So describe random features as random nonlinear projections of simplified version of the ACE basis with correlation order N using extremely simple radial functions.
B.3 Benefit of Random Features
We conduct the following experiment to quantify the benefit of using random nonlinear features over fitting linear model in simplified version of the ACE basis. Using the same hyperparameters in our energy regression experiments, we compute our basis expansion for all the samples in the QM7 dataset. Using the same train seaii/vvvidaation/teee
split, we perform ridge regression by fitting a linear model in our basis B with 2 regularization. We use the validation set to identify the best-performing model and report errors on the test set in Table 4 We note that the best-performing linear model has more than twice the error of the best. performing random features model.
C Representation Theory of the 3D Rotation Group
C.1 Conjugation of Wigner-D Matrices
In the following, derive formula for conjugating clement of a Wigner-D matrix. First, use definition
6.44 in Thompson 1994 for the Wigner-D matrices:
D(0) maa (a., B,7)
- a m mm (3)e tmey
(40)
20

---

Published in Transactions on Machine Learning Research (07/2023)
Method
Test Error (eV)
Spherical CNNs
0.1565
FCHL19
0.0541
Random Features (Ours)
0.0660
Linear Model of B basis (Ours)
0.1542
Table 4: Reported are test errors on the QM7 dataset for Spherical CNNs, FCHL19, and two variants of our method. The first variant the full rotation-invariant random features model. The sccond variant linear model of our basis expansion. We note that while random feature model the second-best performing model on this benchmark, our linear model is the nearly the worst-performing model.
Here; (, are Euler angles parametcrizing rotation, and d m is an element of Wigner-d (small d) matrix. We omit the exact definition of d( m' mm but we note the following symmotry properties:
(3) (
m (-3)
(41)
m' (-3) = m.m' (3)
(
(0)
(42)
d mm (3) (-1)m-m d
m'm (3)
(43)
The relationship between the Wigner matrix elements allows to derive formula for conjugating an element of Wigner D matrix:
D mi (a, e glt)
mn m ( ammy
(44)
=e-i(-m g(0)
am (3) -i{-m))
(45)
= = ---(mmm)j (1) m m (-3)e --)m-i(-m>> -i(-m}r
(46)
=e -e-(mm')o (1) (3)e -if i(-m)y
(47)
= -i(mmm))a (-1) mmm 0 6-m'--m mm (B)e -i(-m)
(48)
= (-1)m'-m (0)
(m,_m(a,,87) 3,)
(49)
m -
C.2 Orthogonality Relations
We use the following orthogonality relationship between elements of Wigner D matrices:
SSOTO ((()) (Q), dQ
8T
2 8m, my 8k de,
(50)
This a well-known fact about the Wigner-D matrices. For a proof, one can derive this relationship from Schur orthogonality relations, which are consequence of Schur's lemma. A direct calculation can be found in section 6.4.2 of Thompson (1994). Combining the two facts from above, we arrive at the following identity, which use to compute rotation-invariant random features.
[S00 (1 1m (122) (Q), dQ d (-1)m² to +
8T²
8 mimms &
(51)
Additional Latency Experiments
In this section we present extra experiments about the prediction latency of our method. Figure compares the effect of molecule size on prediction latency for our method and FCHL19. Importantly, both methods have
21

---

Published in Transactions on Machine Learning Research (07/202) 2023)
FCHL19
Invariant Random Features
Inveriant Rendom Features
0.06
inference timee (seconds)
Inference timee (seconds)
1.75
1.50
0.04
1.25
1.00
0.02
0.75
0.01
0.25
0.00
7.5
15.0 Number Atoms
7.5
10.0
12.5
15.0
17.5
20.0
22.5
(a) Latency for small models
(b) Latency for large models
Figure 4: The number of atoms in a molecule affects the prediction latency for both FCHL19 and the invariant random features method. In Figure 4a we show the prediction latencies for the smallest models tested, corresponding to 250 random features for our method and 200 training samples for FCHL19, For this model size, a quadratic scaling the number of atoms largely determines the prediction latency. Figure 4b shows that for larger models (2,000 random features for our method and 5,000 training samples for FCHL19), the random features method is still in regime where prediction latency is determined the number of atoms, while FCHL19's prediction latency is dominated by the number of inner products required to evaluate the kernel.
a precomputation step that has complexity O(N²) where N is the number of aloms. For our experiments on ModelNet40,, the quadratic dependence on the number of points causes slow prediction latency, but we are able to alleviate this latency by reducing other hyperparameters (Figure 5).
We also investigate whether using deep multilayer perceptrons (MLPs) provides advantageous accuracy- latency tradeoff compared to our original ridge regression method. We first investigate the prediction latency incurred by using MLPs with different widths and depths. In Figure 512 and 1024 only moderately increase prediction latency and are therefore good architecture candidates for the prediction step. Models with higher widths incur up to four times the prediction latency ridge regression. However, in Figure 7 we find that after training these architectures to convergence, their test error does not outperform ridge regression. The input to our MLPs were 2,000 random features from the QM7 dataset. We trained the MLPs predict atomization energy by minimizing the mean squared error using the Adam optimizer. We scarched over learning rate schedules and weight decay hyperparameters.
observe that models with widths
E Hyperparameter Sensitivity Analysis
We perform multiple experiments to evaluate the sensitivity of our random feature models' performance. In Figures g and g we find that our models trained the QM7 dataset most sensitive the standard deviation of the random weights. In Figures 10 and 11 ModelNet40 dataset.
we find the same result for models trained on the
22

---

Published in Transactions on Machine Learning Research (07/202) 2023)
1.0
Maximum L
5
0.8
10
Classification Error
15
0.6
0.4
0.2
0.0
1.0 1.2
1.4
Prediction Latency (s)
Figure 5: When training classification models on the ModelNei400 dataset, we are able to trade off prediction accuracy for prodiction lateney by changing the maximum frequency of spherical harmonics L. We represent samples from the ModelNet440 dataset using point clouds of size N-1,024. Computing random features has computational complexity scaling quadratically in N, which significantly slows down prediction latency. However, the prediction error of our model is relatively insensitive to the maximum frequency L (sce Figure 10ah, so can greatly reduce the prediction latency by decreasing L. We evaluate prediction latency on & machine with 24 physical CPU cores. In the figure, the centers show the median latency sample, and the error bars span the 25th and 75th percentiles.
10
4000
Prediction Latency Increase Factor
3500
3000
2500
Width
2000
1500
1000
500
10
Depth
Figure 6: We measure the prediction latency of computing 2,000 random features and predicting using different MLP architectures. We present increase in latency when compared to ridge regression. A value of 2.0 indicates that predictions take twice as long as predictions using ridge regression.
23

---

Published in Transactions on Machine Learning Research (07 (07/202) 2023)
0.11
0.10
0.10
0.09
0.09
Train/lest MAE
0.08
Train/Test
0.07
0.07
0.06
0.05
Train MAE
0.05
Train MAE
0.04
Test MAE
0.04
Test MAE
Ridge Regression
Ridge Regression
0.03
MLP Depth
MLP Depth
(a) MLPs with width 512
(b) MLPs with width 1024
Figure 7: The ridge regression baseline outperforms MLP architectures with low prediction latency.
Train Test MAE
Train
Test
Train/lest MAE
10
10
Train/fest
10-2
10-2_
10
Maximum frequency
Standard Deviation of randomly-drawn weights a
(a) Effect of maximum frequency
(b) Effeec of random weight distribution W/ o2)
Figure the QM7 experiments, the standard deviation of the weights is the most important hyperparameler. For this sensitivity experiment, we use a standard hyperparameter setting and vary one hyperparameter at a time. The standard setting is 2,000 random features; standard deviation of the random weights the maximum frequency L = 5. In all experiments, parameters, and select the best model using a held-out validation set. In Figure Sa. we vary the maximum frequency of the spherical harmonies, and we see this does not have large effect on the test error. In Figure 8b we vary a, and we see this has a large effect on both the train and test error.
2.0; and search over pre-defined grid of L regularization
24

---

Published in Transactions on Machine Learning Research (07 (07/2023) /203)
Bump width 0.75x
1.0
Train MAE
0.5
0.08
Test MAE
0.0
Train/leest Finfeestt MAE
Bump width
0.06
0.5
0.05
0.0
5
Bump width 1.25x
0.03
0.5
0.02
0.0
1.25x
Radial Function Width
Effect of radial function width
(b) Visualizing different radial function widths
Figure 9: Our results the QM7 dataset are slightly sensitive to the width of the chosen radial functions. For the molecular energy regression experiments, with different widths [2), We train and test models with different width scales: 0.75 x [21,02] and 1.25 x (r,,0].. We show the resulting train and test errors in Figure 9a and visualize the resulting radial functions in Figure 9b
radial functions two Gaussians centered at 1,
Train Error
0.4
Test Error
0,4
Trainffest Enror Eroor
Train/lest
0.2
0.2
Train Error
0.0
0.0
Maximum frequency
Standard Deviation of randomly-drawn weights
(a) Effect of maximum frequency
(b) Effect of random weight distribution W/ o²)
Figure 10: In the ModelNet440 experiments, the standard deviation of the weights the most important hyperparameter. For this sensitivity experiment, use standard hyperparameter setting and vary hyperparameter at time. The standard setting is 15,000 random fcatures; standard deviation of the random weights a = 2.0; and the maximum frequency = 10. In Figure 10a we vary the maximum frequency of spherical harmonics L, and we see that increasing T. slightly improves the performance. In Figure 10b we vary a, and we see the model's performance is sensitive to this parameter.
25

---

Published in Transactions on Machine Learning Research (07/202) 2023)
Radial function width 0.75x
1.0
0.5
0.300
0.8
Train/Test Error
Radial function width
0.250
1.0
Train Error
0.5
Test Error
0.0
0.200
0.0
0.175
Radial function width 1.25x
0.150
0.5
0.125
0.0
0.0
0.2
0.6
0.8
0.75x
1.25x
Radius
Radial Function Width
(a) Effect of radial function width
(b) Visualizing different radial function widths
Figure 11: Our results the ModelNet40 dataset are very slightly sensitive to the width of the chosen radial functions. Our original radial functions are three Gaussians with centers [0, 0.5, 1.0] and width a. We train and Lest models with different radial function widths: 0.75 x 1 x a, and 1.25 x a. Figure 11a shows the resulting train and test errors, and Figure Ilb shows the resulting radial functions.
26

---

Published in Transactions on Machine Learning Research (07/2023)
F
Solving Ridge Regression Problems for Random Features
This section outlines numerical methods for solving ridge regression problems arising in our random features models. In general, we want to solve the following problem:
arg min 4 ull2 + 1
(52)
In our problem instances, Φ E Tg is the matrix of random features. T is dense matrix; there is no structured sparsity. n is fixed; it is the number of samples in the dataset. The number of random features d can be treated as a hyperparameter, but we find that d >> n is required for optimal test error, 80 the system of the regularization parameter. Larger À corresponds to more regularization. Generally, when solving a large system, we approximately know a good value for and want keep it fixed To recap, we have an instance of dense, overdelermined ridge regression problem.
equations is underdetermined.
By taking the gradient of Equation 52 with respect to 3 we arrive at the normal equation:
① Φ + A1)
8" TTy
(53)
F.1
Exact Solution via the Singular Value Decomposition
From Equation
53
we see that the solution to the ridge regression problem
0 ( ( + X)) TTv = -V 2T) - UU y
(54)
where 4 UEVT is the singular value decomposition (SVD). Because (Σ² + AI) diagonal matrix, we can exactly and efficiently compute its inverse. This suggests a method for solving Equation (52): compute the SVD of D, efficiently compute (I² + A and reconstruct 8 via Equation 54p. This solution method is stable and relatively performant for small problem instances (n, d < 10, 000). We use this method the QM7 experiments. Evaluating the full SVD of the feature matrix generated the QM9 experiment is prohibitively costly, so we experimented with other solution methods.
F.2 Approximate Solution via Iterative Methods
Equation 52) can be treated as an optimization problem in the variables E Rd. The objective strongly convex, with smoothness omax (+) and strong convexity parameter A. This motivates the application of standard iterative methods to find approximate solution. Gradient descent, for instance, is known to enjoy lincar convergence (on log-log plot) for smooth and strongly-convex objectives. However, the rate of convergence is (o) which is very slow for typical problem instances.
A family of iterative algorithms known Krylov subspace methods are designed for linear least-squares problems like Equation (52) These methods include the conjugate gradient method and variations. In the QM9 experiment, we use LSQR (Paige & Saunders) 1982a b) implemented in scipy (Virtanen et al. 2020)
While they are empirically faster than applying gradient descent to the objective and are known to converge to the exact solution in finite number of iterations, the conjugate gradient method and LSQR both suffer from poor convergence dependence the quantity ( We have found that our problem instance, LSQR requires approximately 100,000 iterations to converge to a solution with low training error.
F.3 Approximate Solution via Principal Components Regression
We also attempt to approximately solve the ridge regression problem with principal components regression. We compute a truncated SVD with scipy's sparse lincar algebra package (Virtanen et al. 2020 The runtime
2 As opposed to other optimization problems, where increasing is acceptable method of improving the conditioning of the system.
3Tee strong convexity parameter actually max(A, o o2 (4)) but in practice, our random feature matrices are approximately low rank, so 2in (+) &
27

---

Published in Transactions on Machine Learning Research (07 (07/2023)
of this operation depends on k, the number of singular vectors choose to resolve. After computing the truncated SVD, we construct approximate solution:
3. V2 (E) + +1) - EU
Empirically, we find that we need to choose a large & to find a solution with low training error, which incurs a large runtime. We find LSQR arrives an approximate solution faster than principal components regression.
F.4
Future Work
We have identified two different classes of methods from numerical linear algebra as promising candidates for improving our scaling to larger datasets.
F.4.1 Preconditioning Iterative Methods
The runtime of iterative methods like LSQR (Paige & Saunders 1982a b) depend adversely on the problem's 's conditioning _ One way to improve the convergence methods like LSQR is to find a preconditioning matrix M [g where @) < ( Once a preconditioner is formed, one solves the augmented
problem
arg min M Myl f 2||311
using just few iterations of an itcrative solver. There are number of classical proconditioning techniques and review articles, including Wathen (2015):
Benzi
(002)).
LSRN (Meng et al. 2014 offers a promising method of computing a preconditioner designed for dense, overdetermined ridge regression problems. The algorithm constructs a preconditioner by subsampling columns of the matrix & and computing a singular value decomposition the subsampled matrix. We believe that using a preconditioner like LSRN a promising future research direction.
F.4.2 Approximate Solution via Sketching
Matrix sketching methods from the field of randomized numerical linear algebra Drineas & Mahoney 2016 aim to reduce the dimension of linear system while approximately preserving the solution. This can be performed by randomly selecting rows the (possibly preconditioned) matrix & and solving the smaller problem instance. Sketching algorithms specifically for overdetermined ridge regression have been introduced in recent. years Chowdhury et al. (2018); Kacham & Woodruff (2022).
28


Reviews:
[{"review_id": "DBpSE7K78x", "reviewer": "Reviewer_drhY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This maniscript studies an interesting and important problem: how to build rotation-invariant features for point-cloud data. The authors introduce new mathematical tools and show promising initial results.\n\nThe proposed algorithm performs quite well on small-scale dataset such as energy regression in QM7 dataset. This will be particularly attractive and useful for many real-world applications.\n\nThe proposed representation is quite general and task-agnostic, making it suitable for many different tasks.\n\nOn the other hand, it’s quite concering if the proposed method can be scaled up since the feature matrix depends on the number of data points. The author also mentioned the applicability of the proposed method on larger-scale dataset and result in ill-conditioned feature matrix. It remains unknown how to use it in those cases.\n\nIn Table 2, the high-performance methods are all based on neural network. I think the proposed feature can be direclty used as the input of the networks. Is there any technical difficulties in this part. It will be more convincing if the proposed feature can further improve neural network ability.\n\nSimilarly, the performance on modelnet is not very promising. Even though this particular dataset is not the focus on this paper. It is still quite far away from the current state-of-the-art method and is not convincing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a novel approach to constructing rotation-invariant features for point-cloud data using multiple basis defined by spherical harmonics functions. The proposed decomposition method is applicable to various tasks, including energy regression in physics/chemistry and point-cloud shape classification in computer vision. The authors demonstrate state-of-the-art performance in small-scale energy regression experiments and show the applicability of their approach in larger-scale energy regression experiments and shape classification. The contribution of this paper lies in introducing new mathematical tools to the field and providing promising initial results.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Critical Changes:\n\nIt would be a huge benefits if the author could concern how to address the performance issue on larger scale datasets.\n\nAnother concern is although the feature is general, the performance is only attractive in one case: small-scale energy regression. I would suggest the author improve performance on modelnet40 or exploring other cases where the propose methods can be critical or useful.\n\nMinor changes:\n\nThere are some typos in the equations or minor typos. The authors should carefully proofread the text.", "broader_impact_concerns": "Not Applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "25fGUnovSg", "reviewer": "Reviewer_RJHT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written and easy to follow. In the paper, the authors consider transformations w.r.t all possible rotations in SO(3) on the point cloud data into the random feature method proposed by Rahimi & Recht (2007) for rotation-invariant learning, and solve the problem via spherical harmonics with well theoretical derivation, making the method reliable. Experiments conducted on QM7, QM9 and ModelNet40 datasets to evaluate the proposed method in terms of accuracy and efficiency..\n\n\nHowever, results of the proposed method on all the three datasets are less impressive. It’s suggested to include more comparisons and analyses among different methods, e.g., those listed in Spherical CNNs proposed by Cohen et al. and some recent ones for QM7 and ModelNet40 datasets, under different settings, e.g., the test settings of ‘z/z’, ‘SO(3)/SO(3)’ and ‘z/SO(3)’ in [1] for ModelNet40 dataset. I may also suggest including in Fig.3 the inference time of the results in Table 1.\n\nSome references are missed in Sec. 5.1.2 and all the three tables. I may suggest including in the paper some more recent references. \n\nBesides, I could find the notation explanation of  $\\phi(\\cdot;w_j)$  in Sec. 4.\n\n[1] Carlos Esteves, et al., Learning SO(3) Equivariant Representations with Spherical CNNs.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "A rotation-invariant version of the random feature method is proposed in this paper for processing point cloud data, and is applied to the tasks of molecular property prediction and 3D shape classification.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Refer to Strengths And Weaknesses.", "broader_impact_concerns": "The article has not had a broader impact statement, e.g, use for molecular property prediction.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sjS79a7BAb", "reviewer": "Reviewer_cvrh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:  \n(+) The paper tackles an important problem in 3D point cloud analysis, of designing invariant functions. The motivation is clear, the writing is well polished and concise and made self contained.   \n\n(+) The adaptation of random features to the 3D case is quite elegant and well explained\n\n\n**Weaknesses**:  \n(-) While the paper proposes an elegant method its performance falls short compared to some reported baselines. It would be interesting to explore in which scenarios the proposed method could provide an advantage. The authors suggest that it can be used to bound the performance of more sophisticated neural network based methods however given the gaps with baselines, in which cases would that bound be useful? \n\n(-) The proposed method involves choosing radial functions and other weight parameters, which may affect its performance. A sensitivity analysis of these parameters would provide more insights into the model's behavior and could help guide future improvements.\n\n(-) It would be helpful to understand better why the proposed method is not as performant as some other methods. For example, is it because the invariant features are not expressive enough, or because they are used in ridge regression rather than as input to a deep net? Additionally, exploring how the method's performance would be affected if the shapes were aligned to a canonical pose, or if the amortization causes a large gap in representation capacity, could provide useful insights.\n\n(-) The paper could benefit from a discussion of its relation to other recent rotation equivariant works. For example, Frame Averaging is a more general approach to making any function applied to the point cloud equivariant. Highlighting where the proposed method could be preferred over these alternatives would provide more context and a more nuanced understanding of its strengths and limitations.\n\n\n**References**:   \n[1] Frame averaging for invariant and equivariant network design  \n[2] Vector Neurons: A General Framework for SO(3)-Equivariant Networks", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a method to compute learnable rotation-invariant features for the purpose of solving rotational invariant tasks like classification of point clouds. \nThe core idea follows the method of random fourier features, and generates a randomly-weighted combination of volumetric functions which are a composition of radial functions and spherical harmonics functions. These functions are then used to compute features by amortizing their multiplication with the (zero centered) pointcloud function across all possible rotations. Through properties of spherical harmonics, this integral has a closed form solution. Each such operation (composed with a $\\sin$ function) results in an entry inside the feature matrix $\\Phi$. This feature matrix is then used to learn parameters $\\beta$ through ridge regression towards matching target labels $\\hat{y}$. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* suggest practical use-cases\n* comparison to more recent baselines\n* Using the features in a deep net\n* parameters sensitivity analysis\n", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper bears in mind the importance of rotation invariance in many vision and machine learning tasks, and study how rotation invariance plays roles in these learning tasks, especially when deep models are simultaneously used. The paper devises a general-purpose method that learns rotation invariant functions using random features. Experiments show that the method provides a rotation-invariant baseline on some tasks, and is more efficient than other alternatives. 

All reviewers acknowledge that the paper addresses interesting and important problem, and the proposed solution is elegant. Reviewers are also concerned with performance of the proposed method, especially on large-scale data and tasks, how features from the proposed method can be used as input of the deep networks, and other technical and writing issues, including sensitivity analysis of radial functions and weight parameters. 

Authors have revised the paper that has addressed most of these concerns. Please further revise the paper to address the remaining issues, if any, including the writing ones. 

Decision: Accept (poster)