VectorAdam for Rotation Equivariant Geometry Optimization
Authors: Selena Ling
Nicholas Sharp
Alec Jacobson
Venue: NeurIPS 2022 Accept (2022)
Keywords: adam
optimizer
geometry optimization

Abstract: The Adam optimization algorithm has proven remarkably effective for optimization problems across machine learning and even traditional tasks in geometry processing. At the same time, the development of equivariant methods, which preserve their output under the action of rotation or some other transformation, has proven to be important for geometry problems across these domains. In this work, we observe that Adam — when treated as a function that maps initial conditions to optimized results — is not rotation equivariant for vector-valued parameters due to per-coordinate moment updates. This leads to significant artifacts and biases in practice. We propose to resolve this deficiency with VectorAdam, a simple modification which makes Adam rotation-equivariant by accounting for the vector structure of optimization variables. We demonstrate this approach on problems in machine learning and traditional geometric optimization, showing that equivariant VectorAdam resolves the artifacts and biases of traditional Adam when applied to vector-valued data, with equivalent or even improved rates of convergence.

VectorAdam for Rotation Equivariant Geometry Optimization
Selena Ling
Nicholas Sharp University of Toronto nsharpocs toronto..... edu
University of Toronto
selena ling@mail
www.ootooooo... wtw.gamai....ooootttoooooooo
Alec Jacobson
University of Toronto. Adobe Research jacobson8cs, toronto.. edu
Abstract
The Adam optimization algorithm has proven remarkably effective for optimization problems machine learning and even traditional tasks in geometry processing. At the same time, the development of equivariant methods, which preserve their output under the action of rotation some other transformation, has proven be important for geometry problems across these domains. In this work, we observe that Adam when treated as a function that maps initial conditions optimized results - is not rotation equivariant for vector-valued parameters due per-coordinate moment updates. This leads to significant artifacts and biases in practice. We propose to resolve this deficiency with VectorAdam, simple modification which makes Adam rotation-equivariian by accounting for the vector structure of optimization variables. We demonstrate this approach on problems in machine learning and traditional geometric optimization, showing that equivariant VectorAdan resolves the artifacts and biases of traditional Adam when applied to vector- valued data, with equivalent or even improved rates of convergence.
Introduction
Over the last decades, gradient-based optimization has enabled rapid progress in machine learning and related fields. To tackle the problems caused by large variance of network weight gradients, researchers have developed adaptive variants of stochastic gradient descent such Adam (IT) which adaptively rescale the step size based on per-scalar gradient statistics to accelerate convergence.
In the field of computer graphics and geometry processing. we also optimize geometric energies. For example, the progress in differentiable rendering made derivatives rendering parameters such geometry, lighting and texture parameters amenable these gradient- based optimization techniques. Two recent differentiable rendering works 20 14 default to Adam for their experiments. Recently, also the successful use of customized Adam algorithm for optimizing geometric interpolation weights [3I] This work taps into this trend and takes a closer look the of Adam in geometric optimization problems in geometry processing and machine learning.
increasing use of Adam to
Another important concern in geometric learning is the design of equivariant functions, which have the property that rotating the input to the function causes the output to rotate in the same way. This property is natural for many geometric problems, where the choice of coordinate system is arbitrary, In this work we adopt the perspective that the optimization process is itself a function, mapping initial conditions optimized results, and it is likewise expected that this function be equivariant. The
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

widely-used Adam algorithm lacks this property, because its per-scalar statistics do not account for vector structure in the optimization variables. In particular, we show that Adam updates depend on the coordinate system chosen to represent vector-valuee data, even if the minimized loss function is coordinate invariant. This leads to unexpected pordinate-dependent artifacts and biases, even when optimizing simple geometric regularization energies.
We propose VectorAdam, a solution that extends Adam's per-scalar operations vectors. We test VectorAdam on a number of machine learning and geometric optimization problems including geometry and texture optimization through differentiable rendering, adversarial descent, surface parameterization, and Laplacian smoothing. We find that like regular Adam, VectorAdam outperforms traditional first-order gradient descent, yet unlike regular Adam it provably rotation equivariant.
2
Related Work
Many techniques have been proposed for gradient-based optimization in machine learning and in numerical computing more broadly (see (28] for general survey). We will primarily consider first order methods, which take as input only the gradient of the objective function with respect to the unknown parameters, and produce as output updated parameters after the optimization step. For large neural networks, loss gradients are often estimated stochastically on subset of the loss function (25]. however optimizers are generally agnostic to how the input gradients are computed. VectorAdam is no exception and works with stochastic samples or full gradients.
Numerical Optimization The most direct approach is to simply update the parameters directly via (stochastic) gradient descent, but this approach can be improved in many respects. In numerical optimization literature, techniques like momentum [18] and line search 21 offer accelerated conver- gence over basic gradient descent. More advanced techniques such as BFGS [ [6] and conjugate gradient [10] make higher-order approximations of the objective landscape, while still taking only gradients as input.
Optimization in Machine Learning Machine learning, and in particular the training of deep neural networks, demands optimizers which have low computational overhead, scale to very high dimensional problems, avoid full loss computation, and naturally escape from shallow local minima. Here, many optimizers have been developed following the basic pattern of tracking ((1) additional data associated with each parameter to smooth and rescale gradients to generate updates 6 34 30 [1]. All of these methods can be shown to have various strengths, but Adam [I1] has emerged as a particularly popular black-box optimizer for machine learning. The crux of our work is to identify a key flaw in Adam when optimizing geometric problems in machine learning and beyond, and to present a simple resolution. We focus primarily on Adam due to its ubiquity and recent relevance in geometric optimization more broadly, but our core insights could be applied to many other gradient- bradient-based optimization schemes in much the same way.
Variants of Adam A number of variants have been proposed to improve Adam's performance in training neural networks. For example, NAdam (5) incorporates Nestorov momentum into Adam's formulation to improve rate of convergence, and RAdam (17) introduces a term to rectify the variance of the adaptive learning rate, among many others. Most similar to this work, Zhang et al. (37) (also published as (36)) propose ND-Adam, which preserves the direction of weight vectors during updates improve generalization. Like our approach, Zhang al. explicitly leverage the vector structure of optimization variables. However, their approach does so for the sake of generalization and memory savings rather than equivariance, and produces magnitude-on updates to vectors which are unsuitable for geometric problems where points must move freely through space.
Adam in Geometry Recent work in 3D geometry processing has leveraged Adam to great effect, adjusting the algorithm for the problem domain. Nicolet et [19] proposed UniformAdam, which takes the infinity norm of the second momentum term, improving performance for differentiable rendering tasks. Wang et al. 31 customized their own Adam optimizer by resetting the momentum terms regularly in the optimization process, outperforming traditional L-BFGS on challenging benchmark. This work aims to further this trend of Adam as general optimization tool by modifying it to have the eeuivvarinivv properties which are fundamental to geometric computing.

---

Rotation Invariance and Equivariance This work contributes to the fruitful study of invariance and equivariance in geometric machine learning [ When some transformation applied the input, an invariant method produces the same output, while an equivariant produces equivalently- transformed output. In our context, one expects that if the input to a geometric optimization problem is rotated, the output will be identical up to the same rotation- optimizer is equivariant. Many novel invariant and equivariant layers, architectures, losses, and regularizers have been developed across geometric machine learning (see [29] 32 7 12 3 23 35
9 15 among many others). In
this work, we argue that optimizers deserve the same treatment.
Background and Motivation
3.1 Adaptive Moment Estimation (Adam)
Given an objective function f parameterized by 0 and its gradient the first-order optimization algorithm Adam adaptively rescales gradients of its 7th iteration using rolling estimations of first (m) and second moments (v) dictated by the following rules: for each scalar variable O..
t 0f/00;
m, - 31m, + (1 31)g:
8 - 324
32)
0 - 0 -a 1 B
m4
0/
+ +c)
where 22, are Adam's hyperparameter constants 11 and a the specified learning rate.
Adam's optimization step is invariant to any uniform re-scaling of the gradients' magnitudes. The gradient is adjusted take a value roughly =1, bounding the effective step size by the learning rate a. This makes it particularly suitable for problems with gradients of large variance and unknown scaling, such as neural network optimization.
3.2 Problems in Geometry Optimization
In recent years, also increasing use of Adam in geometry optimization problems such as differen- tiable rendering [3] 20 [4]. In this problem domain, the target parameters are usually geometric quantities such as vertex positions of triangle mesh, surface normal vectors, and camera positions. For all of these quantities, the vector form of their gradients encodes important geometric information that are tied to the specific coordinate system in which the problem is em- bedded.
Traditional Adam, originally devised for network weight optimization, does not account for the vector na- ture of gradients in geometry problems. The optimiza- tion process is thus not rotation equivariant-4 funda- mental property expected practice-ane and furthermore optimization may be more unstable. Figure shows an example of traditional Adam's component-we oper- ations creating self-intersections and axis- ali-nal arti- facts, observed in an inverse rendering problem.
Adam
VectorAdam
Figure 1: Geometric optimization results on two sample inputs arising differ- entiable rendering task. Ordinary Adam results in axis-aligned artifacts, which are resolved by using VectorAdam instead.
When we consider gradients vector- wise, regular Adam's 's uniform re- re-scaling does not preserve the norm and the direction of the gradients. In most cases, Adam's 's algorithm re-scales gradient to the magnitude of 11, restricting each optimization step within a trust region of the current parameter defined by the size of the learning rate [1I].
This, however, problematic geometry optimization context. When all components of gradient vector re-scaled to be of magnitude 1, the gradient vector snaps to the diagonal direction of

---

the coordinate system as shown in Figure changing its original direction. As shown in Figure this diagonal bias in early optimization steps easily results in celf-intersections near the axis and introduces geometry changes that are hard to correct in subsequent steps.
In geometry optimization problems, such inverse rendering, self- intersections are a major source of unrecoverable failure. A non-self- intersecting surface forms the well-defined boundary of solid shape (left in inset).Such boundary can be displaced so that one part of the shape passes through another part, causing self-intersection of the mesh (red in inset). Self- ieffintteseccions are symptoms implying that space is no longer well separated into "inside" and "outside" (ambiguous regions shown in solid red in inset).
For example, consider the case where three parameters within form vertex position [x. y, in
a geometry optimization problem. Denote its corresponding gradient vector [Øx- Pv. 92]. The first Adam step = 1 after m, v are initialized with zeroes re-written in terms of this vector-valued quantity as
[Ma myy m2] (1 - 31) By 9z]
(2)
[Uxy Nyy U2] = (1 (1-B2)l93,
32) . oy g-1
and the step update for this vertex the first iteration becomes:
// A ya, y, =| = 9r / 03 t) 9u /W +c) 9x / +6)
(3)
-a aa (sign( sign( sign(g=)]
That for any non-vanishing gradient values, a vertex will be updated along some diagonal of the underlying coordinate system (-a(1, ( 1,
-a[-1, 1, -1]. etc.). This effect creates bias toward the corners of the coordinate system not just on the first iteration, but also throughout the optimization process (see the histograms in Figure o for experimental evidence).
If gradient vector indicates how the algorithm should, for example, move vertex a 3D coordinate system to reach the optimal shape, the original Adam optimizer thus changes not only the magnitude but also the direction of each step. The optimization process then suffers from the loss of important geometric information encoded by the original gradient vectors' directions.
4 VectorAdam
We propose VectorAdam, a first-order rotation- cquivariant optimizer for geometry optimiza- tions. It tackles the aforementioned flaw of Adam for geometry optimization, by modify- ing Adam's 's per-scalar operation to operate on vectors. As shown in Figure [ our proposed VectorAdam rescales the gradients vector-wise while preserving direction, making the optimiza- tion process rotation equivariant and stable.
Original Gradients
Adam
VectorAdam
Figure 2: Adam's per-scalar adaptive momentum scaling results uneven vector norm and direc- tion inconsistency when the gradients have vector structures, while our VectorAdam scales the vector norm evenly and preserves the gradient direction.
More specifically, we note that the problems discussed above with the original Adam algorithm stem from the per-scalar calculation of the second moment v. VectorAdam modifies the second moment update to the square of the vector norm per gradient vector instead of taking the per-scalar square. The algorithm then proceeds with the regular bias correction and updates the gradients vector- wise with corresponding products vector first moments and scalar second moments. This has welcome side-effect of requiring a factor of n less memory for storing moments of n-dimensional vector parameters. Without loss of generality a description of VectorAdam is provided in Algorithm assuming all parameters in 0 can be arranged as an m-long list of n-dimensional vectors. If the optimization involves quantities of varying vector dimension n, the algorithm can applied in tandem to each, noting that for the scalar 1 case VectorAdam reduces to ordinary Adam.

---

Algorithm VectorAdam: minimizing f over parameters, each an re-dimensional vector.
Require: step size
Require: B1, B2 E [0, 1): Adam's 's exponential decay rates for the moment estimates
Require: <>> 0: Adam's numerical fudge factor
Require: f(0): (stochastic) objective function
Require: 0: Initial parameter vector of shape r
m <- (Initialize 1s moment vector of shape x n))
f D (Initialize 2nd moment vector of shape x & + 0 (Initialize timestep)
while 0, not converged do
t - t ++]
g f Vo) (Get current gradients of shape n)
for i E [1 do (Iterate through each vector quantity in rows of f)
m, - f1m; + (1 81)g
Bzu; + (1 32)1
m. m. -3)
G - 0/ -
0. 0. ariit// (v 6))
end for end while return /
Let's reconsider our 3D vertex example above, applied to the first step of optimization after initializing m, With VectorAdam, the first, inttiiiitialized iteration reduces to
[mays Myn ms] = (1 [g 9t 9s]
(4)
v = (1 - 32) [9 Bys 9=]11²
The corresponding VectorAdam step update for this vertex becomes
aax, y, 2] = -a 9y: By: 91lg/ Bys 9=]ll
(5)
that is, the unit-length direction of the vertex's gradient scaled by the learning rate Thus, VectorAdam preserves the direction of original gradients while roughly normalizing their magni- tude. This achieves the vector-wise analogue of the original Adam's 's action on scalar-wise gradients.
4.1 Properties of VectorAdam
The key property which makes VectorAdam suitable for geometric optimization problem is its rotation equivariance: when the input data is rotated, the output of the optimizer will be identical up to the same rotation, at every step of optimization.
To formalize this property, we consider function VectorAdam which evaluates one update step of Algorithm on single parameter vector 0 with first and second moment terms denoted by m,
0. m, + VectorAdam (0, m, v)
i E R" m E R E
(6)
for some given rotation invariant objective f, and internal optimizer parameters a, B2, Proposition (VectorAdam is rotation equivariant). For any rotation SO(n), let
m' v² +- VectorAdam; (0, v) and Ø. ml v' +- VectorAdam (R00 Rm,,))
Then VectorA dam satisfies 0' = RØ² n.' = Rm, and v' = v²" Because each step of VectorA is equivariant, the parameters 0 after any sequence of optimization steps are equivariant as well.
Proof. Since f is rotation invariant, its gradient is rotation equivariant, i.e. (f(RR) = RV Substituting in to the update step then shows the equivariance of m
m' B, Rm + (1 Rg R(3 m + (1 31)g) -mm
5

---

original gradients
Adam
Adam step
After Step 1
After Step 5
After Step
After Step
VectorAdam step
VectorAdam
Figure 3: Laplacian regularization on 2D and 3D spherical meshes demonstrates the coordinate system bias of Adam, which changes gradients directions and magnitude breaking the intrinsic symmetry of this problem setup. VectorAdam maintains symmetry.
and likewise for the invariance of 1
y' = 3 (1 12)-211 B2u (1 32)1 0²
(8)
and the same trivially holds for their rescaled counterparts m. Rmt and o' = 6*. This then yields the equivariance of
0 = RØ - am'' - am /(v 0- + c)) = Rd
Induction these relations with the base case mi = 0 and 0 then furthermore implies that the same equivariance holds for any sequence of successive optimization steps by VectorAdam.
5 Experiments and Applications
We now compare the performance of Adam and VectorAdam in range of optimization problems in machine learning and traditional geometric optimization context. These experiments demonstrate the undesirable coordinate-systen bias and artifacts caused by Adam's per-coordinate update rules, and show how Vector Adam solves this problem while achieving equivalent or even improved quality and convergence rate. All experiments performed in PyTorch
NVIDIA RTX 2080 GPU.
Vector-valued data are especially common in geometric optimization problems, which the gradients, also structured by vectors, carry important geometric signals for the problem at hand. We consider Laplacian regularization, adverserial descent, deformation energy optimization, and several inverse rendering tasks, all of which leverage first-order gradient- based optimization. Our VectorAdam yields stable and rotation equivariant optimization, and often arrives at a better result by preserving the geometric signals encoded by the gradient vectors.
Laplacian Regularization
5.1 Geometric Optimization
Laplacian regularization mesh is widely-used in geometric problems, like 3D reconstruction and generative modeling tasks, especially to discourage mesh self-intersections. Applicable to any mesh or graph problem (e.g.. point clouds k nearest neighbors), this loss term adds smoothness prior on 3D positions by requiring them to as similar as possible to their neighrbors'. For surfaces this loss is often discretized version of Dirichlet energy (f ||V=))',, expressed in the form of u u in which v is the mesh vertices and I. represents the Laplacian matrix. intrinsic and does not depend on the rotation of the shape.

---

In this experiment, we use both Adam and VectorAdam to directly minimize the Laplacian regulariza- tion loss on mesh. We observe that Adam's 's per-coordinaee moment updates introduces undesirable axis-aligned distortion to the Laplacian regularization. In Figure 3 Adam's 's optimization steps shrink the original sphere 3D, or circle in 2D, to a diamond shape after a few optimization steps, which aligns with the diagonal bias we discussed before. In addition, in Figure 4 we show that Adam's s gradient update is not rotation equivariant, resulting in a different updated mesh given a rotated input mesh In contrast, VectorAdam follows the geometric intuition and shrinks the sphere, circle, smoothly across the surface in Figure 3 and updates the rotated input mesh identically in Figure I
Deformation We also experiment with two common deformation loss optimizations in 2D. These energies can be seen as priors defining the smooth deformation. In partieular, imize the as-rigid-as -iissila-- energy and the symmetric Dirichlet energy on a deformed 2D mesh given reference mesh.
min-
The prriiid-as-possibbee energy measures how much a mesh has become distorted relative to an initial "rest" configuration. Given edge connecting and pj in the original mesh, pa and p', in the deformed mesh, the as-rigid- as-possible energy at this edge is defined as ming., (p(p; Pş) Rs P')Il with the best fit rotation Rıg E SO(3). This energy is interesting because it measures the local stretch- ing and squashing of the triangles while fac- toring out local rigid motion (27]. It is a commonly-used deformation energy with its detail-preserving property.
Input
Adam
VectorAdam
Figure 4: We optimize for Laplacian smoothing loss on a canonical mesh and rotated mesh with both Adam and our VectorAdam. The output of ordinary Adam significantly altered by the ro- tation, while our VectorAdam yields an identical update up to rotation, as expected.
We optimize the ARAP energy for 2D triangle mesh and visualize the per-vertex gradient vec- tors for the first optimization step in the top left panel of Figure 6 Gradient Descent has no coordinate-systen bias, but wide range in step scales. If the gradient vector for particular vertex is [x: y), then the first VectorAdam step will normalize this to -ax (c y)- Accordingly, these uniform-length steps show no coordinate-system bias. Meanwhile, the first step of Adam will take a step of -a (siiinn(() (x) sign(y)) thus exhibiting extreme bias to the 45 directions.
Log(E)
Symmetric Dirichlet energy is an alternative way of measuring triangle distortions between a de- formed mesh and the reference mesh. It is based
2.1
2.0
on a computationally efficient form of isomet- ric distortion metric that uses the eigenvalues of the affine transformation between two triangles. Optimizing the Symmetric Dirichlet energy pre- vents local folds of triangles. Similar to the Laplacian regularization, the energy is intrin- sic to the mesh, and therefore rotation invariant, We optimize for symmetrie Dirichlet energy 16 times with the input mesh rotated by a different angle and plot the loss curves in Figure [5] As shown in the Figure 5] VectorA follows the same optimization path regardless of the starting orientation, whereas Adam produces different optimization paths. VectorAdam achieves equiv-
Adam
1.9
Gradient Descent
1.8
VectorAdam
1.7
2
3
5
6
Log(Steps)
Figure 5: Log loss curves with symmetric Dirichlet energy (26] optimization with rotated inputs. We show that our VeatorAdam follow the same descent path as desired while Adam's 's descent path varies across rotated inputs.
alent convergence rate and final quality as traditional Adam.
7

---

2D ARAP Optimization
3D Point Cloud Classification
Adversarial Optimization
Gradient Descent
VectorAdam
Adam
(a) Gradients at first update step
VectorAdam
Adam
(b) 2D gradient direction histogram
Figure 6: On the left, we optimize the ARAP energy for a triangle mesh with 200 vertices for 100 iterations with 1000 randomly rotated initial configurations. (a) shows the gradients at first optimization step from Adam and VecmorAdam. (b) shows the histograms of the angular distribution of these 20M gradient directions. Gradient Descent and Vector Adam exhibit their rotation equivariance by presenting a uniform distribution, while Adam presents a biased distribution with spikes at 45 angles. On the right in adverserially optimize inputs to 3D classifier.
observe similar coordinate-syster bias input point cloud, when
Adversarial Descent Similar challenges occur in the context of geometric machine learning, One such example adversarially optimizing input data minimize or maximize classifier's prediction, process which is vulnerable to Adam's ss coordinate bias. Here, we trained point cloud classifier from the -oution-equivariant Vector Neuron architecture [4] on the ModelNet40 dataset [2]. and adversarially optimize input point cloud to minimize the classifcation loss. In Figure 6 right, Adam's gradient updates exhibit the diagonal bias while VectorAdam preserves the directions.
Inverse Rendering Research progress in dif- ferentiable rendering in recent years has made image-based inverse geometry optimization popular problem. Within this problem domain, we experiment with shading-based
Log(E)
2.1
2.0
Adam
1.9
inverse geometry and texture reconstruction. Both of these problems optimize vector-valued parameters, i.e. vertex positions and RGB color vectors.
1.8
Vector Adam
1.7
50
100
150
200
Steps
For geometry reconstruction, we run the same experiments as in [19], which optimize sphere to match against rendering of a Nerfertiti bust. More specifically, times with different learning rates the range of 9 and le-5, and show the Hausdorff distances between the optimized mesh and the target mesh in Figure during optimization. Quantitatively, VectorAdam's optimized mesh consistently achieves better, i.e. lower, Hausdorff distance with the reference mesh. Qualitatively, we observe more self-intersecting faces in Adam's 's optimized mesh as highlighted in red.
Figure Log loss curves for environment map reconstruction with Adam and VectorAdam, over 200 runs. The idea of VectorAdam also benefits optimization of non-geometric quantities that have vector structures such as RGB colors.
run the optimization 8
We note that an Adam variant called Uniform was proposed [19] which also preserves gradi- ent direction by taking the infinity norm of the second moment estimation. However, UniformAdam is

---

Hausdorff Distance
10
Adam
VectorAdam
10
Self-intersections
2000
4000
6000
Steps
Adam
VectorAdam
Figure 8: For shading- based inverse reconstruction of the Nefertiti bust, Hausdorff distance between optimized mesh and target mesh across multiple runs with learning rates in the range of (9e-6, le-5] shows that VectorAdam consistently outperforms Adam. We observe that qualitatively Adam more often results tangled surface self-intersections which are difficult to recover from (red), whereas this VectorAdan result does not suffer from this problem.
rotation equivariant the per-coordinate infinity norm changes under rotation. Our VectorAdam can easily be modified in a similar way while preserving the property of rotation equivariance by taking the infinity norm over gradient vector norms in the second moment estimation.
In addition, we also extend our experiment to non- non-geon oon-eemetrric rector-valued data with the problem of texture reconstruction. We run the same texture reconstruction script, provided by [13], 200 times with different initializations. In Figure 7 we show VectorAdam consistently arrives at better texture result while achieving better convergence rate. This shows that our insight formulating optimizer by vectors can also benefit other non-geometric problems with vector-valued data. In future work, we interested in investigating the meaning of rotation mn/eequivvarian in color space.
6
Future Work
The above experiments focus mainly coordinate-valued geometric data, where the properties of VectorAdam are most easily ev- ident and well-founded in theory (Section 4.Il.. However, preliminary experiments also show promise for VectorAdam in the more general context of network weight optimization in geo- metric machine learning.
Test Loss
1.00
0.75
0.50
0.25
Test Accuracy
classic task in geometric machine learning is point cloud segmentation. We perform pre-
liminary experiment using the widely-adopted PointNet architecture [24], in which we train the network using both VectorAdam and con- ventional Adam on a subset ShapeNet dataset. More precisely, we apply VectorAdam's equivariant update the rows of PointNet ker- nel gradients. For example, consider the ker- weights in the first convolutional layer in the shape of x 3, which transforms the 3- dimensional point cloud coordinate data into n-dimensional latent vectors;
VectorAdam Adam
Figure 9: Average test loss and accuracy over 10 runs for point cloud segmentation using PointNet, trained with VectorAdam and Adam. The shaded region indicate the between plus and minus one standard deviation of test losses and accuracies over 10 runs.
VectorAdam preserves the direction of each row vector in the kernel. In Figure 99, plot the test accuracy and loss throughout the training process, and observed a significant increase stability using VectorAdam.
These network weights are not coordinate-valued but can be potentially interpreted geometrically as they map the point coordinates into latent spaces to perform certain geometry-aware tasks such as shape classification and segmentation. Therefore, this preliminary experiment points to the potential benefits of geometry-aware optimization in the broader machine learning context. However, this observed stability is not fully understood in theory, and we hope to investigate further in the future.

---

7 Conclusion
Our proposed VectorAdam alleviates the identified mordinate-system dependency flaw of Adam. We proved rotation equivariance of VectorAdam theoretically and demonstrated its implications for popular geometric optimization tasks in geometry processing and machine learning more broadly.
We suspect that networks which take positional coordinate values as input but whose degrees of freedom are layers of weight matrices (e.g.. 22) may implicitly suffer from the same lack of rotation equivariance. Unfortunately. our VectorAdam is only well understood for problems where the degrees of freedom are explicitly vector-valued quantities such as positional coordinates. Neural networks which take such coordinates as imput but whose degrees of freedom are layer weight matrices may also suffer from coordinate-depe deeeedddeee bias under the traditional Adam optimizer. Although our preliminary PointNet experiment shows an increase in test-time stability with VectorAdam, it is not yet clear how the rotational equivariant property of VectorAdam will directly apply these network weight optimization.
Additionally, the theoretical underpinnings for Adam have primarily been investigated in the stochastic optimization case; some of the direct geometric tasks considered here would benefit from more analysis understand why Adam and VectorAdam prove effective. In this work, we have focused on rotations and low-dimensional problems in geometry processing. In future work, we are further interested in investigating invariances and equivariances other transformation groups and high-dimensional vector or tensor spaces.
We hope that our attention to rotation equivariance for optimizers encourages further development of coordinate- geometric optimization within the evolving machinery continuous optimization and deep learning.
References
[1] Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104. 13478, 2021.
[2] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information -ichh 3d model repository. arXiv preprint arXiv:1512.030/1 2015.
[3] Chao Chen, Guanbin Li, Ruijia Xu, Tianshui Chen, Meng Wang, and Liang Lin. Clusternet: Deep hierarchical cluster network with rigorously cloud analysis. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4994-5002, 2019.
representation for point
[4] Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, and Leonidas Guibas. Vector Neurons: A General Framework for SO(3)-Equivarian Networks. arXiv:2104. 12229 [cs], April 2021. arXiv: 1222..
[5] Timothy Dozat. Incorporating nesterov momentum into adam. 2016.
[6] John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
[7] Carlos Esteves, Christine Allen Blanchheeee Ameesh Makadia, and Kostas Daniilidis. Learn- ing so (3) equivariant representations with spherical enns. In Proceedings of the European Conference on Computer Vision (ECCV), pages 52-68, 2018
[8] Roger Fletcher. Practical methods of optimization, 1987. John and Sons, Chichester, 1987.
[9] Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. Se (3)-transformers: 3d roto-translation equivariant attention networks. Advances in Neural Information Processing Systems, 33:1970-1988 2020.
[10] Magnus Hestenes and Eduard Stiefel. Methods of conjugate gradients for solving. Journal of research of the National Bureau of Standards, 49(6):409, 1952.
10

---

[11] Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. arX1v:1412.6980 [cs], January 2017. arXiv: 1412.6980.
[12] Risi Kondor, Zhen Lin, and Shubhendu Trivedi. Clebsch moaschhgordann nets: fully fourier space spherical convolutional neural network. Advances in Neural Information Processing Systems, 31, 2018.
[13] Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, and Timo Aila. Modular primitives for high-performance differentiable rendering. ACM Transactions on Graphics, 39(6):1-14, November 2020.
[14] Tzu-Mao Li, Miika Aittala, Frédo Durand, and Jaakko Lehtinen. Differentiable monte carlo ray tracing through edge sampling. ACM Transactions on Graphics (TOG), 37(6):1-11, 2018.
[15] Xianzhi Li, Ruihui Li, Guangyong Chen, Chi-Wing Fu, Daniel Cohen- Or, and Pheng-Am Ann Heng. A rotation-invarian framework for deep point cloud analysis. IEEE Transactions on Visualization and Computer Graphics, 2021.
[16] Dong C Liu and Jorge Nocedal. On the limited memory bfgs method for large scale optimization. Mathematical progrumming, 45(1):503-528, 1989.
[17] Liyuan Liu, Haoming Jiang, Pengeheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. arXiv preprint arXivv: /908. 03265, 2019.
[18] Yurii E Nesterov. method for solving the convex programming problem with convergence rate O (1 In Dokl. akad. nauk Sssr, volume 269, pages 543-547, 1983.
[19] Baptiste Nicolet, Alec Jacobson, and Wenzel Jakob. Large steps in inverse rendering of geometry. ACM Transactions on Graphics, 1016);1-13, December 2021.
[20] Merlin Nimier-David, Delio Vicini, Tizian Zelıer, and Wenzel Jakob. Mitsuba retargetable forward and inverse renderer. ACM Transactions on Graphics (TOG), 38(6):1-17, 2019.
[21] Jorge Nocedal and Stephen J Wright. Numerical optimization. Springer, 1999.
[22] Jeong Joon Park, Peter Florence, Julian Straub, Richard A. Newcombe, and Steven Lovegrove. Deepsdf:
Learning continuous signed distance functions for shape representation. In IEEE Conference Computer Vision and Puttern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 165-174. Computer Vision Foundation IEEE, 2019.
[23] Adrien Poulenard, Marie-Julie Rakotosaona, Yann Ponty, and Maks Ovsjanikov. Effective rotation-invariant point cnn with spherical harmonics kernels. In 2019 International Conference on 3D Vision (3DV), pages 47-56. IEEE, 2019.
[24] Charles Ruizhongtai Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 77-85.
IEEE Computer Society, 2017.
[25] Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pages 400-407, 1951.
[26] Jason Smith and Scott Schaefer. Bijective parameterization with free boundaries. ACM Transactions on Graphics (TOG), 34(4):1-9, 2015.
[27] Olga Sorkine and Marc Alexa. As-rigid-as-possible surface modeling. In Symposium Geometry processing, volume 4, pages 109-116. 2007.
[28] Shiliang Sun, Zehui Cao, Han Zhu, and Jing Zhao. survey of optimization methods from a machine learning perspective. IEEE transactions on cybernetics, 3068):3668-368) 2019.
[29] Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and vransaaton-equivvriantt neural networks for point clouds. arXiv preprint 18X2v:1802.008 2018.

---

[30] Tijmen Tieleman and G Hinton. Divide the gradient by running average of its recent magnitude. coursera neural netw. Mach. Learn, 6:26-31, 2012.
[31] Yu Wang and Justin Solomon. Fast quasi-harmont weights for geometric data interpolation. ACM Transactions Graphics (TOG), 40(4): 2021.
[32] Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco S Cohen. 3d steerable cnns: Learning rotationally equivariant features in volumetric data. Advances in Neural Information Processing Systems, 31, 2018.
[33] Li Yi, Vladimir G Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Sheffer, and Leonidas Guibas. A scalable active framework for region annotation in 3d shape collections. ACM Transactions on Graphics (ToG), 35(6):1-12, 2016.
[34] Matthew Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv: 2012.
[35] Zhiyuan Zhang, Binh-Son Hua, David W Rosen, and Sai- Kit Yeung. Rotation invariant convolutions for 3d point clouds deep learning. In 2019 International Conference on 3D Vision (3DV), pages 204-213. IEEE, 2019.
[36] Zijun Zhang. Improved Adam Optimizer for Deep Neural Networks. In 2018 IEEE/ACM 26th International Symposium on Quality of Service (IWQoS), pages 1-2, June 2018. ISSN: 1548-615X.
[37] Zijun Zhang, Lin Ma, Zongpeng Li, and Chuan Wu. Normalized direction- preeerinn adam. arXiv preprint 1XXx:1700.... 709604446, 2017.
12


Reviews:
[{"review_id": "zj2EH1o_5v8", "reviewer": "Reviewer_2Wxe", "summary": "The Adam optimization algorithm has proven effective on tasks in geometric optimization we often have vector-valued data). However, the updates we obtain by naively applying Adam does not preserve the original gradient direction (due to magnitude rescaling), and the Adam updates are coordinate dependent (hence not rotation equivariant). This paper proposes *VectorAdam* which preserves the gradient direction and is provably rotation equivariant. The authors also demonstrate empirically on a variety of geometric optimization tasks that, the proposed modification avoids artifacts and biases created by the ordinary Adam, with equivariant or even improved rate of convergence.", "questions": "1. Why Adam optimization is a good algorithm for geometric optimization? Is it significantly better than plain gradient descent (which of course preserves the original gradient)? If true, why that's the case? Do the authors have some intuition about this ?\n2. Neural implicit representation is an emerging area of research with lots of applications, especially inverse rendering. It has a neural network that takes in positional coordinates. Do you think the proposed VectorAdam can be useful?\n\n[1] Implicit Neural Representations with Periodic Activation Functions (https://arxiv.org/abs/2006.09661)\n[2] Representing Scenes as Neural Radiance Fields for View Synthesis (https://arxiv.org/abs/2003.08934)", "limitations": "I have no concern in terms of potential negative societal impact.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. The authors provide a good observation that, while Adam is effective in geometric optimization, it will create biases and artifacts, and the optimization results are coordinate dependent (which is not ideal in the setting of geometric optimization). They propose a simple yet effective modification that solves the issues abovementioned. \n2. All claims in this paper are backed by theoretical proof and sufficient empirical study. \n3. The paper is very well-written, with illustrations and simple examples for clarity. It is a pleasure to read this submission.\n4. Geometric optimization is a useful application that may appear in computational chemistry, physics, computer vision etc. This work provides a modification to the Adam optimization algorithm which has been previously applied to this problem successfully. I believe this work is very useful in related areas of research, and may inspire future research on the underexplored equivariant optimization algorithms.\n\n**Weaknesses**\n1. This work may become a stronger submission if they can provide more analysis regarding why the artifacts and biases will appear in addition to their visual examples. A mathematical quantification and analysis would be good because it may shed some light on future development beyond vector-valued quantities.\n2. The paper might benefit from having more content, such as an exploration of equivariance in colour space (as they mentioned), or explain more about geometric optimization to people who are not familiar with this special type of optimization problem. Currently the paper is slightly more than 8 pages and they have enough space.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "l4XUlIIoG8U", "reviewer": "Reviewer_oBCs", "summary": "The paper suggests a vector-modified version of the popular Adam optimization algorithm. The motivation stems from problems where the optimization variables are coordinates positions and the underlying optimized energy is rotation invariant, a typical setting in geometry processing. The main idea is to consider gradient update steps consisting of vectors (instead of scalars) and modify momentum normalization such that the results step is rotation equivariant.\nThe method is evaluated on various geometry processing tasks, showcasing its merits.\n", "questions": "Please address the weakness stated above. \n\nSome interesting questions I would have liked the paper to address are:\nAre there any limitations to constraining the optimization solution space? Are there settings where it fails? Does it require a more careful initialization design?\n\nIn addition, for not geometry processing experts, it would be beneficial to elaborate on the self-intersection issues and the red lines in figures 1 and 8.\n", "limitations": "The paper does not include such a discussion.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n-------------\nThe paper is very well written and easy to follow.\nThe background and motivation section does a great job of overviewing some of the issues of Adam in geometry optimization.\nThe proposed method is simple yet seems to be effective. It feels like a good starting point to encourage further research.\nThe method is evaluated on an adequate range of problems, showing improved qualitative results. \n\n\nWeaknesses\n----------------\n\nThe paper could better explain the role of equivariance in optimization settings versus the known settings used in modeling functions (networks). I was a bit confused that in contrast to equivariant networks where it is also valuable to constrain the prediction of transformed inputs (the prediction of a rotated point cloud is predictable), here, the inputs to the problem are not expected to be transformed. It is just that by using the equivariant constraint, the class of possible solutions is reduced (which is a strong and useful inductive bias to rotation invariant energies).\n\nMoreover, I think it would be better to define equivariant already at the beginning of the paper, including a statement that the paper assumes rotation invariant energies.\n  \nThe evaluation should include quantitative results. Does the VectorAdam local minimum is lower/higher than Adam solutions (although qualitatively better).\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "W6XFxQzkgZn", "reviewer": "Reviewer_KcD5", "summary": "The authors propose a variant of the gradient-based optimization algorithm Adam, which they denote VectorAdam, which is equivariant with respect to rotation of vector valued data. This is realised by changing the Adam gradient and second moment computations, which assume each parameter is a scalar quantity, with the appropriate quantities for vector valued parameters. \n\nThe authors show in some geometric optimization experiments that VectorAdam is indeed rotationally equivariant, while Adam is not. Further experiments are performed, with the goal of highlighting advantages of VectorAdam: for example, in the task of inverse rendering, they show that Adam gives rise to self-intersecting surfaces, while VectorAdam does not.\n", "questions": "A major improvement to VectorAdam that I can see make it a much more impactful paper would be to extend the algorithm for equivariant optimization of any representation of SO(n). Such representations are part of layers of recent equivariant neural networks (eg, e3nn). Thus, this would also apply to networks that take geometric coordinates as inputs as well.\n", "limitations": "The authors describe quite well the limitations of their approach.\n\nOverall, I think the work is worth of publication in a some more technical venue, but it does not achieve the level of novel and impact to the machine learning community to justify the publication in a venue like NeurIPS. I would encourage the authors to extend the scope of applicability of VectorAdam to, perhaps, TensorAdam, which could cover the full extend of equivariant networks.\n", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is clear and well written. The assumptions are clearly stated and the algorithm is clearly described. The paper addresses an interesting topic that deserves attention. Especially since the recent popularity of equivariant neural networks, which often are difficult to train, having geometry aware optimisers is likely important for the exploiting the full potential of such models.\n\nThe main weaknesses of the paper consists in the fact that it introduces just a simple variation with respect to the original Adam algorithm, where the gradients for a vector are computed instead of scalar parameters. The experiments substantially provide evidence of the claim of rotation equivariance of VectorAdam. Other than that, I’m not sure that such machine learning techniques are relevant enough in the machine learning community to justify the relevance of substantially just an incremental modification of the (very successful) Adam optimizer. In fact, as Figure 5 shows, it can be seen that in some of these optimization problems Adam finds the same minimum as VectorAdam. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HVnbQiFOlB2", "reviewer": "Reviewer_5HeJ", "summary": "In this work, the authors present a vector-based version of the Adam optimizer called VectorAdam. In contrast to Adam, VectorAdam operates on the vector form of the gradients instead of performing the scaling on the independent values within the gradient vector. This conceptual change primarily results in the update to the second-order moment no longer using the square of the individual values but using the norm of the vector instead. The authors show that with this change, VectorAdam will have rotation equivariant weight updates. \n\nThe authors evaluate their proposed optimizer experimentally for experiments; Laplacian Regularisation, Deformation, Adversarial Decent, and Inverse rendering. In all three approaches, VectorAdam shows a benefit over regular Adam.", "questions": "As stated above, the main question arising from the paper relates to the relevance of the proposed optimizer to the machine learning community. As it stands, it is unclear how weight-equivariance relates to output-invariance. This should be clarified in the rebuttal.", "limitations": "The paper is accompanied by a clear code base. Aside from the issues discussed above, there are no further limitations that should have been highlighted in the paper. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The presented paper is well-written, contains an extensive related work section and presents a number of interesting experimental evaluations.  \n\nHowever, based on the current version of the manuscript it is unclear to what extent the proposed VectorAdam optimizer is relevant to the NeurIPS community. Currently, only the adversarial descent experiment is related to machine learning. However, also here the setup is different from the main use case of Adam in machine learning—updating the weights of the model—and instead focuses on determining an adversarial update to the point could points. \n\nMore generally, the main weakness of the paper is that it is unclear how output-space invariance (as we are mostly interested in machine learning) translates to equivariance in the weight-space. Without this clarification, the paper might be more suitable for a  differential geometry-specific venue. \n\nIn addition to that, I would like to highlight some minor comments:\n- In both the definition of Adam and VectorAdam, the initial m_i should be m_{i-1} before the update.\n- In the algorithm, both the shape of the parameter vector and the first order moment uses $m$ as a variable. This is slightly confusing. To stay in line with previous work, I would suggest changing the notation for the shape of the parameter vector.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept