Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this work, the authors present a vector-based version of the Adam optimizer called VectorAdam. In contrast to Adam, VectorAdam operates on the vector form of the gradients instead of performing the scaling on the independent values within the gradient vector. This conceptual change primarily results in the update to the second-order moment no longer using the square of the individual values but using the norm of the vector instead. The authors show that with this change, VectorAdam will have rotation equivariant weight updates. 

The authors evaluate their proposed optimizer experimentally for experiments; Laplacian Regularisation, Deformation, Adversarial Decent, and Inverse rendering. In all three approaches, VectorAdam shows a benefit over regular Adam.
strengths_and_weaknesses: The presented paper is well-written, contains an extensive related work section and presents a number of interesting experimental evaluations.  

However, based on the current version of the manuscript it is unclear to what extent the proposed VectorAdam optimizer is relevant to the NeurIPS community. Currently, only the adversarial descent experiment is related to machine learning. However, also here the setup is different from the main use case of Adam in machine learning—updating the weights of the model—and instead focuses on determining an adversarial update to the point could points. 

More generally, the main weakness of the paper is that it is unclear how output-space invariance (as we are mostly interested in machine learning) translates to equivariance in the weight-space. Without this clarification, the paper might be more suitable for a  differential geometry-specific venue. 

In addition to that, I would like to highlight some minor comments:
- In both the definition of Adam and VectorAdam, the initial m_i should be m_{i-1} before the update.
- In the algorithm, both the shape of the parameter vector and the first order moment uses $m$ as a variable. This is slightly confusing. To stay in line with previous work, I would suggest changing the notation for the shape of the parameter vector.
questions: As stated above, the main question arising from the paper relates to the relevance of the proposed optimizer to the machine learning community. As it stands, it is unclear how weight-equivariance relates to output-invariance. This should be clarified in the rebuttal.
limitations: The paper is accompanied by a clear code base. Aside from the issues discussed above, there are no further limitations that should have been highlighted in the paper.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The Adam optimization algorithm has proven effective on tasks in geometric optimization we often have vector-valued data). However, the updates we obtain by naively applying Adam does not preserve the original gradient direction (due to magnitude rescaling), and the Adam updates are coordinate dependent (hence not rotation equivariant). This paper proposes *VectorAdam* which preserves the gradient direction and is provably rotation equivariant. The authors also demonstrate empirically on a variety of geometric optimization tasks that, the proposed modification avoids artifacts and biases created by the ordinary Adam, with equivariant or even improved rate of convergence.
strengths_and_weaknesses: **Strengths**
1. The authors provide a good observation that, while Adam is effective in geometric optimization, it will create biases and artifacts, and the optimization results are coordinate dependent (which is not ideal in the setting of geometric optimization). They propose a simple yet effective modification that solves the issues abovementioned. 
2. All claims in this paper are backed by theoretical proof and sufficient empirical study. 
3. The paper is very well-written, with illustrations and simple examples for clarity. It is a pleasure to read this submission.
4. Geometric optimization is a useful application that may appear in computational chemistry, physics, computer vision etc. This work provides a modification to the Adam optimization algorithm which has been previously applied to this problem successfully. I believe this work is very useful in related areas of research, and may inspire future research on the underexplored equivariant optimization algorithms.

**Weaknesses**
1. This work may become a stronger submission if they can provide more analysis regarding why the artifacts and biases will appear in addition to their visual examples. A mathematical quantification and analysis would be good because it may shed some light on future development beyond vector-valued quantities.
2. The paper might benefit from having more content, such as an exploration of equivariance in colour space (as they mentioned), or explain more about geometric optimization to people who are not familiar with this special type of optimization problem. Currently the paper is slightly more than 8 pages and they have enough space.

questions: 1. Why Adam optimization is a good algorithm for geometric optimization? Is it significantly better than plain gradient descent (which of course preserves the original gradient)? If true, why that's the case? Do the authors have some intuition about this ?
2. Neural implicit representation is an emerging area of research with lots of applications, especially inverse rendering. It has a neural network that takes in positional coordinates. Do you think the proposed VectorAdam can be useful?

[1] Implicit Neural Representations with Periodic Activation Functions (https://arxiv.org/abs/2006.09661)
[2] Representing Scenes as Neural Radiance Fields for View Synthesis (https://arxiv.org/abs/2003.08934)
limitations: I have no concern in terms of potential negative societal impact.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper suggests a vector-modified version of the popular Adam optimization algorithm. The motivation stems from problems where the optimization variables are coordinates positions and the underlying optimized energy is rotation invariant, a typical setting in geometry processing. The main idea is to consider gradient update steps consisting of vectors (instead of scalars) and modify momentum normalization such that the results step is rotation equivariant.
The method is evaluated on various geometry processing tasks, showcasing its merits.

strengths_and_weaknesses: Strengths
-------------
The paper is very well written and easy to follow.
The background and motivation section does a great job of overviewing some of the issues of Adam in geometry optimization.
The proposed method is simple yet seems to be effective. It feels like a good starting point to encourage further research.
The method is evaluated on an adequate range of problems, showing improved qualitative results. 


Weaknesses
----------------

The paper could better explain the role of equivariance in optimization settings versus the known settings used in modeling functions (networks). I was a bit confused that in contrast to equivariant networks where it is also valuable to constrain the prediction of transformed inputs (the prediction of a rotated point cloud is predictable), here, the inputs to the problem are not expected to be transformed. It is just that by using the equivariant constraint, the class of possible solutions is reduced (which is a strong and useful inductive bias to rotation invariant energies).

Moreover, I think it would be better to define equivariant already at the beginning of the paper, including a statement that the paper assumes rotation invariant energies.
  
The evaluation should include quantitative results. Does the VectorAdam local minimum is lower/higher than Adam solutions (although qualitatively better).


questions: Please address the weakness stated above. 

Some interesting questions I would have liked the paper to address are:
Are there any limitations to constraining the optimization solution space? Are there settings where it fails? Does it require a more careful initialization design?

In addition, for not geometry processing experts, it would be beneficial to elaborate on the self-intersection issues and the red lines in figures 1 and 8.

limitations: The paper does not include such a discussion.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The authors propose a variant of the gradient-based optimization algorithm Adam, which they denote VectorAdam, which is equivariant with respect to rotation of vector valued data. This is realised by changing the Adam gradient and second moment computations, which assume each parameter is a scalar quantity, with the appropriate quantities for vector valued parameters. 

The authors show in some geometric optimization experiments that VectorAdam is indeed rotationally equivariant, while Adam is not. Further experiments are performed, with the goal of highlighting advantages of VectorAdam: for example, in the task of inverse rendering, they show that Adam gives rise to self-intersecting surfaces, while VectorAdam does not.

strengths_and_weaknesses: The paper is clear and well written. The assumptions are clearly stated and the algorithm is clearly described. The paper addresses an interesting topic that deserves attention. Especially since the recent popularity of equivariant neural networks, which often are difficult to train, having geometry aware optimisers is likely important for the exploiting the full potential of such models.

The main weaknesses of the paper consists in the fact that it introduces just a simple variation with respect to the original Adam algorithm, where the gradients for a vector are computed instead of scalar parameters. The experiments substantially provide evidence of the claim of rotation equivariance of VectorAdam. Other than that, I’m not sure that such machine learning techniques are relevant enough in the machine learning community to justify the relevance of substantially just an incremental modification of the (very successful) Adam optimizer. In fact, as Figure 5 shows, it can be seen that in some of these optimization problems Adam finds the same minimum as VectorAdam. 

questions: A major improvement to VectorAdam that I can see make it a much more impactful paper would be to extend the algorithm for equivariant optimization of any representation of SO(n). Such representations are part of layers of recent equivariant neural networks (eg, e3nn). Thus, this would also apply to networks that take geometric coordinates as inputs as well.

limitations: The authors describe quite well the limitations of their approach.

Overall, I think the work is worth of publication in a some more technical venue, but it does not achieve the level of novel and impact to the machine learning community to justify the publication in a venue like NeurIPS. I would encourage the authors to extend the scope of applicability of VectorAdam to, perhaps, TensorAdam, which could cover the full extend of equivariant networks.
review_title: null

Meta-review: metareview: Ratings: 5/8/4/7.
Confidence: 4/3/4/4.
Discussion among reviewers: Yes.

Summary: this paper introduces a variant of Adam where instead of keeping EMAs of the squared individual gradients, the algorithm keeps an EMAs of the squared L2 norm of vectors of gradients. This EMA of the squared L2 norm is then used as normalizer, making sure that the L2 norm of weight updates are normalized. This has as advantage that the algorithm becomes equivariant to rotations in parameter space, which is crucial for certain types of problems. The reviewers noted the clear presentation and the encouraging results.

There's one reviewer with a reject rating, whose main reason for rejection is that the proposed algorithm itself is simple, in the sense that it's only a small change from baseline Adam. Other reviewers disagree that this is a reason for rejection, and I agree with the other reviewers here. Simplicity isn't bad if the method is novel, the motivation is clear, the results are good, and the empirical results are strong.

Recommendation: I recommend to accept this paper.
confidence: Certain
Decision: Accept