Title: Riemannian Adaptive Optimization Methods
Authors: Gary Becigneul
Octavian-Eugen Ganea
Venue: ICLR 2019
Keywords: Riemannian optimization
adaptive
hyperbolic
curvature
manifold
adam
amsgrad
adagrad
rsgd
convergence

Abstract: Several first order stochastic optimization methods commonly used in the Euclidean domain such as stochastic gradient descent (SGD), accelerated gradient descent or variance reduced methods have already been adapted to certain Riemannian settings. However, some of the most popular of these optimization tools - namely Adam, Adagrad and the more recent Amsgrad - remain to be generalized to Riemannian manifolds. We discuss the difficulty of generalizing such adaptive schemes to the most agnostic Riemannian setting, and then provide algorithms and convergence proofs for geodesically convex objectives in the particular case of a product of Riemannian manifolds, in which adaptivity is implemented across manifolds in the cartesian product. Our generalization is tight in the sense that choosing the Euclidean space as Riemannian manifold yields the same algorithms and regret bounds as those that were already known for the standard algorithms. Experimentally, we show faster convergence and to a lower train loss value for Riemannian adaptive methods over their corresponding baselines on the realistic task of embedding the WordNet taxonomy in the Poincare ball.

TL;DR: Adapting Adam, Amsgrad, Adagrad to Riemannian manifolds. 

Review discussion:
This paper presents Riemannian versions of adaptive optimization methods, including ADAGRAD, ADAM, AMSGRAD and ADAMNC. There are no natural coordinates on a manifold.
Therefore, the authors resort to product of manifolds and view each manifold component as a coordinate. Convergence analyses for those methods are given. The the theoretical results and their Euclidean versions coincide. An experiment of embedding a tree-like graph into a Poincare model is used to show the performance of the Riemannian versions of the four methods.
This paper is well-written except a few flaws (see below). I do not have time to read the proofs carefully. The proposed methods are potentially important in some applications.
Therefore, I suggest publish this paper after addressing the comments below.
Remarks:
*) P1, line 2: it particular -> in particular.
*) P3, line 9: Is R_x(v) = x + v most often chosen? A manifold is generally nonlinear. A simple addition would not give a point in the manifold.
*) P5, in Assumptions and notations paragraph: what are T and [T]? Is T the number of total iterations or the number of functions in the function family. The subscript of the function f_t seems to be an index of the functions.
But its notation is also related to the number of iterations, see (8) and the algorithms in Figure 1.
*) P5, Figure 1: does a loop for the index $i$ missing?
*) Section 5: it would be clearer if the objective function is written as L:(D^n)^m \to R: \theta-> , where m is the number of nodes.
Otherwise, it is not obvious to see the domain.
*) P7, last paragraph: Tables 2 and 3 -> Figures 2 and 3.
*) Besides the application in the experiments, it would be nice if more applications, at least references, are added.
Thank you for your detailed feedback. We have updated our paper according to your suggestions.
We reply below to each of your remarks, more specifically.
*) Typo corrected.
*) Retraction: indeed, choosing the retraction R_x(v)=x+v requires having immersed the manifold into an ambient Euclidean space: note that we only say that the retraction is “most often chosen as” such, not that this choice is always a valid one. We mention it here because it is the one we used in our experiments.
*) T is the number of iterations, [T] denotes the set of integers from 1 to T. We use same notations as in [1]: Each f_t is the objective function of the parameters to be optimized, evaluated at the batch taken at time t.
For instance, when training a neural network, one could alternatively write f_t(x) = \sum_{y\in S_t} L(x,y), where x is the set of parameters of the model, L is the loss, each y is an input to the network, and S_t the (mini)batch taken at time t.
*) yes, these are coordinate-wise operations. We did not write explicitly the loop over i to not influence the reader into implementing this algorithm with a loop over i. In most languages, such as python or C++, coordinate-wise operations such as adding vectors are highly optimized in the standard library. One could rewrite the algorithm without the “i”, with coordinate wise operations on vectors.
*) We added a footnote clarifying the domain of the loss function in Section 5.
*) Tables 2 & 3 -> figures 2 & 3: Thank you, we corrected this typo.
*) Other potential applications include any optimization-based graph or word embedding method on a manifold. Note that the product-structure assumption is natural, since if one needs to embed n nodes into a manifold, the parameter space is a product of n manifolds.
Following your suggestions, we have added a few references [2,3,4] as suggestions for further experiments, at the beginning of the experiment section.
[1] On the convergence of Adam and beyond, Reddi et al., ICLR 2018
[2] Representation trade-offs for hyperbolic embeddings, De Sa et al., ICML 2018
[3] Hyperbolic entailment cones for learning hierarchical embeddings, Ganea et al., ICML 2018
[4] Learning continuous hierarchies in the Lorentz model of hyperbolic geometry, Nickel & Kiela, ICML 2018
I have enjoyed reading this paper. The paper is accessible in most cases and provides a novel optimization technique. Having said this, I have a few concerns here,
- I am not sure why the notion of product manifolds is required in developing the technique. To me, all the arguments follow without that.
Even if the authors are only interested in manifolds that can be constructed in a product manner (say R^n from R),  the development can be done without explicitly going along that path.
Nevertheless I may have missed something so please elaborate why product manifolds. I have to add that in many cases, the underlying Riemannian geometry cannot be derived as a product  space.
For example, the SPD manifold cannot be constructed as a product space of lower dimensional geometries.
- I have a feeling that finding the operator \Pi in many interesting cases is not easy. Given the dependency of the developments on this operator, I am wondering if the method can be used to address problems on other manifolds such as SPD, Grassmannian or Stiefel. Please provide the form of this operator for the aforementioned manifolds and comment on how the method can be used if such an operator is not at our disposal.
- While I appreciate the experiments done in the paper,  common tests (e.g., Frechet means) are not presented in the paper (see my comment below as well).
- last but not least, the authors missed the work of   Roy et. al., "Geometry Aware Constrained Optimization Techniques for Deep Learning", CVPR'18 where RSGC with momentum and Riemannian version of RMSProp are developed. This reference should be considered and compared.
Aside from the above, please
- define v and \hat{v} for Eq.(5)
- provide a reference for the claim at l3-p4 (claim about the gradient and Hessian)
- maybe you want to mention that \alpha -> 0 for |g_t^i| at the bottom of p4
- what does [.] mean in the last step of the algorithm presented in p7
- what is the dimensionality of the Hn in the experiments
Thank you for your interest and professionalism. We reply below to each of your concerns.
Product structure:
(i) The product structure is natural for any optimization-based graph or word embedding method: if one wants to embed n nodes into a manifold M, then the parameter space is M^n.
In particular, this would apply also if M is a PSD manifold.
(ii) We noticed recently that other very recent approaches propose to also embed each point into a product of spaces, arguing that it allows the embeddings to benefit from the metric properties of each space [1,2].
(iii) Our proof arguments would not hold without this product structure. This is easier to see from the convergence proof of Euclidean AMSgrad [3], appendix D, Eq.(18), where the last equality exploits the Euclidean coordinate system to expand the squared norms. This is not possible on a general Riemannian manifold.
However, with a product structure, one can expand squared distances in the product manifold, as the sum of squared distances in each manifold of the product.
Pi operator:
-The presence of the projection operator is mostly useful for the convergence proof, to guarantee that the learning trajectory in parameter space is bounded, hence the presence of D_\infty in the bounds.
-Note that this operator is also required to obtain theoretical bounds for Euclidean AMSgrad/Adam, even though it is often omitted in practice.
-Note that in [4, section 3], it is assumed to be given, as a “projection oracle”.
Also note that for many applications of interest, such as computing Karcher means on PSD manifolds (as done in [4, section 4]), a projection operator is not used nor needed for convergence, since the trajectory is trivially bounded (formally, this amounts to choosing a trivial projection into a ball containing the trajectory).
-In the Poincaré ball, if X is a ball centered at the origin (as in our experiments), then the projection is naturally given by the parametrization in the Euclidean ambient space.
Fréchet means:
-This is an interesting suggestion that we will keep in mind for future work.
Related work:
-Thank you for pointing us to this relevant reference. We have added it to the related work section.
Other remarks:
In Eq.(5), \hat{v} is defined from v, which is defined as for Adam, defined just above. We have added a footnote explaining this.
Added a reference for the claim line 3, page 4, about gradient and Hessian.
At the bottom of page 4, \alpha is not required to go to 0: we consider here an (R)SGD update of fixed size \alpha.
If by [.] you refer to gyr[. , .], this square bracket comes from the notation of the gyro-operator, for which we provide a reference. We use it in our experiments to efficiently compute parallel transport in the Poincaré ball.
The dimension we use in our experiments is 5, as suggested in [5]. Added to experiments section.
[1] Learning mixed curvature representations in product spaces,
https://openreview.net/forum?id=HJxeWnCcF7
[2] Poincaré Glove: hyperbolic word embeddings,
https://openreview.net/forum?id=Ske5r3AqK7
[3] On the convergence of Adam and beyond, Reddi et al., ICLR 2018
https://openreview.net/forum?id=ryQu7f-RZ
[4] First order methods for geodesically convex optimization, Zhang & Sra, JMLR 2016
proceedings.mlr.press/v49/zhang16b.pdf
[5] Poincaré embeddings for learning hierarchical representations, Nickel & Kiela, NIPS 2017
https://arxiv.org/abs/1705.08039
The paper extends Euclidean optimization methods, Adam/Amsgrad, to the Riemannian setting, and provides theoretical convergence analysis which includes the Euclidean versions as a special case. To avoid breaking the sparsity, coordinate-wise updates are performed on product manifolds.
The empirical performance seems not very good, compared to RSGD which is easier to use.
Thank you for reviewing our work.
Even though RSGD is indeed slightly easier to use, we will make our code available to facilitate the use of our algorithms.
We would like to thank all three reviewers for their work and their interest in our work.
We have incorporated the modifications suggested by reviewers and are open to further comments.
Please find more detailed responses below each review.

Meta-review:
role: AC
data: metareview: Dear authors,

All reviewers agreed that your work sheds new light on a popular class of algorithms and should thus be presented at ICLR.

Please make sure to implement all their comments in the final version.

Decision: Accept (Poster)