Venue: ICLR 2018 conference

Reviews:

final_score: rating: null
review_content: The authors present a neural embedding technique using a hyperbolic space.
The idea of embedding data into a space that is not Euclidean is not new.
There have been attempts to project onto (hyper)spheres.
Also, the proposal bears some resemblance with what is done in t-SNE, where an (exponential) distortion of distances is induced. Discussing this potential similarity would certainly broaden the readership of the paper.

The organisation of the paper might be improved, with a clearer red line and fewer digressions.
The call to the very small appendix via eq. 17 is an example.
The position of Table in the paper is odd as well.
The order of examples in Fig.5 differs from the order in the list.

The experiments are well illustrative but rather small sized.
The qualitative assessment is always interesting and it is completed with some label prediction task.
Due the geometrical consideretations developed in the paper, other quality criteria like e.g. how well neighbourhoods are preserved in the embeddings would give some more insights.

All in all the idea developed in the paper sounds interesting but the paper organisation seems a bit loose and additional aspects should be investigated. 
review_title: The paper somehow overlooks literature from dimensionality reduction

final_score: rating: 7: Good paper, accept
review_content: This paper proposes tree vertex embeddings over hyperbolic space. The conditional predictive distribution is the softmax of <v1, v2>_H = ||v1|| ||v2|| cos(theta1-theta2), and v1, v2 are points  defined via polar coordinates (r1,theta1), and (r2,theta2).
To evaluate, the authors show some qualitative embeddings of graph and 2-d projections, as well as F1 scores in identifying the biggest cluster associated with a class. 

The paper is well motivated, with an explanation of the technique as well as its applications in tree embedding in general. I also like the evaluations, and shows a clear benefit of this poincare embedding vs euclidean embedding.

However, graph embeddings are now a very well explored space, and this paper does not seem to mention or compare against other hyperbolic (or any noneuclidean) embedding techniques. From a 2 second google search, I found several sources with very similar sounding concepts:

Maximilian Nickel, Douwe Kiela, Poincaré Embeddings for Learning Hierarchical Representations

A Cvetkovski, M Crovella, Hyperbolic Embedding and Routing for Dynamic Graphs

Yuval Shavitt, Tomar Tankel, Hyperbolic Embedding of Internet Graph for Distance Estimation and Overlay Construction

Thomas Bläsius, Tobias Friedrich, Anton Krohmer, andSören Laue. Efficient Embedding of Scale-Free Graphs in the Hyperbolic Plane

I think this paper does have some novelty in applying it to the skip-gram model and using deep walk, but it should make more clear that using hyperbolic space embeddings for graphs is a popular and by now, intuitive construct. Along the same lines, the benefit of using the skip-gram and deep-walk techniques should be compared against some of the other graph embedding techniques out there, of which none are listed in the experiment section. 

Overall, a detailed comparison against 1 or 2 other hyperbolic graph embedding techniques would be sufficient for me to change my vote to accept. 



review_title: nice paper, good concepts, not enough experiments. 

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors present a neural embedding technique using a hyperbolic space.
The idea of embedding data into a space that is not Euclidean is not new.
There have been attempts to project onto (hyper)spheres.
Also, the proposal bears some resemblance with what is done in t-SNE, where an (exponential) distortion of distances is induced. Discussing this potential similarity would certainly broaden the readership of the paper.

The organisation of the paper might be improved, with a clearer red line and fewer digressions.
The call to the very small appendix via eq. 17 is an example.
The position of Table in the paper is odd as well.
The order of examples in Fig.5 differs from the order in the list.

The experiments are well illustrative but rather small sized.
The qualitative assessment is always interesting and it is completed with some label prediction task.
Due the geometrical consideretations developed in the paper, other quality criteria like e.g. how well neighbourhoods are preserved in the embeddings would give some more insights.

All in all the idea developed in the paper sounds interesting but the paper organisation seems a bit loose and additional aspects should be investigated. 
review_title: Nice idea but missing literature, poor paper organisation, and experiments that could be more complete

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors present a method to embed graphs in hyperbolic space, and show that this approach yields stronger attribute predictions on a set of graph datasets. I am concerned by the strong similarity between this work and Poincaré Embeddings for Learning Hierarchical Representations (https://arxiv.org/abs/1705.08039). The latter has been public since May of this year, which leads me to doubt the novelty of this work.

I also find the organization of the paper to be poor.
- There is a surprisingly high number of digressions.
- For some reason, Eq 17 is not included in the main paper. I would argue that this equation is one of the most important equations in the paper, given that it is the one you are optimizing.
- The font size in the main result figure is so small that one cannot hope to parse what the plots are illustrating.
- I am not sure what insights the readers are supposed to gain from the visual comparisons between the Euclidean and Poincare embeddings. 

Due to the poor presentation, I actually have difficulty making sense of the evaluation in this paper (it would help if the text was legible). I think this paper requires significant work and it not suitable for publication in its current state.

As a kind of unrelated note. It occurs to me that papers on hyperbolic embeddings tend to evaluate evaluate on attribute or link prediction. It would be great if authors would also evaluate these pretrained embeddings on downstream applications such as relation extraction, knowledge base population etc.
review_title: Concerns about novelty

final_score: rating: 4: Ok but not good enough - rejection
review_content: == Preamble ==

As promised, I have read the updated paper from scratch and this is my revised review. My original review is kept below for reference. My original review had rating "4: Ok but not good enough - rejection".

== Updated review ==

The revised improves upon the original submission in several ways and, in particular, does a much better job at positioning itself within the existing body of literature. The new experiments also indicate that the proposed model offer some improvement over Nickel & Douwe, NIPS 2017).

I do have remaining concerns that unfortunately still prevent me from recommending acceptance:

- Throughout the paper it is argued that we should embed into a hyperbolic space. Such a space is characterized by its metric, but the proposed model do not use a hyperbolic metric. Rather it relies on a heuristic similarity measure that is inspired by the hyperbolic metric. I understand that this may be a practical choice, but then I find it misleading that the paper repeatedly states that points are embedded into a hyperbolic space (which is incorrect). This concern was also raised on this forum prior to the revision.

- The resulting optimization is one of the key selling points of the proposed method as it is unconstrained while Nickel & Douwe resort to a constrained optimization. Clearly unconstrained optimization is to be preferred. However, it is not entirely correct (from what I understand), that the resulting optimization is indeed unconstrained. Nickel & Douwe work under the constraint that |x| < 1, while the proposed model use polar coordinates (r, theta): r in (0, infinity) and theta in (0, 2 pi]. Note that theta parametrize a circle, and therefore wrapping may occur (this should really be mentioned in the paper). The constraints on theta are quite easy to cope with, so I agree with the authors that they have a more simple optimization problem. However, this is only true since points are embedded on the unit disk (2D). Should you want to embed into higher dimensional spaces, then theta need to be confined to live on the unit sphere, i.e. |theta| = 1 (the current setting is just a special-case of the unit sphere). While optimizing over the unit sphere is manageable it is most definitely a constrained optimization problem, and it is far from clear that it is much easier than working under the Nickel & Douwe constraint, |x| < 1.

Other comments:
- The sentence "even infinite trees have nearly isometric embeddings in hyperbolic space (Gromov, 2007)" sounds cool (I mean, we all want to cite Gromov), but what does it really mean? An isometric embedding is merely one that preserves a metric, so this statement only makes sense if the space of infinite trees had a single meaningful metric in the first place (it doesn't; that's a design choice).

- In the "Contribution" and "Conclusion" sections it is claimed that the paper "introduce the new concept of neural embeddings in hyperbolic space". I thought that was what Nickel & Douwe did... I understand that the authors are frustrated by this parallel work, but at this stage, I don't think the present paper can make this "introducing" claim.

- The caption in Figure 2 miss some indication that "a" and "b" refer to subfigures. I recommend "a" --> "a)" and "b" --> "b)".

- On page 4 it is mentioned that under the heuristic similarity measure some properties of hyperbolic spaces are lost while other are retained. From what I can read, it is only claimed that key properties are kept; a more formal argument (even if trivial) would have been helpful.


== Original Review ==

The paper considers embeddings of graph-structured data onto the hyperbolic Poincare ball. Focus is on word2vec style models but with hyperbolic embeddings. I am unable to determine how suitable an embedding space the Poincare ball really is, since I am not familiar enough with the type of data studied in the paper. I have a few minor comments/questions to the work, but my main concern is a seeming lack of novelty:
The paper argues that the main contribution is that this is the first neural embedding onto a hyperbolic space. From what I can see, the paper

  Poincaré Embeddings for Learning Hierarchical Representations
  https://arxiv.org/abs/1705.08039

consider an almost identical model to the one proposed here with an almost identical motivation and application set. Some technicalities appear different, but (to me) it seems like the main claimed novelties of the present paper has already been out for a while. If this analysis is incorrect, then I encourage the authors to provide very explicit arguments for this in the rebuttal phase.

Other comments:
*) It seems to me that, by construction, most data will be pushed towards the boundary of the Poincare ball during the embedding. Is that a property you want?
*) I found it rather surprising that the log-likelihood under consideration was pushed to an appendix of the paper, while its various derivatives are part of the main text. Given the not-so-tight page limits of ICLR, I'd recommend to provide the log-likelihood as part of the main text (it's rather difficult to evaluate the correctness of a derivative when its base function is not stated).
*) In the introduction must energy is used on the importance of large data sets, but it appears that only fairly small-scale experiments are considered. I'd recommend a better synchronization.
*) I find visual comparisons difficult on the Poincare ball as I am so trained at assuming Euclidean distances when making visual comparisons (I suspect most readers are as well). I think one needs to be very careful when making visual comparisons under non-trivial metrics.
*) In the final experiment, a logistic regressor is fitted post hoc to the embedded points. Why not directly optimize a hyperbolic classifier?

Pros:
+ well-written and (fairly) well-motivated.

Cons:
- It appears that novelty is very limited as highly similar work (see above) has been out for a while.


review_title: Done before?

Meta-review: This paper does not meet the acceptance bar this year, and thus I must recommend it for rejection.
Decision: Reject