Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper develops a MLR based on hyperbolic geometry. The idea is based on well-known concept of horocycle and horospheres which are known to be hyperbolic counterpart of line and plane in Euclidean geometry (see Coxter). Then the authors show the universal approximation which kind of follows similarly from the Euclidean counterpart. In fact we can probably conject that this universal approximation holds for any manifolds with constant sectional curvature.

Strength: To the best of my knowledge, this is the first paper to deal with linear models on hyperbolic spaces by borrowing geometric tools like horocycles.



Major weakness:

The ideas are borrowed from well-known geometric tools, although this is not a weakness but the theorems closely follow Euclidean counterpart. This essentially reduces the ``````"novelty" of the paper. Moreover, the experiments are ``"synthetic", there is no motivation to use such a construction in real experiment. It will be good to see the authors discuss in which real cases we need to use such a hyperbolic MLR.

1) The work should be better motivated, for example what is the motivation of using Horocycle layer and Poisson neuron layer?
2) In section 6.2, the 2D output after 4 convolutional layers seems very less expressive, why not increase the dimension? Also what is the motivation to map it to H^2?
3) In Theorem 2, eq. 9, why the inner product is Euclidean instead of hyperbolic? 
4) The universal approximation theorem in Theorem 2 almost follows from the Euclidean counterpart, e.g., see https://cbmm.mit.edu/sites/default/files/publications/CBMM-Memo-054.pdf
5) What is the additional consequence of Corollary 1 other than showing we can approximate any function, similar as Theorem 2?
6) The statement in section 6.3 stating "t is the best hyperbolic geometry related MNIST classifier" does not carry much weight, e.g., what is the motivation of using MNIST images for MLR using hyperbolic geometry? 
7) There is not much point for section 6.4. In most practical cases, the 1-dimensional reduction is not meaningful as it can not carry much information.
8) Section 6.5 seems very rushed including the Fig. 8 and experiment of Flowers. This section seems more like placeholder.
review_title: Nice idea taken from well-known geometric tools but falls short with experiments

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:

This paper proposes new neural models for hyperbolic space, which unlike previous hyperbolic NN works, relies on the notion of horocycle in the Poincare disk. This novel framework has connections to spectral learnig in hyperbolic space. Representation theorems alla Cybenko for layers constructed from these neurons are presented. Finally, various experiments on clustering and classifying datasets using these neurons to generate hyperbolic embeddings are presented. 

With the caveat that this paper is outside my main area of expertise, I must say that I have mixed feelings about it. On the one hand, I want to like it - the topic is quite interesting and timely, the theoretical connections are intriguing, the representation results seem quite remarkable, and the experiments seem to suggest (modulo some questions I have, see below) that this is a promising approach. On the other hand, the writing, dry exposition, utter lack of discussion or intuition for most results, and the confusing setup of the experiments make it hard to produce a confident assessment. In addition, these drawbacks probably imply that the paper might not be accessible but to a few niche in the community, and might have a very limited impact. 

For the reasons above, I'm leaning towards rejection, but I think that this could be a very solid paper if: (i) the results hold, (ii) the writing and exposition is improved, and (iii) the results are better discussed and motivated .

Strengths:
* Interesting problem in a flourishing but not-yet-too-crowded corner of the representation learning literature
* Seemingly very strong theoretical results (representation theorems for neural nets in hyperbolic space)
* Seemingly very convincing experimental results, outperforming alternative methods by wide margins

Weaknesses:
* The paper needs thorough rewriting. There's various typos, confusing grammatical choices, and overall, confusing writing. 
* Besides grammar, etc, the paper needs to be written with an ICLR audience in mind, most of which might not be experts in hyperbolic geometry, so more hand holding is needed. 
* The paper needs restructuring. Too much space is devoted to listing prior results without further explanation or discussion (e.g. Theorem 1 - what's the importance, implication of this result?). In turn, the contribution of this paper, mostly contained in Section 4.2, could benefit from more detailed discussion and motivation. In particular, I find sentences like "Suppose this Poisson neuron is non-trainable ... " very confusing. I have no idea what this whole sentence is trying to convey.
* The results in Section 5 need more discussion. Theorem 2 at least is reminiscent of other representation theorems in the NN literature, but what is reader supposed to take away from Lemma 1 and Corollary 1? Instead of provding a full proof of Theorem 2, I would suggest deferring that to the appendix, and using the additional space to discuss the importance of all these results.
* The experiments seem quite impressive, but then again, I'm not sure whether I can gauge their soundness with confidence. There are many details about the experimental setting that are either missing or not well explained. For example:
    * Are the G/S/H models in Table 1 all directly comparable? Do they have a similar number of parameters? Similar training?
    * The reported advantage of H over G/S seems to be mostly prominent in low dimensions of the Poincare ball. I would like to see a discussion on why this is the case.
    * Given how much variance the results in Table 1 seem to have, standard deviation or error bars should be reported along with the means
    * It is not clear what exactly is meant by test error on a clustering task in Section 6.2. How are the train / test samples used? 
    * Many experimental/design choices are not well justified - e.g., why is the input layer scaled down with PCA in 6.3?

Other issues:
* The notion of end prototypes seems quite interesting, but I feel like it could be better explained / elaborated on, e.g., at the end of Section 4.
* In Theorem 2, it isn't clear where K is coming into play in the definition of F (given that the density argument is on $L_p(K,\mu)$, I suppose F is only defined for $x in K$?). 
review_title: Obscure, intriguing, confusing, and ultimately not ready for publication

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper introduced a new hyperbolic neuron based on horocycles (hyperbolic counterparts of hyperplanes). The authors proved that these neurons in H^n are as useful as traditional neurons in R^n through theoretical arguments and demonstrated they can significantly improve learning in hyperbolic embeddings of tree datasets and MNIST/CIFAR datasets.

Quality:
This contribution has both theoretical and practical strengths. Theoretically, they proved that the proposed hyperbolic neurons are universal approximators (Theorem 2). Practically, they introduced a new kind of hyperbolic neuron, with its difference with existing literature clearly demonstrated through formulations and density plots. It shows supervisor performance improvements in several examples.

Clarity:
The language is well polished. The formulations and statements are clear and consistent. The presentation has high clarity with good intuitions through illustrations.

Originality:
The proposed method is mostly related to hyperbolic neural networks constructed using Mobius arithmetic operators. Their difference is demonstrated both intuitively and empirically through experiments. The relationship with previous works is clearly stated in section 2. The references are proper with page numbers mentioned.

Significance:
This paper establishes a new connection between hyperbolic geometry and deep learning. Therefore it should be interesting to the large group of audiences in those areas.

My main concern and questions are listed as follows:

Most importantly, the introduction and the theorem are based on equation (2); while the experiments are based on the Poisson layer introduced in section 4.2. I can see some inconsistency here: clearly they are different functions. Please fill this gap in the rebuttal and next version.

Are there any explanations and technical arguments of the good empirical performance?

Clearly, the hyper-parameter epsilon is important to maintain numerical stability. There should be some demonstrations in the main text or the  supplementary material to show the robustness to instability (e.g. by setting epsilon=0)

Finally, I summarize the pros and cons as follows.

Pro:
- new hyperbolic deep learning
- proof of representation power on H^n
- strong empirical results

Con:
- missing connection between equation(2) and Poisson layer

Overall, based on the above assessment measures I recommend strong acceptance.

Here are more comments for the authors' revision:

abstract: "MLR" is the abbrev of?

Introduction: introduce notation T_p(H)

After theorem 1 there have to be some remarks to explain the statement.

Same for Lemma/Corollary 1.

Theorem 2 is referred to before the statement.

The volume element "dm" is a bit hard to read

Where are the notation h_x(y) used?

Figure 8 x-axis and y-axis are not clear

-----
After rebuttal:

Thank you for the revision and the clarifications.

It is now clear that this work actually proposed two different neurons: the horocycle neuron defined on H^n and the Position neuron defined on R^n (removing one point). After the revision, they are proved to satisfy the universal approximation property. They share similar level sets (although the density of the level sets is different). It would be interesting to see their relationships through formal arguments and a more careful empirical comparison.

This work needs background knowledge in hyperbolic geometry and may not be easy to read at the beginning. That could explain the criticism regarding clarity. Overall, I believe this paper developed important tools along the line of hyperbolic deep learning and still recommends strong acceptance.
review_title: A new connection between hyperbolic geometry and deep learning

final_score: rating: 4: Ok but not good enough - rejection
review_content: **Review summary**

The proposed models are theoretically sound, as it is as expressive as satisfying the universality theorem. Also, the proposed methods are empirically superior to existing methods. However, I think there is much room for improvement in the presentation of the paper. Besides, it is unclear to me what the research question is and how the proposed methods solve the problem. I would recommend reconsidering the organization of the paper. 

**Summary of the paper**

This paper proposed a horocycle neuron that acts on the hyperbolic space. It uses the Hyperbolic Poisson kernel in place of the standard Euclid norm on the Euclidean space. This paper proposed an architecture called horocycle MLR, which used a horocycle neuron as a building block, and Poisson neural MLR. They showed the universality of a model with a single hidden layer of horocycle neurons or $f^1_{a, p}$, which has been used in the existing literature. They applied the horocycle feature to a subtree classification task of Poincare embedding, a horocycle MLR to a clustering task of 2D embedding, and horocycle and Poisson MLRs to classification tasks on image datasets.

**Claim**

If I understand correctly, this paper claims that the horocycle and Possison neurons are theoretically sound and empirically effective. However, it is not clear to me the research question that this paper addressed and how the theoretical and empirical properties of the proposed methods answer the question. It is true that they discussed the heuristic connection between the universal approximation property and the integral representation of the form (8) of a horocycle neuron. However, I think it is not a research question but supporting evidence that the universal approximation property is likely to hold.

**Soundness of the claims**

Can theory support the claim?
- The authors proved the universal approximation theorem for horocycle and $f^1_{a, p}$. Although it is not a constructive proof due to the Hahn-Banach theorem's nature, as the paper pointed out, it gives an affirmative answer for the theoretical justification and is a good first step to study the expressive power. 
- If I do not miss any information, the Poisson neuron model (Section 4.2, Paragraph 4) is introduced without its motivation nor justification. In addition, this paper does not provide the theoretical superiority of the model. For example, Theorem 1 and Theorem 2 does not apply to the Poisson neuron model. I want to know if there are theoretical justifications for the Poisson model.

Can empirical evaluation support the claim?
- Section 6.1: I confirm that the horocycle model's overall performance is better than Ganea et al. (2018a) and Shimizu et al. (2020). Especially, the proposed method significantly outperforms them when the embedding dimension is two, or the subtree is "worker.n.01". 
- Section 6.2--6.4: I confirm that the proposed method's error rate is smaller than the existing methods.
- Section 6.5: I could not understand the motivation for the experiments of the CIFAR-10 and Fashion-MNIST datasets in this section. Figure 8 claimed that Poisson MLR shows good generalization in the Flowers dataset. However, this paper does not provide such a comparison in the CIFAR-10 and Fashion-MNIST datasets. Also, the performance in these datasets is not as good as the SOTA models (I referenced [1] for CIFAR-10 and [2] for Fashion-MNIST]). Therefore, I think these results do not support the empirical superiority of Poisson MLR.
[1] https://paperswithcode.com/sota/image-classification-on-cifar-10
[2] https://paperswithcode.com/sota/image-classification-on-fashion-mnist


**Significance and novelty**

Novelty
- To the best of our knowledge, this is the first study that proves the universal approximation theorem for a single-hidden layer model on a hyperbolic space.

Relation to previous work
- Although this paper mentioned Ganea et al. (2018a) and Shimizu et al. (2020), with which this paper compare the proposed method in the experiment, it did not compare the methodological difference (especially novelty and superiority) of the proposed method from the two. Same is true of the baseline methods in Table 2 and the method by Ontrup & Ritter (2005) and Grattarola et al. (2019) in Table 3. I would like to recommend to make it clear what is the drawback of the existing model.


**Correctness**
Is the theory correct?
- Yes, So far as I check the proof, Theorem 1 and Corollary 1 (universality of horocycle neurons and the function $f^1_{a, p}$) are correct.

Is the experimental evaluation correct?
- Yes, I did not find any methodologically incorrect point in the experimental procedures. In Table 1, ideally, we should compare three methods with the same train/test partitions because the class label is highly imbalanced (e.g., 1115/82114 is positive in the case of worker.n.01 ); I am wondering if the performance variance caused by the randomness of data partition could be high.

Reproducibility of the experiments
- Yes. It explains experimental settings in detail in the appendix. Also, it has a runnable code with trained parameters.

**Clarity**

I would say that there is much improvement in the clarity of the paper.
     First, I took some time to understand how sections are related and how paragraphs in a section are related. I think adding discourse markers and organizing sentences so that readers can do paragraph reading could make the paper more understandable. Take the introduction section as an example. I feel there is a large gap between the third and fourth paragraphs. In addition, I could not understand that the fourth paragraph intends to explain the horocycle neuron until I reached the end of the paragraph. Also, although the function $f^1_{a, p}$ is introduced in the fifth paragraph, the introduction does not mention it in the remaining part and goes back to the explanation of horocycle neurons. 
     Another problem is that the tables and figures are not prepared appropriately. For example, Table 3 is inserted within a paragraph. Also, captions and legends of figures are tiny and hard to read.

**Additional feedback**
- Abstract: The acronym MLR is used without what it stands for. So, I recommend writing the meaning of MLR without abbreviation.
- Section 1, Paragraph 3: This paper study → studies
- Section 1, Second bullet: Although this sentence mentioned the Poisson neuron and the horocycle MLR, they were not mentioned before this sentence. Similarly, the term "end-based" is used in the introduction but is explained in Section 4.2 for the first time. I would recommend writing their explanation before they are used.
- Section 2 Paragraph 3 (Hyperbolic deep learning): I could not see what this paper intended to mean by the term "prototype" at first reading. This wording may need some definition.
- Section 4.1, Paragraph 3 (Neuron models): What does the following sentence mean?: We accept the representation properties of eigenfunctions on compact manifolds. 
- Section 4.2, Paragraph 5 (End-based clusters, end prototypes): It was hard for me to understand the relationship between sentences in the paragraph. For example, it is not clear at first sight how RBF is related to the discussion of clustering algorithms. I would recommend reconsidering the organization of the paragraph.
- Section 5 (9): $\langle x, \omega_i \rangle$ → $\langle x, \omega_i\rangle_H$
- Section 6.1, Table 1: Could you explain what $H^2$, $H^3$, etc. means?
- Section 6.1: The task (Ganea et al., 2018a) is to classify all other nodes as [...]. → It is not clear what "other" nodes mean solely from the main text. I understand it after I read the first sentence of Section A.11.
- Appendix A.15: This paper says that it adds a loss to distinguish the prototypes of 4 and 9. However, looking at the code, it seems the algorithm randomly selects two classes and adds the loss from prototypes of these classes. I want to confirm if my understanding is correct and recommend explaining the procedure if it is correct.

review_title: Review Comment #1

Meta-review: Reviewers generally appreciate the contributions of the paper, namely the horocycle neuron, Poisson neuron, and the universal approximation properties. However, there are concerns, especially by R4 and R5, that the presentation is confusing, lacks clarity, and should be substantially improved.

Note: Theorem 1.7 in (Helgason, 1970) is proved explicitly for the case n=2, not for general n as claimed in (9).  Thus the Laplacian eigenspace motivation needs to be re-written/re-examined.
Decision: Reject