Beyond Euclidean embeddings
By: Desh Raj
Tags: Word Vectors, Naturallanguageprocessing, Machine Learning, Representation Learning, Deep Learning

Poincare embeddings for hierarchical representations

This paper [2] proposes embeddings in hyperbolic spaces, such as the Poincare sphere. Before we get into the method itself, I think it would be best to give a brief overview of hyperbolic geometry itself.

Hyperbolic geometry

In his book Elements, Euclid provided a rigourous framework for axioms, theorems and postulates for all geometrical knowledge at the time. He stated 5 axioms which were to be assumed true. The first 4 were quite self-evident, and were:

(1) Any two points can be connected by a line.

(2) Any line segment can be extended indefinitely.

(3) Given a line segment, a circle can be drawn with center at one of the endpoints and radius equal to the length of the segment.

(4) Any two right angles are congruent.

However, the fifth axiom, also known as Playfair’s axiom, is much less axiom.

Playfair’s axiom: Given a line L and a point P, there exists at most one line through P that is parallel to L.

Euclid himself wasn’t very fond of this axiom and his first 28 postulates depended only on the first 4 axioms, which are the “core” of Euclidean geometry. Even 2000 years after his death, mathematicians tried to derive the fifth axiom from the first 4. While using “proof by contradiction” for this purpose, they assumed the negation of the fifth axiom (Given a line L and a point P not on L, there are at least two distinct lines that can be drawn through P that are parallel to L) and tried to arrive at a contradiction. However, while the derived results were strange and very different from those in Euclidean geometry, they were consistent within themselves. This was a turning point in mathematics as such a bifurcation in geometry had never been expected before. The geometry that arose from these explorations is known as hyperbolic geometry.

With this knowledge, let us now look at how embeddings may be computed in this new model.

The Poincare sphere model of hyperbolic space is particularly suitable for representing hierarchies. Consider a knowledge base which can be visualized as a tree. For any branching factor b, the number of leaf nodes increases exponentially as the number of levels increases. If we try to replicate this construction in a Euclidean disk(sphere), it would not be possible since the area(volume) of a disk(sphere) increases only quadratically(cubically) with increase in radius. This requires that we increase the number of dimensions exponentially.

However, the Poincare sphere embeds such hierarchies easily: nodes that are exactly l levels below the root are placed on a sphere in hyperbolic space with radius r ∝ l and nodes that are less than l levels below the root are located within this sphere. This type of construction is possible as hyperbolic disc area and circle length grow exponentially with their radius. In the paper, the authors used a sphere instead of disk since more degrees of freedom implies better representation of latent hierarchies.

Distances in the hyperbolic space are given as

Here, hierarchy is represented using the norm of the embedding, while similarity is mirrored in the norm of vector difference. Furthermore, the function is differentiable, which is good for gradient descent.

For optimization, the update term is the learning rate times the Riemannian gradient of the parameter. The Riemannian gradient itself is computed by taking the product of the Poincare ball matrix inverse (which is trivial to compute) with the Euclidean gradient (which depends on the gradients of the distance function). The loss function used in the paper is a softmax with negative sampling.