Probing BERT in Hyperbolic Spaces
Authors: Boli Chen
Yao Fu
Guangwei Xu
Pengjun Xie
Chuanqi Tan
Mosha Chen
Liping Jing
Venue: ICLR 2021 Poster (2021)
Keywords: Hyperbolic
BERT
Probe
Syntax
Sentiment

Abstract: Recently, a variety of probing tasks are proposed to discover linguistic properties learned in contextualized word embeddings. Many of these works implicitly assume these embeddings lay in certain metric spaces, typically the Euclidean space. This work considers a family of geometrically special spaces, the hyperbolic spaces, that exhibit better inductive biases for hierarchical structures and may better reveal linguistic hierarchies encoded in contextualized representations. We introduce a $\textit{Poincaré probe}$, a structural probe projecting these embeddings into a Poincaré subspace with explicitly defined hierarchies. We focus on two probing objectives: (a) dependency trees where the hierarchy is defined as head-dependent structures; (b) lexical sentiments where the hierarchy is defined as the polarity of words (positivity and negativity). We argue that a key desideratum of a probe is its sensitivity to the existence of linguistic structures. We apply our probes on BERT, a typical contextualized embedding model. In a syntactic subspace, our probe better recovers tree structures than Euclidean probes, revealing the possibility that the geometry of BERT syntax may not necessarily be Euclidean. In a sentiment subspace, we reveal two possible meta-embeddings for positive and negative sentiments and show how lexically-controlled contextualization would change the geometric localization of embeddings. We demonstrate the findings with our Poincaré probe via extensive experiments and visualization. Our results can be reproduced at https://github.com/FranxYao/PoincareProbe

Published as a conference paper at ICLR 2021
PROBING BERT IN HYPERBOLIC SPACES
Boli Chen Yao F
Guangwei Xu', Pengjun Xie', Chuanqi Tan', Mosha Chen', Liping Jing Alibaba Group University of Edinburgh 'Bijjin Jiaotong University boli.
com, yao. fu@ed.ac.uk,
(kunka xgw, chengchen wwwppppiiabaaaaaaa.a...a.aaaaaaa [chuanqi. tcq, chenmosha. cms @ ieaiiiaaabiae inc. com, ipjing@bjtu.eduu
com,
ABSTRACT
Recently, variety of probing tasks proposed to discover linguistic proper- ties learned in contextualized word embeddings. Many of these works implicitly assume these embeddings lay in certain metric spaces, typically the Euclidean space. This work considers a family of geometrically special spaces, the hyper- bolic spaces, that exhibit better inductive biases for hierarchical structures and may better reveal linguistic hierarchies encoded in contextualized representations. We introduce Poincaré probe, a structural probe projecting these embeddings into a Poincaré subspace with explicitly defined hierarchies. We focus on two probing objectives: (a) dependency trees where the hierarchy defined as head- dependent structures; (b) lexical sentiments where the hierarchy is defined as the polarity of words (positivity and negativity). We argue that a key desideratum of a probe is its sensitivity the existence of linguistic structures. We apply probes on BERT, a typical contextualized embedding model. In a syntactic subspace, our probe better recovers tree structures than Euclidean probes, revealing the possi- bility that the geometry of BERT syntax may not necessarily be Fuclidean. a sentiment subspace, we reveal two possible meta-embeddings for positive and negative sentiments and show how lexically-controlled contextualization would change the geometric localization of embeddings. We demonstrate the findings with our Poincaré probe via extensive experiments and visualizatior
INTRODUCTION
Contextualized word representations with pretrained language models have significantly advanced NLP progress (Peters et al., 2018a| Devlin et al. 2019). Previous works point out that abundant linguistic knowledge implicitly exists in these representations (Belinkov et al., 2017: Peters al., 20 Tenney et al.] 2019). This paper is primarily inspired by Hewitt Manning (2019) who propose structural probe to recover dependency trees encoded under squared Euclidean distance in a syntactic subspace. Although heing an implicit assumption, there is no strict evidence that the geometry of these syntactic subspaces should be Euclidean, especially under the fact that the Euclidean space has intrinsic difficulties for modeling trees (Linial et al. 1995).
We propose impose and explore different inductive biases for modeling syntactic subspaces. The hyperbolic space, special Riemannian space with constant negative curvature, is good candidate because of its tree-likeness (Nickel & Kiela 2017, Sarkar 2011). We adopt generalized Poincaré Ball, a special model of hyperbolic spaces, to construct a Poincaré probe for contextualized embed- dings. Figure T (A, B) give an example of tree embedded in the Poincaré ball and compare the Fuclidean counterparts. Intuitively, the volume of Poincaré ball grows exponentially with its ra- dius, which is similar to the phenomenon that the number of nodes of full tree grows exponentially with its depth. This would give "enough space" to embed the tree. In the meantime, the volume of the Euclidean ball grows polynomially and thus has less capacity to embed tree nodes.
* "qqual contribution. Work was done during an internship Alibaba DAMO Academy. CCoresponding author. at httpa/////iiiioo/////ooooo//////
Our results can be

---

Published as a conference paper at ICLR 2021
good
luvely
excellent
awesome
IPOSI
Thig
js
INEG]
sample
awful
terrible
tree
poor
.ad
syntax
(A) Euclidean tree
(B) Poincaré tree
(C) Syntax
(D) Sentiment
Figure Visualization of different spaces. (A, B) Comparison between trees embedded Eu- clidean space and hyperbolic space. We use geodesics, the analogy of straight lines in hyperbolic spaces, to connect nodes in (B). Line/geodesic segments connecting nodes are approximately of the same length in their corresponding spaces. Intuitively, nodes embedded in Euclidean space look more "crowded", while the hyperbolic space allows sufficient capacity to embed trees and enough distances between leaf nodes. (C) A syntax tree embedded in a Poincaré ball. Hierarchy levels correspond to syntactical depths. The higher level a word is in a syntax tree, the closer it is to the origin. (D) Sentiment words embedded in Poincaré ball. Hierarchy defined as the sentiment po- larity. We assume two meta [POS] and [NEG] embeddings at the highest level. Words with stronger sentiments are closer their corresponding embedddgns...
Before going any further, it is crutial to differentiate probe and supervised parser (Hall Maudslay et al. 2020), and ask what makes good probe. Ideally, probe should correctly recover syntac- tie information intrinsically contained in the embeddings, rather than being a powerful parser by itself. So it is important that the probe should have restricted modeling power but still be sensitive enough to the existence of syntax. For embeddings without strong syntax information (e.g.. ran- domly initialized word embeddings), probe should not aim assign high parsing scores (because this would overestimate the existence of syntax), while a parser aims for high scores no matter how bad the input embeddings are. The quality of a probe defined by its sensitivity to syntax.
Our work of probing BERT in hyperbolic spaces is exploratory. As opposed to the Euclidean syn- tactic subspaces in Hewitt Manning (2019), we consider the Poincaré syntactic subspace, and show its effectiveness for recovering syntax. Figure [ (C) gives example of the reconstructed dependency tree embedded in the Poincaré ball. In our experiments, we highlight two important ob- servations of our Poincaré probe: (a) it does not give higher parsing scores to baseline embeddings (which have no syntactic information) than Euclidean probes, meaning that it is not a better parser; (b) it reveals higher parsing scores, especially for deeper trees, longer edges, and longer sentences, than the Euclidean probe with strictly restricted capacity. Observation (b) can be interpreted from two perspectives: (1) it indicates that the Poincaré probe might be more sensitive the existence of deeper synlax; (2) the structure of syntactic subspaces of BERT could be different than Euclidean, especially for deeper trees. Consequently, the Fuclidean probe may underestimate the syntactic pability of BERT, and BERT may exhibit stronger modeling power for deeper syntax in some special metric space, our case, Poincaré ball.
To best exploit the inductive bias for hierarchical structures of hyperbolic space, we generalize our Poincaré probe sentiment analysis. We construct Poincaré sentiment subspace by predicting sentiments of individual words using vector geometry (Figure D). We assume two meta represen- tations for the positive and negative sentiments as the roots in the sentiment subspace. The stronger word's polarity is, the closer it locates to its corresponding meta embedding. In our experiments, with clearly different geometric properties, the Poincaré prohe shows that BERT encodes sentiments for each word in very fine-grained way. We further reveal how the localization of word embed- dings may change according to cexically-controlle contextualization, i.c., how different contexts would affect the geometric location of the embeddings in the sentiment subspace.
In summary, we present an Poincaré probe to reveal hierarchical linguistic structures encoded in results indicate the possibility of using
BERT. From hyperbolic deep learning perspective, Poincaré models for learning better representations of linguistic hierarchies. From a linguistic per- spective, reveal the geometric properties of linguistic hierarchies encoded in BERT and posit that
2

---

Published as a conference paper at ICLR 2021
BERT may encode linguistic information in special metric spaces that are not necessarily Euclidean. We demonstrate the effectiveness of our approach with extensive experiments and visualization.
2 RELATED WORK
Recently, there are increasing interests in finding linguistic information encoded in BERT (Rogers al. 2020). One typical line of work is the structural probes aiming to reveal how syntax trees are encoded geometrically in BERT embeddings (Hewitt & Manning 2019 Reif et al. 2019). Our Poincaré probe generally follows this line and exploits the geometric properties of hyperbolic spaces for modeling trees. Again, we note that the goal of syntactic probes is to find syntax trees with strictly limited capacity, i.e., probe should not be a parser (Hewitt & Man- ning |019: Kim et al. (020). and strictly follow this restriction in our experiments. Other probing tasks consider a variety of linguistic properties, including morphology (Belinkov et al. 2017), word sense (Reif et al. 2019), phrases (Kim et al. 2020), semantic fragments (Richardson et al. 2020), and other aspects of syntax and semantics (Tenney et al., 2019). Our extended Poincaré probe for sentiment can be viewed as one typical semantic probe that reveals how BERT geometrically en- codes word sentiments.
Probing BERT
Hyperbolic Deep Learning Recently, methods using hyperbolicc geometry have been proposed for several NLP tasks due its better inductive bias for capturing hierarchical information than Eu- clidean space. Poincaré embeddings (Nickel& & Kiela 2017) and POINCARÉGLOVS (Tifrea et al.) 2019) learn embeddings of hierarchies using Poincaré models and exhibit impressive results, espe- cially in low dimension. These works show the advantages of hyperbolic geometry for modeling trees while we focus on using hyperbolic spaces for probing contextualized embeddings. To learn models in hyperbolic spaces, previous works combine the formalism of Möbius gyrovector spaces with the Riemannian geometry, derive hyperbolic versions of important mathematical operations such as Möbius matrix-vector multiplication, and use them to build hyperbolic neural networks (Ganea et al. 2018). Riemannian adaptive optimization methods (Bonnabel) 2013 Bécigneul & Ganea 2019) are proposed for gradient- prseenttbbsed optimization. Techniques in these works are used as the infrastructure in this work for training Poincaré probes.
3 POINCARÉ PROBE
We begin by reviewing the basics of Hyperbolic Geometry. We follow the notations from Ganea ct al. (018). A generalized Poincaré ball a typical model of hyperbolic space, denoted as (D Iz) for c > 0, where D = fx E F" |||>|| < 1} is Riemannian manifold, o = (X=)² 2, is the metric tensor, X = 2/(11 c|| I|2) is the conformal factor and e is the negative curvature of the hyperbolic space. We will use the term hyperbolic and Poincaré interchangeably according to the context. Our Poincaré probe uses the standard Poincaré ball D with 1. The distance function for x, y D is:
(1) where G denotes the Möbius addition, the hyperbolic version of the addition operator. Note that we O. Additionally,
dr (a. y) = (2/ tanh tanh- x G. y/l),
recover the Fuclidean space Th when matrix-vector multiplication for a linear map M TRn - TRm which is the hyperbolic version of linear transforms. We use expa to denote the exponential map, which maps vectors in the tangent space (in our case, a space projected from the BERT embedding space) to the hyperbolic space. Their foosed-form formulas are detailed in Appendix C
M @ æ denote the Möbius
Our probes consist two simple linear maps P and Q that project BERT embeddings into Poincaré syntactic/sentiment subspace. Formally, let M denote pretrained language model that produces sequence of distributed representations hit given sentence of 1. words w1.- We train linear map P TR= + R* n being the dimension of contextualized embeddings and l: being the probe rank, that projects the distributed representations to the tangent space. Then the exponential map projects the tangent space to the hyperbolic space. In the hyperbolic space, we construct the Poincaré syntactic/sentiment subspace via another linear map Q R - R The equations are;
p: = expo (Ph;)
(2)
9: Q &c P.
(3)
3

---

Published as a conference paper at ICLR 2021
Table 1: Results of tree distance and depth probes. We highlight two observations of Poincaré probes compared with Euclidean probes: (a) they do not assign higher scores to embeddings without syntactic information (ELMo0 and LINEAR), meaning that they do not form a parser; (b) they recover higher scores (colored cells) for contextualized embeddings with smaller probe capacity, meaning that they are more sensitive to the existence syntax in contextualized embeddings.
Distance
Depth
Euclidean
Poincaré
Euclidean Root %
Poincaré Root %
Method
UUAS
DSpr.
UUAS
DSpr.
NSpr.
NSpr.
EL
26.8
0.44
25.8
0.44
54.3
0.56
53.5
0.49
LINEAR
48.9
0.58
45.7
0.58
2.9
0.27
4.5
0.26
ELMol
77.0
0.83
79.8
0.87
86.5
0.87
88.4
0.87
BERTBASE7
79.8
0.85
83.7
0.88
88.0
0.87
91.3
0.88
BERTLARGEIS
82.5
0.86
85.1
0.89
0.88
0.88
BERTLARGEI6
81.7
0.87
0.90
90.1
0.89
91.7
0.89
Here P maps the original BERT embedding space to the tangent space of the origin of the Poincaré ball. Then (-po(-) maps the tangent space inside the Poincaré bal Consequently, in equation 3 we use the Möbius matrix-vector multiplication as the linear transformation in the hyperbolic space
PROBING SYNTAX
Following Hewitt & Manning (209), we aim to test if there exists hyperbolic subspace trans- formed from the original BERT embedding space with simple parameterization where squared dis- tances between embeddings or squared norms embeddings approximate tree distances or node depths, respectively. The goal of the probe is to recover syntactic information intrinsically tained in the embeddings. To this end, probe should not assign high parsing scores to baseline non-contextualized embeddings (otherwise would become a parser, rather than being probe). So is crucial for the probe to have restricted modeling power (in our case, two linear transforms P and Q) but still being sensitive enough for syntactic structures. We further test if the Poincaré probe is able to discover more syntactic information for deeper trees due to its intrinsic bias for modeling trees.
Similar to Hewitt & Manning (2019). we use the squared Poincaré distance to recreate tree distances between word pairs and the squared Poincaré distance to the origin recreate the depth of word:
Ldistance
Σ
[dr(ws, w;) - dp (qi, (qi:9;)² 9
(4)
t²
1.JE(L....f)
Cuepth
dd(wx)) dp (qi, 2
(5)
iE(L.....)
where dr dr(w;,w, denotes the distance between word i, j their dependency tree, i.e., number of edges linking word i to j and (w;) denotes the depth of word in the dependency tree. For optimization, we use the Adam (Kingma & Ba 2014) initialized learning rate 0.001 and train up to 40 epochs. We decay the learning rate and perform model selection based on the dev loss.
Thh choice of tangent space the origin, instead of other points, follows previous works Ganea et al. 2018 Mathieu al. (201)) for its mathematical simplexity and optimization convenience.
This transformation is theoretically redundent, we use it primarily for numerical stability during optimiza-
tion. We further note that such optimization stability is still an open problem in hyperholic deep learning (Math-)
ieu al. 2019. We leave detailed investigation to future work.
4

---

Published as a conference paper at ICLR 2021
85%
6.9
85%
0.9
80%
75%
0.85
0.85
75%
05%
0.8
0.8
5550
0.75
Eucldean CUAS
CUAS
Peinart Dögr
45%
D5pt.
0.7
Enclidean DSpe
Baclidenn DSpr.
55%
077
35%
0.65
10
11
12
16
32
64
125
256
(A) BERTBASE layer index
(B) Probe maximum rank
100%
1.0
9005
0,9
90%
0.9
0.8
0.8
7056
0.7
80%
D8pc
Poincari UUAS
Poincaré UUAS
50%
Eucličen UCAS
0.5
...... Fudiidore 1JIAS
Poincari DSpr
Poincard
Eudiidean DSpr.
BucliOen
30%
0.3
83
4001
J633
Ab-4
-665
2
7
12
17
22
27
32
47
52
59
64
(C) Sentence length
(D) Curvature
Figure 2: Comparison between the two probes. (A) Middle layered embeddings show richer syntac- tic information. (B) All probes recover syntax best at approximately rank 64 and Poincaré probes especially better at low ranks. (C) Poincaré probes recover syntax better for longer sentences. (D) As the curvature goes closer to 0, Poincaré probes behave more similar to Euclidean probes.
4.1 EXPERIMENTAL. SETTINGS
Our experiments aim to demonstrate that the Poincaré probe better recovers deeper syntax in BERT without becoming parser. We denote the probes in Hewitt & Manning (2019) Euclidean probes and follow their datasets and major baseline models. Specifically, we use the Penn Treebank dataset (Mareus & Marcinkiewicz 1993) and reuse the data processing code in Hewitt & Man- ning (2019) to convert the data format to Stanford Dependency (de Marneffe et al., (006). For baseline models, we use (a) ELMo0: strong character-level word embeddings with contextual information. All probes should not find syntax trees from these embeddings.
(b) LINEAR: left-
to-right structured trees that only contain positional information. (c) ELMOl and BERT*; strong contextualized embeddings with rich syntactic information. All probes should accurately recover all parse trees encoded in them.
Since the goal of probing is to recover syntax trees in a strict notion, we restrict our Poincaré probe 64 dimension, i.e. &t 64 for P and Q, which is at the same level, or smaller than the effective rank of the Euclidean probes reported in Hewitt & Manning (2019). We also emphasize that the parameters of our Poincaré probe are simply two matrices, which is again significantly less than a typical deep neural network parser (Dozat& & Manning 2016).
To evaluate the tree distance probes, we report: (a) undirected unlabeled attachment scores (UUAS), the scores showing if unlabeled edges are correctly recovered, against the gold undirected tree; (b) distance Spearman Correlation (DSpr), the scores showing how the recovered distances (either Euclidean or Hyperbolic) correlate with gold tree distances. To evaluate the tree depth probes, we report: (a) norm Spearman Correlation (NSpr). the scores showing how the true depth ordering correlates with the predicted depth ordering; (b) the corretly identified root nodes (root%), the scores showing to what extent the probes can identify sentence roots.
4.2 RESULTS
Table I shows the major results for tree distance and depth probing. We that Poincaré probes do not give higher scores than Euclidean probes for ELMo0 and LINEAR (i.e., they are not parsers. Also appendix A.T how GRU transforms probes to be parsers), yet recovers higher parsing
5

---

Published as a conference paper at ICLR 2021
alod 89
Edge Lype Edge Lengiht
c ccomp:7.67
6-10
cvubj.7.06
Tree Type
Length
11-15
conj
Gold
16-20
Poincare
Euclideum
21-25
some4444 45
021.64
10'0
10-1
10
0.2
0.4
Count
Recall
Figure 3: Left: comparison of edge length distributions. Distribution of the Poincaré probe aligns better with the ground truth than the Euclidean probe. Right: edge prediction recall of top longest edge types. The Poincaré probe is especially better at recovering edges of longer average length.
day,
wild
gnetoy
<tt was
wild
day
piotty
day
wild pritty
Was
CITAS 0.20
was
EIIAS 1.00
pretty
Toori Depilt 2.01
Rost Depiit 9.55
Room Depth 0.39
(A) Syntax tree
(B) ELMo0
(C) BERTBASE7
(D) BERTBASE7
Figure 4: PCA projection of dependency trees for the sentence it was a pretty wild day. Yellow lines/geodesics denote the ground truth and blue dashed lines/geodesies are predicted by the probe. Blue points denote root words of sentences. Word depths are clearly organized the Poincaré ball (D) than the Euclidean space (C). The closer word is to the origin, the upper level it is in the tree.
scores for embeddings with deep contextualization. The results indicate that: (a) Poincaré probes might be more sensitive to the existence of syntactic information; (b) the syntactic information encoded in contextualized embeddings may be underestimated by the Euclidean probes due to their incapacity of modeling trees. We further give detailed analysis of the two probes from various perspectives on the distance task, which is harder than the depth task.
Different BERTBASE Layers Figure 2A and Euclidean probe trained on each layer of BERTBASE. The two are consistent with each other and show similar tendency that syntax primarily exists in the middle layers.
reports the distance scores of both the Poincaé probe
Probe Rank Figure 2B reports the distance scores of probes with different rank & for BERT- that the scores do not increase after 64 di- mension with the Poincaré probe. We posit there might be some intrinsic syntactic subspace whose dimension is close 64, but leave further investigation future work.
BASE7. Similar to Hewitt & Manning (2019),
Sentence and Edge Lengths Figure 2C reports the distance scores of sentences with differ- ent lengths for BERTBASE7. The result that Poincaré probes give higher scores for longer sen- tences than Euclidean probes, meaning that they better reconstruct deeper syntax. Figure 3 (left) compares edge distance (length) distributions between ground truth edges and edges predicted by Euclidean and Poincaré probes, respectively. The distribution from the Poincaré probes are closer the ground truth distribution, especially for longer trees. Figure 3 (right) shows that Poincaré probes consistently achieve better recall for edge types with longer average length. Figure [| in the Appendix further compares minimum spanning tree results from predicted squared distances on BERTBASE7 (we randomly sample 12 instances from the dev set). These observations would be particularly interesting from a linguistic perspective as the syntactic structure for longer sentences more complicated and challenging than those for shorter sentences. A promising future direction would be using hyperbolic spaces for parsing.
Curvature of the Hyperbolic Space We further characterize the structure of the hyperbolic syntactic subspace with the curvature parameter, which measures how "curved" the space is (Fig- ure 2D). We see that the optimal curvature is about -1, which is the curvature of a standard Poincaré

---

Published as a conference paper at ICLR 2021
Table 2:
Classification accuracy on Movie Review dataset. Both Euclidean and Poincaré give 48.4 (nearly random guess) to baseline LINEAR embeddings, meaning neither of them form classifier.
BERTBASE9
BERTBASEIO
BiL.STM
Fuclidcan
Poincaré
Enclidean
Poincaré
Trainable
81.7
84.9
83.5
84.2
Accuracy 79.7
Fixed 78.4 (-3.3)
84.2 (-0.7)
79.1 (-4.4)
84.5 (+0.3)
ball. Additionally, if we gradually change the curvature O, the space would be "less curved" and more similar to the Euclidean space (more "flat"). Consequently, the Poincaré scores converge to the Euclidean scores. When the curvature is O, we recover the Euclidean probe.
Visualization of Syntax Trees To illustrate the structural differences between Euclidean and
Poincaré syntactic subspaces, we visualize the recovered dependency trees in Figure A To simul- taneously visualize edges and tree depth, we jointly train the two probing objectives in equations 4 and for the same probe. PCA projection then used to visualize the syntax trees, similar to Reif et al. (2019). As is shown in Figure E compared with the Euclidean probe, the Poincaré probe shows a more organized word hierarchy: the root word day takes the position at the origin and is surrounded by other words according to tree depth. We further note that embeddings of ELMo0 contains no syntactic information and the corresponding tree looks meaningless.
4.3 SUMMARY ON SYNTACTIC PROBE
The manifold of BERT embeddings could be complicated and exhibit special geometric properties. Our probe essentially serves as a well-defined, differentiable, surjective function that maps the BERT embedding space to low dimensional, moderately curved Poincaré ball (rather than Euclidean space), and consequently leads to better reconstruction of syntax trees. This indicates that the mani- fold of BERT syntax may be geometrically more similar to the Poincaré ball than Euclidean space. And of course, we cannot conclude that the syntactic subspaces are indeed hyperbolic. Rather than being conclusive, we aim explore alternative models for BERT syntax and reveal the underlying geometric structures. Such geometric properties of BERT are still far from well-understood and there are still many open problems worth studying.
5 PROBING SENTIMENT
Having validated the effectiveness of reconstructing syntax trees of our Poincaré probe, in this sec- tion, we generalize the probe to semantic hierarchies and focus on sentiment. We first recover a Poincaré sentiment subspace with two meta positive and negative embeddings taking the topmost hierarchy then identify word localizations in this subspace. We highlight that our probe reveals that BERT encodes fine-grained sentiments (positive, negative and neutral) for each word, the probe is trained with sentence-level binary labels. To further examine how the context a sentence may affect its word embeddings, we perform a qualitative lexically-controlled contextual- ization, i.e., to change the sentiment of a sentence by carefully changing the word choice according common linguistic rules, and visualize how the localization of embeddings changes accordingly.
though
We use the same probe architecture described in Section 3 to construct the sentiment subspace. Again, a probe parameterized by two matrices P and Q that project a BERT embedding into a Poincaré ball. We adopt sentiment labels for sentences as our supervision and use the Movie Review dataset (Pang & Lee 2005) with simple binary labels (positive and negative). Details of this dataset are in Appendix B Given a sentence with t words Wi:s. we project its BERT embeddings according to equation 2 and and obtain qi D To interpret the classification procedure in terms of vector geometry, we set two trainable meta representations for positive and negative labels as Cpos: Creg E D The logits for the two classes lpos, lucg are obtained by summing over the Poincaré
'Although the use of PCA in hyperbolic spaces would lead to certain levels of distortion, we note that it is empirically effective for visualization. As there is no perfect analogy of PCA in hyperbolic spaces (Pennec) al. 2018), leave the investigation of dimension reduction for hyperbolic spaces future work.
7

---

Published as a conference paper at ICLR 2021
Table 3: Top sentiment words recovered by Euclidean and Poincaré probes. Colored words align better with human intuition (orange for positive and blue for negative).
latch, horne, testify, birth, opened, landau, cultivation, bern, willingly, visit, cub, carr, iced, meetings, awake, awakening, cddy, wryly, protective, fencing
POS
Euclidean
worthless, useless, frustrated, fee, inadequate, rejected, equipped, schedule, useful, outdated, discarded, equipment, pointless, sounded, weakened
NEG
funky, lively, connects, merry, documented, vivid, dazzling, etched, infectious, relaxing, evenly, robust, wonderful, volumes, capturing, splendid, floats, sturdy
POS
Poincaré
inactive, stifled, dissatisfied, discarded, insignificant, insufficient, erratic, indifferent, fades, wasting, arrogance, robotic, stil, trails, poorly, inadequate
NEG
ultimately 1.-
pointless
good [NEG]
lpoking
[POS]
gointless
and
pitimafe
NEG
Polncare
with
good with TPOS]
Trae: INI
80
True: [N]
Rii STM Reseline
Fred: [N]
Layer
(A) Euclidean: BERTBASHIO
(B) Poincaré: BERTBASEIO
(C) BERTBASE layerwise accuracy
Figure 5: (A, B) PCA projection of sentence a good-looking but ultimately pointless political thriller with plenty of action and almost no subslance. Words are connected to closer meta- embeddings. Words with dashed lines mean that the differences between their distances to two embeddings not significant (neutral words). (C) Layerwise accuracy. Sentiment emerge at deeper layers (aroung layer 9) than syntax (around layer 7).
distance between each word and the opposite meta embeddings:
Apss
Σ dap (qt, Cneg) 1=1
Inog
dps (q:: Gpos)
(6)
Since we know that the two classes are contrary to each other, we also consider assigning fixed positions for the two meta embeddings. For the Euclidean meta embeddings, we use Cpos = (1/VR)-
1 and Cneg Cpos in the experiments, where k is the space dimension. For the Poincaré meta embeddings, Cpos expo((1/Vk)) 1) and Cneg -Cos-
For training, we use cross-entropy after a Softmax as the loss function. An analog method is used for Euclidean Probes. We use Riemannian (Bécigneul Ganea 2019) for all trainable parameters in the Poincaré ball, notably the two meta embeddings, and vanilla Adam for Euclidean parameters.
5.1 WORD POLARITIES BASED ON GEOMETRIC DISTANCES
Table reports the classification accuracy. Firstly, both probes give better accuracy than a BiLSTM classifier meaning that there exists rich sentiment information in contextualized embeddings. We note that when fixing the class meta representations, the Euclidean probe receives a large loss of performance, while the Poincaré probe can even perform better. This observation strongly suggests that the sentiment information may be encoded in some special geometric way. We also report the top sentiment words ranked by the distance gap between two meta-embeddings in Table 3 and see that the words recovered by the Poincaré probe align better with human intuition (colored words). More comprehensive word list are in Appendix B Classification results for each BERT layer is shown in Figure [C] Sentiment emerges at deeper layers (around 9) than syntax (around layer 7, comparing with Figure 2A), making it an evidence for the well-known assumption that syntax serves the scaffold of semanties.
8

---

Published as a conference paper at ICLR 2021
15-
15+
13-
50-
12-
POST
PPSS]
g06d
[POS]
migiee
trily
awesomee
[POS]
gobd
[NEG]
this
08:
bad
4
[NEG]
this
mogie
[EEG]
NEEG]
Troe: [P] Prod: [P]
True: PP]
True: IN]
True: [N]
Prod: PP]
Pred: [N]
Pred: [N]
(A) Positive
(B) Positive++
(C) Negation
(D) Negative
Figure 6: cexically-controlie contextualization. (A) This good movie. awesome movie. (C) This is not good movie. (D) This is bad movie.
(B) This is truly
12-
50-
13-
not
moric
NEEG]
PPS]
G0OD movie
not
IEEG]
JPOS]
movie
Uhal
Fare
NEG POS
the ausaibll
NEG
PPOSI
True: [P] Prod: [P]
Trae: [P]
True: [?]
Trun: Pred: [N]
Prod: [N]
Pred: [N]
(A) Negation
(B) Double negation
(C) Ambiguous
(D) Satire
Figure 7: Special cases of lextcally-controlled contextualization. (A) This a bad movie. (B) I do not mean that movie is not good. (C) A movie like that is rare. (D) How can they possibly make movie like that.
5.2 VISUALIZATION AND LEXICALLY-CONTROUE CONTEXTUALIZATION
Figure illustrates how sentences embedded in the spaces. We see that: (a) both probes distinguish very fine-grained word sentiment, as one can infer if each word is positive, negative, or neutral, even if the probes are trained on sentence-level binary labels; (b) the Poincaré probe separates two meta-embeddinngs more clearly than the Euclidean probe and gives more compact embeddings. We emphasize that the observation in F represents general pattern, rather than being a special case. More visualization can be found in Appendix B
To see how different contextualization may change the localization of embeddings, we carefully change the input words to control the sentiment. As is shown in Figure 6 we see that: (a) sentiment affects localization: stronger sentiments would result closer distances to the meta- embeddings (Subfigure A v.s. B); (b) contextualization affects localization: when the sentence sentiment changes to negative (Subfigure A v.s. D), all words will be more close to the negative embedding, even when such change induced by simple negation (Subfigure A v.s. C).
We further study more complicated cases to explore the limit of BERT embeddings (Figure 7 and find out: (a) BERT fails at double negation (Subfigure B), which shows that the logical reasoning behind double negation may be challenging for BERT; (b) BERT gives reasonable localization for an ambiguous sentence (Subfigure C) most words do not significantly closer to meta em- beddings (dashed lines); (c) BERT gives correct localization for a sentence with satire (Subfigure
D). More cases can be found in Appendix B We further encourage the reader to run the Jupyter Notebook in supplementary materials to discover more visualization results.
6 CONCLUSION
In this paper, we present Poincaré probes that recover hyperbolic subspaces for hierarchical infor- mation encoded in BERT. Our exploration brings up new analytical tools and modeling possibilities about the geometry of BERT embeddings in hyperbolic spaces with detailed discussions about tree structures, localizations and their interactions with the contextualization.

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
We thank all the reviewers for their valuable suggestions. This work is supported by Alibaba Group through Alibaba Research Intern Program.
REFERENCES
Gary Bécigneul and Octavian-Eugen Ganea. Riemannian adaptive optimization methods. In Pro- ceedings of the Seventh International Conference on Learning Representations, 2019.
Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. What do neural machine translation models learn about morphology? In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 861-872, July
2017.
Silvere Bonnabel. Stochastic gradient descent on riemannian manifolds. IEEE Transactions on Automatic Control, 58(9):2217-2229) 2013.
MariieCatteriiee de Marneffe, Bill MacCartney, and Christopher D Manning. Generating typed dependency parses from phrase structure parses. Proceedings of the Fifth International Con- ference Language Resources and Evaluation (LREC'06), 2006.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171-4186, 2019.
Timothy Dozat and Christopher D Manning, Deep biaffine attention for neural dependency parsing. arXiv preprint 161 2016.
Octavian Ganea, Gary Becigneul, and Thomas Hofmann. Hyperbolic neural networks. In Advances in neural information processing systems 31, pp. 5345-5355. 2018.
Rowan Hall Maudslay, Josef Valvoda, Tiago Pimentel, Adina Williams, and Ryan Cotterell. A tale of a probe and a parser. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 7389-7395. Online, July 2020. Association for Computa- Bathtts::/ /www wwwwebe.bb........... org/
tional Linguistics. doi: 10. 1/ll0/..cm.mm..mmin...
anthology/2020.co acl main. 659
John Hewitt and Christopher D. Manning. A structural probe for finding syntax word representa- tions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computationall Linguistics: Human Language Technologies, Volume (Long and Short Pa- pers), pp. 4129-4138, June 2019.
Taeuk Kim, Jihun Choi, Daniel Edmiston, and Sang-goo Lee. Are pre-trained language models aware of phrases? simple but strong basclines for grammar induction. In International Conference on Learning Representations, 2020.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Nathan Linial, Eran London, and Yuri Rabinovich. The geometry of graphs and some of its algo- rithmic applications. Combinatorica, (5(2):2152255, 1995.
Mitchell P Marcus and Mary Ann Marcinkiewicz. Building a large annotated corpus of english: The penn treebank. Computational Linguistics, 19(2), 1993.
Emile Mathieu, Charline Le Lan, Chris Maddison, Ryota Tomioka, and Yee Whye Teh. Contin- uous hierarchical representations with poincaré variational auto-encoders. In Advances in neural information processing systems, pp. 12565-12576. 2019.
Maximillian Nickel and Douwe Kiela. Poincaré embeddings for learning hierarchical representa- tions. In Advances in Neural Information Processing Systems 30, pp. 6338-6344 2017.
10

---

Published as a conference paper at ICLR 2021
Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd Anmual Meeting on Association for Computational Linguistics, pp. 115-124, 2005.
Xavier Pennec al. Barycentric subspace analysis on manifolds. The Annals of Statistics, 46(6A): 2711-2746, 2018.
Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global vectors for word rep- resentation. In Proceedings of the 2014 Conference on Empirical Methods Natural Language Processing (EMNLP), pp. 1532-1543, 2014.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Con- ference of the North American Chapter the Association for Computational Linguisties: Human Language Technologies, Volume I (Long Papers), pp. 2227-2237, 2018a.
Matthew Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. Dissecting contextual word embeddings: Architecture and representation. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pp. 1499-19995500 1509, 2018b.
Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. Visualizing and measuring the geometry of bert. In Advances in Neural Information Processing Systems 32, pp. 8594-8603. 2019.
Kyle Richardson, Hai Hu, Lawrence S Moss, and Ashish Sabharwal. Probing natural language inference models through semantic fragments. In AAAI, pp. 8713-87211 2020.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842-866 866, 2020. doi: 10. (.016 Wtl.httss: tacl 1.54
/wwp://wwwwww/////////// www ac web.
Rik Sarkar. Low distortion delaunay embedding trees in hyperbolic plane. In International Symposium Graph Drawing, pp. 355-366. Springer, 2011.
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang. Adam Poliak, Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. What do you learn from context? probing for sentence structure in contextualized word representations. In International Conference on Learning Representations, 2019.
Alexandru Tifrea, Gary Becigneul, and Octavian-Euged Ganea. Poincaré glove: Hyperbolic word embeddings. In International Conference on Learning Representations, 2019.

---

Published as a conference paper at ICLR 2021
APPENDIX
A ADDITIONAL RESULTS ON PROBING SYNTAX
Table shows the score results of training Euclidean and Poincaré probes for distance and depth tasks simultaneously. More plots for longer sentences and deeper trees in the dev set are given (Figure 8 9 TOP to illustrate the syntax subspaces (described Section 3) (B) in these figures is
produced by the 768d Euclidean Probe, and (C) (D) (E) are produced by the 64d Poincaré probe.
Table 4: Results of training probes for distance and depth tasks simultaneously.
Fuclidean
Hyperbolic
Distance
Depth
Distance
Depth
Method
UUAS
DSpr.
Root %
NSpr.
UUAS
DSpr.
Root
NSpr.
ELMo0
20.1
0.41
55.0
0.51
19.3
0.41
56.3
0.53
LINEAR
43.4
0.57
8.6
0.27
45.1
0.57
7.0
0.27
69.8
0.79
85.6
0.85
73.3
0.84
88.1
0.87
ELMol
(-7.2)
(-0.04)
(-0.9)
(-0.02)
(-6.5)
(-0.03)
(-0.3)
(-0.00)
73.8
0.81
83.3
0.84
78.2
0.86
88.6
0.87
BERTRASE7
(-6.0)
(-0.04)
(-4.7)
(-0.03)
(-5.5)
(-0.02)
(-2.7)
(-0.01)
Sald
20
notifeed
he
had
glomfany on Bannhah
@Dutch
eeson
02-
la Thegtoiing
Llls
60-
Increosing
-60
company
sotified
dis
-04-
blake
-86-
The Dutch
Burmah
Root: Trae Root Depth:
0.57
the
-B77s -000
1.00
1.25
(B) BART7
(A) Syntax tree
bn
Da
0.1
Tse
company
Dutch
company
company
Jarmah
02-
abe
notified
poligahi
Lake
epcressing op
Juhicaln
02-
02-
adi
02-
hild
UUAS:
Jn
UUAS:
Root: True
Root: Trao
Root Depth:
0.05
Root Degich 0.13
Root Depth: 0.33
-36
-14
-10
0.
32.
(C) ELMO0
(D) ELMol
(E) BERTBASE7
Figure 8: The Dutch company hadn't notified Burmah of its reason for increasing the stake, he said.
A.1 A GRU LAYER TRANSFORMS A PROBE TO BE A PARSER
We show how contextualization transforms a probe to a parser by adding a GRU layer to the probes. For Fuclidean probe, hidden state of each word in GRU is used for encoding syntax, i.e.
12

---

Published as a conference paper at ICLR 2021
sald
bellars
10
withdraw
Mid
6lt
from
surfac
purfacees
off
bid
Memotec
guid
better
etee
With
Pom
Root: Trae Root Depth: 0.12
Memotec
-1.0
Lh
HD db
3.00
0.3b
0.50
0.75
1.50
B BERTBASE7
(A) Syntax tree
Momotec
04-
with
JS1
Momootamm
aurfacess
alattb wotibrreg
dho
Jran
Mematee
wentoorra sai
02-
02-
Rjjcctor
ULJAS:
UUAS: 0.67
UUAS: 0.87
Book: True
Root: Trae
0.05
Rooi Deptth: 0.14
Rool Depth: 0.06
-22
0.2
(C) ELMO0
(D) ELMol
(E) BERTBASE7
Figure 9: ISI said it withdraw from the merger agreement with Memotec if a better bid surfaces.
cover gouglniild but-
abe
Apdpresent
UUAS:
The
of
expanssoo
675
Pypansion cum
Root: Trae
Root Depth: 0.63
peereffaahte.m ool
da gemall
Jan
0.0
spreseat
lights
pige Rights
6:3-
bours
5.0-
Woll saver
Jwo
of
2.25
flghhs
two
2.00
portentage
Jban
ghan
Joss
2.59
Jess
60
20
(B) BERTBASE7
(A) Syntax tree
0E-
Hights
would
epecsion
Ven
cban Alghass
02
Lo payhiion
0.-
RiGhts
0.-
Aogir lith
Grpbercentage
exrrent
percentage
malemall
an Peetitaciin
UUAS: 0.00
Rook: False
True
Root:
Depth:
Rost Depth: 0.28
Root Depth: 0.42
0.5
-06
-2A
<02
25
(C) ELMO0
(D) ELMo1
(L) BERTBASE7
Figure 10: The curbs would cover all but small percentage of flgght,, and represent an expansion of the current ban on flights of less than two hours.
13

---

Published as a conference paper at ICLR 2021
q: Gh,( As for Poincaré probe, the equations are:
p: expo (GRU()))),))
q, = Q &c P:-
(8)
Table 5 reports the scores, which are generally higher than probes based upon linear map. We note both of Poincaré probe and Euclidean probe obtain much higher scores ELMo0, which indicates that these two probes can learn syntax trees, rather than probe from deep models.
Table 5: Results of training probes using local information for distance and depth tasks simultane- ously.
Euclidean w/ GRU
Hyperbolic w/ GRU
Distance
Depth
Distance
Depth Root %
Method
UUAS
DSpr.
Root %
NSpr.
UUAS
DSpr.
NSpr.
ELMo0 LINEAR
0.81
73.3
0.84
86.4
0.86
46.7
0.57
7.8
0.27
46.3
0.58
9.4
0.27
ELMol
87.8
0.91
95.6
0.93
87.4
0.91
96.2
0.93
BERTBASE7
79.3
0.87
93.0
0.92
86.8
0.91
96.0
0.92
B ADDITIONAL RESULTS ON PROBING SENTIMENT
Movie Review balanced sentiment analysis dataset with the same number of positive and negative sentences. Since there is no official split of this dataset, we randomly split 10% as dev and test set separately. The statistics can be found in T
Table 6: Statistics of Movie Review dataset.
Dataset
#Train
#Dev
#Test
Avg.
Max Len
Movie Review
8528
1067
1067
21
59
Table 8 is the extended version of Table 3] Since some neutral words can be equally close to the two meta embeddings, we reports top sentiment words ranked by the distance gap between two meta-embeddings.. All subwords (begin with "##"" and numbers are ignored.
More plots for sentences the Movie Review dev set are given (Figure 13 neutral words with dashed lines are omitted for clarity. We note that in Figure the positive part Poincaré probe correctly realizes "but" is turing to positive, but Euclidean Probe notes it slightly negative, which used most cases. A similar pattern can be found Figure [4] Figure [15 demonstrates the case that neither Poincaré probe Fuclidean probe correctly classify the sentence, they may be disturbed by the name entity.
5 16). Some
more distinctly produced by Poincaré probe than Euclidean probe. In Figure 13
C CLOSED-FORM FORMULAS OF MÖBIUS OPERATIONS
We restate the definitions of fundamental mathematical operations for the generalized Poincaré ball model. We refer readers Ganea et al. (2018) for more details
Möbius Addition
The Möbius addition for æ, y E D is defined as:
(1 + 20(x, y) + + cllyl ) + (1
xe y
+ y
(9)
+ 2c
httpp:///www
corne11 wwwmeeel.ddd/pepppeeeee wttph//peoppee/ppp/oooooeeeeeeee ottp:///e.../popoooooooooo/oeear
14

---

Published as a conference paper at ICLR 2021
Table Accuracy for probing on GLOVE and LINEAR.
Euclidean
Poincaré
LINEAR GLOVE
48.4
48.4
75.9
76.7
Table 8: Top 100 sentiment words by Euclidean and Poincaré probes.
POS
latch, horne, testify, birth, opened, landau, cultivation, bern, willingly, visit, cub, iced, meetings, awake, awakening. eddy, wryly, protective, fencing, clears, bail, levy, shy, doyle, belong. grips, regis, flames, initiation, macdonald, woolf, concludes, hostages, coco, elf, battista, darkly, intimate, closure, caine, ridley, beaches, pol, possession, para- normal, enhancing, gains, playground, bonds, alfonso, prom, pry. wonderland, cozy, warmth, axel, alex, duval, seal, cass, open, versa, openly, benign, stirred, parental, deeply, babics, fiercely, nick, ecstasy, lara, alexandre, fiery, serra, interiors, ile, intimacy, claw, jacob, merry. secretly, eerie, maternal, kidnapping. transitions, jacques, claude, hierarchy, vibrant, warmly, illumination, benoit, wolf, fed, operative, interior, interference
Euclidean
NEG
worthless, useless, frustrated, fee, inadequate, rejected, equipped, schedule, useful, outdated, discarded, equipment, pointless, sounded, weakened, undeveloped, received, ought, heaviest, conceived, recycled, hates, inactive, improper, muttering, pathetic, gee, unnecessary, humiliated, insignificant, timed, dated, sacrificed, bucks, drowned, inef- fective, tired, compiled, wasted, hurried, exaggerated. needed, rational, hired, maga- zine, eighties, junk, biased, garbage, shall, weakly, redundant, flopped, insulting, thou, lifeless, interchange, flatly, hampered, poorly, inability, styled, solution, lame, indiffer- ent, failed, rubbish, generie, accumulated, packaged, truncated, defeating, failure, trails, month, stale, deploy, could, bland, died, gymnastics, cared, graveyard. obsolete. unsuc- cessful, baked, suited, scripts, weathered, worst, amateur, disappointing, labelled, lazy, verse, chair, equation. directions, preliminary, calculated
POS
funky, lively, connects, merry, documented. vivid, dazzling, etched, infectious, relax- ing, evenly, robust, wonderful, volumes, capturing, splendid, floats, sturdy, immensely, potent, reflective, graphical, energetic, inviting, illuminated, vibrant, delightful, illumi- nating, rewarded, stirring, charting. bursting, absorbing. enhancing, admired, recreated, enjoyable, stellar, refreshing. richly, swung, woven, demonstration, captures, sterling, guarantees, glorious, emblem, reflected, tidy, encouraging, stunning, beautifully, reward, flond, gorgeous, charming, fiery, feast, radiant, clears, powerful, searing, shimmering, compassionate, brave, warm, finely, exquisite, successfully, eminent, superh, playful, ef- fortlessly, fiereely, neatly, evan, witty, confirming, candi, teasing, sincerely, excellent, piercing, lyrical, entertaining, reminder, enriched, beautiful, luminous, tremendous, en- gaging. arresting, irresistible, illustrates, fascinating, bracing, pleasant, exceptional, re- vive
Poincaré
NEG
inactive, stifled, dissatisfied, discarded, insignificant, insuflicient, erratic, indifferent, fades, wasting, arrogance, robotic, stil, trails, poorly, inadequate, disappointment, infe- rior, dissipated, useless, meaningless, fails, pathetic, lifeless, threw, flies, meek, flopped, intimidated, tainted, redundant, fee, failing, drained, prevents, hopeless, bland, disap- pointing, ruse, weighs, lacking. squash, hampered, fail, lazy, weaker, dripping, eroded, lame, neglect, failure, worthless, limp, nowhere, undermine, weighted, lax, washed, hol- low, flaw, failed, weak, ruined, inappropriate, boring, inability, mud, miserable, disgusted, uneven, rebellious, sloppy, tired, lacks, defeating, weakly, coarse, relegated, baked, col- lapses, stranded, dissolve, dull, pointless, unnecessary. rejected, scarcely, drain, ted, slug. idiots, unsuccessful, dazed, thinner, exhausted, clumsy, ill, tires, losing. foolish
Möbius Matrix-vector Multiplication
For linear map M IR" - R and Var E Di if Mæ * O, then the Möbius matrix-vector multiplication is defined as:
M - x (1/Vc) tanh
|Maa |
tanh ((Ivc=l1)
Mæ
(10)
|M
and M 0ff Mæ 0.
15

---

Published as a conference paper at ICLR 2021
Exponential and Logarithmic Maps exponential map expx (.) TeDx + D for v o is defined as:
Let TalD" denote the tangent space of D æ. The
exp - a Bc [aanh (vo (
(11)
2
Vdlm
As the inverse of exp (-), the logarithmic map log (.): D - Tx for + x is defined as:
2 VC
t y
log (y)
tanh
te yll)
(12)
x B y
D ADDITIONAL RESULTS ON EUCLIDEAN PROBE VARIATIONS
We extend Euclidean probes to two linear transforms with non-linearity in between, which would make it more fair comparing to Poincaré Probe. The additional results are reported in Table
Table 9: Results of training Euclidean probe variations for distance task.
Euclidean
Poincaré
Method
No non- non-linyy
ReLU
Sigmoid
Tanh
UUAS
18.7
21.5
22.0
22.2
19.9
RANDOM
DSpr.
0.39
0.41
0.40
0.41
0.40
UUAS
26.7
29.5
29.8
29.6
25.8
ELMo0
DSpr.
0.44
0.45
0.45
0.45
0.44
UUAS
48.3
48.5
47.8
48.2
45.7
LINEAR
DSpr.
0.57
0.58
0.57
0.57
0.58
UUAS
79.9
84.0
84.5
84.0
83.7
BERTBASE7
DSpr.
0.84
0.88
0.88
0.88
0.88
E ADDITIONAL RESULTS ON CURVATURE
To visualize the embedded trees directly in 2d hyperbolic spaces, train 2d probes with differ- ent curvatures for both distance and depth tasks simultaneously on BERTBASE7. The scores are reported in Table 10 The circles in (B), (C), (D) of Figure 17 18 19 show the boundary of the Poincaré models.
Table 10: Results of 2d probes with different curvatures for both distance and depth tasks BERT- BASE7.
Distance
Depth Root
Curvature
UUAS
DSpr
NSpr.
Euclidean
0
24.3
0.51
77.6
0.80
-0.1
38.7
0.68
80.6
0.83
Poincaré
00.5
39.1
0.69
80.8
0.83
-1
40.5
0.70
81.5
0.83
16

---

Published as a conference paper at ICLR 2021
Is this where private markets approving of Washington 's bashing of Wall Street
Is this case where private markets are approving of Washington bashing of Wall Street
Is this case where private markets are approving of Washington 's bashing of Wall Street
giee leep getammentt spon
up World
up /
Baceall dei
Radall det gistosestil
lou up with Francioo
World weekeal
Indeed the Dow Jones Transportation Average plunged 102.06 points its second-worst drop in history
Indeed the Dow Jones Transportation Average plunged 102.06 points its second-worst drop in history
Indeed Dow Jones Transportation Average plunged 102.06 points
second-worst drop history
Radrew canzn Delibe
mondee
Disive regeistoon
carin beldie
pary
rank mnedes
man
Setaiie agveres
Basiown regeration
canzin Deiblic
which
thidd
nends sasendes
(4)
17

---

Published as a conference paper at ICLR 2021
Ushikubo genoral manager the investment research department
Toyo Trust
Banking Co.
also optimistic
Ushikaba general manager the investment research depariment
Toyo
Banking Co.
was optimistic
Isa0 Ushikubo general manager the investment reseurch department
Toyo Trust
Banking Co.
also was optimistic
In the U.S. market the reoognition the Manpower name infinitely stronger than Blue Arrow
Mr. Fromstein said
In U.S. market the recognition the Manpower name infinitely stronger than Blue Arrow
Mr. Fromstein said
In the U.S. market the recognition of the Manpower inlinitely stronger than Blue Arrow
Mr. Fromstein suid
(6)
The MMJ has gone beter
stouted one vider about 2:15 London time the U.S. Major Maket Indés contact suddentt indioused turnabout
The MMI has better
stouted about 3:15 London
U.S. Major Madets lndes subdeill indisueed tutudona
gone shouted
Major connact suddenly indicased turtabout
18

---

Published as a conference paper at ICLR 2021
For the moment at least euphoria has replaced anxiety on Wall Street
For the moment least euphoria has replaced anxiety on Wall Street
For the moment least euphoria has replaced anxiety on Wall Street
We weat
down
point
minutes
right
before lunch
then after lunch went up 3/4 point
minutes
sand
We down
3/4 point
minutes
right before lunch
then after lunch
up 3/4 minutes said
We went down 3/4 point in 10 minutes right before lunch
then after lunch we went 3/4 point in 12 minutes be said
The percentage rates are calculated on 360-day year while the coupon-oquivalent yield is based on 365-day
The percentage rates are calculated on a 360-day year while the coupon-oquivalcon yield is based on 365-day year
19

---

Published as a conference paper at ICLR 2021
percentage calculated
360-day year
while the coupon-cquivalent yield based 365-day year
(10)
Mr. Farley followed similar pattern when be acquired Northwest Industries Inc. and then sold much of its assets
Mr. Farley followed similar pattern when he acquired Northwest Industries Inc. and then sold much of
Farley followed similar pattern when he aequired Northwest Industries Inc. and then much of assets
(11)
Among Big Board specialists cay Pull your meining specialists expected get tigher prices shares
Among Big Board specialists the was Pull your ofters meaning that specialists soon expected higher prices for thiir shares
Among Big Board specialists the was
Pull your offers
meaning that specialists soon espected to get higher prices for their shares
(12)
Figure 11: Minimum spanning trees resultant from predicted squared distances on BERTBASE7. Black edges are the gold parse, red edges are predicted by Euclidean probes, blue edges are predicted by Poincaré probes. The two probes primarily differs at long edges.
apt
1.5
s #ly
beautifully
pamed
0.5
shimmering
[POS]
beautifully
Pamed
INEGI Boesn
the
and CecimmerPOSS ge
worr
Santymee INEG
True: [N]
-1.0-
True: [N]
Gimed
groduction
Pred: [N]
-7.
5.
7.5
10.0
-1.5
-1.0
-0.5
8.0
0.5
10
1.5
(A) Euclidean probe: BERTRASE10
(B) Poincaré probe DERTBASENT
Figure 12: Aptly named, this shimmering, beautifully costumed and filmed production doesn't work for me.
20

---

Published as a conference paper at ICLR 2021
1.-
6-
Juify dbe
very
1.0
ol
POLY
[POS]
out
0-
ho
at
[POS]
witth Gareful
welcome
-05
[NEG]
-4-
NEGGgggeiinn pavie d #making
-100-
-8-
True:
True:
ottention
Pred: (P)
Pred: (P)
-1.5
-1.0
-0.5
0.5
1.0
(A) Euclidean probe: BERTBASEIO
(B) Poincaré probe BERTBASEIO
Figure 13: It ss traditional moviemaking all the way, but it's done with a lot of careful period attention as well as some very welcome wit.
8-
1.0
amusing
0.5
JPOSI
2-
amusing
NEGI
0-
[POS] mildly
-0.5
[NEG]
recommend
True: [N]
True: [N] Pred: [N]
-6-
but
ger
Pred: [N]
-1.5
-1.0
-0.5
0.5
1.
15
(A) Euclidean probe: BERTBASEI0
(B) Poincaré probe: BERTBASEIO
Figure 14: It's mildly amusing, but I certainly can't recommend it.
10
however
1.-
1.-
loking
whaa
0.5
origina
EPOSEO
esn
IPPSI
0.0
0-
NEG
INEG]
-05
original
-1.0-
True: [N]
True: [N]
-6-
Grama
ju
Pred: [P]
Prod: [P]
-1.
-5.0
-2.5
0.0
25
50
100
12.
15.0
-1.5
-1.0
-0.5
(A) Euclidean probe: BERTBASELO
(B) Poincaré probe: BERTBASEI0
Figure 15: An uplifting drama...whar what Antwone Fisher isn't I,',, however, is original.
21

---

Published as a conference paper at ICLR 2021
1.0
1.0
[NEG]
movie
an
[NEG]
this
0.0
awesome
an
PPSII
0.0
awful
thi
movie
PPSI]
-1.0
True:
-1.0
True: [N]
Pred: [P]
Prod: [N]
-1.5
-1.0
-1.0
-0.5
0.5
1.5
(A) Positive+
(B) Negative+
1.5
1.0
1.0
good
pot
this
[NEG]
0.0
awfful
extremel
NEG
movie
0.0
nean
[POS]
[POS]
movie
-0.5
0.5
-1.0
True: [N]
-1.0
True: [N]
Pred: [N]
Pred: [N]
-1.5
-1.
1.5
(C) Negative++
(D) Negation
1.5
1.5
0.5
0.5
like
that
[POS]
NEE
YOU
mogeln
never
[NEG]
carry
watching
PPSS
-0.5
-0.
True: [?]
True: [N]
Pred: [N]
Pred: [P]
-1.5
-1.5
1.5
-1.0
-0.5
0.0
0.3
1.0
1.5
1.5
-1.0
-0.5
0.0
0.b
1.0
1.5
(E) Ambiguous
(F) Satire
Figure 16: Additional special cases of lexically contextualization. (A) This is an awesome movie. (B) This is awful movie. (C) This is extremely awful movie. (D) I do not mean it is good movie. (E) have never seen a movie like that. (F) You will carry on watching?
22

---

Published as a conference paper at ICLR 2021
grOw
onlly
Expeesee
hhe
period
0.00
perio
-0.25
Non-interest
-0.50-
ULAS: 0.75
Root: True
the
-0.75
Carvature:
Root Depth: 0.05
only
Curvature -1
(A) Syntax tree
poriod
ah perood
Nov Infgrest
eneenest TOW
-65-
UJAS; 0.50
UUNS: 0.50 True
UUAS:
Root: True
Root: True
Roat Deptlt 0.08
Root Depth: 0.01
Curveture: 0.1
Buclidean
-1a
-15
60
35
(C) Curvature -0.5
(D) Curvature -0.1
(E) Euclidean
Figure 17: Non-interest expense grew only 4% in the period.
expeci
desei ford
Investors
Co.
bid
Muttor
spnect
here
0.00
Sgguar
Ford
Mittor
Corp.
too
-0.25
-0.50
UCAS: 0.61
Root: True
Generial
Motors
-0.5
Carvature: -1
Root Depth: 0.10
Jagua
(B) Curvature -1
(A) Syntax tree
Outeessslooa Soquar
Motoco ford
bortes
Cuinanl gel Sang
harg GTL expect divestoss
-05-
UCAS: 0.64
UUAS: 0.50
UUAS:
Rook:Tou Tuee
-3
Trum
Root: True
Roat Depdle
Root Depth: 0.07
-0.5
Curveture: 2..
Buclidean
-1a dis
60
35
(C) Curvature -0.5
(D) Curvature -0.1
(E) Euclidean
Figure 18: Investors here still expect Ford Motor or General Motors Corp. bid for Jaguar.
23

---

Published as a conference paper at ICLR 2021
banker
honyyoefoontact
mestmmntt
0.50
Hiter of sons hiin
Jnvestmeet
BMA
buyers
0.25
outho Hamesiiae Sugter
0.00
font Centonaaaoon nii
Alex
Inc.
hlee for
-0.2
-0.50
UCAS: 0.56
Sons
aunit
Root: True
Root Depth: 0.18 Carvature: -1
Llbe
10 -0:00
(B) Curvature
(A) Syntax tree
unit
ffyeers
1.5
Sons
posssble
idfthiffsed
contact ivveetentt Saahherr
nnvestmeni BMA
00
-05
UUAS: 0.56
UUAS Boot: Trus
Boot: True
Root Depth: 0.25
Curveture:
Dupili:
1.5
Buclidean
Rove Depth 0.47
69
(C) Curvature = -0.5
(D) Curvature -0.1
(E) Euclidean
Figure 19: BMA's investment banker, Alex. Brown & Sons Inc., has been authorized to contact possible buyers for the unit.
24


Reviews:
[{"review_id": "vkO5cKM4U60", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary:\n\nIn the same vein as Hewitt & Manning 2019, the authors present an extremely lightly parametrized “probe” model to determine the presence of syntactic structure in the embedding space of BERT models. While Hewitt & Manning examine the Euclidean distance between linearly transformed token embeddings and its correlation with parse tree distance and depth, this work examines a different distance function based on distances in hyperbolic space. They find that this distance measure, for an equivalent or lesser number of parameters, better reproduces the syntactic properties. This suggests that the BERT model may operate simply, but on a non-Euclidean manifold, in order to work with syntactic information.\n\n\n#### Reasons for score:\n\n\nThe paper presents a simple method to examine syntactic structure in contextual embedding models, as a nice extension of the seminal work on Euclidean probing, and should be directly of interest to the ICLR audience. The paper seems to be well-executed experimentally, with a variety of analyses.\n\nHowever, I would like to see the authors give more geometric rigor to their model and argument, and clarify the hypothesis which they are testing. There is a lot of appeal to intuition about hyperbolic space and its ability to encode trees, but it is not clear what that has to do with their proposed probe in a concrete way. \n\nThe paper would be greatly improved by a simple 1 to 3 dimensional example of their proposed probe and how it could discover an embedded submanifold with hyperbolic structure.\n\n#### Positives:\n\n- This is a mathematically (and superficially conceptually), simple generalization of the work of Hewitt & Manning, which offers improvements to and new avenues for exploration of these probing ideas.\n- The work convincingly shows that their probe offers better performance, especially in UUAS, compared to Euclidean probes, which is quite interesting.\n- Generally, I believe that there is a good idea here, but it would be much improved by additional mathematical rigor.\n\n#### Comments/Concerns:\n\n- It is hard to understand this paper without having read Hewitt & Manning, it could do with being a bit more self-contained. For example, the metrics like “DSpr”, etc, should be explained.\n\n- It is not clear what hypothesis is being tested here. What does it mean for the embedding space to have a hyperbolic structure in this sense? The Nash embedding theorem ensures the existence of an isometric embedding of d-1 dimensional hyperbolic space in d dimensional Euclidean space, but that embedding can hardly be accomplished in general by the simple linear transform + exponential map + moebius multiplication described here.\n\n- In this vein, there should be much more rigor as to describing what exactly the Poincaré probe is doing. The paper talks about mapping to “the tangent space”. What tangent space is this? Is it the tangent space of a Poincaré ball at the origin? Why is this a reasonable thing to do if the goal is to discover hyperbolic structure? When should this be expected to work?\n\n- An example in low dimensions with visuals of a hyperbolic submanifold, and a Poincaré probe interacting with it, should feature prominently.\n\n- Along these same lines, it would be very helpful for the paper to give an example of how the neural network might access this information on a hyperbolic submanifold, though this may be beyond the scope of the work.\n\n- In the introduction, the authors note the ability of hyperbolic space to encode exponentially larger volumes vs Euclidean space, but of course the Poincaré ball they are examining is embedded in Euclidean space.  What does this mean concretely in the context of the BERT transformer network?\n\n#### Minor comments:\n\n- middle of page 7, “RiemannianAdam” is missing the space ", "extra_scores": {}, "extra_text": {}}, {"review_id": "W_0oI5YMVT", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\n\nThis work examines some of the syntactic and semantic information present in contextual word embeddings by training probes for dependency parsing and sentiment classification. The probes take the form of a low-dimensional projection of the word embeddings, and obtain the dependency parse and sentiment by considering the distance between the pairs of embeddings or between the embeddings and a class embedding respectively.\n\nThis is closely related to the approach of (Hewitt and Manning, 2019), whose experimental setting this work reproduces, but the innovation here consists in using hyperbolic rather than Euclidean distances: the embedding space is identified with the tangent space at the origin and projected to the Poincare ball using its exponential map, allowing the probe to use the hyperbolic distance.\n\nThe hyperbolic version of the probes consistently out-perform the Euclidean one, which is encouraging, and the authors provide some useful visualizations of the learned projections. However, the experiments fail to account for one possibly relevant difference between their Hyperbolic and Euclidean setting. The paper is also at times difficult to follow on its own, as it relies a bit too much on cited work.\n\n## Clarity\n\nEven if the experimental setting is the same as (Hewitt and Manning, 2019), a quick summary would be welcome here: currently, the name of the parsing dataset is not even mentioned, nor are the meanings of the Spearman correlation metrics (the abbreviations are not self-explanatory).\n\nThere are also some open questions: for parsing, do you train a single model with both objectives? From reading the code, it looks like you alternate doing an epoch of each. Is that right? How does that compare to learning both individually? \n\n## Correctness\n\nMy main issue with the comparison lies in the second projection which the hyperbolic (but not the Euclidean) method uses, as defined in Equation (3)\n\nFirst, doing the projection in the hyperbolic space is redundant. Remember that the Mobius matrix-vector simply applies the logarithmic map at the origin followed by a linear transformation in the tangent space followed by the exponential map at the origin. Hence, Equations (2) and (3) could simply be summarized as:\n$$q_i = \\exp_0 (Q P h_i) $$\n\n(Same remark for the GRU in the appendix.)\n\nWhile this would be equivalent to having a single projection matrix if we could do global optimization, given the geometry of the loss function the hyperbolic setting with its additional parameter might be easier to optimize than the Euclidean one. In particular, this probably accounts for the difference in sentiment classification with fixed label embeddings.\n\nThankfully, the difference in the same task with learned embedding seems to indicate that there is more to the result, and I would consider increasing my score if all the results hold with either an additional linear transformation in the Euclidean setting or a single projection in the hyperbolic.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "PGvac6_6PWd", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes probes based on hyperbolic embedding spaces, and compares them to the behaviour of Euclidean probes from recent work. The main result is that these probes allow for better recovery of syntactic properties of sentences from contextualized word embeddings compared to context-independent ones, when comparing them to euclidean probes. Similar results are presented  on sentiment analysis, even though no results are presented for context-independent word embeddings.\n\nOn the whole the paper is well-written with great visualizations. My main concern is that the findings are not strong enough at this point. As per the introduction, the first main finding is that the results indicate the possibility that hyperbolic models help us construct more sensitive probes. But how do we know a probe is not too sensitive or not sensitive enough? E.g. looking at the results in table 1, how do we know that the differences in the scores when using contextualised embeddings are for a good reason? Some of the differences are small, especially in the case of the depth probe. Similarly, for the second finding about BERT might be encoding information in non-Euclidean way, how do we know this is the case? One way would be to somehow modify BERT and make it \"more Euclidean\", and then this would help strengthen the hypothesis. But now as it stands, it is interesting but rather speculative.\n\nSome other points:\n- Statistical significance would help in reporting the results as many differences are small in Table 1. Also confidence intervals in the sentence length experiment would be nice.\n- It would be better to explain the scores used in Table 1, instead of referring to Hewitt and Manning\n- For syntactic parsing it is argued that the probe shouldn't be a parser, but in the sentiment analysis the probe is a better parser than the baseline model used. Shouldn't the probe not be a good sentiment analysis model in itself?\n- In the ends of section 5, the referring labels for subfigures in figure 6 are off.\n\nPost-author response: I appreciate the extra experiment on GLoVe vs Linear on sentiment analysis, it is what I was asking for. I have raised my score in response to that, as well as the additional reporting on the results. The discussion in Bayesian terms was interesting, I think it would help. Nevertheless, I was thinking is it that it should be possible to construct embeddings that have known syntactic vs semantic properties. E.g. one could increase/decrease the context size, perhaps to extreme values in the case of models such as GLoVe. And then we could actually have a much stronger prior. If the paper is accepted, I think such an experiment would be very informative.", "extra_scores": {}, "extra_text": {}}, {"review_id": "4yFfqozK15s", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes probing BERT representations by projecting them into a Poincare subspace. The proposed approach is used to probe ELMO and BERT for both syntax and sentiment in comparison with the conventional Euclidean probes.\n\nI am ambivalent about this paper. On the positive side, I think that it is a quite solid work, with extensive experimentation, additional supporting results in the appendix, and an accompanying code that can be used to reproduce results and obtain additional visualizations. The paper is also well written and the authors are rigorous when discussing their results rather than trying to oversell. \n\nOn the negative side, I have some reservations about the relevance of this study. What do we learn from it? It is true that the Poincare probes obtain generally higher scores than the Euclidean probes, but it doesn't look like they lead to any new insight about how BERT works. If the message here is that Poincare probes are more appropriate than their Euclidean counterparts, I would have liked to see instances were Euclidean probes lead to erroneous or at least different conclusions when compared to Poincare probes. In the absence of that, we can expect that practitioners will stick with Euclidean probes given that they are simply easier to use.\n\nMoreover, I am not sure if the comparison with Euclidean probes is entirely fair. If my understanding is correct, the Poincare probes learn two linear transformations (P and Q), whereas Euclidean probes learn a single one. Unless I am missing something, it could be that the Poincare probes obtain higher scores simply because the transformation they are learning is more expressive, and not because of the underlying geometric space. In order to test this hypothesis, I think that the authors should try learning two linear transformations for Euclidean probes, with a non-linearity like ReLU in between.\n\nFinally, I feel that some of the analyses did not follow a systematic methodology and some of the interpretations seem subjective and possibly questionable. For instance, I don't see any clear difference between the Poincare and Euclidean probes when it comes to the sentence length (Figure 2C), except for very short sentences. For sentence length > 12 the curves look very similar to me, except that the absolute values for Poincare are higher. Similarly, the visualizations, although interesting, provide a rather anecdotal evidence, in particular since the examples seem to be cherry-picked.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a new method to probe contextualized word embeddings for syntax and sentiment properties using hyperbolic geometry. The paper is written well and relevant to the ICLR community. Reviewers highlight that the proposed Poincaré probe offers solid results, extensive experiments that support the benefits of the approach, and proposes a new approach to analyze the geometry of BERT models. The revised version clarified various concerns of the initial reviews and improved the manuscript (comparison to Euclidean probes, low dimensional examples, new results on edge length distributions etc.). Overall, the paper makes valuable contributions to probing contextualized word embeddings and the majority of reviewers and the AC support acceptance for its contributions. Please revise your paper to take feedback from reviewers after rebuttal into account (especially to further improve clarity and discussion of the method).
Decision: Accept (Poster)