Probing BERT in Hyperbolic Spaces

1 INTRODUCTION . Contextualized word representations with pretrained language models have significantly advanced NLP progress ( Peters et al. , 2018a ; Devlin et al. , 2019 ) . Previous works point out that abundant linguistic knowledge implicitly exists in these representations ( Belinkov et al. , 2017 ; Peters et al. , 2018b ; a ; Tenney et al. , 2019 ) . This paper is primarily inspired by Hewitt & Manning ( 2019 ) who propose a structural probe to recover dependency trees encoded under squared Euclidean distance in a syntactic subspace . Although being an implicit assumption , there is no strict evidence that the geometry of these syntactic subspaces should be Euclidean , especially under the fact that the Euclidean space has intrinsic difficulties for modeling trees ( Linial et al. , 1995 ) . We propose to impose and explore different inductive biases for modeling syntactic subspaces . The hyperbolic space , a special Riemannian space with constant negative curvature , is a good candidate because of its tree-likeness ( Nickel & Kiela , 2017 ; Sarkar , 2011 ) . We adopt a generalized Poincaré Ball , a special model of hyperbolic spaces , to construct a Poincaré probe for contextualized embeddings . Figure 1 ( A , B ) give an example of a tree embedded in the Poincaré ball and compare the Euclidean counterparts . Intuitively , the volume of a Poincaré ball grows exponentially with its radius , which is similar to the phenomenon that the number of nodes of a full tree grows exponentially with its depth . This would give “ enough space ” to embed the tree . In the meantime , the volume of the Euclidean ball grows polynomially and thus has less capacity to embed tree nodes . ∗Equal contribution . Work was done during an internship at Alibaba DAMO Academy . †Corresponding author . 1Our results can be reproduced at https : //github.com/FranxYao/PoincareProbe . Before going any further , it is crutial to differentiate a probe and supervised parser ( Hall Maudslay et al. , 2020 ) , and ask what makes a good probe . Ideally , a probe should correctly recover syntactic information intrinsically contained in the embeddings , rather than being a powerful parser by itself . So it is important that the probe should have restricted modeling power but still be sensitive enough to the existence of syntax . For embeddings without strong syntax information ( e.g. , randomly initialized word embeddings ) , a probe should not aim to assign high parsing scores ( because this would overestimate the existence of syntax ) , while a parser aims for high scores no matter how bad the input embeddings are . The quality of a probe is defined by its sensitivity to syntax . Our work of probing BERT in hyperbolic spaces is exploratory . As opposed to the Euclidean syntactic subspaces in Hewitt & Manning ( 2019 ) , we consider the Poincaré syntactic subspace , and show its effectiveness for recovering syntax . Figure 1 ( C ) gives an example of the reconstructed dependency tree embedded in the Poincaré ball . In our experiments , we highlight two important observations of our Poincaré probe : ( a ) it does not give higher parsing scores to baseline embeddings ( which have no syntactic information ) than Euclidean probes , meaning that it is not a better parser ; ( b ) it reveals higher parsing scores , especially for deeper trees , longer edges , and longer sentences , than the Euclidean probe with strictly restricted capacity . Observation ( b ) can be interpreted from two perspectives : ( 1 ) it indicates that the Poincaré probe might be more sensitive to the existence of deeper syntax ; ( 2 ) the structure of syntactic subspaces of BERT could be different than Euclidean , especially for deeper trees . Consequently , the Euclidean probe may underestimate the syntactic capability of BERT , and BERT may exhibit stronger modeling power for deeper syntax in some special metric space , in our case , a Poincaré ball . To best exploit the inductive bias for hierarchical structures of hyperbolic space , we generalize our Poincaré probe to sentiment analysis . We construct a Poincaré sentiment subspace by predicting sentiments of individual words using vector geometry ( Figure 1 D ) . We assume two meta representations for the positive and negative sentiments as the roots in the sentiment subspace . The stronger a word ’ s polarity is , the closer it locates to its corresponding meta embedding . In our experiments , with clearly different geometric properties , the Poincaré probe shows that BERT encodes sentiments for each word in a very fine-grained way . We further reveal how the localization of word embeddings may change according to lexically-controlled contextualization , i.e. , how different contexts would affect the geometric location of the embeddings in the sentiment subspace . In summary , we present an Poincaré probe to reveal hierarchical linguistic structures encoded in BERT . From a hyperbolic deep learning perspective , our results indicate the possibility of using Poincaré models for learning better representations of linguistic hierarchies . From a linguistic perspective , we reveal the geometric properties of linguistic hierarchies encoded in BERT and posit that BERT may encode linguistic information in special metric spaces that are not necessarily Euclidean . We demonstrate the effectiveness of our approach with extensive experiments and visualization . 2 RELATED WORK . Probing BERT Recently , there are increasing interests in finding linguistic information encoded in BERT ( Rogers et al. , 2020 ) . One typical line of work is the structural probes aiming to reveal how syntax trees are encoded geometrically in BERT embeddings ( Hewitt & Manning , 2019 ; Reif et al. , 2019 ) . Our Poincaré probe generally follows this line and exploits the geometric properties of hyperbolic spaces for modeling trees . Again , we note that the goal of syntactic probes is to find syntax trees with strictly limited capacity , i.e. , a probe should not be a parser ( Hewitt & Manning , 2019 ; Kim et al. , 2020 ) , and strictly follow this restriction in our experiments . Other probing tasks consider a variety of linguistic properties , including morphology ( Belinkov et al. , 2017 ) , word sense ( Reif et al. , 2019 ) , phrases ( Kim et al. , 2020 ) , semantic fragments ( Richardson et al. , 2020 ) , and other aspects of syntax and semantics ( Tenney et al. , 2019 ) . Our extended Poincaré probe for sentiment can be viewed as one typical semantic probe that reveals how BERT geometrically encodes word sentiments . Hyperbolic Deep Learning Recently , methods using hyperbolic geometry have been proposed for several NLP tasks due to its better inductive bias for capturing hierarchical information than Euclidean space . Poincaré embeddings ( Nickel & Kiela , 2017 ) and POINCARÉGLOVE ( Tifrea et al. , 2019 ) learn embeddings of hierarchies using Poincaré models and exhibit impressive results , especially in low dimension . These works show the advantages of hyperbolic geometry for modeling trees while we focus on using hyperbolic spaces for probing contextualized embeddings . To learn models in hyperbolic spaces , previous works combine the formalism of Möbius gyrovector spaces with the Riemannian geometry , derive hyperbolic versions of important mathematical operations such as Möbius matrix-vector multiplication , and use them to build hyperbolic neural networks ( Ganea et al. , 2018 ) . Riemannian adaptive optimization methods ( Bonnabel , 2013 ; Bécigneul & Ganea , 2019 ) are proposed for gradient-based optimization . Techniques in these works are used as the infrastructure in this work for training Poincaré probes . 3 POINCARÉ PROBE . We begin by reviewing the basics of Hyperbolic Geometry . We follow the notations from Ganea et al . ( 2018 ) . A generalized Poincaré ball is a typical model of hyperbolic space , denoted as ( Dnc , gDx ) for c > 0 , where Dnc = { x ∈ Rn | c‖x‖2 < 1 } is a Riemannian manifold , gDx = ( λcx ) 2In is the metric tensor , λcx = 2/ ( 1 − c‖x‖2 ) is the conformal factor and c is the negative curvature of the hyperbolic space . We will use the term hyperbolic and Poincaré interchangeably according to the context . Our Poincaré probe uses the standard Poincaré ball Dnc with c = 1 . The distance function for x , y ∈ Dnc is : dD ( x , y ) = ( 2/ √ c ) tanh−1 ( √ c‖ − x⊕c y‖ ) , ( 1 ) where⊕c denotes the Möbius addition , the hyperbolic version of the addition operator . Note that we recover the Euclidean space Rn when c → 0 . Additionally , we use M ⊗c x to denote the Möbius matrix-vector multiplication for a linear map M : Rn → Rm , which is the hyperbolic version of linear transforms . We use expcx ( · ) to denote the exponential map , which maps vectors in the tangent space ( in our case , a space projected from the BERT embedding space ) to the hyperbolic space . Their closed-form formulas are detailed in Appendix C. Our probes consist two simple linear maps P and Q that project BERT embeddings into a Poincaré syntactic/sentiment subspace . Formally , let M denote a pretrained language model that produces a sequence of distributed representations h1 : t given a sentence of t words w1 : t. We train a linear map P : Rn → Rk , n being the dimension of contextualized embeddings and k being the probe rank , that projects the distributed representations to the tangent space . Then the exponential map projects the tangent space to the hyperbolic space . In the hyperbolic space , we construct the Poincaré syntactic/sentiment subspace via another linear map Q : Rk → Rk . The equations are : pi = exp0 ( Phi ) ( 2 ) qi = Q⊗c pi ( 3 ) Here P maps the original BERT embedding space to the tangent space of the origin of the Poincaré ball . Then exp0 ( · ) maps the tangent space inside the Poincaré ball2 . Consequently , in equation 3 we use the Möbius matrix-vector multiplication as the linear transformation in the hyperbolic space3 . 4 PROBING SYNTAX . Following Hewitt & Manning ( 2019 ) , we aim to test if there exists a hyperbolic subspace transformed from the original BERT embedding space with simple parameterization where squared distances between embeddings or squared norms of embeddings approximate tree distances or node depths , respectively . The goal of the probe is to recover syntactic information intrinsically contained in the embeddings . To this end , a probe should not assign high parsing scores to baseline non-contextualized embeddings ( otherwise it would become a parser , rather than being a probe ) . So it is crucial for the probe to have restricted modeling power ( in our case , two linear transforms P and Q ) but still being sensitive enough for syntactic structures . We further test if the Poincaré probe is able to discover more syntactic information for deeper trees due to its intrinsic bias for modeling trees . Similar to Hewitt & Manning ( 2019 ) , we use the squared Poincaré distance to recreate tree distances between word pairs and the squared Poincaré distance to the origin to recreate the depth of a word : Ldistance = 1 t2 ∑ i , j∈ { 1 , ... , t } |dT ( wi , wj ) − dDn ( qi , qj ) 2| ( 4 ) Ldepth = 1 t ∑ i∈ { 1 , ... , t } |dD ( wi ) − dDn ( qi,0 ) 2| ( 5 ) where dT ( wi , wj ) denotes the distance between word i , j on their dependency tree , i.e. , number of edges linking word i to j and dD ( wi ) denotes the depth of word i in the dependency tree . For optimization , we use the Adam ( Kingma & Ba , 2014 ) initialized at learning rate 0.001 and train up to 40 epochs . We decay the learning rate and perform model selection based on the dev loss . 2The choice of tangent space at the origin , instead of other points , follows previous works ( Ganea et al. , 2018 ; Mathieu et al. , 2019 ) for its mathematical simplexity and optimization convenience . 3This transformation is theoretically redundent , we use it primarily for numerical stability during optimization . We further note that such optimization stability is still an open problem in hyperbolic deep learning ( Mathieu et al. , 2019 ) . We leave a detailed investigation to future work .

Summary: In the same vein as Hewitt & Manning 2019, the authors present an extremely lightly parametrized “probe” model to determine the presence of syntactic structure in the embedding space of BERT models. While Hewitt & Manning examine the Euclidean distance between linearly transformed token embeddings and its correlation with parse tree distance and depth, this work examines a different distance function based on distances in hyperbolic space. They find that this distance measure, for an equivalent or lesser number of parameters, better reproduces the syntactic properties. This suggests that the BERT model may operate simply, but on a non-Euclidean manifold, in order to work with syntactic information.