Regularly varying representation for sentence embedding

1 INTRODUCTION . Representing the meaning of natural language in a mathematically grounded way is a scientific challenge that has received increasing attention with the explosion of digital content and text data in the last decade . Relying on the richness of contents , several sentence embeddings have been proposed ( Peters et al . ( 2018 ) ; Radford et al . ( 2018 ) ; Devlin et al . ( 2018 ) ) with demonstrated efficiency for the considered tasks . These embeddings , learnt on massive datasets , are commonly used to perform downstream tasks such as classification or data generation . In a classification context , e.g . when the goal is to distinguish between positive and negative sentiments in user reviews , most classifiers are based on Empirical Risk Minimization ( ERM ) strategies and variants of it . However , nothing guarantees that such classifiers perform satisfactorily on the tails of the explanatory variables , i.e . in regions attached to rare events or even outside the domain of observed data . The present paper builds upon the methodological framework proposed by ( Jalalzai et al . ( 2018 ) ) performing classification in extreme regions with guarantees concerning the excess risk of the ERM classifier . Their approach relies on multivariate extreme value theory ( EVT ) and is valid under regularity assumptions concerning the multivariate distributional tail which appear to be an homogeneity property above large thresholds : the tail behaviour at infinity is similar to the behaviour of polynomial function near infinity ( see equation 1 below ) . To wit , it is necessary for the multivariate survival function of the considered random vector to be approximately homogeneous of degree −1 , up to appropriate marginal standardization . Although there is experimental evidence that their framework improves the performance of baseline classifiers in extreme regions on low dimensional data , there is no reason to assume that the previously mentioned text embeddings satisfy the required regularity assumptions . The aim of the present work is to apply Jalalzai et al . ( 2018 ) ’ s methodology to text data represented by state of the art embeddings . To achieve this , we train the algorithm to perform a transformation that allows it to map any text embedding X onto a random vector Z which satisfies the aforementioned regularity assumptions . The transformation is learnt by an adversarial strategy ( Goodfellow et al . ( 2016 ) ) . As a by-product , we obtain a novel data augmentation mechanism which takes advantage of the scale invariance properties of Z to generate synthetic sentences that keep invariant the attribute of the original sentence . The work detailed in this paper can be connected to other articles ( Socher et al . ( 2013 ) ; Guu et al . ( 2018 ) ; Siffer et al . ( 2017 ) ) , though our framework is different . Our contribution is twofold . We introduce ( i ) a novel classification algorithm ( Orthrus ) , taking as input an embedded sentence to map it onto a heavy-tailed vector in the latent space satisfying Equation 1 . ( ii ) a semi-supervised data augmentation method ( Hydra ) , which leverages the homothetic invariance property of the embedding learnt by Orthrus to produce new sentences with prescribed label . Label preserving data augmentation , which increases the number of training samples in machine learning systems , is an effective solution to data insufficiency and is an efficient pre-processing step in computer vision ( Wang & Perez ( 2017 ) ) . Due to the discrete nature of text , data augmentation is a much harder problem in natural language processing ( NLP ) than in computer vision . Current work in natural language ( NL ) dataset augmentation with label preservation mainly relies either on back-translation ( Shleifer ( 2019 ) ) , on synonym replacements ( Kobayashi ( 2018 ) ) , on slot filling ( Hou et al . ( 2018 ) ) or on the use of handcrafted heuristics ( Wei & Zou ( 2019 ) ) ( e.g swap , deletion , synonym replacements ) . Unlike other state of the art solutions , Hydra can generate multiple sentences with a considered sentiment using a unique input sentence . Hydra does not require any additional resources ( e.g . synonym dictionary ) , nor does it rely on hand-crafted heuristics , or use style transfer methods ( Hu et al . ( 2017 ) ; Colombo et al . ( 2019 ) ) . In this paper , we work with output vectors issued by BERT sentence embedding as inputs ( Devlin et al . ( 2018 ) ) . BERT is currently one of the state-of-the art algorithms for sentence embedding . We demonstrate on both synthetic and real datasets that the embedding learnt by Orthrus on top of BERT indeed follows a heavy-tail distribution , and that Orthrus is able to outperform a state-of-the art classifier built on BERT . On the dataset augmentation task , quantitative and qualitative experiments demonstrate the ability of Hydra to generate new sentences while preserving labels . The rest of this paper is organized as follows . Section 2 introduces the necessary background in multivariate extremes and adversarial learning . The methodology we propose is detailed at length in section 3 . Illustrative numerical experiments on both synthetic and real data are gathered in section 4 . Additional experimental results are available in the supplementary material . 2 BACKGROUND . 2.1 HEAVY TAILS AND REGULAR VARIATION . By definition , heavy-tail phenomena are those which are ruled by very large values , occurring with a far from negligible probability and with significant impact on the system under study . When the phenomenon of interest is described by the distribution of a univariate random variable , the theory of regularly varying functions provides the appropriate mathematical framework for the study of heavy-tailed distributions . For the sake of clarity and in order to introduce notations we recall some related theoretical background . One may refer to ( Resnick ( 2013 ) ) for an excellent account of the theory of regularly varying functions and its application to the study of heavy-tailed distributions . A random variable X with cumulative distribution function ( c.d.f . ) F is heavy-tailed of index α > 0 if and only if for any fixed x > 0 : nP { X/F−1 ( 1− 1/n ) > x } −−−−→ n→∞ x−α where F−1 ( u ) = inf { t : F ( t ) ≥ u } denotes F ’ s generalized inverse . Based on this characterization , the heavy-tail model is classically extended to the multivariate setup as follows . Consider now a d-dimensional random vector X = ( X ( 1 ) , . . . , X ( d ) ) taking its values in Rd+ . Then X is said to be heavy-tailed with tail index α > 0 if there exists a positive Radon measure µ on the punctured set [ 0 , ∞ ] d\ { 0 } and a function b ( t ) →∞ such that : tP { X/b ( t ) ∈ A } −−−→ t→∞ µ ( A ) ( 1 ) For any Borelian set A ⊂ [ 0 , ∞ ] d which is bounded away from 0 and such that the measure µ of the boundary ∂A is zero . In such a case , µ fulfills the homogeneity property µ ( tA ) = t−αµ ( A ) for all t > 0 and any set A in [ 0 , ∞ ] d\ { 0 } . Using the homogeneity property , one may show that µ can be decomposed into a radial component and an angular component Φ , which are independent from each other . Indeed , for all x = ( x1 , . . . , xd ) ∈ Rd , set R ( x ) = ‖x‖ Θ ( x ) = ( x1 R ( x ) , . . . , xd R ( x ) ) ∈ S , ( 2 ) where S is the positive orthant of the unit sphere in Rd for the chosen norm ‖.‖ . The choice of the norm is unimportant as all norms are equivalent in Rd . Then , for every B ⊂ S , we have : µ [ { x : R ( x ) /b ( t ) > t , Θ ( x ) ∈ B } ] = t−αΦ ( B ) , where Φ is referred to as the spectral or angular measure . 2.2 CLASSIFICATION IN EXTREME REGION . In the standard setup of binary classification , ( X , Y ) ∈ Rd+ × { −1 , 1 } is a random pair defined on a certain probability space with unknown joint probability distribution P . Now , equipped with the concepts introduced in the previous section we shall assume regular variation for both classes with the same tail index . Stated otherwise ( as formulated in ( Jalalzai et al . ( 2018 ) ) ) we shall make the assumption that : tP { X ∈ tA | Y = σ1 } → µσ ( A ) where σ ∈ { −1 , +1 } , A ⊂ Rd+ , 0 6∈ ∂A . If one considers ( X∞ , Y∞ ) a random pair defined by P { X∞ ∈ A , Y∞ = +1 } = limt→∞ , P { X ∈ tA , Y = +1 | ||X|| > t } , P { Y∞ = +1 } = limt→∞ P { Y = +1 | ||X|| > t } , then one can prove ( see Theorem 1 in Jalalzai et al . ( 2018 ) ) that the Bayes regression function η∞ ( x ) = P { Y∞ = 1|X∞ = x } for the limiting pair is constant along rays i.e . is a function of Θ ( x ) ∈ S only . The same is true of the Bayes classifier for the asymptotic pair . As a consequence , the optimal classifiers on extreme regions are based on indicator functions of truncated cones on the kind { ||x|| > r , Θ ( x ) B } , where B ⊂ S. As a consequence , the optimal classifiers on extreme regions are based on indicator functions of truncated cones on the kind { ‖x‖ > r , Θ ( x ) ∈ B } , where B ⊂ S ( see Figure 1 ) . This observation justifies an ERM strategy using the extreme points of the dataset to train an angular classifier . Let { ( Xi , Yi ) } ni=1 be n i.i.d copies of ( X , Y ) . By sorting the training observations by decreasing order of magnitude , we introduce X ( i ) ( with corresponding sorted label Y ( i ) ) the i-th order statistics , i.e . ||X ( 1 ) || ≥ . . . ≥ ||X ( n ) || . Let GS be a class of classifiers defined on the sphere S with finite VC dimension VGS < ∞ . Thus for any g ∈ GS and any θ ∈ S , g ( θ ) ∈ { −1 , 1 } . By extension , for any x ∈ Rd+ we define g ( x ) = g ( Θ ( x ) ) ∈ { −1 , 1 } . Let tτ be the quantile at level ( 1− τ ) of the r.v ||X|| where τ > 0 corresponds to a small fraction of extreme observations : P { ||X|| > tτ } = τ . Set k = bnτc and consider the empirical risk dedicated to the extreme samples : L̂k ( g ) = 1 k k∑ i=1 1 { Y ( i ) 6= g ( Θ ( X ( i ) ) ) } , which corresponds to the empirical version of Ltτ , the loss at level tτ . Now consider solutions to the minimization problem : min g∈GS L̂k ( g ) ( 3 ) Thence the authors of Jalalzai et al . ( 2018 ) provide guarantees concerning the classification risk in out-of-sample regions . Theorem 1 ( Jalalzai et al . ( 2018 ) ) Let ĝk be any solution of equation 3 . Recall k = bnτc . Then , for δ ∈ ( 0 , 1 ) , ∀n ≥ 1 , we have with probability larger than 1− δ : Ltτ ( ĝk ) − L∗tτ ≤ 1√ k ( √ 2 ( 1− τ ) log ( 2/δ ) + C √ VGS log ( 1/δ ) ) + 1 k ( 5 + 2 log ( 1/δ ) + √ log ( 1/δ ) ( C √ VGS + √ 2 ) ) + { inf g∈GS Ltτ ( g ) − L∗tτ } , where C is a constant independent from n , τ and δ .

Summary: This paper proposes a new embedding method for sentences that aims to preserve dilation invariance.  Much of the methodology is justified by results for extremal point classification under particular assumptions, and then the authors try and encourage these assumptions to be met via penalty terms introduced in their embeddings/augmentation models.  However, while the proposed methodology seems interesting/novel, it remains conceptually unclear why it should be superior to standard text classification methods (ie. exactly what assumptions are being exploited to improve performance and how exactly do those assumptions help should be made more explicit). In particular, why is dilation invariance even a good idea?