Year: 2019

Abstract:
The dominant approaches to sentence representation in natural language rely on learning embeddings on massive corpuses. The obtained embeddings have desirable properties such as compositionality and distance preservation (sentences with similar meanings have similar representations). In this paper, we develop a novel method for learning an embedding enjoying a dilation invariance property. We propose two algorithms: Orthrus, a classification algorithm, constrains the distribution of the embedded variable to be regularly varying, i.e. multivariate heavy-tail. and uses Extreme Value Theory (EVT) to tackle the classification task on two separate regions: the tail and the bulk. Hydra, a text generation algorithm for dataset augmentation, leverages the invariance property of the embedding learnt by Orthrus to generate coherent sentences with controllable attribute, e.g. positive or negative sentiment. Numerical experiments on synthetic and real text data demonstrate the relevance of the proposed framework.

heading: 1 INTRODUCTION
text: Representing the meaning of natural language in a mathematically grounded way is a scientific challenge that has received increasing attention with the explosion of digital content and text data in the last decade. Relying on the richness of contents, several sentence embeddings have been proposed (Peters et al. (2018); Radford et al. (2018); Devlin et al. (2018)) with demonstrated efficiency for the considered tasks. These embeddings, learnt on massive datasets, are commonly used to perform downstream tasks such as classification or data generation.
In a classification context, e.g. when the goal is to distinguish between positive and negative sentiments in user reviews, most classifiers are based on Empirical Risk Minimization (ERM) strategies and variants of it. However, nothing guarantees that such classifiers perform satisfactorily on the tails of the explanatory variables, i.e. in regions attached to rare events or even outside the domain of observed data. The present paper builds upon the methodological framework proposed by (Jalalzai et al. (2018)) performing classification in extreme regions with guarantees concerning the excess risk of the ERM classifier. Their approach relies on multivariate extreme value theory (EVT) and is valid under regularity assumptions concerning the multivariate distributional tail which appear to be an homogeneity property above large thresholds: the tail behaviour at infinity is similar to the behaviour of polynomial function near infinity (see equation 1 below). To wit, it is necessary for the multivariate survival function of the considered random vector to be approximately homogeneous of degree −1, up to appropriate marginal standardization. Although there is experimental evidence that their framework improves the performance of baseline classifiers in extreme regions on low dimensional data, there is no reason to assume that the previously mentioned text embeddings satisfy the required regularity assumptions. The aim of the present work is to apply Jalalzai et al. (2018)’s methodology to text data represented by state of the art embeddings. To achieve this, we train the algorithm to perform a transformation that allows it to map any text embedding X onto a random vector Z which satisfies the aforementioned regularity assumptions. The transformation is learnt by an adversarial strategy (Goodfellow et al. (2016)). As a by-product, we obtain a novel data augmentation mechanism which takes advantage of the scale invariance properties of Z to generate synthetic sentences that keep invariant the attribute of the original sentence.
The work detailed in this paper can be connected to other articles (Socher et al. (2013); Guu et al. (2018); Siffer et al. (2017)), though our framework is different. Our contribution is twofold. We
introduce (i) a novel classification algorithm (Orthrus), taking as input an embedded sentence to map it onto a heavy-tailed vector in the latent space satisfying Equation 1. (ii) a semi-supervised data augmentation method (Hydra), which leverages the homothetic invariance property of the embedding learnt by Orthrus to produce new sentences with prescribed label.
Label preserving data augmentation, which increases the number of training samples in machine learning systems, is an effective solution to data insufficiency and is an efficient pre-processing step in computer vision (Wang & Perez (2017)). Due to the discrete nature of text, data augmentation is a much harder problem in natural language processing (NLP) than in computer vision. Current work in natural language (NL) dataset augmentation with label preservation mainly relies either on back-translation (Shleifer (2019)), on synonym replacements (Kobayashi (2018)), on slot filling (Hou et al. (2018)) or on the use of handcrafted heuristics (Wei & Zou (2019)) (e.g swap, deletion, synonym replacements). Unlike other state of the art solutions, Hydra can generate multiple sentences with a considered sentiment using a unique input sentence. Hydra does not require any additional resources (e.g. synonym dictionary), nor does it rely on hand-crafted heuristics, or use style transfer methods (Hu et al. (2017); Colombo et al. (2019)).
In this paper, we work with output vectors issued by BERT sentence embedding as inputs (Devlin et al. (2018)). BERT is currently one of the state-of-the art algorithms for sentence embedding. We demonstrate on both synthetic and real datasets that the embedding learnt by Orthrus on top of BERT indeed follows a heavy-tail distribution, and that Orthrus is able to outperform a state-of-the art classifier built on BERT. On the dataset augmentation task, quantitative and qualitative experiments demonstrate the ability of Hydra to generate new sentences while preserving labels.
The rest of this paper is organized as follows. Section 2 introduces the necessary background in multivariate extremes and adversarial learning. The methodology we propose is detailed at length in section 3. Illustrative numerical experiments on both synthetic and real data are gathered in section 4. Additional experimental results are available in the supplementary material.

heading: 2 BACKGROUND

heading: 2.1 HEAVY TAILS AND REGULAR VARIATION
text: By definition, heavy-tail phenomena are those which are ruled by very large values, occurring with a far from negligible probability and with significant impact on the system under study. When the phenomenon of interest is described by the distribution of a univariate random variable, the theory of regularly varying functions provides the appropriate mathematical framework for the study of heavy-tailed distributions. For the sake of clarity and in order to introduce notations we recall some related theoretical background. One may refer to (Resnick (2013)) for an excellent account of the theory of regularly varying functions and its application to the study of heavy-tailed distributions. A random variable X with cumulative distribution function (c.d.f.) F is heavy-tailed of index α > 0 if and only if for any fixed x > 0:
nP { X/F−1(1− 1/n) > x } −−−−→ n→∞ x−α
where F−1(u) = inf{t : F (t) ≥ u} denotes F ’s generalized inverse. Based on this characterization, the heavy-tail model is classically extended to the multivariate setup as follows. Consider now a d-dimensional random vector X = (X(1), . . . , X(d)) taking its values in Rd+. Then X is said to be heavy-tailed with tail index α > 0 if there exists a positive Radon measure µ on the punctured set [0, ∞]d\{0} and a function b(t)→∞ such that:
tP {X/b(t) ∈ A} −−−→ t→∞ µ(A) (1)
For any Borelian set A ⊂ [0,∞]d which is bounded away from 0 and such that the measure µ of the boundary ∂A is zero. In such a case, µ fulfills the homogeneity property µ(tA) = t−αµ(A) for all t > 0 and any set A in [0,∞]d\{0}. Using the homogeneity property, one may show that µ can be decomposed into a radial component and an angular component Φ, which are independent from each other. Indeed, for all x = (x1, . . . , xd) ∈ Rd, set R(x) = ‖x‖ Θ(x) = ( x1 R(x) , . . . , xd R(x) ) ∈ S, (2)
where S is the positive orthant of the unit sphere in Rd for the chosen norm ‖.‖. The choice of the norm is unimportant as all norms are equivalent in Rd. Then, for every B ⊂ S, we have:
µ[{x : R(x)/b(t) > t,Θ(x) ∈ B}] = t−αΦ(B),
where Φ is referred to as the spectral or angular measure.

heading: 2.2 CLASSIFICATION IN EXTREME REGION
text: In the standard setup of binary classification, (X,Y ) ∈ Rd+ × {−1, 1} is a random pair defined on a certain probability space with unknown joint probability distribution P . Now, equipped with the concepts introduced in the previous section we shall assume regular variation for both classes with the same tail index. Stated otherwise (as formulated in (Jalalzai et al. (2018))) we shall make the assumption that:
tP {X ∈ tA | Y = σ1} → µσ(A)
where σ ∈ {−1,+1}, A ⊂ Rd+, 0 6∈ ∂A. If one considers (X∞, Y∞) a random pair defined by P {X∞ ∈ A, Y∞ = +1} = limt→∞, P {X ∈ tA, Y = +1 | ||X|| > t} , P {Y∞ = +1} = limt→∞ P {Y = +1 | ||X|| > t}, then one can prove (see Theorem 1 in Jalalzai et al. (2018)) that the Bayes regression function η∞(x) = P {Y∞ = 1|X∞ = x}for the limiting pair is constant along rays i.e. is a function of Θ(x) ∈ S only. The same is true of the Bayes classifier for the asymptotic pair. As a consequence, the optimal classifiers on extreme regions are based on indicator functions of truncated cones on the kind {||x|| > r,Θ(x)B}, where B ⊂ S. As a consequence, the optimal classifiers on extreme regions are based on indicator functions of truncated cones on the kind {‖x‖ > r,Θ(x) ∈ B}, where B ⊂ S (see Figure 1). This observation justifies an ERM strategy using the extreme points of the dataset to train an angular classifier.
Let {(Xi, Yi)}ni=1 be n i.i.d copies of (X,Y ). By sorting the training observations by decreasing order of magnitude, we introduce X(i) (with corresponding sorted label Y(i)) the i-th order statistics, i.e. ||X(1)|| ≥ . . . ≥ ||X(n)||. Let GS be a class of classifiers defined on the sphere S with finite VC dimension VGS <∞. Thus for any g ∈ GS and any θ ∈ S, g(θ) ∈ {−1, 1}. By extension, for any x ∈ Rd+ we define g(x) = g ( Θ(x) ) ∈ {−1, 1}.
Let tτ be the quantile at level (1− τ) of the r.v ||X|| where τ > 0 corresponds to a small fraction of extreme observations: P {||X|| > tτ} = τ . Set k = bnτc and consider the empirical risk dedicated to the extreme samples:
L̂k(g) = 1
k k∑ i=1 1{Y(i) 6= g(Θ(X(i)))},
which corresponds to the empirical version of Ltτ , the loss at level tτ . Now consider solutions to the minimization problem:
min g∈GS L̂k(g) (3)
Thence the authors of Jalalzai et al. (2018) provide guarantees concerning the classification risk in out-of-sample regions.
Theorem 1 (Jalalzai et al. (2018)) Let ĝk be any solution of equation 3. Recall k = bnτc. Then, for δ ∈ (0, 1), ∀n ≥ 1, we have with probability larger than 1− δ:
Ltτ (ĝk)− L∗tτ ≤ 1√ k
(√ 2(1− τ) log(2/δ) + C √ VGS log(1/δ) ) + 1
k
( 5 + 2 log(1/δ) + √ log(1/δ)(C √ VGS + √ 2) ) + { inf g∈GS Ltτ (g)− L∗tτ } ,
where C is a constant independent from n, τ and δ.

heading: 2.3 ADVERSARIAL LEARNING
text: Adversarial networks, introduced in (Goodfellow et al. (2014)), form in a system where two neural networks are competing. A first model called the generator generates samples as close as possible to the input dataset. A second model called the discriminator aims at distinguishing samples produced by the generator from the input dataset. The goal of the generator is to maximize the probability of the discriminator making a mistake. Hence, if Pinput is the distribution of the input dataset then the adversarial network intends to minimize the distance (as measured by the Jensen-Shannon divergence) between the distribution of the generated data PG and Pinput. Auto-encoders and derivations (Goodfellow et al. (2016); Laforgue et al. (2018); Fard et al. (2018)) shape a subclass of neural networks whose purpose is to learn a suitable representation by learning encoding and decoding functions which capture the core properties of the input data. An adversarial auto-encoders (see Makhzani et al. (2015)) is a specific kind of auto-encoder where the code plays the role of the generator of an adversarial network. Thus the latent code is forced to follow a given distribution while containing information relevant to reconstructing the input.

heading: 3 DILATION INVARIANT REPRESENTATION

heading: 3.1 LEARNING A DILATION INVARIANT REPRESENTATION
text: We now introduce Orthrus, a novel algorithm for classification of text data represented by high dimensional vectors as issued by pre-trained embeddings such as BERT which is assumed to not satisfy the regular variation assumption. The idea behind Orthrus is to modify the output X of BERT in order to increase the information carried by the resulting representation Z = ϕ(X) regarding the label Y , including in the upper tail regions of Z corresponding to low probability sentences, in order to improve the performance of a downstream classifier. This is achieved by training an encoding function ϕ in such a way that (i) the marginal distribution q(z) of the code Z be close to a user-specified heavy tailed distribution p satisfying the regularity property (1) ; and (ii) the classification loss of a multilayer perceptron trained on the code Z be small. A major difference distinguishing Orthrus from existing auto-encoding schemes is that the target distribution on the latent space is not chosen as a Gaussian distribution but as a heavy-tailed one.
As the Bayes classifier in the extreme region has a potentially different structure from the Bayes classifier on the bulk (recall from Section 2 that the regression function at infinity depends on the angle Θ(x) only), Orthrus trains two different classifiers, Cext on the extreme region of the latent space on the one hand, and Cbulk on its complementary set on the other hand. Given a high threshold t, the extreme region of the latent space is defined as the set {z : ‖z‖ > t}. In practice, the threshold t is chosen as an empirical quantile of order (1− κ) (for some small, fixed κ) of the norm of encoded data ‖Zi‖ = ‖ϕ(Xi)‖. In the end, the classifier issued by Orthrus is of the kind C(z) = Cext(z)1{‖z‖ > t}+ Cbulk(z)1{‖z‖ ≤ t} . Our goal is to minimize the weighted risk
R(ϕ,Cext, Cbulk) = λ1P { Y 6= Cext(Z), ‖Z‖ ≤ t } + λ2P { Y 6= Cbulk(Z), ‖Z‖ > t } + λ3D(q(z), p(z)),
(4)
where Z = ϕ(X), d is the Jensen-Shannon distance between the heavy tailed target distribution p and the code distribution q, and λ1, λ2, λ3 are positive weights. Following common practice in the adversarial literature, the Jensen-Shannon distance is approached (up to a constant term) by the
empirical proxy L̂(q, p) = supD∈Γ L̂(q, p,D)), with
L̂(q, p,D) = 1
m m∑ i=1 logD(Zi) + log(1−D(Z̃i))
where Γ is a wide class of discriminator functions, and where independent samples Zi, Z̃i are respectively sampled from the target distribution and the code distribution q. In the end, Orthrus solves the following min-max problem infCext,Cbulk,ϕ supD R̂(ϕ,C ext, Cbulk, D) where
R̂(ϕ,Cext, Cbulk) = λ1 k k∑ i=1 1{Y(i) 6= Cext(Z(i))}+ λ2 n− k n−k∑ i=1 1{Y(i) 6= Cbulk(Z(i))}+ · · ·
· · ·λ3 L̂(q, p) ) ,
where {Z(i) = ϕ(X(i)), i = 1, . . . , n} are the encoded observations with associated labels Y(i) sorted by decreasing magnitude of ‖Z‖ (i.e. ‖Z(1)‖ ≥ · · · ≥ ‖Z(n)‖), and k = bκnc is the number of extreme samples among the n encoded observations.
This minimization problem results in Algorithm 1 detailed below, where m is the batch size, ρ is the proportion of the points given to train the extreme classifier and the cost function ` is the negative log-likelihood.

heading: 3.2 A DILATION INVARIANT REPRESENTATION FOR DATASET AUGMENTATION
text: We now introduce Hydra, a data augmentation algorithm, which relies on the dilation invariance of labels on the extreme embeddings learnt by Orthrus. Let U = (u1, . . . , uT ) be a sequence of inputs of length T . The dataset augmentation problem consists in generating multiple sentences {U ′i}i∈N such that each sequence U ′i is label coherent with U . Hydra follows the seq2seq approach (Sutskever et al. (2014)) and learns two decoders with attention (Bahdanau et al. (2014)), one for each region of the representation previously learnt by Orthrus. On the whole, a decoder takes an input Z (the latent code) and generates an output sequence U ′i = (u ′ i1 , . . . , u′iT ′ ), where each word u ′ ik
is in the vocabulary. To generate an output word u′k, the decoder iteratively takes as input the previously generated word u′k−1 (u ′ 0 being a start symbol), updates its internal state, and returns the next word with highest probability. This process is repeated until the decoder generates either a stop symbol or the length of the generated sequence reaches the maximum sequence length.
Hydra is a semi-supervised algorithm in that it requires a first labeled dataset Dn to learn embeddings through Orthrus and a second dataset Dgn (not necessarily labeled) to train a decoder Gext (resp Gbulk) for the extremes (resp non extremes) on Dgn . The learning is carried out by optimising the classical negative log likelihood of individual tokens `gen.A detailed description of Hydra is provided in Algorithm 2 below.
After Hydra has been trained, the data augmentation proceeds as follows: For each input X with extreme embedding, multiple sentences with constant label are obtained by applying a λ dilation (see Figure 2b). In other words, if Gext is a decoder learnt by Hydra and applicable to the extreme region of the representation, synthetic sentences of the form Gext(λz) can be obtained, with the same attribute as z.
Algorithm 1 Orthrus INPUT: Coef. λ1, λ2, λ3 > 0, Training dataset Dn = {(U1, Y1), . . . , (Un, Yn)}, proportion κ ≤ 1 of extreme observations, batch size m ≤ n. Initialization: parameters of the encoder ϕτ , classifiers, Cextθ , C bulk θ′ , decoders and discriminator Dγ
while (τ, θ, θ′) not converged do Sample {(U1, Y1) . . . , (Um, Ym)} from Dn. Sample {Z1, . . . , Zm} from the prior PZ . Sample Z̃i from ϕτ (Z|Xi) for i ∈ {1, . . . ,m}.
Update Dγ by ascending:
λ3 m m∑ i=1 logDγ(Zi) + log(1−Dγ(Z̃i)).
Sort {Z̃i}i∈{1,...,m} by decreasing order of magnitude ||Z̃(1)|| ≥ . . . ≥ ||Z̃(m)||. Update Cextθ by descending:
Lext def= λ1bκmc bκmc∑ i=1 ` ( Y(i), C ext θ (Z̃(i)) ) .
Update Cbulkθ′ by descending:
Lbulk def= λ2 m− bκmc m∑ i=bκmc+1 ` ( Y(i), C bulk θ′ (Z̃(i)) ) .
Update ϕτ by descending:
1
m m∑ i=1 −λ3 · logDγ(Z̃i) + Lext + Lbulk.
end while Compute {Z̃i}i∈{1,...,n} = ϕ(Xi)i∈{1,...,n} Sort {Z̃i}i∈{1,...,n} by decreasing order of magnitude ||Z̃(1)|| ≥ . . . ||Z̃(bκnc)|| ≥ . . . ≥ ||Z̃(n)||. OUTPUT: encoder ϕ, classifiers Cextθ applicable on the region {x : ||ϕ(x)|| ≥ ||Z̃(bκnc)||} and Cbulkθ applicable on the region {x : ||ϕ(x) ) || <
||Z̃(bκnc)||}.
Algorithm 2 Hydra
INPUT: Coef. λ1, λ2, λ3 > 0, Training datasets Dn = {(U1, Y1), . . . , (Un, Yn)}, Dgn = {Ug1 , . . . , Ugn} batch size m ≤ n,
Initialization: parameters of the encoder ϕ, classifiers, Cextθ , C bulk θ′ , decoders, G ext ψ , G bulk ψ′ and dis-
criminator Dγ
Optimization: ϕ, Cext, Cbulk = Orthrus(λ3, Dn,) while (ψ,ψ′) not converged do Sample {(Xg1) . . . , (Xgm)} from the training set Dgn . Sample Z̃i from ϕ(Z|Xgi) for i ∈ {1, . . . ,m}.
Sort {Z̃i}i∈{1,...,m} by decreasing order of magnitude
||Z̃(1)|| ≥ . . . ≥ ||Z̃(m)||.
Update Gextψ by descending:
Lextg def = λ1 bρmc bρmc∑ i=1 `gen. ( Y(i), G ext ψ (Z̃(i)) ) .
Update Gbulkψ′ by descending:
Lbulk def= λ2 m− bρmc m∑ i=bρmc+1 `gen. ( Y(i), G bulk ψ′ (Z̃(i)) ) .
end while Compute {Z̃i}i∈{1,...,n} = ϕ(Xi)i∈{1,...,n} Sort {Z̃i}i∈{1,...,n} by decreasing order of magnitude ||Z̃(1)|| ≥ . . . ||Z̃(k)|| ≥ . . . ≥ ||Z̃(n)||.
OUTPUT: encoder ϕ, decoder Gext applicable on the region {x : ||ϕ(x)|| ≥ ||Z̃(bκnc)||} and Gbulk applicable on the region {x : ||ϕ(x)|| < ||Z̃(bκnc)||}.
Algorithm: Orthrus and Hydra. For both algorithm X is the embedding provided by BERT to an input sentence U .

heading: 4 NUMERICAL EXPERIMENTS
text: Heavy-tail distribution: In the remaining of this paper, the regularly varying target distribution is a logistic distribution with parameter δ = 0.9 (see Appendix B.2). The logistic distribution is widely used in the context of extreme values analysis. This distribution is simulated according to Stephenson (2003) which is defined in Rd with parameter δ ∈ (0, 1] by its parametric c.d.f. F (x) = exp { − ( ∑d j=1 x (j) 1δ )δ }
. Figure 4 in Appendix illustrates this distribution with various values of δ. Experimental Settings: Classifiers Cbulk, Cext are Multi Layer Perceptrons (MLP). For each dataset, one-fourth of data is retained as a test set while the remaining data points are used as the train set. We denote by T the extreme test set region as selected with Algorithm 1. The considered norm is the `∞ norm. For the generation task, the dilation coefficient varies between 1 and 1.5 (λ ∈ [1, 1.5]).

heading: 4.1 APPLICATION TO SENTIMENT ANALYSIS
text: We assess Othrus ability to perform sentiment classification tasks using two balanced binary labeled datasets: Yelp, Amazon introduced by (Kotzias et al. (2015)). Further results on the datasets are reported in the Appendix. Each dataset is balanced and is composed of 1000 sentences and each sentence expresses a strong positive or negative sentiment. Baseline: Orthrus is compared with a state of the art classifier: a MLP trained on the initial BERT embedding (NN ) taking as input the same training data. Classification performance: Table 1 illustrates the performance of considered models in terms of Hamming loss on the test set. Performance of Orthrus is better than NN on both dataset on the whole test set. The difference in terms of performance is even greater when considering the extreme samples.

heading: 4.2 APPLICATION TO DATASET AUGMENTATION FOR LABELLED SENTENCES
text: Comparison to related work: We compare Hydra with two state of the art methods for dataset augmentation: EDA (Wei & Zou (2019)) and round-trip translation (Shleifer (2019)). Compared to EDA which uses heuristics and a synonym dictionary, Hydra generates grammatically correct sentences and conserve the labels. Compared to round-trip translation Hydra can generate multiple sentences for a single input and conserves sentence labels. We also compare Hydra to a Vanilla Sequence to Sequence to demonstrate the validity of our approach. In this section, we validate Orthrus using the Yelp dataset. To train Orthrus the dataset Dn is the Yelp dataset introduced in (Kotzias et al. (2015)). Dgn is a subset of the full Yelp Corpus composed of 5 · 105 sentences. Evaluation: Automatic evaluation of generative models for text is still an open research problem. We evaluate our models through three criteria (C1, C2, C3). C1 measures Cohesion (Crossley & McNamara (2010)) (Are the generated sentences grammatically and semantically consistent?). C2 (named Sent. in Table 1a.) evaluates label conservation (Does the expressed sentiment in the generated sentence match the sentiment of the input sentence?). C3 measures the diversity of the sentences (Is the dataset augmented with diverse sentences?)
We carry out two types of evaluations: a qualitative one based on a user study – that provides evaluation through C1 and C2 – and a quantitative one – that provides evaluation through C3 measures. We measure diversity (C3) with two automatic methods: (1) Distinct n, the diversity measure introduced by (Li et al. (2015)) which relies on a n-grams count; (2) a F1 score that reports the improvement induced by the data augmentation while training a fastText (Joulin et al. (2016)) classifier. This improvement reflects the information added by newly generated sentences.
Qualitative Evaluation: We compare the different methods on the extreme set of sentences according to Hydra. As previously, the extreme test set T consists of 92 sentences. For each sentence, we randomly sample λ ∈ [1, 1.5] and generate a new sentence with Hydra. To compare our method, we use Eda, round-trip translation and a Vanilla Seq2seq to generate new sentences. Results of the qualitative evaluation are reported in Table 2a. As expected Hydra exhibits comparable performance with all other models except Eda in terms of cohesion. Such a result is expected since Eda’s use of random swaps or permutations may result in ungrammatical sentences. In terms of expressed sentiments, Hydra outperforms Seq2seq and round-trip translation, demonstrating the relevance of the proposed approach.
Quantitative Evaluation: We report in (Table 2b) the results of evaluation for C3. We observe that Hydra outperforms all other methods in term of distinct 1 and distinct 2. Table 1b shows that improvement in F1 score induced by dataset augmentation by Hydra beats all other methods and is only equalled by EDA.
Example of generated sentences: We report in Table 3 sentences generated by Hydra. While we make no claim that these generated sentences are better than sentences generated by existing methods, we believe that these samples are at least competitive with the generative models in the literature and highlight the potential of the introduced framework.

heading: 5 DISCUSSION & PERSPECTIVES
text: The approach promoted in this paper relies on learning a regularly varying representation by minimizing an empirical proxy of an objective function. The latter writes as a weighted sum of a classification risk and a regularization term penalizing the distance between the representation distribution and a heavy-tailed target. Our experiments show that the obtained representation allows to diminish the classification error compared to a MLP with comparable complexity.The obtained representation used for a text data augmentation task, is competitive with existing data augmentation methods. The attribute invariance under code dilation is the key to generate meaningful sentences with prescribed attribute. Future work will investigate the possibility to train an auto-encoder satisfying this dilation invariance property without relying on a pretrained classifier such as Orthrus.

heading: A REMARKS
text: Remark 1 (Selection of k) To the best of our knowledge, selection of k in Algorithm 1 and Algorithm 2 is still a vivid problem in EVT which is not solved yet. As k gets large the number of extreme points increases including samples which are not large enough and deviates from the asymptotic distribution of extremes. Smaller values of k increase the variance of the classifier/generator. This bias-variance trade-off is beyond the scope of this paper.
Remark 2 In Figure 3a selecting the extreme samples on the input space is not a straightforward step since the tails of the input clusters are not heavy-tail. It was decided to standardise the input data (Xi)i∈{1,··· ,n} by applying the rank-transformation:
T̂ (x) = ( 1/ ( 1− F̂j(x) ))
j=1,··· ,d
for all x = (x1, · · ·xd) ∈ Rd where F̂j def = 1n+1 ∑n i=1 1{X j i ≤ x} is the jth empirical marginal distribution. It comes ∀i ∈ {1, · · · , n}, Vi = T̂ (|Xi|). Vi margins follow a standard Pareto distribution. Extreme samples are thus {Vi, ||Vi|| ≥ n/k} with k = √ n by convention. One may notice that extremes in Figure 3c are more label balanced compared to Figure 3a.

heading: B EXPERIMENTS
text: Further synthetic experiments were conducted to show the improvement of the classifier.

heading: B.1 SYNTHETIC DATA
text: Dataset description: Synthetic data was generated using Scikit-Learn (Pedregosa et al. (2011)). Focus is made on the classification dataset where clusters of points are normally distributed for each class (see Figure 3a). Convergence to a logistic: In Figure 3b we visualise the latent space of the output of the encoder ϕ (see Figure 2). In Orthrus, the adversarial approach to minimize the discrepancy between the distribution of the output of the encoder and a logistic distribution. Notice the similarity between Figure 3b and Figure 4a. Extremes selection: Extreme samples from Figure 3a are assessed directly on the input data while extremes from Figure 3c are obtained from the latent space. Extremes from the latent space (output of ϕ after the training phase) are located on the border of the distribution which comforts the correct behaviour of the training. We also note similarities while comparing in the input space: extreme samples selected in the input space and extreme ones selected in the learnt latent space. In Figure 3a no extreme observation appear on the negative class whereas labels of extremes of Figure 3c are balanced.

heading: B.2 LOGISTIC DISTRIBUTION
text: Figure 4 illustrates the logistic distribution with varying parameter δ.

heading: B.3 ADDITIONAL EXPERIMENT SETTINGS FOR REAL DATA
text: We use BERT pretrained models and code from the library pytorch-transformers. All models were implemented using Pytorch. The output of BERT is a R768 vector, Z is a R50 vector. The dimension of Z (50) was obtained by cross validation. D, Cbulk and Cext are MLP composed of 3 layers and uses a ReLu as activation function. Decoders Dbulk and Dext are GRU composed of 6 layers. For training the batch size is set to 64 and we set λ1 = (1 − P̂(||Z|| ≥ ||Zbκnc||)−1,λ2 = P̂(||Z|| ≥ ||Zbκnc||)−1,λ3 = 10−3. All neural networks have been optimized with Adam (Kingma & Ba (2014)). For generation the maximum dilation factor λ is set to 1.5 since we observe that with greater values a stuttering phenomenon appears.

heading: B.4 ADDITIONAL EXPERIMENTS ON YELP DATA
text: Dilation invariance of Cext : Figure 5a shows the evolution of the prediction for each sample of the test set applying the dilation. Predictions of Cbulk change for some samples resulting in bichromatic columns. Figure 5b focuses on the samples from T and is exclusively composed with monochromatic columns demonstrating that the predicted labels from Cext do not change, Cext is homothetic invariant. With no assumption on Cext, this classifier enjoys favorable theoretical properties since it belongs to the class of angular classifiers at the end of the training phase.

heading: B.5 ADDITIONAL EXPERIMENTS ON AMAZON DATA
text: Experiments on the Amazon dataset were conducted similarly to to the ones from the Yelp dataset from subsection 4.1, similar conclusions can be drawn from Figure 6.

heading: B.6 QUALITATIVE EVALUATION
text: For the qualitative evaluation, we compute the Krippendorff’s alpha coefficient Krippendorff (2018) to evaluate the inter-annotator agreement. We obtain α = 0.41 for the grammar analysis and α = 0.53 for the sentiment analysis. If α = 1 it indicates perfect reliability; if α = 0 it indicates the absence of reliability. Such values of α can be considered as relatively high given our huge number of annotators (126 for the grammar and 134 for the sentiment analysis).

heading: B.7 ADDITIONAL GENERATED SENTENCES
text: We present in Table 4 additional extreme sentences.

heading: C A LINGUISTIC POINT OF VIEW OF EXTREMES
text: In this section, we provide a comparative linguistic analysis of the extreme samples in case of sentiment classification through the affect scores provided by the lexicon for English words from
Mohammad (2017). To better understand the content of extreme and non extreme text samples in experiments conducted in section 4, the maximum and the minimum of the affect score for each sentence are studied (see Figure 7). For both Yelp and Amazon datasets, one can notice that the violin plot of the maximum of the affect score is wider among the extremes than in the bulk. This illustrates that extreme tend to have both a more positive and a more negative affect score than the non extreme observations. Same goes for the minimum affect score of extremes whose violin plot tends to be larger among the extremes compared to the violin plot of the minimum affect score on the bulk.

Review summaries:
This paper proposes a new embedding method for sentences that aims to preserve dilation invariance.  Much of the methodology is justified by results for extremal point classification under particular assumptions, and then the authors try and encourage these assumptions to be met via penalty terms introduced in their embeddings/augmentation models.  However, while the proposed methodology seems interesting/novel, it remains conceptually unclear why it should be superior to standard text classification methods (ie. exactly what assumptions are being exploited to improve performance and how exactly do those assumptions help should be made more explicit). In particular, why is dilation invariance even a good idea?