Title: REGULARLY VARYING REPRESENTATION FOR SEN-TENCE EMBEDDING

Abstract: The dominant approaches to sentence representation in natural language rely on learning embeddings on massive corpuses. The obtained embeddings have desirable properties such as compositionality and distance preservation (sentences with similar meanings have similar representations). In this paper, we develop a novel method for learning an embedding enjoying a dilation invariance property. We propose two algorithms: Orthrus, a classification algorithm, constrains the distribution of the embedded variable to be regularly varying, i.e. multivariate heavy-tail. and uses Extreme Value Theory (EVT) to tackle the classification task on two separate regions: the tail and the bulk. Hydra, a text generation algorithm for dataset augmentation, leverages the invariance property of the embedding learnt by Orthrus to generate coherent sentences with controllable attribute, e.g. positive or negative sentiment. Numerical experiments on synthetic and real text data demonstrate the relevance of the proposed framework.

1. INTRODUCTION:

Representing the meaning of natural language in a mathematically grounded way is a scientific challenge that has received increasing attention with the explosion of digital content and text data in the last decade. Relying on the richness of contents, several sentence embeddings have been proposed (Peters et al. (2018) ; Radford et al. (2018) ; Devlin et al. (2018) ) with demonstrated efficiency for the considered tasks. These embeddings, learnt on massive datasets, are commonly used to perform downstream tasks such as classification or data generation.

In a classification context, e.g. when the goal is to distinguish between positive and negative sentiments in user reviews, most classifiers are based on Empirical Risk Minimization (ERM) strategies and variants of it. However, nothing guarantees that such classifiers perform satisfactorily on the tails of the explanatory variables, i.e. in regions attached to rare events or even outside the domain of observed data. The present paper builds upon the methodological framework proposed by (Jalalzai et al. (2018) ) performing classification in extreme regions with guarantees concerning the excess risk of the ERM classifier. Their approach relies on multivariate extreme value theory (EVT) and is valid under regularity assumptions concerning the multivariate distributional tail which appear to be an homogeneity property above large thresholds: the tail behaviour at infinity is similar to the behaviour of polynomial function near infinity (see equation 1 below). To wit, it is necessary for the multivariate survival function of the considered random vector to be approximately homogeneous of degree -1, up to appropriate marginal standardization. Although there is experimental evidence that their framework improves the performance of baseline classifiers in extreme regions on low dimensional data, there is no reason to assume that the previously mentioned text embeddings satisfy the required regularity assumptions. The aim of the present work is to apply Jalalzai et al. (2018) 's methodology to text data represented by state of the art embeddings. To achieve this, we train the algorithm to perform a transformation that allows it to map any text embedding X onto a random vector Z which satisfies the aforementioned regularity assumptions. The transformation is learnt by an adversarial strategy (Goodfellow et al. (2016) ). As a by-product, we obtain a novel data augmentation mechanism which takes advantage of the scale invariance properties of Z to generate synthetic sentences that keep invariant the attribute of the original sentence.

The work detailed in this paper can be connected to other articles (Socher et al. (2013) ; Guu et al. (2018) ; Siffer et al. (2017) ), though our framework is different. Our contribution is twofold. We introduce (i) a novel classification algorithm (Orthrus), taking as input an embedded sentence to map it onto a heavy-tailed vector in the latent space satisfying Equation 1. (ii) a semi-supervised data augmentation method (Hydra), which leverages the homothetic invariance property of the embedding learnt by Orthrus to produce new sentences with prescribed label. Label preserving data augmentation, which increases the number of training samples in machine learning systems, is an effective solution to data insufficiency and is an efficient pre-processing step in computer vision (Wang & Perez (2017) ). Due to the discrete nature of text, data augmentation is a much harder problem in natural language processing (NLP) than in computer vision. Current work in natural language (NL) dataset augmentation with label preservation mainly relies either on back-translation (Shleifer (2019) ), on synonym replacements (Kobayashi (2018) ), on slot filling (Hou et al. (2018) ) or on the use of handcrafted heuristics (Wei & Zou (2019) ) (e.g swap, deletion, synonym replacements). Unlike other state of the art solutions, Hydra can generate multiple sentences with a considered sentiment using a unique input sentence. Hydra does not require any additional resources (e.g. synonym dictionary), nor does it rely on hand-crafted heuristics, or use style transfer methods (Hu et al. (2017) ; Colombo et al. (2019) ).

In this paper, we work with output vectors issued by BERT sentence embedding as inputs (Devlin et al. (2018)) . BERT is currently one of the state-of-the art algorithms for sentence embedding. We demonstrate on both synthetic and real datasets that the embedding learnt by Orthrus on top of BERT indeed follows a heavy-tail distribution, and that Orthrus is able to outperform a state-of-the art classifier built on BERT. On the dataset augmentation task, quantitative and qualitative experiments demonstrate the ability of Hydra to generate new sentences while preserving labels.

The rest of this paper is organized as follows. Section 2 introduces the necessary background in multivariate extremes and adversarial learning. The methodology we propose is detailed at length in section 3. Illustrative numerical experiments on both synthetic and real data are gathered in section 4. Additional experimental results are available in the supplementary material.

2. BACKGROUND:



2.1. HEAVY TAILS AND REGULAR VARIATION:

By definition, heavy-tail phenomena are those which are ruled by very large values, occurring with a far from negligible probability and with significant impact on the system under study. When the phenomenon of interest is described by the distribution of a univariate random variable, the theory of regularly varying functions provides the appropriate mathematical framework for the study of heavy-tailed distributions. For the sake of clarity and in order to introduce notations we recall some related theoretical background. One may refer to (Resnick (2013) ) for an excellent account of the theory of regularly varying functions and its application to the study of heavy-tailed distributions. A random variable X with cumulative distribution function (c.d.f.) F is heavy-tailed of index α > 0 if and only if for any fixed x > 0:

nP X/F -1 (1 -1/n) > x ----→ n→∞ x -α

where F -1 (u) = inf{t : F (t) ≥ u} denotes F 's generalized inverse. Based on this characterization, the heavy-tail model is classically extended to the multivariate setup as follows. Consider now a d-dimensional random vector X = (X (1) , . . . , X (d) ) taking its values in R d + . Then X is said to be heavy-tailed with tail index α > 0 if there exists a positive Radon measure µ on the punctured set [0, ∞] d \{0} and a function b(t) → ∞ such that:

EQUATION

For any Borelian set A ⊂ [0, ∞] d which is bounded away from 0 and such that the measure µ of the boundary ∂A is zero. In such a case, µ fulfills the homogeneity property µ(tA) = t -α µ(A) for all t > 0 and any set A in [0, ∞] d \{0}. Using the homogeneity property, one may show that µ can be decomposed into a radial component and an angular component Φ, which are independent from each other. Indeed, for all x = (x 1 , . . . , where S is the positive orthant of the unit sphere in R d for the chosen norm . . The choice of the norm is unimportant as all norms are equivalent in R d . Then, for every B ⊂ S, we have:

EQUATION

µ[{x : R(x)/b(t) > t, Θ(x) ∈ B}] = t -α Φ(B),

where Φ is referred to as the spectral or angular measure.

2.2. CLASSIFICATION IN EXTREME REGION:

In the standard setup of binary classification, (X, Y ) ∈ R d + × {-1, 1} is a random pair defined on a certain probability space with unknown joint probability distribution P . Now, equipped with the concepts introduced in the previous section we shall assume regular variation for both classes with the same tail index. Stated otherwise (as formulated in (Jalalzai et al. (2018) )) we shall make the assumption that: Jalalzai et al. (2018) ) that the Bayes regression function η ∞ (x) = P {Y ∞ = 1|X ∞ = x}for the limiting pair is constant along rays i.e. is a function of Θ(x) ∈ S only. The same is true of the Bayes classifier for the asymptotic pair. As a consequence, the optimal classifiers on extreme regions are based on indicator functions of truncated cones on the kind {||x|| > r, Θ(x)B}, where B ⊂ S. As a consequence, the optimal classifiers on extreme regions are based on indicator functions of truncated cones on the kind { x > r, Θ(x) ∈ B}, where B ⊂ S (see Figure 1 ). This observation justifies an ERM strategy using the extreme points of the dataset to train an angular classifier.

tP {X ∈ tA | Y = σ1} → µ σ (A) where σ ∈ {-1, +1}, A ⊂ R d + , 0 ∈ ∂A. If one considers (X ∞ , Y ∞ ) a random pair defined by P {X ∞ ∈ A, Y ∞ = +1} = lim t→∞ , P {X ∈ tA, Y = +1 | ||X|| > t} , P {Y ∞ = +1} = lim t→∞ P {Y = +1 | ||X|| > t}, then one can prove (see Theorem 1 in

Let {(X i , Y i )} n i=1 be n i.i.d copies of (X, Y ). By sorting the training observations by decreasing order of magnitude, we introduce X (i) (with corresponding sorted label Y (i) ) the i-th order statistics, i.e. ||X (1) || ≥ . . . ≥ ||X (n) ||. Let G S be a class of classifiers defined on the sphere S with finite VC dimension V G S < ∞. Thus for any g ∈ G S and any θ ∈ S, g(θ) ∈ {-1, 1}. By extension, for any

x ∈ R d + we define g(x) = g Θ(x) ∈ {-1, 1}

. Let t τ be the quantile at level (1 -τ ) of the r.v ||X|| where τ > 0 corresponds to a small fraction of extreme observations: P {||X|| > t τ } = τ . Set k = nτ and consider the empirical risk dedicated to the extreme samples:

L k (g) = 1 k k i=1 1{Y (i) = g(Θ(X (i) ))},

which corresponds to the empirical version of L tτ , the loss at level t τ . Now consider solutions to the minimization problem:

EQUATION

Thence the authors of Jalalzai et al. (2018) provide guarantees concerning the classification risk in out-of-sample regions.

Theorem 1 (Jalalzai et al. (2018) ) Let g k be any solution of equation 3. Recall k = nτ . Then, for δ ∈ (0, 1), ∀n ≥ 1, we have with probability larger than 1 -δ:

L tτ ( g k ) -L * tτ ≤ 1 √ k 2(1 -τ ) log(2/δ) + C V G S log(1/δ) + 1 k 5 + 2 log(1/δ) + log(1/δ)(C V G S + √ 2) + inf g∈G S L tτ (g) -L * tτ ,

where C is a constant independent from n, τ and δ.

2.3. ADVERSARIAL LEARNING:

Adversarial networks, introduced in (Goodfellow et al. ( 2014)), form in a system where two neural networks are competing. A first model called the generator generates samples as close as possible to the input dataset. A second model called the discriminator aims at distinguishing samples produced by the generator from the input dataset. The goal of the generator is to maximize the probability of the discriminator making a mistake. Hence, if P input is the distribution of the input dataset then the adversarial network intends to minimize the distance (as measured by the Jensen-Shannon divergence) between the distribution of the generated data P G and P input .

Auto-encoders and derivations (Goodfellow et al. (2016) ; Laforgue et al. (2018) ; Fard et al. ( 2018)) shape a subclass of neural networks whose purpose is to learn a suitable representation by learning encoding and decoding functions which capture the core properties of the input data. An adversarial auto-encoders (see Makhzani et al. (2015) ) is a specific kind of auto-encoder where the code plays the role of the generator of an adversarial network. Thus the latent code is forced to follow a given distribution while containing information relevant to reconstructing the input.

3. DILATION INVARIANT REPRESENTATION:



3.1. LEARNING A DILATION INVARIANT REPRESENTATION:

We now introduce Orthrus, a novel algorithm for classification of text data represented by high dimensional vectors as issued by pre-trained embeddings such as BERT which is assumed to not satisfy the regular variation assumption. The idea behind Orthrus is to modify the output X of BERT in order to increase the information carried by the resulting representation Z = ϕ(X) regarding the label Y , including in the upper tail regions of Z corresponding to low probability sentences, in order to improve the performance of a downstream classifier. This is achieved by training an encoding function ϕ in such a way that (i) the marginal distribution q(z) of the code Z be close to a user-specified heavy tailed distribution p satisfying the regularity property (1) ; and (ii) the classification loss of a multilayer perceptron trained on the code Z be small. A major difference distinguishing Orthrus from existing auto-encoding schemes is that the target distribution on the latent space is not chosen as a Gaussian distribution but as a heavy-tailed one.

As the Bayes classifier in the extreme region has a potentially different structure from the Bayes classifier on the bulk (recall from Section 2 that the regression function at infinity depends on the angle Θ(x) only), Orthrus trains two different classifiers, C ext on the extreme region of the latent space on the one hand, and C bulk on its complementary set on the other hand. Given a high threshold t, the extreme region of the latent space is defined as the set {z : z > t}. In practice, the threshold t is chosen as an empirical quantile of order (1 -κ) (for some small, fixed κ) of the norm of encoded data Z i = ϕ(X i ) . In the end, the classifier issued by Orthrus is of the kind

C(z) = C ext (z)1{ z > t} + C bulk (z)1{ z ≤ t} .

Our goal is to minimize the weighted risk

EQUATION

where Z = ϕ(X), d is the Jensen-Shannon distance between the heavy tailed target distribution p and the code distribution q, and λ 1 , λ 2 , λ 3 are positive weights. Following common practice in the adversarial literature, the Jensen-Shannon distance is approached (up to a constant term) by the empirical proxy L(q, p) = sup D∈Γ L(q, p, D)), with

L(q, p, D) = 1 m m i=1 log D(Z i ) + log(1 -D( Zi ))

where Γ is a wide class of discriminator functions, and where independent samples Z i , Zi are respectively sampled from the target distribution and the code distribution q. In the end, Orthrus solves the following min-max problem inf

C ext ,C bulk ,ϕ sup D R(ϕ, C ext , C bulk , D) where R(ϕ, C ext , C bulk ) = λ 1 k k i=1 1{Y (i) = C ext (Z (i) )} + λ 2 n -k n-k i=1 1{Y (i) = C bulk (Z (i) )} + • • • • • • λ 3 L(q, p) ,

where This minimization problem results in Algorithm 1 detailed below, where m is the batch size, ρ is the proportion of the points given to train the extreme classifier and the cost function is the negative log-likelihood.

{Z (i) = ϕ(X (i) ), i = 1, . . . ,

3.2. A DILATION INVARIANT REPRESENTATION FOR DATASET AUGMENTATION:

We now introduce Hydra, a data augmentation algorithm, which relies on the dilation invariance of labels on the extreme embeddings learnt by Orthrus. Let U = (u 1 , . . . , u T ) be a sequence of inputs of length T . The dataset augmentation problem consists in generating multiple sentences {U i } i∈N such that each sequence U i is label coherent with U . Hydra follows the seq2seq approach (Sutskever et al. (2014) ) and learns two decoders with attention (Bahdanau et al. (2014) ), one for each region of the representation previously learnt by Orthrus. On the whole, a decoder takes an input Z (the latent code) and generates an output sequence U i = (u i1 , . . . , u i T ), where each word u i k is in the vocabulary. To generate an output word u k , the decoder iteratively takes as input the previously generated word u k-1 (u 0 being a start symbol), updates its internal state, and returns the next word with highest probability. This process is repeated until the decoder generates either a stop symbol or the length of the generated sequence reaches the maximum sequence length.

Hydra is a semi-supervised algorithm in that it requires a first labeled dataset D n to learn embeddings through Orthrus and a second dataset D gn (not necessarily labeled) to train a decoder G ext (resp G bulk ) for the extremes (resp non extremes) on D gn . The learning is carried out by optimising the classical negative log likelihood of individual tokens gen .A detailed description of Hydra is provided in Algorithm 2 below.

After Hydra has been trained, the data augmentation proceeds as follows: For each input X with extreme embedding, multiple sentences with constant label are obtained by applying a λ dilation (see Figure 2b ). In other words, if G ext is a decoder learnt by Hydra and applicable to the extreme region of the representation, synthetic sentences of the form G ext (λz) can be obtained, with the same attribute as z. 

X ϕ(X) Z Y ∈ {-1, 1} C ext C bulk Y ∈ {-1, 1} (a) Orthrus G ext X ϕ(X) G bulk Z λ N Z . . . λ 1 Z s N . . . s 1 s (b) Hydra

L ext def = λ1 κm κm i=1 Y (i) , C ext θ ( Z(i) ) .

Update C bulk θ by descending:

L bulk def = λ2 m -κm m i= κm +1 Y (i) , C bulk θ ( Z(i) ) .

Update ϕτ by descending: Update G ext ψ by descending:

1 m m i=1 -λ3 • log Dγ( Zi) + L ext + L bulk .

L ext g def = λ1 ρm ρm i=1 gen. Y (i) , G ext ψ ( Z(i) ) .

Update G bulk ψ by descending: Algorithm: Orthrus and Hydra. For both algorithm X is the embedding provided by BERT to an input sentence U .

L bulk def = λ2 m -ρm m i= ρm +1 gen. Y (i) , G bulk ψ ( Z(i) ) .

Heavy-tail distribution: In the remaining of this paper, the regularly varying target distribution is a logistic distribution with parameter δ = 0.9 (see Appendix B.2). The logistic distribution is widely used in the context of extreme values analysis. This distribution is simulated according to Stephenson (2003) which is defined in R d with parameter δ ∈ (0, 1] by its parametric c.d.f. 4 in Appendix illustrates this distribution with various values of δ. Experimental Settings: Classifiers C bulk , C ext are Multi Layer Perceptrons (MLP). For each dataset, one-fourth of data is retained as a test set while the remaining data points are used as the train set. We denote by T the extreme test set region as selected with Algorithm 1. The considered norm is the ∞ norm. For the generation task, the dilation coefficient varies between 1 and 1.5 (λ ∈ [1, 1.5]).

F (x) = exp -( d j=1 x (j) 1 δ ) δ . Figure

4.1. APPLICATION TO SENTIMENT ANALYSIS:

We assess Othrus ability to perform sentiment classification tasks using two balanced binary labeled datasets: Yelp, Amazon introduced by (Kotzias et al. (2015) ). Further results on the datasets are reported in the Appendix. Each dataset is balanced and is composed of 1000 sentences and each sentence expresses a strong positive or negative sentiment. Baseline: Orthrus is compared with a state of the art classifier: a MLP trained on the initial BERT embedding (N N ) taking as input the same training data. Classification performance: Table 1 illustrates the performance of considered models in terms of Hamming loss on the test set. Performance of Orthrus is better than N N on both dataset on the whole test set. The difference in terms of performance is even greater when considering the extreme samples. 1a ) and Amazon (Table 1b ) datasets. The first column transcribes the loss on the whole test set while the second column transcribes the loss on the extreme samples.

4.2. APPLICATION TO DATASET AUGMENTATION FOR LABELLED SENTENCES:

Comparison to related work: We compare Hydra with two state of the art methods for dataset augmentation: EDA (Wei & Zou (2019) ) and round-trip translation (Shleifer (2019) ). Compared to EDA which uses heuristics and a synonym dictionary, Hydra generates grammatically correct sentences and conserve the labels. Compared to round-trip translation Hydra can generate multiple sentences for a single input and conserves sentence labels. We also compare Hydra to a Vanilla Sequence to Sequence to demonstrate the validity of our approach. In this section, we validate Orthrus using the Yelp dataset. To train Orthrus the dataset D n is the Yelp dataset introduced in (Kotzias et al. (2015) ). D gn is a subset of the full Yelp Corpus composed of 5 • 10 5 sentences. Evaluation: Automatic evaluation of generative models for text is still an open research problem. We evaluate our models through three criteria (C1, C2, C3). C1 measures Cohesion (Crossley & McNamara (2010) ) (Are the generated sentences grammatically and semantically consistent?). C2 (named Sent. in Table 1a .) evaluates label conservation (Does the expressed sentiment in the generated sentence match the sentiment of the input sentence?). C3 measures the diversity of the sentences (Is the dataset augmented with diverse sentences?)

We carry out two types of evaluations: a qualitative one based on a user study -that provides evaluation through C1 and C2 -and a quantitative one -that provides evaluation through C3 measures. We measure diversity (C3) with two automatic methods: (1) Distinct n, the diversity measure introduced by (Li et al. (2015) ) which relies on a n-grams count;

(2) a F1 score that reports the improvement induced by the data augmentation while training a fastText (Joulin et al. (2016) ) classifier. This improvement reflects the information added by newly generated sentences. 1a ) and with automatic metrics (Table 1b ). Table 3 : Sentences generated by Hydra for extreme embeddings implying label (sentiment) invariance for generated sentence. λ is the dilation factor.

Qualitative Evaluation: We compare the different methods on the extreme set of sentences according to Hydra. As previously, the extreme test set T consists of 92 sentences. For each sentence, we randomly sample λ ∈ [1, 1.5] and generate a new sentence with Hydra. To compare our method, we use Eda, round-trip translation and a Vanilla Seq2seq to generate new sentences. Results of the qualitative evaluation are reported in Table 2a . As expected Hydra exhibits comparable performance with all other models except Eda in terms of cohesion. Such a result is expected since Eda's use of random swaps or permutations may result in ungrammatical sentences. In terms of expressed sentiments, Hydra outperforms Seq2seq and round-trip translation, demonstrating the relevance of the proposed approach.

Quantitative Evaluation: We report in (Table 2b ) the results of evaluation for C3. We observe that Hydra outperforms all other methods in term of distinct 1 and distinct 2. Table 1b shows that improvement in F1 score induced by dataset augmentation by Hydra beats all other methods and is only equalled by EDA.

Example of generated sentences: We report in Table 3 sentences generated by Hydra. While we make no claim that these generated sentences are better than sentences generated by existing methods, we believe that these samples are at least competitive with the generative models in the literature and highlight the potential of the introduced framework.

5. DISCUSSION & PERSPECTIVES:

The approach promoted in this paper relies on learning a regularly varying representation by minimizing an empirical proxy of an objective function. The latter writes as a weighted sum of a classification risk and a regularization term penalizing the distance between the representation distribution and a heavy-tailed target. Our experiments show that the obtained representation allows to diminish the classification error compared to a MLP with comparable complexity.The obtained representation used for a text data augmentation task, is competitive with existing data augmentation methods. The attribute invariance under code dilation is the key to generate meaningful sentences with prescribed attribute. Future work will investigate the possibility to train an auto-encoder satisfying this dilation invariance property without relying on a pretrained classifier such as Orthrus.

APPENDIX:



A REMARKS:

Remark 1 (Selection of k) To the best of our knowledge, selection of k in Algorithm 1 and Algorithm 2 is still a vivid problem in EVT which is not solved yet. As k gets large the number of extreme points increases including samples which are not large enough and deviates from the asymptotic distribution of extremes. Smaller values of k increase the variance of the classifier/generator. This bias-variance trade-off is beyond the scope of this paper.

Remark 2 In Figure 3a selecting the extreme samples on the input space is not a straightforward step since the tails of the input clusters are not heavy-tail. It was decided to standardise the input data (X i ) i∈{1,••• ,n} by applying the rank-transformation: 

T (x) = 1/ 1 -F j (x) j=1,••• ,d for all x = (x 1 , • • • x d ) ∈ R d where F j def = 1 n+1 n i=1 1{X j i ≤ x} is the j th empirical marginal dis- tribution. It comes ∀i ∈ {1, • • • , n}, V i = T (|X i |). V i margins

B EXPERIMENTS:

Further synthetic experiments were conducted to show the improvement of the classifier.

B.1 SYNTHETIC DATA:

Dataset description: Synthetic data was generated using Scikit-Learn (Pedregosa et al. (2011) ). Focus is made on the classification dataset where clusters of points are normally distributed for each class (see Figure 3a ). Convergence to a logistic: In Figure 3b we visualise the latent space of the output of the encoder ϕ (see Figure 2 ). In Orthrus, the adversarial approach to minimize the discrepancy between the distribution of the output of the encoder and a logistic distribution. Notice the similarity between Figure 3b and Figure 4a . Extremes selection: Extreme samples from Figure 3a are assessed directly on the input data while extremes from Figure 3c are obtained from the latent space. Extremes from the latent space (output of ϕ after the training phase) are located on the border of the distribution which comforts the correct behaviour of the training. We also note similarities while comparing in the input space: extreme samples selected in the input space and extreme ones selected in the learnt latent space. In Figure 3a no extreme observation appear on the negative class whereas labels of extremes of Figure 3c 

B.3 ADDITIONAL EXPERIMENT SETTINGS FOR REAL DATA:

We use BERT pretrained models and code from the library pytorch-transformers. All models were implemented using Pytorch. The output of BERT is a R 768 vector, Z is a R 50 vector. The dimension of Z (50) was obtained by cross validation. D, C bulk and C ext are MLP composed of 3 layers and uses a ReLu as activation function. Decoders D bulk and D ext are GRU composed of 6 layers. For training the batch size is set to 64 and we set λ 1 = (1 -P(||Z|| ≥ ||Z κn ||) -1 ,λ 2 = P(||Z|| ≥ ||Z κn ||) -1 ,λ 3 = 10 -3 . All neural networks have been optimized with Adam (Kingma & Ba (2014)). For generation the maximum dilation factor λ is set to 1.5 since we observe that with greater values a stuttering phenomenon appears. Table 4 : Supplementary sentences generated by Hydra for extreme embeddings implying label (sentiment) invariance for generated sentence. λ is the dilation factor.

B.5 ADDITIONAL EXPERIMENTS ON AMAZON DATA:

Experiments on the Amazon dataset were conducted similarly to to the ones from the Yelp dataset from subsection 4.1, similar conclusions can be drawn from Figure 6 . 

B.6 QUALITATIVE EVALUATION:

For the qualitative evaluation, we compute the Krippendorff's alpha coefficient Krippendorff (2018) to evaluate the inter-annotator agreement. We obtain α = 0.41 for the grammar analysis and α = 0.53 for the sentiment analysis. If α = 1 it indicates perfect reliability; if α = 0 it indicates the absence of reliability. Such values of α can be considered as relatively high given our huge number of annotators (126 for the grammar and 134 for the sentiment analysis).

B.7 ADDITIONAL GENERATED SENTENCES:

We present in Table 4 additional extreme sentences.

C A LINGUISTIC POINT OF VIEW OF EXTREMES:

In this section, we provide a comparative linguistic analysis of the extreme samples in case of sentiment classification through the affect scores provided by the lexicon for English words from

:



Figures and Tables:

Figure 1: Colored cones correspond to a given label from the classifier on the simplex.

n} are the encoded observations with associated labels Y (i) sorted by decreasing magnitude of Z (i.e. Z (1) ≥ • • • ≥ Z (n) ), and k = κn is the number of extreme samples among the n encoded observations.

Figure 2: Pipeline of the introduced algorithms.

Coef. λ1, λ2, λ3 > 0, Training dataset Dn = {(U1, Y1), . . . , (Un, Yn)}, proportion κ ≤ 1 of extreme observations, batch size m ≤ n. Initialization: parameters of the encoder ϕτ , classifiers, C ext θ , C bulk θ , decoders and discriminator Dγ while (τ, θ, θ ) not converged do Sample {(U1, Y1) . . . , (Um, Ym)} from Dn. Sample {Z1, . . . , Zm} from the prior PZ . Sample Zi from ϕτ (Z|Xi) for i ∈ {1, . . . , m}. Update Dγ by ascending: λ3 m m i=1 log Dγ(Zi) + log(1 -Dγ( Zi)). Sort { Zi}i∈{1,...,m} by decreasing order of magnitude || Z(1) || ≥ . . . ≥ || Z(m) ||. Update C ext θ by descending:

end while Compute { Zi}i∈{1,...,n} = ϕ(Xi) i∈{1,...,n} Sort { Zi}i∈{1,...,n} by decreasing order of magnitude || Z(1) || ≥ . . . || Z(k) || ≥ . . . ≥ || Z(n) ||. OUTPUT: encoder ϕ, decoder G ext applicable on the region {x : ||ϕ(x)|| ≥ || Z( κn ) ||} and G bulk applicable on the region {x : ||ϕ(x)|| < || Z( κn ) ||}.

! λ = 1.2 the penne vodka is excellent! λ = 1.3 the penne vodka is excellent! λ = 1.5 the penne vodka is perfect! input awful service. λ = 1 bad maintenance. λ = 1.1 a terrible service. λ = 1.3it's a terrible service. λ = 1.5 horrible service. input i love their fries and their beans. λ = 1 i love their fries and their bananas. λ = 1.1 i like their pies and their bananas. λ = 1.3 i enjoy her fries and her beans. λ = 1.5 i enjoy enjoy enjoy and beans. input awesome selection of beer. λ = 1 great selection of beer. λ = 1.1 a superb selection of beer. λ = 1.3 an amazing selection of beer. λ = 1.5 great choice of beer.

follow a standard Pareto distribution. Extreme samples are thus {V i , ||V i || ≥ n/k} with k = √ n by convention. One may notice that extremes in Figure 3c are more label balanced compared to Figure 3a.

Figure 4 illustrates the logistic distribution with varying parameter δ.

Figure 6: Evolution of the prediction of Orthrus on the whole test set (a) and on the extreme test set (b). Colors represent predicted labels for each sample (λ x test ) with λ being the dilatation factor (Y-axis).

OUTPUT: encoder ϕ, classifiers C ext θ applicable on the region {x : ||ϕ(x)|| ≥ || Z( κn ) ||} and C bulk θ applicable on the region {x : ||ϕ(x) || < || Z( κn ) ||}. Y1), . . . , (Un, Yn)}, Dg n = {Ug 1 , . . . , Ug n } batch size m ≤ n,

Table content: <table><tr><td colspan="2">Algorithm 2 Hydra</td><td/></tr><tr><td colspan="3">INPUT: Coef. λ1, λ2, λ3 &gt; 0, Training datasets</td></tr><tr><td colspan="3">Dn = {(U1, Initialization: parameters of the encoder ϕ, classi-</td></tr><tr><td>fiers, C ext θ , C bulk θ</td><td>, decoders, G ext ψ , G bulk ψ</td><td>and dis-</td></tr><tr><td>criminator Dγ</td><td/><td/></tr><tr><td>Optimization:</td><td/><td/></tr><tr><td>ϕ, C</td><td/><td/></tr></table>

Hamming loss (Hloss) on the Yelp (Table

Table content: <table><tr><td>Model</td><td colspan="2">Hloss All Hloss Extreme</td><td>Model</td><td colspan="2">Hloss All Hloss Extreme</td></tr><tr><td>NN</td><td>0.30</td><td>0.30</td><td>NN</td><td>0.33</td><td>0.32</td></tr><tr><td>Orthrus</td><td>0.26</td><td>0.24</td><td>Orthrus</td><td>0.29</td><td>0.22</td></tr><tr><td/><td>(a)</td><td/><td/><td>(b)</td><td/></tr></table>

Evaluation of Hydra with user evaluation (Table

Table content: <table/>

Peer review (iclr2020):
This paper proposes a new embedding method for sentences that aims to preserve dilation invariance. Much of the methodology is justified by results for extremal point classification under particular assumptions, and then the authors try and encourage these assumptions to be met via penalty terms introduced in their embeddings/augmentation models. However, while the proposed methodology seems interesting/novel, it remains conceptually unclear why it should be superior to standard text classification methods (ie. exactly what assumptions are being exploited to improve performance and how exactly do those assumptions help should be made more explicit). In particular, why is dilation invariance even a good idea? Overall, I find the paper a bit mathematically dense in Secs 2.1-2.2, which would not be a bad thing if the math were necessary to justify why the proposed methodology works well, but it in this case seems mainly presented as background material (as if it were a prerequisite to understand the method itself, which it is certainly not). Why not instead present an explicit theorem providing some statistical guarantees for the proposed methodology in Sec 3.1 based on the constant-along-rays result (which would be nice to have regardless), and then follow up the theorem with background math from 2.1-2.2 which is necessary to understand the proof? As it is currently written the paper is a bit too dense in terminology, and opaque names like Hydra and Orthrus used to describe straightforward concepts that are essentially a neural classifier (of a particular form) and a seq2seq-based data augmentation procedure (which would be good to describe in language more familiar to the ML audience). In particular, the goals of Hydra and Orthrus should first be intuitively described before delving into their various components. - Why did the authors never evaluate the overall sentiment prediction performance of Orthrus + Hydra used together vs other classifiers + data augmentation strategies? - If the goal of dilation invariance is to help the classifiers better generalize to out-of-distribution test sentences, then why not verify this happened, eg. by training on Yelp and testing on Amazon? - The authors should better justify the assumption of Jalalzai et al, and why this is appropriate for the MLP classifier used later in the paper. - This statement needs to be clarified and have citation: *Such classifier whose output solely depends on the angle ?(x) of the considered input, with provable guarantees concerning the classification risk in out-of-sample regions scaling as the square root of the number of extreme points used at the training step* - The authors should explain Equation (1) in English rather than referring to it so early on the paper (pg 1: *satisfying Equation 1*). I had no idea what this was supposed to mean as a first time reader. - The way figure 4 is presented is a bit opaque and took me a while to understand (have to look closely at Fig 4a to see the columns are not monochromatic). - *We also compare Hydra to a Vanilla Sequence to Sequence to demonstrate the validity of our approach* How *Vanilla Sequence to Sequence* (word *model* is missing) is used for dataset augmentation needs to be clarified here. - A figure demonstrating an example of the phenomenon explained in Sec 2.2 would be helpful to aid reader*s intuition. - Typo: *eugmentation*