Regularly varying representation for sentence embedding
Authors: Hamid Jalalzai
Pierre Colombo
Chloé Clavel
Eric Gaussier
Giovanna Varni
Emmanuel Vignon
Anne Sabourin
Venue:  (2020)
Keywords: extreme value theory
classification
supvervised learning
data augmentation
representation learning

Abstract: The dominant approaches to sentence representation in natural language rely on learning embeddings on massive corpuses. The obtained embeddings have desirable properties such as compositionality and distance preservation (sentences with similar meanings have similar representations). In this paper, we develop a novel method for learning an embedding enjoying a dilation invariance property. We propose two algorithms: Orthrus, a classification algorithm, constrains the distribution of the embedded variable to be regularly varying, i.e. multivariate heavy-tail. and uses Extreme Value Theory (EVT) to tackle the classification task on two separate regions: the tail and the bulk. Hydra, a text generation algorithm for dataset augmentation, leverages the invariance property of the embedding learnt by Orthrus to generate coherent sentences with controllable attribute, e.g. positive or negative sentiment. Numerical experiments on synthetic and real text data demonstrate the relevance of the proposed framework.


Under review as a conference paper at ICLR 2020
REGULARLY VARYING REPRESENTATION FOR SEN- TENCE EMBEDDING
Anonymous authors Paper under double-blind review
ABSTRACT
The dominant approaches to sentence representation in natural language rely on learning embeddings on massive corpuses. The obtained embeddings have desirable properties such as compositionality and distance preservation (sentences with similar meanings have similar representations). In this paper, we develop a novel method for learning an embedding enjoying a dilation invariance property. We propose two algorithms: Orthrus, a classification algorithm. constrains the distribution of the embedded variable to be regularly varying, í.e. multivariate heavy-tail. and uses Extreme Value Theory (EVT) to tackle the classification task separate regions: the tail and the bulk. Hydra, text generation algorithm for dataset augmentation, leverages the invariance property of the embedding learnt by Orthrus generate coherent sentences with controllable attribute, e.g. positive or negative sentiment. Numerical experiments on synthetic and real text data demonstrate the relevance of the proposed framework.
INTRODUCTION
Representing the meaning of natural language in mathematically grounded way is scientific challenge that has received increasing attention with the explosion of digital content and text data in the last decade. Relying on the richness of contents, several sentence embeddings have been proposed (Peters et al. (2018); Radford et al. (2018); Devlin et al. (2018)) with demonstrated efficiency for the considered tasks. These embeddings, learnt on massive datasets, are commonly used to perform downstream tasks such as classification or data generation.
In a classification context, e-g. when the goal is to distinguish between positive and negative sentiments in user reviews, most classifiers are based on Empirical Risk Minimization (ERM) strategies and variants of However, nothing guarantees that such classifiers perform satisfactorily on the tails of the explanatory variables, i.e. in regions attached to rare events or even outside the domain of observed data. The present paper builds upon the methodological framework proposed by (Jalalzai et al. (2018)) performing classification in extreme regions with guarantees concerning the excess risk the ERM classifier. Their approach relies on multivariate extreme value theory (EVT) and is valid under regularity assumptions concerning the multivariate distributional tail which appear to be an homogeneity property above large thresholds: the tail hehaviour at infinity is similar to the behaviour of polynomial function near infinity (see equation [ below). To wit, it is necessary for the multivariate survival function of the considered random vector to be approximately homogeneous of degree- 1, up appropriate marginal standardization. Although there is experimental evidence that their framework improves the performance of baseline classifiers in extreme regions on low dimensional data, there is no reason to assume that the previously mentioned text embeddings satisfy the required regularity assumptions. The aim of the present work is to apply Jalalzai et al. (2018) 8)'s methodology to text data represented by state the art embeddings. To achieve this, we train the algorithm to perform transformation that allows it to map any text embedding X onto random vector Z which satisfies the aforementioned regularity assumptions. The transformation is learnt by adversarial strategy (Goodfellow et al. (2016)). As by-product, obtain novel data augmentation mechanism which takes advantage the scale invariance properties of Z to generate synthetic sentences that keep invariant the attribute of the original sentence.
The work detailed in this paper can be connected to other articles (Socher et al. 2013); Guu et al. (2018); Siffer al. (2017)), though framework is different. Our contribution is twofold. We

---

Under review as a conference paper at ICLR 2020
introduce (i) novel classification algorithm (Orthrus), taking as input an embedded sentence to map it onto a heavy-tailed vector in the latent space satisfying Equation []] (ii) a sumi-supervised data augmentation method (Hydra), which leverages the homothetic invariance property of the embedding learnt by Orthrus to produce new sentences with prescribed label.
Label preserving data augmentation, which increases the number of training samples in machine learning systems, is an effective solution to data insufficiency and is an efficient pre-processing step in computer vision (Wang& & Perez (2017). Due to the discrete nature of text, data augmentation is a much harder problem in natural language processing (NLP) than in computer vision. Current work in natural language (NL) dataset augmentation with label preservation mainly relies either on buck-translatee (Shleifer (2019)), on synonym replacements (Kobayashi (2018), on slot filling (Hou) et al. ) or on the use of handcrafted heuristics (Wei & Zou (2019)) (e.g swap, deletion, synonym replacements). Unlike other state of the art solutions, Hydra can generate multiple sentences with a considered sentiment using unique input sentence. Hydra does not require any additional resources
(e.g. synonym dictionary), nor does it rely on hand-crafted heuristics, use style transfer methods (Hu et al. (2017); Colombo et al. (2019).
In this paper, we work with output vectors issued by BERT sentence embedding as inputs (Devlin) ct al. (2018). BERT is currently one of the sfate-of-the art algorithms for sentence embedding. We demonstrate on both synthetic and real datasets that the embedding learnt by Orthrus on top of BERT indeed follows a heavy-tail distribution, and that Orthrus is able to outperform a state-of-the art classifier built on BERT. On the dataset augmentation task, quantitative and qualitative experiments demonstrate the ability of Hydra to generate new sentences while preserving labels.
The rest of this paper is organized as follows. Section 2 introduces the necessary background in multivariate extremes and adversarial learning. The methodology we propose is detailed at length in section 3) Illustrative numerical experiments on both synthetic and real data are gathered in ssctiionn Additional experimental results are available in the supplementary material.
2
BACKGROUND
2.1 HEAVY TAILS AND REGULAR VARIATION
By definition, heavy-tail phenomena are those which are ruled by very large values, occurring with a far from negligible probability and with significant impact on the system under study. When the phenomenon of interest is described by the distribution of univariate random variable, the theory of regularly varying functions provides the appropriate mathematical framework for the study of heavy-tailed distributions. For the sake of clarity and in order introduce notations we recall some related theoretical background. One may refer to (Resnick (2013)) for an excellent account of the theory of regularly varying functions and its application the study of heavy-tailed distributions.
A random variable X with cumulative distribution function (c.d.f)) F is heavy-tailed of index >> 0 if and only if for any fixed x >
nPP (X/F 1/n)
where (u) inf(t: : F(t) 2 n) denotes E"s generalized inverse. Based on this characterization. the heavy-tail model is classically extended to the multivariate setup as follows. Consider now a d-dimensional random vector X =
x(d) taking its values in R4. Then X is said to be heavy-tailed with tail index a if there exists positive Radon measure y on the punctured set [0, (00 and a function b(t) + co such that:
UP (X/b(1) E A) 11000 m(A)
For any Borelian set A C [0, oc] which is bounded away from 0 and such that the measure of the boundary OA is zero. In such a case, /r fulfills the homogeneity property μ(tA) = t- p(A) for all Using the homogeneity property, one may show that y can be
1> 0 and any set A in [0, xx]{{(0 decomposed into radial component and an angular component t which are independent from each
other. Indeed, for all = (I1, .... Ia) E Rd, set
R(x)= = ||a|||
21
Fd
(2)
0(xx) =
E ES,
R(x))
R(x)
2

---

Under review as a conference paper at ICLR 2020
Figure I: Colored cones correspond to given label from the classifier on the simplex.
where S is the positive orthant of the unit sphere in Rd for the chosen norm ||-I]. The choice of the norm is unimportant as all norms are equivalent in Rd. Then, for every C S, we have:
ala R(x)/b(t) t, (x) e B)] =
where & is referred to as the spectral or angular measure.
2.2 CLASSIFICATION EXTREME REGION
In the standard setup of binary classification, (X E R4 x (-1,1} is a random pair defined on a certain probability space with unknown joint probability distribution P. Now, equipped with the concepts introduced in the previous section shall assume regular variation for both classes with the same tail index. Stated otherwise (as formulated in (Jalalzai et al. (2018)) we shall make the assumption that:
tP E E Y Y o - (A()
where E +11,+11, AC C Rd. R4,0 & OA. one considers (X Yoo) a random pair defined by P (Xxx x E A, = +1] lim P E LA, +1 ||||| > 4. P D0 +1) lim; [{== the Bayes regression function (x)) = {Y> = 1
|| > f}. then one can prove (see Theorem I in Jalalzai et al. (018)) that Hor the limiting pair is constant along rays i.e. is a function of ((x) E S only. The same true of the Bayes classifier for the asymptotic pair. As consequence, the optimal classifiers extreme regions based indicator functions of truncated cones on the kind |||||| > (xx) B). where B C S. As a consequence, the opti- mal classifiers extreme regions are based indicator functions of truncated cones the kind [||=|| O(x) E B}, where B c S (see Figure I This observation justifies an ERM strategy using the extreme points of the dataset to train an angular classifier.
Let {X,, r)) be i.i.d copies of (X, Y). By sorting the training observations by decreasing order of magnitude, we introduce ((e) (with corresponding sorted label Y(s) the i-th order statisties, i.e. [1X 2 ... 2 (x)))]- Let Gs be class of classifiers defined on the sphere S with finite VC dimension Vgg
Thus for any g E Gs and any 0 S.9 E 1).. By extension, for any E Rd. we define g(x) = s(0(z)) E (-1,1)
Let 4, be the quantile at level (1 7) of the r.v ||| where 7 > 0 corresponds to a small fraction of extreme observations: P [||-X|] } r. Set k [nr] and consider the empirical risk dedicated to the extreme samples:
k
(x(g) = k Σ IY ())),
which corresponds to the empirical version of T,,, the loss at level f,,, Now consider solutions the minimization problem:
min T. (g)
9E9s
Thence the authors of Jalalzai et al. (2018) provide guarantees concerning the classification risk in out-of-sample regions.

---

Under review as a conference paper at ICLR 2020
Theorem (Jalalzai et al. (2018) Let 9k be any solution of equation 3 Recall k [nT]. Then, for d E (0,1), vn 2 I, we have with probability larger than 1 6:
1 (V2(1- - (00(1/) C CV Vos log(1/8)
-ig 2 Lt, (Tx)- -
Vk
5 + 2log(1 + (log(1/5)(C//Vos + v2))
inf Lt, (g) Li 9E99
where C is a constant independent from n, 7 and
2.3 ADVERSARIAL LEARNING
Adversarial networks, introduced in (Goodfellow et al. (2014)), form in a system where two neural networks are competing. A first model called the generator generates samples as close possible to the input dataset. A second model called the discriminator aims at distinguishing samples produced by the generator from the input dataset. The goal of the generator is to maximize the probability of the discriminator making a mistake. Hence, if P ippu is the distribution of the input dataset then the adversarial network intends to minimize the distance (as measured by the censen-Shannon divergence) between the distribution of the generated data Pg and P Pimpul-
Auto-encoders and derivations (Goodfellow et al.] (2016); Laforgue et al. (2018); Fard et al. (2018))
shape subclass of neural networks whose purpose is to learn a suitable representation by learning encoding and decoding functions which capture the core properties of the input data. An adversarial auto-encee encoders (see Makhzani et al. (2015)) is specific kind of auto-encoder where the code plays the role of the generator of an adversarial network. Thus the latent code is forced to follow a given distribution while containing information relevant reconstructing the input.
3 DILATION INVARIANT REPRESENTATION
3.1 LEARNING A DILATION INVARIANT REPRESENTATION
We now introduce Orthrus, a novel algorithm for classification of text data represented by high dimensional vectors as issued by pre-trained embeddings such as BERT which is assumed to not satisfy the regular variation assumption. The idea behind Orthrus is to modify the output X of BERT in order to increase the information carried by the resulting representation Z = (XX) regarding the label Y, including in the upper tail regions of Z corresponding to low probability sentences, in order to improve the performance of downstream classifier. This is achieved by training an encoding function in such a way that (i) the marginal distribution g(2) of the code Z be close to a user-specified heavy tailed distribution p satisfying the regularity property ① ; and (ii) the classification loss of a multilayer perceptron trained on the code / be small. A major difference distinguishing Orthrus from existing auto-encoding schemes is that the target distribution on the latent space is not chosen as Gaussian distribution but as a heavy-tailed one.
As the Bayes classifier in the extreme region has a potentially different structure from the Bayes classifier on the bulk (recall from Section 2 that the regression function at infinity depends on the angle (xx) only), Orthrus trains two different classifiers, eext on the extreme region of the latent space the one hand, and Chulk on its complementary set on the other hand. Given a high threshold f, the extreme region of the latent space is defined as the set {2 : ||ж||| > t}. In practice, empirical quantile of order (1 x) (for some small, fixed k) of the norm of encoded data || o (X)]. In the end, the classifier issued by Orthrus is of the kind C(z) C [I] > f} + Chulk (1)]
the threshold Ê is chosen
< t} Our goal is to minimize the weighted risk
R(y,, C Chhlkk
AP CC (Z), || <t) + Xxp (Y holk (Z), |2| t
(4)
+x,0(g(2),
where = (X),, d is the ShasennShannon distance between the heavy tailed target distribution p and the code distribution q, and '₁ A2, Ag are positive weights. Following common practice in the adversarial literature, the Lensen-Shannol distance is approached (up constant term) by the
4

---

Under review as a conference paper at ICLR 2020
empirical proxy E(q.p) p) supDer L(q. D)), with
m
L(q.p, D)
Σ m
log D(Z.) + log(1 D(2))
where is a wide class of discriminator functions, and where independent samples Z, Z, are respectively sampled from the target distribution and the code distribution q. In the end, Orthrus solves the following min-max problem inf Cankk supp R(yp, Ces C D) where
h
A
A2 K
P((y,, C Cau,, Cruuk))
H((0) + C (20))
11 # C (2()) +...
k
2m
A3 (qq
where (Z() (())). , are the encoded observations with associated labels Y(s) sorted by decreasing magnitude of || (i.e. [ 2 ... 2 (z) [). and & [kR]] is the number of extreme samples among the n encoded observations.
This minimization problem results in Algorithm | detailed below, where m is the batch size, P is the proportion of the points given train the extreme classifier and the cost function & is the negative log-likelihood
3.2 DILATION INVARIANT REPRESENTATION FOR DATASET AUGMENTATION
We now introduce Hydra, data augmentation algorithm, which relies the dilation invariance of labels on the extreme embeddings learnt by Orthrus. Let U = (ui...., ur) be a sequence of inputs of length T. The dataset augmentation problem consists generating multiple sentences {U[ such that each sequence U! is label coherent with U. Hydra follows the seq2seq approach (Sutskever al. (0144) and learns two decoders with attention (Bahdanau al. (2014)) one for each region of the representation previously learnt by Orthrus. On the whole, a decoder takes an input Z (the latent code) and generates an output sequence U = (u/,.... u/y,,> ). where each word u/, is in the vocabulary. To generate an output word u/z, the decoder iteratively takes as input the previously generated word 14-1 (ug heing start symbol), updates its internal state, and returns the next word with highest probability. This process is repeated until the decoder generates either a stop symbol or the length of the generated sequence reaches the maximum sequence length.
Hydra is a semi-supervised algorithm in that it requires a first labeled dataset D,, to learn embeddings through Orthrus and second dataset Dg.. (not necessarily labeled) train decoder Gest (resp Gbulk) for the extremes (resp non extremes) Dy.. The learning is carried out by optimising the classical negative log likelihood of individual tokens f .gen.A A detailed description of Hydra is provided in Algorithm 2 below.
After Hyydra has been trained, the data augmentation proceeds as follows: For each input X with extreme embedding, multiple sentences with constant label obtained by applying a dilation (see Figure 2b). In other words, if GeM is decoder learnt by Hydra and applicable to the extreme region of the representation, synthetic sentences of the form Gext (Xz) can be obtained, with the same attribute as
Y (-(11,1)
A Z
81
Gext
X ((x)
Cext
Ax
SN
x(X)
Chulk
X
7
YE(-111]
Cbulk
8
(a) Orthrus
(b) Hydra
Figure 2: Pipeline of the introduced algorithms.
5

---

Under review as a conference paper at ICLR 2020
Algorithm 1 Orthrus
Algorithm Hydra
INPUT: Coef. λ2, As Training dataset DA {lU,,Yi Yi). .... (U,, Yu),. proportion K S of extreme observations. batch size
INPUT: Coef. A2, D. ((U1: Vi). .... (U,, Yn),, D {U .... Ugn batch size S
Training datasets
Initialization: parameters of the encoder yru classi- fiers, Cg Ce Chulk, decoders and discriminator D, while (r, 0,0') not converged do
Initialization: parameters of the encoder Φ. classi- fiers, C5a Crucl decoders, G G and dis- criminator D,
Sample ((Us, Yi) .... (Um: Vm)) from Du- Sample (Zi..... Zm) from the prior Pz.
Sample 2, from şr (2),.) for m..
Optimization:
y,, C C. =Othrus(AAg D.,))
Update D, by ascending:
while (v), 2f) not converged do from the training set
Sample ((X,))..,, Dgn.
λy
log Dz(Z.) lug(1 D2(2.))
VT8 1=1
Sample 2, from x(Z[X,,) for [l,...,] m).
Sort (2, (e ().....) by decreasing order of magni- tude [| ... Zmmll
Sort (2she( by decreasing order of magni- tude
| 2 (12
Update Cg by descending:
God
A1 km
Σ
Update G by descending:
Update Co by descending:
del
A1
Laara
Egen. (Y Call (2()))
ppm
chllk
Ag
Σ
m
(VYo, Cp (2w0).
Update Ghell by descending:
mn
Updute y- by descending:
Chikk
A2
Egen. (Y Ghulk (Z(a))
t= 1=mm ++1
As log D2(2i) cm L
end while
end while
Compute (2.)e ((Xx))
Compute (2:).et)...)) = (e(t))
Sort (2, (2....).....))) by decreasing order magni-
Sort { (2:).e))....)))) by decreasing order of magni- tude [1 2 ... |ل || 2 ... 2 || OUTPUT: encoder classifiers Of applicable the region [|{(œ)]] Chilt applicable on the region (ar [|<()))) [|]]))])
tude |1 | 2 (1)l--
OUTPUT: encoder decoder G applicuble the region {a [|{())]] 2 [| |} and Ghilk ap- plicable on the region (a [|{[x)]] < [|
[[] and
Algorithm: Orthrus and Hydra. For both algorithm X is the embedding provided by BERT to an input sentence U.

---

Under review as a conference paper at ICLR 2020
4 NUMERICAL EXPERIMENTS
Heavy-tail distribution: In the remaining of this paper, the regularly varying target distribution is logistic distribution with parameter ò = 0.9 (see Appendix B.2 The logistic distribution is widely used in the context extreme values analysis. This distribution is simulated according to Stephenson (2003) which is defined in R with parameter E (0, 1] by its parametric c.d.f.
F(x) = exp
Figure in Appendix illustrates this distribution with various
values of 6.
Experimental Settings: Classifiers Chalk C are Multi Layer Perceptrons (MLP). For each dataset, one-fourth data is retained as a test set while the remaining data points are used as the train set. We denote by T the extreme test set region as selected with Algorithm [ The considered norm is the Eo norm. For the generation task, the dilation coefficient varies between and 1.5 (A E [1, 1.5)).
4.1 APPLICATION TO SENTIMENT ANALYSIS
We assess Othrus ability to perform sentiment classification tasks using two balanced binary labeled datasets: Yelp, Amazon introduced by (Kotzias et al. (2015)). Further results on the datasets are reported in the Appendix. Each dataset balanced and is composed of 1000 sentences and each sentence expresses a strong positive or negative sentiment.
Baseline: Orthrus is compared with state of the classifier: MLP trained the initial BERT emhedding (N N) taking as input the same training data.
Classification performance: Table illustrates the performance of considered models in terms of Hamming loss on the test set. Performance of Orthrus is better than NN on both dataset on the whole test set. The difference in terms of performance even greater when considering the extreme samples.
Model NN 0.30
Hloss All
Hloss Extreme
Model NN
Hloss All
Hloss Extreme
0.30
0.33
0.32
Orthrus
0.26
0.24
Orthrus
0.29
0.22
(a)
(b)
Table 1: Hamming loss (Hloss) on the Yelp (Table la) and Amazon (Table 1b) datasets. The first
column transcribes the loss the whole test while the second column transcribes the loss on the extreme samples.
4.2 APPLICATION TO DATASET AUGMENTATION FOR LABELLED SENTENCES
Comparison to related work: We compare Hydra with two state of the art methods for dataset augmentation: EDA (Wei Zou (209)) and round-trip translation (Shleifer (2019)). Compared to EDA which uses heuristics and synonym dictionary, Hydra generates grammatically correct sentences and conserve the labels. Compared to round-trip translation Hydra can generate multiple sentences for single input and conserves sentence labels. We also compare Hydra Vanilla Sequence to Sequence to demonstrate the validity of our approach. In this section, we validate Orthrus using the Yelp dataset. To train Orthrus the dataset D, is the Yelp dataset introduced in Kotzias et al. (2015)). D,,, is subset of the full Yelp Corpus composed of 5 10 sentences.
Evaluation: Automatic evaluation of generative models for text still an open research problem. We evaluate our models through three criteria (C1, C2, C3). C1 measures Cohesion (Crossley McNamara (2010)) (Are the generated sentences grammatically and semantically consistent?).
C2 (named Sent. in Table la.) evaluates label conservation (Does the expressed sentiment in the generated sentence match the sentiment of the input sentence?). C3 measures the diversity of the sentences (Is the dataset augmented with diverse sentences?)
We carry out two types of evaluations: a qualitative one based on a user study - that provides evaluation through CI and C2 and quantitative one that provides evaluation through C3 measures. We measure diversity (C3) with two automatic methods: (1) Distinct n, the diversity measure introduced by (Li et al. (0155) which relies n-grams count: (2) Fl score that reports the improvement induced by the data augmentation while training a fastText (Joulin et al.; (2016)) classifier. This improvement reflects the information added by newly generated sentences.
7

---

Under review as a conference paper at ICLR 2020
Model
Sent
Cohesion
Model
F1
Distinet
Distinct 2
Raw
0.90
0,85
Raw
0.87
0.385
0,817
Back
0.78
0,84
Back
0.87
0.405
0,816
Eda
0.87
0,68
Eda
0.95
0.387
0.813
Seq2seq
0.60
0.87
Seq2seq
0.807
0.375
0.813
Hydra
0.88
0.90
Hydra
0.95
0.457
0.844
(a)
(b)
Table 2: Evaluation of Hydra with user evaluation (Table 1a) and with automatic metrics (Table lb).
imput
penne vodka excellent!
input
love their fries and their beans.
penne vodka splendid! A= 1.2
À
love their fries and their bananas,
the penne vodka excellent!
À
like their pies and their bananas.
A= = 1.3
the penne vodka excellent!
À = 1.3
i enjoy her fries and her beans.
A = 1.5
the penne vodka is perfect!
== 1.5
enjoy enjoy enjoy and beans.
input
awful service. maintenance. a terrible service.
imput
awesome selection of beer.
À =
great selection of beer.
A= 1.1
X 1.1
a superb selection of beer.
À = 1.3
it's terrible service.
X x 1.3
an amazing selection of beer. great choice of beer.
À 1.5
horrible service.
A= 1.5
Table 3: Sentences generated by Hydra for extreme embeddings implying label (sentiment) invariance for generated sentence. À is the dilation factor.
Qualitative Evaluation: We compare the different methods the extreme set of sentences according to Hydra. As previously, the extreme test set consists of 92 sentences. For each sentence, we randomly sample À [1, 1.5] and generate new sentence with Hydra. To compare we use Eda, round-trip translation and a Vanilla Seq2seq to generate new sentences. Results of the qualitative evaluation
method,
reported in Table 2a. As expected Hydra exhibits comparable performance with all other models except Eda in terms of cohesion. Such result expected since Eda's 's use of random swaps permutations may result in ungrammatical sentences. In terms of expressed sentiments, Hydra outperforms Seq2seq and round-trij translation, demonstrating the relevance of the proposed approach.
Quantitative Evaluation: We report in (Table 2b) the results of evaluation for C3. We observe that Hydra outperforms all other methods in term of distinct 1 and distinct 2. Table 1b shows that improvement in F1 score induced by dataset augmentation by Hydra beats all other methods and is only equalled by EDA.
Example of generated sentences: We report in Table sentences generated by Hydra. While we make no claim that these generated sentences are better than sentences generated by existing methods, we believe that these samples least competitive with the generative models in the literature and highlight the potential of the introduced framework.
DISCUSSION & PERSPECTIVES
The approach promoted in this paper relies on learning a regularly varying representation by minimiz- ing an empirical proxy of objective function. The latter writes as weighted sum of classification risk and regularization term penalizing the distance between the representation distribution and heavy-tailed target. Our experiments show that the obtained representation allows to diminish the classification error compared to MLP with comparable complexity. The obtained representation used for a text data augmentation task, is competitive with existing data augmentation methods. The attribute invariance under code dilation is the key to generate meaningful sentences with prescribed attribute. Future work will investigate the possibility to train an auto-encoder satisfying this dilation invariance property without relying pretrained classifier such as Orthrus.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Dzmitry Bahdanau. Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:/409.004 2014.
Pierre Colombo, Wojciech Witon, Ashutosh Modi, James Kennedy, and Mubbasir Kapadia. Affect- driven dialog generation. arXiv preprint arXiv:/904.00 1904.0273,, 2019.
Scott Crossley and Danielle McNamara. Cohesion, coherence, and expert evaluations of writing proficiency. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 32, 2010.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
Maziar Moradi Fard, Thibaut Thonet, and Eric Gaussier. Deep k-maass: Jointly clustering with k-means and learning representations. arXiv preprint arXiv: 10069, 2018.
Ian Goodfellow, Jean Pouget-Abadics Mehdi Mirza, Bing Xu, David Marde-Farley Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- tion processing systems, pp. 2672-2680, 2014.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http:
Kelvin Guu, Tatsunori
Hashimoto, Yonatan Oren, and Percy Liang. Generating sentences by editing prototypes. Transactions of the Association for Computational Linguistics, 6:437-450, 2018.
Yutai Hou, Yijia Liu, Wanxiang Che, and Ting Liu. Sequence-to-sequeee data augmentation for dialogue language understanding. arXiv preprint
2018.
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric Xing. Toward controlled generation of text. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1587-1596. JMLR. org, 2017.
Hamid Jalalzai, Stephan Clémençon, and Anne Sabourin. On binary classification in extreme regions. In Advances Neural Information Processing Systems, pp. 3092-3100, 2018.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.0175/ 2016.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXhv:1412.6980, 2014.
Sosuke Kobayashi. Contextual augmentation: Data augmentation by words with paradigmatic relations. arXiv preprint arXiv: 1805.06201, 2018.
Dimitrios Kotzias, Misha Denil, Nando De Freitas, and Padhraic Smyth. From group individual labels using deep features. In Proceedings of the 2 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 597-606. ACM, 2015.
Klaus Krippendorff. Content analysis: An introduction to its methodology. Sage publications, 2018
Pierre Laforgue, Stephan Clémengon, and Florence d' Alché Buc. Autoencoding any data through kernel autoencoders. arXiv preprint arXiv 1008, 2018.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity-promoting objective function for neural conversation models. arXiv preprint 1r1.v:15100030555 2015.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders. arXiv preprint arXiv: 1511::441 2015.
Saif M Mohammad. Word affect intensities. arXiv preprint arXiv: 1rXvv01704.08799. 2017.

---

Under review as a conference paper at ICLR 2020
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830 2011.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proc. of NAACL, 2018.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. URL https://s3-u-wwest-2 amazonaws. com/openai- http:///wwbbbbeeeeuuuuuuuuueeeee
understanding paper. pdf. 2018.
Sidney Resnick. Extreme values, regular variation and point processes. Springer, 2013.
Sam Shleifer. Low resource text classification with ulmfit and backtranslation. arXiv preprint vrXiv:1903.0924: 2019.
Alban Siffer, Pierre-Alain Fouque, Alexandre Termier, and Christine Largouet. Anomaly detection in streams with extreme value theory. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1067-1075. ACM, 2017.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp.
1631-1642, 2013.
A. Stephenson. Simulating multivariate extreme value distributions of logistic type. Extremes, 6(1): 49-59, 2003.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances neural information processing systems, pp. 3104-3112, 2014.
Jason Wang and Luis Perez. The effectiveness of data augmentation in image classification using deep learning. Convolutional Neural Networks Vis. Recognit, 2017.
Jason W Wei and Kai Zou. Eda: Easy data augmentation techniques for boosting performance on text classification tasks. arXiv preprint 17X1v:190.11199, 2019.
10

---

Under review as a conference paper at ICLR 2020
APPENDIX
A REMARKS
Remark (Selection of k) To the best of our knowledge, selection of lo in A and Algo- rithm 2 still vivid problem EVT which is not solved yet. As gets large the number of extreme points increases including samples which are not large enough and deviates from the asymptotic distribution of extremes. Smaller values of & increase the variance of the classifier/generator This bias-variame variance trade-off is beyond the scope of this paper.
In Figure seleccting the extreme samples on the input space is not straightforward step since the tails of the imput clusters are not heavy-tail. It was decided standardise the input data (X,))e(),:r.a)) by applying the aunk-trasfformatin
Remark 2
T(x) = (1((1-1(11)
1-1,-..
for all x (x), .....=))===).)))). E Rd where F def 1 Ei= l( S < is the j empirical marginal dis- n+I
tribution. It comes Vi E (1, 11,...................... ,n), ), V, T(X.I). V, margins follow standard Pareto distribution.
Extreme samples are thus (V, |[V 2 n/k} with k / by convention.
One may notice that extremes in Figure 3c are more label balanced compared to Figure 3a
B
EXPERIMENTS
Further synthetic experiments were conducted to show the improvement of the classifier.
B. SYNTHETIC DATA
Dataset description:
Synthetic data was generated using Scikit-Learn (Pedregosa et al. (2011
Focus is made on the classification dataset where clusters of points are normally distributed for each class (see Figure 3a).
Convergence to a logistic: In Figure 3b 3blwe we visualise the latent space of the output of the encoder + (see Figure 2). In Orthrus, the adversarial approach to minimize the discrepancy between the distribution of the output the encoder and a logistic distribution. Notice the similarity between Figure 3b and Figure 4a
Extremes selection: Extreme samples from Figure 3a are assessed directly on the input data while extremes from Figure 3c are obtained from the latent space. Extremes from the latent space (output of yp after the training phase) are located on the border of the distribution which comforts the correct behaviour of the training. We also note similarities while comparing the input space: extreme samples selected in the input space and extreme ones selected in the learnt latent space. In Figure 3a no extreme observation appear on the negative class whereas labels of extremes Figure 3c are balanced.
B.2 LOGISTIC DISTRIBUTION
Figure iillsstrate the logistic distribution with varying parameter ò.
B.3 ADDITIONAL EXPERIMENT SETTINGS FOR REAL DATA
We BERT pretrained models and code from the library pytorch-transformerss All models were implemented using Pytorch. The output of BERT is a R768 vector, Z is a R50 vector. The dimension of Z (50) was obtained by cross validation. D, Cailk and Cextt are MLP composed of layers and uses a ReLu as activation function. Decoders budk and Dew are GRU composed of 6 layers. For
training the batch size is set to 64 and we set λ₁ (1 P(]I Z ||] .λ2 P[[
IZ jon] [1)-1.x .Ag 10 3 All neural networks have been optimized with Adam (Kingma & Ba (2014)). For generation the maximum dilation factor is set to 1.5 since we observe that with greater
values a stuttering phenomenon appears.
11

---

Under review as a conference paper at ICLR 2020
30
not tremm 0
extreme 1
x(21
X(2)
bulk 0
bulk ecteme
ectremn C
uxtreme
extreme
x(1)
x(1)
x(1)
(a)
(b)
(c)
Bivariate synthetic samples normally distributed and designed for binary classification. Extreme observations are based on a standardisation of the observations in the input space (refer the Supplementary for details about the standardisation) Figure 3b) Bivariate outputs
Figure Figure 3a
of the encoder &o on the dataset visualised in Figure 3al Figure 3c Input space where extreme
observations are based on || where Z is the output of yp
300
200
206
1900
1800
(a) independence
(b) low dependence
(c) high dependence
Figure 4: Illutration of the direction @(V) obtained with bivariate samples V generated from logistic model with different coefficients of dependence from no dependence Figure 4a = 0.9) to high dependence Figure ( 0.1) through low dependence Figure 4b) 0.5) Non extreme samples are in gray, extreme samples are black and directions ((V) (extreme samples projected the sup norm sphere) in red. Note that not all extremes shown since the plot was truncated for a better visualization. However all projections on the sphere are shown.
B.4 ADDITIONAL EXPERIMENTS ON YELP DATA
dilation
dilation
10.
15
samples
samples
(a)
(b)
Figure 5: Evolution of the prediction of Orthrus on the whole test set (a) and on the extreme test set (b). Colors (black and white) represent predicted labels for each sample (A Frest) with À being the dilation factor (Y-axis).
Dilation invariance of Cex Figure 5a shows the evolution the prediction for each sample the test set applying the dilation. Predictions of Obblk change for some samples resulting in bichromatic columns. Figure 5b focuses on the samples from and is exclusively composed with monochromatic columns demonstrating that the predicted labels from C do not change, C is homothetic invariant. With no assumption on Ceest this classifier enjoys favorable theoretical properties since belongs to the class of angular classifiers at the end of the training phase.
12

---

Under review as a conference paper at ICLR 2020
imput
It wasa' I busy either), the building was cold. (it was not occupied cither), the bailding was cold.
Input
seriously killer hot chai latte.
==
'i'ss a real killer.
1.1
wasn't busy either), the building was frozen.
A 1.2
he is real killer.
1.3
also, the building was freezing.
A 1.3
he likes to kill.
1.5
plus. the building was colder than ioc.
1.5
murders.
Imput
food quality has been homible.
iuput
all of the lapus dishes were delicious
A
1
food quality has been lerrible.
A = 1
all the tapas was delicious.
1.1
the quality the food
horrible.
A 1.1
all tapas dishes were delicious!
A
1.3
the quality of the food has been horrible.
A= 1.3
all the tapas dishes were delicious!
1.5
the quality of food was terrible.
1.5
the tapas were great!
Input
overall like there food and the service
imput
there hardly any meal.
A
i love food and the service.
x= 1
there was almost no meat.
1.1
on whole, like
and service.
= L11
was practically no meat.
1.3
in general, i like the food and the service
A= 1.3
there was almost meat.
1.5
in general, i like food and service.
A= 1.5
there was no meat.
input
the desserts were bit strange
Inpot
waiter was a jerk.
A 1
the desserts were little weird.
A
the waiter was jerk. awaiter was poor guy. waiter an idior. waiter was such an idiot.
1.1
the desserts were very strange.
A= 1.1
1.3
the desserts terrible.
A= 1.3
= 1.5
the desserts were terrible.
A 1.5
input
we definitely enjoyed ourselves.
Input
cating here?
A 1
enjoyed ourselves,
A
i don't 1 eat here. don't cat here!
= 1.1
we had a lot of fun. we've really enjoyed each other. we certainly had fun.
X 1.1
1.3
X = 1.3
i'm not going eat here!
A = 1.5
A 1.5
i will never going to cat here!
Table 4: Supplementary sentences generated by Hydra for extreme embeddings implying label (sentiment) invariance for generated sentence. is the dilation factor.
B.5 ADDITIONAL EXPERIMENTS ON AMAZON DATA
Experiments on the Amazon dataset were conducted similarly to the ones from the Yelp dataset from /omm/susectionn 4.1 similar conclusions can be drawn from Figure 6
samples
samples
(b)
Figure 6: Evolution of the prediction of Orthrus on the whole test set (a) and on the extreme test set (b). Colors represent predicted labels for each sample (A Trest) with À being the dilatation factor (Y-axis).
B.6 QUALITATIVE EVALUCATION
For the qualitative evaluation, we compute the Krippendorff's alpha coefficient KrippeenoottKipppoooorrooo (2018) to evaluate the inter-annotator agreement. We obtain a 0.41 for the grammar analysis and = 0.53 for the sentiment analysis. If
it indicates perfect reliability; if a 0 it indicates the absence of reliability. Such values of a can be considered as relatively high given our huge number of annotators (126 for the grammar and 134 for the sentiment analysis).
B.7 ADDITIONAL GENERATED SENTENCES
We present in Table 4 additional extreme sentences.
C A LINGUISTIC POINT OF VIEW OF EXTREMES
In this section, we provide a comparative linguistic analysis of the extreme samples in case of sentiment classification through the affect scores provided by the lexicon for English words from
13

---

Under review as a conference paper at ICLR 2020
Mohammad (2017).
To better understand the content of extreme and non extreme text samples in experiments conducted
in section 4 the maximum and the minimum of the affect score for each sentence are studied (see
Figure 7).
notice that the violin plot of the maximun of the affect score is wider among the extremes than in the bulk. This illustrates that extreme tend to have both more positive and more negative affect score than the non extreme observations. Same goes for the minimum affect score of extremes whose violin plot tends to be larger among the extremes compared the violin plot of the minimum affect score on the bulk.
For both Yelp and Amazon datasets,
1.0
1.0
0.8
affect score
affect score
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
max extremes max bulk min extremes min bulk
max extreme max bulk min extreme min bulk Yelp
Amazon
(h)
Figure 7: Illutration of the affect score on the Amazon dataset (Figure 7a) and on the Yelp dataset (Figure 7b) dataset). The blue (respectively orange) violin plot represents the maximum affect score of sentences in the extreme (respectively non extreme) samples. The green (respectively red) violin plot represents the minimum affect score sentences in the extreme (respectively non extreme) samples.
14


Reviews:
[{"review_id": "rye-_wpCFS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new embedding method for sentences that aims to preserve dilation invariance.  Much of the methodology is justified by results for extremal point classification under particular assumptions, and then the authors try and encourage these assumptions to be met via penalty terms introduced in their embeddings/augmentation models.  However, while the proposed methodology seems interesting/novel, it remains conceptually unclear why it should be superior to standard text classification methods (ie. exactly what assumptions are being exploited to improve performance and how exactly do those assumptions help should be made more explicit). In particular, why is dilation invariance even a good idea?\n\nOverall, I find the paper a bit mathematically dense in Secs 2.1-2.2, which would not be a bad thing if the math were necessary to justify why the proposed methodology works well, but it in this case seems mainly presented as background material (as if it were a prerequisite to understand the method itself, which it is certainly not).\n\nWhy not instead present an explicit theorem providing some statistical guarantees for the proposed methodology in Sec 3.1 based on the constant-along-rays result (which would be nice to have regardless), and then follow up the theorem with background math from 2.1-2.2 which is necessary to understand the proof?\n\nAs it is currently written the paper is a bit too dense in terminology, and opaque names like Hydra and Orthrus used to describe straightforward concepts that are essentially a neural classifier (of a particular form) and a seq2seq-based data augmentation procedure (which would be good to describe in language more familiar to the ML audience). In particular, the goals of Hydra and Orthrus should first be intuitively described before delving into their various components.\n\n- Why did the authors never evaluate the overall sentiment prediction performance of Orthrus + Hydra used together vs other classifiers + data augmentation strategies?\n\n- If the goal of dilation invariance is to help the classifiers better generalize to out-of-distribution test sentences, then why not verify this happened, eg. by training on Yelp and testing on Amazon?\n\n- The authors should better justify the assumption of Jalalzai et al, and why this is appropriate for the MLP classifier used later in the paper.\n\n- This statement needs to be clarified and have citation: \"Such classifier whose output solely depends on the angle Θ(x) of the considered input, with provable guarantees concerning the classification risk in out-of-sample regions scaling as the square root of the number of extreme points used at the training step\"\n\n- The authors should explain Equation (1) in English rather than referring to it so early on the paper (pg 1: \"satisfying Equation 1\"). I had no idea what this was supposed to mean as a first time reader.\n\n- The way figure 4 is presented is a bit opaque and took me a while to understand (have to look closely at Fig 4a to see the columns are not monochromatic). \n\n- \"We also compare Hydra to a Vanilla Sequence to Sequence to demonstrate the validity of our approach\" How \"Vanilla Sequence to Sequence\" (word 'model' is missing) is used for dataset augmentation needs to be clarified here.\n\n- A figure demonstrating an example of the phenomenon explained in Sec 2.2 would be  helpful to aid reader's intuition.  \n\n- Typo: \"eugmentation\"", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1evAPSTtS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper explores learning dilation-invariant sentence representations, with a goal of improving downstream task performance on rare events. A pre-trained embedding is encoded as a latent variable Z, which is constrained to be multi-variate heavy tailed. Separate classifiers are trained on the head and tail of the distribution. Similarly, separate sentence generators are trained on the head and tail of the distribution, in order to allow data augmentation (creating diversity in the outputs by scaling the representation). While the high level motivation and algorithm is interesting, I found the paper very hard to follow, and the experiments are weak.\n\nI have quite a few concerns:\n- The algorithm takes a sentence embedding from BERT as input. BERT produces contextualized word representations, not sentence embeddings, so I don't know what the authors did here (the intro also claims that ELMo and GPT learn sentence embeddings, which is also confusing).\n- The paper argues that with empirical risk minimization, \"nothing guarantees that such classifiers perform satisfactorily\non the tails of the explanatory variables\". However, I could not follow what such guarantees the proposed method offers, if any.\n- Experiment 4.1 is impossible to follow without reading the appendix. This section should be expanded, or completely moved to the appendix.\n- The authors claim without evidence that a baseline of a neural network trained on top of the \"BERT embedding\" is state-of-the-art for sentiment classification. While there isn't enough information to know what was done, most state-of-the-art approaches involve fine-tuning BERT.\n- No comparisons are made with any other work, despite the method attempting a very general and well studied problem of text classification.\n- The submission claims that \"Applying a dilation is equivalent to assess the generalization of classifiers outside\nthe envelope of both training and testing samples.\". It isn't obvious to me that dilation captures the variation in embeddings you'd get from out-of-domain training samples.\n- The authors compare their data augmentation results to \"backtranslation\". The citation for the method appears to be a class project, and in fact does round-trip translation for paraphrasing, and not back translation.\n- No attempt is made to show if the data augmentation approach actually improves end task performance.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "HyeJgFbiFS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presented two methods for augmenting sentiment classification from the perspective of applying the Extreme Value Theory (EVT), including:\n\n1) A classification algorithm which has an adversarial classifier to enforce the intermediate representations of a neural network to be similar to one EVT distribution, logistic distribution;\n2) An encoder-decoder model that is able to generate grammatically coherent sentences with the same sentiment as the given input sentence.\n\nQuestions:\n\n--\n1) the Fisher–Tippett–Gnedenko theorem states that it is possible that the maximum value of a set of iid samples converges to one of three plausible distributions, and the chosen logistic distribution falls into the Weibull distribution category. I have a couple concerns about this choice:\n\n1.1) In order to show that the EVT indeed helps empirically in the way that an adversarial classifier enforces the inf-norm of vectors follow the Generalised Extreme Value (GEV) distribution, at least three plausible distributions from each form of the GEV distribution needs to be checked. The logistic distribution is interesting, but the marginal improvement gained by enforcing the lengths of the produced vectors to follow the logistic distribution could be a result of hyper-param tuning, which shouldn't be a piece of supporting evidence.\n\n1.2) From the perspective of applying the EVT, recent successful work from the best of my knowledge is on Anomaly Detection [1], where the EVT enables the system to learn from samples in only one class and also adjust the threshold for detecting the abnormal behaviour of samples. It is also theoretically grounded as the error variable of a logistic regression follows a Gumbel distribution which is one form of the GEV distribution, therefore, applying EVT for binary classification case makes sense.\n\n1.3) From the perspective of learning representations with structured priors, there exists an interesting work on decomposing vector representations into lengths and directions and enforcing lengths to follow a uniform distribution and directions a Von Mises–Fisher (vMF) distribution as in [2]. It would be interesting to see if the proposed method is indeed better than the way that structured priors are enforced in [2].\n\n1.4) Linguistically, given the distributional hypothesis, the length of learnt vectors tends to be highly correlated with the frequency information of available concepts and the direction of them matters more. The argument is also presented by the paper. However, in sentiment analysis, the length could contain the information about how strong the sentiment of the input sentence is, so I am not convinced that the proposed method would be applicable in fine-grained sentiment analysis, such as Stanford Sentiment Treebank [3]. \n\n\n--\n2) A soft approximation over the inf-norm of a set of iid samples is log-sum-exp function, and it is the cdf of softmax function, which is also theoretically grounded in EVT for classifications. It could be a nicer story than the current one as the choice of the logistic distribution seems to be too intend. \n\n\n--\n3) The construction of the two datasets seems to be very arbitrary given that there exists a large number of sentiment analysis datasets and many with lots of samples, I am not sure that the results on the chosen constructed two datasets are sufficient enough to support the claim.\n\n3.1) The size of the datasets is too small. Given that, the marginal improvement against the NN baseline could be a result of a specific initialisation, which doesn't generalise to other random initialisations.\n\n3.2) The dimension of vector representations is also too small. Normally, commonly used word embeddings are of 300 dimensions, and contextualised ones are of higher than 1200 dimensions. The chosen 50 dimension could prevent the NN baseline model to perform well and IMO, it is helpful for picking a suitable logistic prior than it is in a very high dimensional space.\n\n3.3) There are many straightforward distributions that could be applied as a prior on the lengths of vector representations, e.g. the Rayleigh distribution in 2D and the Chi-squared distribution in higher-dimension. Then again, the distribution gets flatter and becomes similar to a uniform distribution when the dim goes higher, which is a common issue. It goes back to my concern or doubt on the usability of a prior on the norm of high dimensional vectors.\n\n\n--\n4) I am still interested in seeing EVT being applied in various domains, but I'd be in favor of more justifiable approaches. \n\n[1] Siffer, Alban, et al. \"Anomaly detection in streams with extreme value theory.\" Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 2017.\n[2] Guu, Kelvin, et al. \"Generating sentences by editing prototypes.\" Transactions of the Association for Computational Linguistics 6 (2018): 437-450.\n[3] Socher, Richard, et al. \"Recursive deep models for semantic compositionality over a sentiment treebank.\" Proceedings of the 2013 conference on empirical methods in natural language processing. 2013.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Three reviewers recommend rejection. After a good rebuttal, the first reviewer is more positive about the paper yet still feels the paper is not ready for publication. The authors are encouraged to strengthen their work and resubmit to a future venue.
Decision: Reject