Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes a new embedding method for sentences that aims to preserve dilation invariance.  Much of the methodology is justified by results for extremal point classification under particular assumptions, and then the authors try and encourage these assumptions to be met via penalty terms introduced in their embeddings/augmentation models.  However, while the proposed methodology seems interesting/novel, it remains conceptually unclear why it should be superior to standard text classification methods (ie. exactly what assumptions are being exploited to improve performance and how exactly do those assumptions help should be made more explicit). In particular, why is dilation invariance even a good idea?

Overall, I find the paper a bit mathematically dense in Secs 2.1-2.2, which would not be a bad thing if the math were necessary to justify why the proposed methodology works well, but it in this case seems mainly presented as background material (as if it were a prerequisite to understand the method itself, which it is certainly not).

Why not instead present an explicit theorem providing some statistical guarantees for the proposed methodology in Sec 3.1 based on the constant-along-rays result (which would be nice to have regardless), and then follow up the theorem with background math from 2.1-2.2 which is necessary to understand the proof?

As it is currently written the paper is a bit too dense in terminology, and opaque names like Hydra and Orthrus used to describe straightforward concepts that are essentially a neural classifier (of a particular form) and a seq2seq-based data augmentation procedure (which would be good to describe in language more familiar to the ML audience). In particular, the goals of Hydra and Orthrus should first be intuitively described before delving into their various components.

- Why did the authors never evaluate the overall sentiment prediction performance of Orthrus + Hydra used together vs other classifiers + data augmentation strategies?

- If the goal of dilation invariance is to help the classifiers better generalize to out-of-distribution test sentences, then why not verify this happened, eg. by training on Yelp and testing on Amazon?

- The authors should better justify the assumption of Jalalzai et al, and why this is appropriate for the MLP classifier used later in the paper.

- This statement needs to be clarified and have citation: "Such classifier whose output solely depends on the angle Θ(x) of the considered input, with provable guarantees concerning the classification risk in out-of-sample regions scaling as the square root of the number of extreme points used at the training step"

- The authors should explain Equation (1) in English rather than referring to it so early on the paper (pg 1: "satisfying Equation 1"). I had no idea what this was supposed to mean as a first time reader.

- The way figure 4 is presented is a bit opaque and took me a while to understand (have to look closely at Fig 4a to see the columns are not monochromatic). 

- "We also compare Hydra to a Vanilla Sequence to Sequence to demonstrate the validity of our approach" How "Vanilla Sequence to Sequence" (word 'model' is missing) is used for dataset augmentation needs to be clarified here.

- A figure demonstrating an example of the phenomenon explained in Sec 2.2 would be  helpful to aid reader's intuition.  

- Typo: "eugmentation"
review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: The paper explores learning dilation-invariant sentence representations, with a goal of improving downstream task performance on rare events. A pre-trained embedding is encoded as a latent variable Z, which is constrained to be multi-variate heavy tailed. Separate classifiers are trained on the head and tail of the distribution. Similarly, separate sentence generators are trained on the head and tail of the distribution, in order to allow data augmentation (creating diversity in the outputs by scaling the representation). While the high level motivation and algorithm is interesting, I found the paper very hard to follow, and the experiments are weak.

I have quite a few concerns:
- The algorithm takes a sentence embedding from BERT as input. BERT produces contextualized word representations, not sentence embeddings, so I don't know what the authors did here (the intro also claims that ELMo and GPT learn sentence embeddings, which is also confusing).
- The paper argues that with empirical risk minimization, "nothing guarantees that such classifiers perform satisfactorily
on the tails of the explanatory variables". However, I could not follow what such guarantees the proposed method offers, if any.
- Experiment 4.1 is impossible to follow without reading the appendix. This section should be expanded, or completely moved to the appendix.
- The authors claim without evidence that a baseline of a neural network trained on top of the "BERT embedding" is state-of-the-art for sentiment classification. While there isn't enough information to know what was done, most state-of-the-art approaches involve fine-tuning BERT.
- No comparisons are made with any other work, despite the method attempting a very general and well studied problem of text classification.
- The submission claims that "Applying a dilation is equivalent to assess the generalization of classifiers outside
the envelope of both training and testing samples.". It isn't obvious to me that dilation captures the variation in embeddings you'd get from out-of-domain training samples.
- The authors compare their data augmentation results to "backtranslation". The citation for the method appears to be a class project, and in fact does round-trip translation for paraphrasing, and not back translation.
- No attempt is made to show if the data augmentation approach actually improves end task performance.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The paper presented two methods for augmenting sentiment classification from the perspective of applying the Extreme Value Theory (EVT), including:

1) A classification algorithm which has an adversarial classifier to enforce the intermediate representations of a neural network to be similar to one EVT distribution, logistic distribution;
2) An encoder-decoder model that is able to generate grammatically coherent sentences with the same sentiment as the given input sentence.

Questions:

--
1) the Fisher–Tippett–Gnedenko theorem states that it is possible that the maximum value of a set of iid samples converges to one of three plausible distributions, and the chosen logistic distribution falls into the Weibull distribution category. I have a couple concerns about this choice:

1.1) In order to show that the EVT indeed helps empirically in the way that an adversarial classifier enforces the inf-norm of vectors follow the Generalised Extreme Value (GEV) distribution, at least three plausible distributions from each form of the GEV distribution needs to be checked. The logistic distribution is interesting, but the marginal improvement gained by enforcing the lengths of the produced vectors to follow the logistic distribution could be a result of hyper-param tuning, which shouldn't be a piece of supporting evidence.

1.2) From the perspective of applying the EVT, recent successful work from the best of my knowledge is on Anomaly Detection [1], where the EVT enables the system to learn from samples in only one class and also adjust the threshold for detecting the abnormal behaviour of samples. It is also theoretically grounded as the error variable of a logistic regression follows a Gumbel distribution which is one form of the GEV distribution, therefore, applying EVT for binary classification case makes sense.

1.3) From the perspective of learning representations with structured priors, there exists an interesting work on decomposing vector representations into lengths and directions and enforcing lengths to follow a uniform distribution and directions a Von Mises–Fisher (vMF) distribution as in [2]. It would be interesting to see if the proposed method is indeed better than the way that structured priors are enforced in [2].

1.4) Linguistically, given the distributional hypothesis, the length of learnt vectors tends to be highly correlated with the frequency information of available concepts and the direction of them matters more. The argument is also presented by the paper. However, in sentiment analysis, the length could contain the information about how strong the sentiment of the input sentence is, so I am not convinced that the proposed method would be applicable in fine-grained sentiment analysis, such as Stanford Sentiment Treebank [3]. 


--
2) A soft approximation over the inf-norm of a set of iid samples is log-sum-exp function, and it is the cdf of softmax function, which is also theoretically grounded in EVT for classifications. It could be a nicer story than the current one as the choice of the logistic distribution seems to be too intend. 


--
3) The construction of the two datasets seems to be very arbitrary given that there exists a large number of sentiment analysis datasets and many with lots of samples, I am not sure that the results on the chosen constructed two datasets are sufficient enough to support the claim.

3.1) The size of the datasets is too small. Given that, the marginal improvement against the NN baseline could be a result of a specific initialisation, which doesn't generalise to other random initialisations.

3.2) The dimension of vector representations is also too small. Normally, commonly used word embeddings are of 300 dimensions, and contextualised ones are of higher than 1200 dimensions. The chosen 50 dimension could prevent the NN baseline model to perform well and IMO, it is helpful for picking a suitable logistic prior than it is in a very high dimensional space.

3.3) There are many straightforward distributions that could be applied as a prior on the lengths of vector representations, e.g. the Rayleigh distribution in 2D and the Chi-squared distribution in higher-dimension. Then again, the distribution gets flatter and becomes similar to a uniform distribution when the dim goes higher, which is a common issue. It goes back to my concern or doubt on the usability of a prior on the norm of high dimensional vectors.


--
4) I am still interested in seeing EVT being applied in various domains, but I'd be in favor of more justifiable approaches. 

[1] Siffer, Alban, et al. "Anomaly detection in streams with extreme value theory." Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 2017.
[2] Guu, Kelvin, et al. "Generating sentences by editing prototypes." Transactions of the Association for Computational Linguistics 6 (2018): 437-450.
[3] Socher, Richard, et al. "Recursive deep models for semantic compositionality over a sentiment treebank." Proceedings of the 2013 conference on empirical methods in natural language processing. 2013.
review_title: Official Blind Review #3

Meta-review: Three reviewers recommend rejection. After a good rebuttal, the first reviewer is more positive about the paper yet still feels the paper is not ready for publication. The authors are encouraged to strengthen their work and resubmit to a future venue.
Decision: Reject