Learning Entailment-Based Sentence Embeddings from Natural Language Inference
Authors: Rabeeh Karimi Mahabadi*
Florian Mai*
James Henderson
Venue:  (2020)
Keywords: sentence embeddings
textual entailment
natural language inference
interpretability

Abstract: Large datasets on natural language inference are a potentially valuable resource for inducing semantic representations of natural language sentences.  But in many such models the embeddings computed by the sentence encoder goes through an MLP-based interaction layer before predicting its label, and thus some of the information about textual entailment is encoded in the interpretation of sentence embeddings given by this parameterised MLP.  
In this work we propose a simple interaction layer based on predefined entailment and contradiction scores applied directly to the sentence embeddings.  This parameter-free interaction model achieves results on natural language inference competitive with MLP-based models, demonstrating that the trained sentence embeddings directly represent the information needed for textual entailment, and the inductive bias of this model leads to better generalisation to other related datasets.

Under review as a conference paper at ICLR 2020
LEARNING ENTAILMENT-BASED SENTENCE EMBED- DINGS FROM NATURAL LANGUAGE INFERENCE
Anonymous authors Paper under double-blind review
ABSTRACT
Large datasets natural language inference potentially valuable resource for inducing semantic representations of natural language sentences. But in many such models the embeddings computed by the sentence encoder goes through MI.P-based interaction layer before predicting its label, and thus some of the in- formation about textual entailment is encoded in the interpretation of sentence embeddings given by this parameterised MLP. In this work we propose a sim- ple interaction layer based on predefined entailment and contradiction scores ap- plied directly to the sentence embeddings. This parameter-free interaction model achieves results on natural language inference competitive with MLP-based mod- els, demonstrating that the trained sentence embeddings directly represent the in- formation needed for textual entailment. The inductive bias of this model leads better generalisation to other natural language inference datasets, and the resulting sentence embeddings are better for other semantic tasks.
INTRODUCTION
After the enormous success of learning representations of words, learning representations of sen- tences has become one the most important challenges in natural language understanding. Many models have been proposed for embedding sentences in vector space, but the methods for inter- preting those representations are dominated by two approaches: using the dot product (or cosine) to measure the similarity between vectors, training a parameterised function to do the in- terpretation for specific tasks (e.g. (Conneau et al., (0177). In this paper we consider different form of interpretation, based neither on similarity nor a parameterised function. We investigate embedding sentences into vector representations of information inclusion and contradiction.
1.1 MOTIVATION
Information inclusion, called entailment, is fundamental concept in theories of the semanties of natural language. This motivated the proposal of recognising entailment in text as a generic task which captures very wide range of issues in natural language understanding (Dagan et al.. 2005), which has been demonstrated by the conversion of a large number of semantic datasets into a single textual entailment task (Poliak al.) (0088).. The fundamental nature of this task has motivated the development of many datasets for textual entailment (Poliak et al. (018a). The more recent textual entailment datasets have extended this task include annotations for contradiction, which are often referred to as natural language inference (NLI) datasets.
Modelling these semantic relationships between sentences not only crucial problem in natu- ral language processing, but it also has wide variety of applications, such as question answer- ing (Harabagiu Hickl, (006), producing abstract summaries (Lacatusu et al. (0066] Yan et al., b)) and machine translation evaluation (Padó et al., (009). is possible to transfer the predic- tions of an NLI model directly to the task, but has been shown that NLI models trained on large
NLI benchmarks such as SNLI (Bowman et al.) 2015) and MNLI (Williams et al. (018a) do not transfer well to other NLI benchmarks (Talman & Chatzikyriakidis 2019). Despite recent efforts to improve generalisation (Belinkov al. 2019b),, this remains open question. A more flexible and powerful approach is to transfer the knowledge learned about the NLI task to other tasks with representation learning. For example, Conneau al. (207) train sentence embeddings the NLJ

---

Under review as a conference paper at ICLR 2020
task and demonstrate their usefulness other tasks. But there is still no clear interpretation of these embeddings in terms of entailment, other than via a trained parameterised function.
In this work we propose novel approach to learning sentence embeddings from NLI data, where the sentence embeddings have a direct interpretation in terms of entailment and contradiction. Anal- ogously to the use of the dot product cosine to define similarity over a vector space, we define entailment and contradiction as two parameter-free frree operators over pairs of vectors. The entailment operator provides a score of how well one vector includes all the information in the other vector. The contradiction operator provides a score of to what extent two vectors contradict each other. We use these operators to train sentence encoders which embed sentences into vector space which has these interpretations, using data for the Natural Language Inference task.
1.2 OVERVIEW
Natural language inference involves predicting whether a premise entails the given hypothesis, whether they contradict each other, or neither (Bowman et al., 2015). Due to the availability of large-scale datasets for natural language inference (Bowman et al. 2015 Williams et al.] 2018b), neural network models have become popular for textual entailment. We can characterise these mod- els in terms of three stages. First, the encoding stage computes d-dimensional embeddings of the premise and the hypothesis. Then the interaction layer models the interactions between the two sentences. Finally classifier layer makes the prediction of entailment, contradiction or neutral. While most of the literature has been concerned with improving the encoding stage, in this paper we focus on the interaction layer.
For the interaction layer, passing the sentence embeddings plus their heuristic matching features (Mou et al.] 2016) through multi-layer perceptron (MIP) has become the standard practice for textual entai entailment ment models (Conneau et al.
2017] [Bowman et al. (2016) Kim et al.] 2019 Pan et al.) Conneau et al., 2018b). The heuristic matching features are component wimp measures of similarity between premise and hypothesis, most commonly imple- mented as the absolute difference and the component-wit wise multiplication. However, textual entail- ment is not only about similarity, since it reflects an information inclusion relationship. Moreover, most similarity features are symmetric, whereas entailment an intrinsically asymmetric relation. It is only after the MLP that the sentence embeddings and similarity- -based features get transformed into a representation which can directly mapped to entailment, contradiction and neutral. Often the MLP plays a significant role in the model's 's architecture, for example requiring 800k of the total 2. Im model parameters in the state-of-the-arr Dynamic Self-Attentinn model YYonn et al. 2018).
2018 Yoon al., 2018 Kiela al. [2018]
In this paper, propose parameter-free interaction layer consisting of only a few scores, including entailment and contradiction, defined in terms of the entailment-vent vectors framework of Henderson & Popal (016)), which are passed to a linear classification layer. This forces the sentence em- beddings to have a direct interpretation in terms of entailment and contradiction, and imposes an inductive bias towards learning entailment and contradiction.
Despite drastically reducing the number of trained parameters, this simple interaction layer is em- pirically competitive with MLP-based models. Because it is parameter-free, all of the information learned about entailment and contradiction must be encoded directly in the sentence embeddings, rather than indirectly via its interpretation by the trained MLP. Thus information about entailment or contradiction can be transferred to other tasks simply by transferring the sentence embeddings. An ablation analysis indicates that the entailment and contradiction scores are indeed the main factors in the model's success. When we test our trained models on related datasets that have not been used for training, we find that the inductive bias of our proposed model results in better transfer performance. When we test our induced sentence embeddings on other tasks, we find better transfer performance on semantic tasks.
In summary, we propose novel, theoretically grounded, parameter-free model of the interactions between premise and hypothesis for entailment and contradiction. Evaluation on benchmark natural language inference datasets show this model is competitive with modelling interaction with an MLP. The resulting sentence embeddings have a direct entailment based interpretation, and the model's inductive bias results in better transfer different IIII datasets and different semantic tasks.
2

---

Under review as a conference paper at ICLR 2020
2 MODELLING ENTAILMENT AND CONTRADICTION RELATIONS
The task of textual entailment takes premise and hypothesis sentences and classifies their relation- ship into entai Iment, contradic ct ion, neutral. To train sentence embeddings, adopt the widely used approach of first encoding each sentence in an embedding, then modelling the interactions between the premise and hypothesis, followed by softmax classifier which estimates the probability of each label (Conneau et al. 2018b).
2.1 ENCODING STAGE
The encoding stage embeds the premise sl and hypothesis s2 into two d'-dimensional vector rep- resentations p.h = enc(s1, where enc is sentence encoder, and and h are the premise and hypothesis embeddings respectively. There are two types of encoders; intra-sentence encoders only look each sentence individually, while inter-sentence encoders consider both sentences. In our first set of experiments, we consider models with both single-sentence and inter-sentence encoders, specifically the two baselines from the GLUE benchmark (Wang al. (019). The single-sentence encoder a bidirectional LSTM (Bi-LSTM) with max-pooling, which Conneau et al. (2077) found work best among diverse set of encoders. The inter-sentence encoder adds attention over the other sentence (Wang et al., 2019). In our second set of experiments, we consider models with the single-sentence encoder of (Reimers & Gurevych] 2019), based on pre-trained BERT model Devlin et al. 2019
2.2 MLP INTERACTION STAGE
Most previous neural network approaches to NLI model the interaction between sentences with a non-linear MLP. The input to this MLP includes the embeddings of the hypothesis h and premise obtained from the encoding stage. Mou et al. (2016) proposed adding heuristic matching features in order to capture the similarity of the two sentences, which have since been widely adopted (Conneau
et al. 2017 Chen et al. 2017 [owmman et al.) 2016 Kim et al. 2019) Pan et al.. 2018 Yoon et al. 2018 Kiela al. 2018). These features most commonly the element-wing product and absolute difference between the two sentence embeddings, giving the input vector m.
m [p; h; p h1: o h]
The MLP then computes a hidden layer, from which the softmax classifier predicts the labels.
class softmax We tanh(W m ba) be)
where We E 3 b 33 We Rnx4d be E R", and n is the size of the hidden layer. Because of the high dimensionality of the sentence embeddings, the number of parameters in this interaction layer (W,) can be large. Following Wang et al. (201)), we set n 512.
2.3 THE PROPOSED INTERACTION STAGE
We propose alternative interaction stage with parameters. The input is the two sentence beddings, and the output is just 5 scores, which are then input to the softmax classifier. These predefined scores provide strong inductive bias to the model, and force the sentence embeddings to have the interpretation required by these scores.
Normalisation In the scores, the sentence embeddings are first passed through sigmoid func- tion, and we want them to use the full output range of this function. We therefore apply layer normalisation the sentence embeddings make their mean zero and their standard deviation s, where s is a hyperparameter set to 6 in all experiments. We use to refer to the embeddings p, h respectively after applying normalisation.
Entailment Vectors Framework Henderson & Popal (2016) propose vector-space framework for entailment which explicitly models the information known about a word by interpreting each dimension of a word embedding the probability that a certain feature is known to be present. So zero is interpreted as unknown, and not as necessarily false. Entailment when all the features that are known about the entailed vector are also known about the entailing vector. They propose
3

---

Under review as a conference paper at ICLR 2020
different operators for computing the probability of this entailment. Previous work only applied this framework to lexical entailment (Henderson & Popa 2016), but we apply it to rextual entailment.
Modelling Contradictory Information The original entailment vectors framework only models binary entailment. We extend this representation also model contradiction. We split entailment vectors into two blocks, one 1: for modelling features f and another 2idd for their negation -f (as suggested by Henderson & Popa (016)) for a different purpose). So if æk. E {1...., 4 represents that something known be frue, then #*+ 1 represents that it is known be false, and thus having both these features active a contradiction.
2
We compute the entailment score between two sentences using the factorised entailment operator (y=x) proposed by Heneersoon & Popa (016)), which measures the probability of entailment the k-th dimension as 55 (entail] x, y) entailment to be probable in every dimension, giving us:
Entailment Score:
(a)o())), and which requires
S(entail y) = CII (o)o)))))(
1=
Contradiction Score: We consider two sentences contradict each other if their respective em- beddings have contradictory features in at least one dimension. Our contradiction score for the k-th dimension E {1..... 2 thus:
Sy (cCotrrraee y) o (xx))o(ye+ ( ))(())))))))))) (xx)) )0(4 ())()))
where the first term models that the feature is known to be true in the hypothesis and known to be false in premise, the second term the reverse, and the third term both. The total score of contradiction is the complement of not having contradiction in any dimension:
$
S(contradict|z, y) =1 II (1 - Ss (contrradii y)) ==1
Neutral Score: We define neutral score the non-negative complement of the contradiction and entailment scores, computed as:
S(neutral æ y) = ReLU(1 S(entail| æ. y) S(contradict y)),
The ReLU function avoids negative scores, and its nonlinearity makes this score non-redundant in the log-linear softmax classifier.
Similarity Scores:
Since similarity information is known to help textual entailment (Mou et al.) 2016), we employ two scores can be viewed as condensed versions of the heuristic matching Teatures, namely averages of element-wise multiplication and absolute difference, respectively:
d
sisTimut (c, y)
aa (x)))
d 1=
simay (z, y) d Σ (1o(#x) (vv)))
Initialisation For the success of our models, we found it critical to initialise the output layer in such a way that the scores are associated with their respective class label. To this end, we add I to the random initial weight connecting the entailment score to the entailment class, the contradiction score to the contradiction class, and the neutral score the neutral class, respectively. We observed that without this initialisation the model takes several epochs to train before achieving better than random performance.
4

---

Under review as a conference paper at ICLR 2020
3 EVALUATION ON NLI DATASETS
We first evaluate our approach to learning sentence embeddings from NLI data without using any resources external the NLI datasets themselves. We train and test our models on two popular large- scale datasets for textual entailment, SNLI (Bowman et al. 2015) and MNLI (Williams et al. 201 8a). In addition evaluating classification performance, we conduct an ablation study to determine whether our sentence embeddings and associated scores have the interpretations claimed.
3.1 EXPERIMENTAL SETUP
For the LSTee-based sentence encoders, we always use Bi-LSTM state vectors with 512 dimensions. With larger models (2048 dimensions) the results were only slightly better and the pattern of results did not change, while the computational cost was substantially higher. We train these models for 20 epochs, use validation accuracy for early stopping, and use mini-batch SGD with batch size of 64. The learning rate is selected among {0.06, 0.08, 0.1, 0.3, 0.5) based on the validation scores.
For MNLI, we tuned the models on the mismatched validation set, and report results for both mis- matched and matched validation sets, only reporting test set results for the best baseline and our best models.
3.2 BASELINES
compared our model with several baselines: 1) p.h: Using only sentence embeddings followed features input MLP (i.e. the baseline from Conneau ct al. (2017 ); 3) Random: compute five random scores followed by a linear softmax classifier. For baseline (3), use untrained nonlinear projections of the embeddings p, h of the form:
by an MLP; 2) HM:
(W go (W.(p; h] b.) by).
where the weight matrices W; E Rdx2d W, E gaxd and their corresponding biases are randomly generated using Glorot Initialisation (Glorot & Bengio 2010). This baseline evaluates how much performance being gained by employing our specific five scores, with their inductive bias.
Validation and test accuracies (for MNLI, on matched/mismatcheed sets), and num- ber of parameters in the encoder (#enc) and MLP and/or classifier (#mlp).
Table 1:
Table 2: Ablation results for the scores (C)ontradiction, (E)ntailment, (N)eutral, and (S)imilarity, on SNLI and MNLI (matched/mismathhed) validation sets.
Model
#ene
#mlp
SNLI
MNLI
Used scores
SNLI MNLI
Random
3.3m
18
79.07
65.91 / 65.88
C,
83.47
69.97 / 70.51
p.h
3.3m
1.3m
78.70
64.70/ 65.69
E,C.N
83.14
69.19 / 69.97
E.C
78.02
69.49 / 69.66
HM
3.3m
2.4m
84.82
71.23 71.46
S
75.48
63.03 63.31
Ours
3.3m
18
83.47
69.97 / 70.51
E
78.62
63.57 63.92
HM+attn
13.8m
2.4m
86.46
74.81 / 74.81
74.7
58.19 / 58.96
Ours+attn
13.8m
18
86.28
74.21 74.41
Test set results
HM
3.3m
2.4m
84.79
71.22 / 70.36
Ours
3.3m
18
83.09
69.79 68.96
HM+attn
13.8m
2.4m
85.52
74.37 73.60
Ours+attn
13.8m
18
85.49
73.14 / 72.74
3.3
RESULTS
Table I shows the results. First, note that random scores already achieve reasonable performance, at about the level of the MLP-based baseline p.h. This suggests that the encoder so powerful that generate embeddings suitable for virtually any kind of interaction model. Nonetheless, the
5

---

Under review as a conference paper at ICLR 2020
performance of our interaction model is over 4 points better than random scores, indicating that our proposed scores impose an inductive bias which is well-suited for textual entailment.
For single-sentence encoders, interaction model performs slightly worse than the MLP-based model. The fact that our models' performance close indicates that our model is capturing nearly the same amount of information about NLI, and it has the advantage that all this learned infor- mation encoded in the sentence embeddings, not in the MLP. When we move to inter-sentence encoders (+attn), the difference reduces in every case, and virtuallyy eliminated for SNLI. Since an inter-sentence encoder is able to model some interaction in the encoder itself, this suggests that the performance difference in part due the reduced power of our parameter-free interaction model compared to an MLP. For example, our interaction model with a single-sentence encoder does not seem powerful enough to model known artefacts in the datasets (Gururangan et al. 2018), which we will discuss further in Section 4
3.4 ABLATION STUDY
To better understand the effect of each the scores, we performed an ablation study by evaluating models with subsets of our proposed scores, shown in Table The two similarity scores alone (S) perform rather poorly, indicating that similarity information alone is not enough to model entailment. In contrast, using only the three entailment scores (E,C,N) achieves very good performance, yielding within 0.66 points of the full model on MNLI and within 0.33 points of the full model on SNLI. Removing the neutral score had surprisingly large impact on SNLI, but otherwise the neutral score is less important than the contradiction score, which is less important than the entailment score.
To further understand the roles of the different scores, consider the trained weights of the final classification layer for the model with only the entailment, contradiction and neutral scores (E,C,N):
Sg
SN
Se
E
+41.3
+0.2
-2.0
26.
W: N
10.8
-3.
35.0
b,
+21.
+5.3
C
29.5
44.1
+60.0
where the rows denote the labels and the columns denote the scores. The very large weights in the first and last columns indicate that indeed the entailment score predicts entailment and the contradic- tion score predicts contradiction. The neutral score weights interact with the biases to compensate for when the entailment and contradiction scores are both high.
4 EVALUATION OF TRANSFER PERFORMANCE
One motivation for using sentence embeddings which are directly interpretable in terms of entail- ment and contradiction that the resulting interaction model imposes stronger inductive bias towards learning entailment and contradiction. Such inductive biases are often unhelpful when eval- uating within the domain of a given dataset, particularly because the large NLI datasets, used the previous section, contain annotation artefacts which allow high performance to be achieved without actually modelling entailment and contradiction (Gururangan et al., 2018). To evaluate the useful- ness of this inductive bias for learning the more general regularities behind textual entailment, evaluate our proposed model on its ability to generalise to a large number of different NLI datasets which do not share these same annotation biases.
We conduct these transfer experiments using sentence encoders which have been pretrained on very large amounts of unannotated text using a language modelling objective, namely the BERT model (Devlin et al. 2019). This pretraining has previously been demonstrated to help general- isation large number of tasks (Devlin al., 2019), including state of the art results in NLI. In this section. we first evaluate our approach to learning sentence embeddings from NLI data by fine-tuning a pretrained BERT model, replicating the results from the previous section on the perfor- mance of the resulting NLI m We then evaluate the resulting sentence embeddings on their model's s ability to generalise to large number of other related tasks. addition, we evaluate the resulting sentence embeddings transfer to sentence classification tasks.
We do not replicate the full range of experiments in this context because of the high computational cost of training with BERT pretrained models.

---

Under review as a conference paper at ICLR 2020
4.1 MODELS
The original BERT model of NLI (Devlin et al. 2019) encodes the premise and hypothesis together and uses the embedding of the CLS token at the last layer as input classifier. But this model does not produce individual sentence embeddings for the premise and hypothesis. To get a sen- tence embedding model based on BERT, we use the Sentence- Berece--TTTT-e (SBERT) model proposed by Reimers & Gurevych| (219). Reimers & Gurevych| (2019) encode each sentence individually using a pretrained BERT model, then obtain the final fixed-sized embedding by average- serage-poolling over the outputs at the last layer. Average pooling found to be better than max pooling or using the CLS token embedding.
For the baseline model (HM+SBERT), after obtaining embedding for the premise and the hypoth- esis, we proceed similarly to Section 2.2 First the baseline computes heuristic matching features from the sentence embeddings, but instead of using MIP classifier, it uses a log-linear classifier in accordance with Reimers & Gurevych (2019 Our model (Ours+SBERT) computes the 5 scores defined above and uses them in a log-linear classifier.
4.2 EXPERIMENTAL SETUP
We use BERT-Large as the foundation and finetune it with AdamW (Loshchilov & Hutter 2019) We generally use universal initial learning rate of 2e-5 for all heuristic matching models. When training our entailment- the parameters of the classifier. On SNLL, found learning rate of 2e-5 for training the BERT parameters and not initialising the output layer as described in Section 2.3 to work best. On MNLI, learning rate of 2e-6 with initialisation worked best. In all cases, there is warm-up phase lasting for 30% of the training dataset. For SNLI, we used a batch size of 8 whereas on MNLI we had to reduce the batch size 4 because of limited memory.
model, a significantly larger learning rate of 0.006 was necessary for
4.3 NLI RESULTS
When training and testing on the same NLI corpora, finetuning a pretrained BERT model gives us the same pattern of results in Section 3.3 shown in Table B] We see a small decrease in accuracies for both SNLI and MNLI for our much simpler interaction model, exacerbated by the fact that there is no interaction in the encoding stage. We hypothesise that this decrease largely due to the inability of our simpler model to capture the annotation biases in these datasets. In the next section, investigate this issue by testing on datasets which do not share the same annotation biases.
Table Validation accuracies(for MNLI, on matched/miismtched sets), and number of parameters in the encoder (#enc) and MIP and/or classifier (#mlp).
Model
#enc
#mlp
SNLI
MNLI
HM+SBERT
13.8m
12.3k
87.2
76.1 / 76.2
SBERT
13.8m
18
85.1
74.6 73.7
Test set results
HM+SBERT
13.8m
12.3k
86.0
75.2 75.6
OuIs+SBERT
13.8m 18
85.1
73.8 / 74.2
4.4 TRANSFER PERFORMANCE
Our model introduces a strong inductive bias for textual entailment that forces almost all the in- formation about entailment to be contained in the sentence embeddings rather than the classifier. This is potentially very useful for transfer learning in two ways: transfer of the whole model to
We found that using ML. instead of a log-linear classifier resulted in the same validation performance on SNLI and worse validation performance on MNL., particularly for the mismatched set.
7

---

Under review as a conference paper at ICLR 2020
out-of-domain NLI datasets, and transfer of the sentence embeddings features for downstream tasks.
4.4.1 RESULTS ON OUT-OF-DOMAIN NLI DATASETS
To evaluate how well the baseline and proposed models generalise to solving textual entailment in domains which do not share the same annotation biases the large NLI training sets used above, we take trained NLI models and test them on a number of different NLI datasets.
Datasets: We consider total of 11 different NLI datasets. We use the 10 datasets studied by Poliak al. (2018b). These datasets include MNLI, SNLI, SciTail (Khot et al. 2018). AddOneRTE
(Pavlick & Callison-Burch 2)16). Johns Hopkins Ordinal Commonsense Inference (JOCI) (Zhang)
al. 2017), Multiple Premise Entailment (MPE) (Lai et al. 2017). Sentences Involving Composi-
tional Knowledge (SICK) (Marelli et al. 2014), and three datasets from White et al. (2017) which
are automatically generated from existing datasets for other NLP tasks including: Semantic Proto-
Roles (SPR) (Reisinger et al. 2015), Definite Pronoun Resolution (DPR) (Rahman & Ng 2012)
and FrameNet Plus (FN+) (Pavlick et al. 2013). We additionally consider the Recognising Textual Entailment dev dataset (RTE) from GLUE benchmark |Wang et al. (2019), and the Quora Question Pairs (QQP) d where the task is to determine whether two given questions are semantically matching (duplicate) not. As in Gong et al. (2017). we interpret duplicate question pairs as an entailment relation and neutral otherwise. We use the same split ratio mentioned by Wang et al. 2017
Since the datasets considered have different label spaces, when evaluating on each target dataset, we map the model's labels to the corresponding target dataset's space. We train all models on MNLI and evaluate their performance on other target datasets. MNLI contains three labels, contradiction, neutral, and entailment. Some of the datasets we consider contain only two labels. In the case of labels entailed and not-entailed, as in DPR, we map contradiction and neutral to the not-entailed class. In the case labels entailment and neutral, as SciTail, we map contradiction to neutral.
Table 4: Accuracy results of models transferring to new datasets. All models are trained on MNLI and tested on target test sets. A are absolute differences between our method and the baseline.
Methods
Target Test Dataset
Baseline
Ours
A Ours
RTE
48.38
64.98
+16.6
JOCI
41.14
45.58
+4.44
SCITAIL
68.02
71.59
+3.57
SPR
50.84
53.74
+2,9
QQP
68.8
69.7
+0.9
DPR
49.95
49.95
FN+
43.04
42.81
-0.23
SICK
56.57
54.03
-2.54
MPE
48.1
41.0
-7.10
ADD-ONE-RTE
29.2
17.05
-12.15
SNLI
64.96
54.14
-10.82
Results Table shows the transfer performance of the bascline and proposed SBERT models to other NLI datasets. As shown in prior work (Belinkov et al.) 2019a), the SNLI dataset has very similar annotation biases the MNLI data which the models were trained on, so we do not expect any improvement in the relative performance of the two models for SNLI. Also, for the ADD-ONE- RTE dataset, both models essentially fail generalise to this data, so the comparison not very meaningful. For the remaining datasets, our proposed model performs better than the baseline on datasets, the same on one, and worse on 3. On average, this represents a substantial improvement
https://date quora com/ rst Data---atatteee. Release Quest Pnpairs
8

---

Under review as a conference paper at ICLR 2020
in generalisation by using the proposed inductive bias, especially given that model started out from a lower accuracy on the in-domain data.
4.4.2 RESULTS ON DOWNSTREAM TASKS
Because our model does not powerful MIP classifier top of the sentence embeddings, has to encode almost all the semantics about entailment in the sentence embeddings themselves. This should make the sentence embeddings more useful when used features in downstream tasks.
To test this hypothesis, we evaluate the same SBERT-based sentence encoders from Section 4.4.1 on downstream tasks from the SentEval framework (Conneau & Kiela 2018), which has previ- ously been used to evaluate sentence embeddings trained on the textual entailment task (Conneau al. 2017 Reimers & Gurevych] 2019). We use the default evaluation configuration, where for the single-eentence classification tasks a logistic regression classifier is trained on top of the given sentence embedding. We exclude the supervised sentence-pair tasks (STS-B, SICK-E, SICK-R, and MRPC) from our analysis because the SentEval implementation uses heuristic matching fea- tures for these experiments and thus is incompatible with our model. For related tasks, see the NLI experiments in section 4.4.
We report the downstream performance on supervised tasks Table 5 and on unsupervised tasks in Table In 10 out of 12 datasets, the sentence embeddings from model achieve a substantial im- provement over the sentence embeddings from the baseline. In the remaining two cases (TREC and STS15), the performance is approximately the same. These results support the hypothesis that model produces sentence embeddings that carry more semantic information that are useful features in downstream tasks.
Table 5: Scores supervised downstream tasks from SentEval toolkit attained by our model and the baseline.
Model
MR
CR
MPQA
SUBJ
SST2
SST5
TREC
Ours+
84.76
90.57
89.88
98.57
90.50
82.6 86.44
49.14
HM+SBERT
80.27
88.77
88.07
90.74
46.56
83.0
A
+4.49
+1.8
+1.81
+2.83
+4.06
+2.58
-0.4
Table Scores on unsupervised downstream tasks from SentEval toolkit attained by our model and the baseline. Numbers reported are Pearson correlations x 100.
Model
STS12
STS13
STS14
STS15
STS16
SBERT
61.25
60.58
66.18
66.85
67.40
HM+SBERT
53.39
50.65
62.89
66.53
63.51
A
+7.86
+9.93
+3.29
+0.32
+3.89
To get better understanding of what type of features our sentence embeddings encode, we eval- uate them on the 10 linguistic probing tasks introduced by Conneau et al. (2018a). These tasks are grouped into three categories. The first two are 1) whether the embedding encodes surface information like word content (WC) or sentence length (SentLen), and 2) whether it encodes syn- tuctic information like word order (BShift) or information about the parse tree structure (TopConst, Depth). The last category tests for semantic information. However, [onneau al. (2018a) note that three of these tasks can arguably be solved mainly by considering surface information at the sub- word level (which are available to BERT), namely identifying the tense (Tense), and identifying the number of the subject (SubjNum) and object (ObjNum). The last two tasks require solid semantic understanding of the encoded sentences, namely whether random noun was replaced fodd-man-out (OMO)) and whether the coordinate clause was swapped with the main clause (CoordInv).
The results of our model in comparison with the baseline are shown in Table With the exception of BShift, the baseline encodes substantially more surface and syntactic information. In contrast,

---

Under review as a conference paper at ICLR 2020
model is superior at the semantically challenging tasks OMO and CoordInv, further supporting the hypothesis that our sentence embeddings encode more useful semantic features than the baseline.
Table 7: Scores on the probing tasks attained by our model and the baseline.
Model
Length
WC
Depth
TopConst
BShift
Tense
SubjNum
ObjNum
OMO
Coordiny
OuIS+SBERT
54.94
46.60
28.27
55.66
79.58
82.56
74.16
73.89
63.33
66.42
HM+SBERT
75.19
55.14
33.86
65.76
70.78
83.67
78.70
78.43
55.72
62.50
-20.25
-8.54
-5.59
-10.1
+8.8
-1.11
-4.54
-4.54
+7.61
+3.92
CONCLUSION
Both the ablation study and the full NLI results demonstrate that our proposed entailment and con- tradiction scores are effective for modelling textual entailment. This parameter-free model of the relationship between premise and hypothesis forces training to put all information about entailment and contradiction in the sentence embeddings, rather than in the parameters of an MLP classifier, and gives those sentence embeddings direct entailment based interpretation. We also show that the inductive bias given by forcing the learned sentence embeddings to have this entailment- based inter- pretation results in better generalisations to other NLI datasets and more useful sentence embeddings for other semantic tasks.
REFERENCES
Yonatan Belinkov, Adam Poliak, Stuart Shieber, Benjamin Van Durme, and Alexander Rush. Don't take the premise for granted: Mitigating artifacts in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, 2019a.
Yonatan Belinkov, Adam Poliak, Stuart M Shieber, Benjamin Van Durme, and Alexander Rush. On adversarial removal of hypothesis-only bias in natural language inference. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics, 2019b.
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 2015.
Samuel R. Bowman, Jon Gauthier, Abhinav Rastogi, Raghav Gupta, Christopher D. Manning, and Christopher Potts. fast unified model for parsing and sentence understanding. In 54th Annual Meeting of the Association for Computational Linguistics, 2016.
Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. Enhanced Istm for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, 2017.
Alexis Conneau and Douwe Kiela. Senteval: An evaluation toolkit for universal sentence repre- sentations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018), 2018
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. Supervised learning of universal sentence representations from natural language inference data. In Proceed- ings of the 2017 Conference on Empirical Methods Natural Language Processing, 2017.
Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. What you can eram into a single vector: Probing sentence embeddings for linguistic properties. arXiv preprint 1805.0 2018a.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk. and Veselin Stoyanov. XNLI: evaluating cross-lingual sentence representations. In EMNLP, 2018b
10

---

Under review as a conference paper at ICLR 2020
Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL recognising textual entailment challenge. In MLCW, Lecture Notes in Computer Science. Springer, 2005.
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, volume 9 of JMLR Proceedings, pp. 249-256. JMLR.org. 2010.
Yichen Gong, Heng Luo, and Jian Zhang. Natural language inference over interaction space. In International Conference on Learning Representations, 2017.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A Smith. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguis- fies: Human Language Technologies, 2018.
Sanda M. Harabagiu and Andrew Hickl. Methods for using textual entailment in open-domain question answering. In ACL. The Association for Computer Linguistics, 2006.
James Henderson and Diana Nicoleta Popa. A vector space for distributional semantics for entail- ment. In ACL (1). The Association for Computer Linguistics, 2016.
Tushar Khot, Ashish Sabharwal, and Peter Clark. Scitail: A textual entailment dataset from science question answering. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Douwe Kiela, Changhan Wang, and Kyunghyun Cho. Dynamic meta-embeddings for improved sentence representations. In Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing, 2018.
Seonhoon Kim, Jin--Huk Hong, Inho Kang, and Nojun Kwak. Semantic sentence matching with densely-connected recurrent and co-attentive information. In AAAI, 2019.
Finley Lacatusu, Andrew Hickl, Kirk Roberts, Ying Shi, Jeremy Bensley, Bryan Rink, Patrick Wang. and Lara Taylor. Lec's gistexter at due 2006: Multi-strategy multi-document summarization. In DUC*06, 2006.
Alice Lai, Yonatan Bisk, and Julia Hockenmaier. Natural language inference from multiple premises. In Proceedings of the Eighth International Joint Conference on Natural Language Processing, 2017.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Confer- ence on Learning Representations, 2019.
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. A sick cure for the evaluation of compositional distributional semantic models. In Nicoletta Calzolari (Conference Chair), Khalid Choukri, Thierry Declerck, Hrafn Loftsson, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceed- ings of the Ninth International Conference on Language Resources and Evaluation 14), Reykjavik, Iceland, may 2014. European Language Resources Association (ELRA). ISBN 978- 2-9517408-8-4
Lili Mou, Rui Men, Ge Li, Yan Xu, Lu Zhang, Rui Yan, and Zhi Jin. Natural language inference by tree- bree-basd convolution and heuristic matching. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, 2016.
Sebastian Padó, Michel Galley, Dan Jurafsky, and Chris Manning. Robust machine translation evaluation with entailment features. In Proceedings of the Joint Conference of the 47th Anmual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP. 2009.
11

---

Under review as a conference paper at ICLR 2020
Boyuan Pan, Yazheng Yang, Zhou Zhao, Yueting Zhuang, Deng Cai, and Xiaofei He. Discourse marker augmented network with reinforcement learning for natural language inference. In ACL (1), 2018.
Ellie Pavlick and Chris Callison-Bureis Most" babies" are" little" and most" problems" are" huge": Compositional entailment in adjective-nouns. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, 2016.
Ellie Pavlick, Travis Wolfe, Pushpendre Rastogi, Chris Callison-Burch, Mark Dredze, and Benjamin Van Durme. Framenet+: Fast paraphrastic tripling of framenet. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), volume 2, pp. 408-413, 2015.
Adam Poliak, Aparajita Haldar, Rachel Rudinger, J. Edward Hu, Ellic Pavlick, Aaron Steven White, and Benjamin Van Durme. Collecting Diverse Natural Language Inference Problems for Sentence Representation Evaluation. In Empirical Methods in Natural Language Processing, 2018a.
Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. Hypothesis only baselines natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, 2018b.
Altaf Rahman and Vincent Ng. Resolving complex cases of definite pronouns: the winograd schema challenge. In Proceedings of the 2012 Joint Conference Empirical Methods in Natural Lan- guage Processing and Computational Natural Language Learning. 2012.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. CoRR, abs/1908. 1at.//9088.0004,,.. 2019.
Drew Reisinger, Rachel Rudinger, Francis Ferraro, Craig Harman, Kyle Rawlins, and Benjamin Van Durme. Semantic proto-roles. Transactions of the Association for Computational Linguistics, 2015.
Aame Talman and Stergios Chatzikyriakidis. Testing the generalization power of neural network models across NLI benchmarks. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Ana- lyzing and Interpreting Neural Networks for NLP, 2019.
Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2019.
Zhiguo Wang, Wael Hamza, and Radu Florian. Bilateral multi- perspective matching for natural language sentences. In Proceedings of the 26th International Joint Conference on Artificial Intel- ligence, 2017.
Aaron Steven White, Pushpendre Rastogi, Kevin Duh, and Benjamin Van Durme. Inference is everything: Recasting semantic resources into unified evaluation framework. In Proceedings of the Eighth International Joint Conference on Natural Language Processing, 2017.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies. 201 la.
Adina Williams, Nikita Nangia, and Samuel R. Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, 2018b.
Rui Yan, Liang Kong, Congrui Huang, Xiaojun Wan, Xiaoming Li, and Yan Zhang. Timeline generation through evolutionary trans-temporal summarization. In Proceedings of the Conference Empirical Methods in Natural Language Processing, 20.
12

---

Under review as a conference paper at ICLR 2020
Rui Yan, Xiaojun Wan, Jahna Otterbacher, Liang Kong, Xiaoming Li, and Yan Zhang. Evolution- ary timeline summarization: a balanced optimization framework via iterative substitution. In Proceedings of the 34th international ACM SIGIR conference Research and development in Information Retrieval, 2011b.
Deunsol Yoon, Dongbok Lee, and SangKeun Lee. Dynamic self-attention: Computing attention over words dynamically for sentence embedding, arXiv preprint arXiv:1808.07383 2018.
Sheng Zhang, Rachel Rudinger, Kevin Duh, and Benjamin Van Durme. Ordinal common-sense inference. Transactions of the Association for Computational Linguistics, 2017.
13


Reviews:
[{"review_id": "ryx2DriFtr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a few heuristic scorers to model entailment and contradiction, based on encoded sentence embeddings.\n\nThese scores include an entailment score, a contradiction score, a neutral score, and two similarity scores. They are defined heuristically, e.g., entailment score = geometric avg of such thing: 1 - sigma(premise not satisfied) * sigma(hypothesis satisfied). This is similar to fuzzy logic (for example, https://en.wikipedia.org/wiki/Fuzzy_logic) and some citations are needed in this regard.\n\nDifferent from fuzzy logic, this paper learns whether an anonymous feature is true or false by NN encoders end-to-end. Thus, the model actually has enough power to extract those features suitable for fuzzy logic-like heuristic matching.\n\nThe experiments are well designed. I especially appreciate the comparison to random matching heuristics, which already exhibits non-trivial performance. This is very reasonable because the neural network underlying random matching heuristics is still learnable. However, the proposed matching heuristics achieve 7% improvement compared with random ones, showing the effectiveness of the approach. \n\nThe authors also have ablation test and experiments on out-of-domain datasets. \n\nI have two concerns:\n\n1. One limitation of this paper is that the heuristic matching scorers are pretty ad hoc to the inference task. The two similarity scores are not too novel, for example, sim_diff is the L1-distance between two vectors. Entailment, contradiction, and neutral scores are interesting, but hardly generalize to other sentence matching tasks (e.g., various IR applications). \n\n2. I have a feeling that the importance of NLI is over-estimated. While logical reasoning is important in AI, NLI datasets are somehow degenerated, and existing solutions are basically connecting neural edges. As mentioned in the paper, NLI models do not transfer well to out-of-domain NLI samples, not to mention non-NLI tasks. It would be interesting to see if the well-designed heuristic matching scores could ease the underlying model, so that it learns more generic sentence embeddings in general. \n\n\nMinor:\n\nIn references: Williams, Nagnia, Bowman: duplicate entry\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "HJgW7iEhYr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n***Update***\nI'd like to thank the authors for responding to my questions and for the additional experiments. I think the new sentence embedding experiments make the paper quite a bit stronger - it would be interesting to scale them up to using SNLI + MNLI to see how much further they can go (right now they are still below Sentence-BERT which also was trained on SNLI in addition to MNLI). I think this paper is an interesting idea, and my only other main concern is the transfer results to other NLI datasets. I think it would be a good idea to confirm that the difference between the two approaches is due to biases perhaps though an error analysis. I am borderline on this paper, but I feel enough improvement has been made to raise my rating.\n\n\nThis paper proposes a new way to train sentence embedding models using NLI data such that very few parameters are used for classification. This is in contrast to prior work where an MLP is used. They define an interaction layer where operations are applied to the sentence embeddings to produce just 5 scores, which are then fed into a softmax layer for the final prediction.\n\nThe reason for their approach is they hypothesize that the the classification layers are encoding some of the information, presumably lessening the amount of information in the sentence embeddings and also preventing them from having a direct interpretation.\n\nTheir model does surprisingly well on the entailment datasets, only about 1-1.5 points or so worse than using a MLP, and so they indeed demonstrate that the sentence embeddings contain a lot of entailment information. However, I do have some concerns about their claims that this helps in transfer learning to other NLI tasks. Their main results show transfer performance comparing their approach and using an MLP, but it seems that overall, on all datasets, their approach transfers more poorly. Two datasets that they do worse on they try to discount for either having terrible, below random performance (which is true) and the other for having the same biases as MNLI. However, if that was the case, I don't see why their model would perform worse since both theirs and the baseline would benefit from having these biases, but their model performs about 11 points less.\n\nSo therefore, I don't find the transfer experiments convincing, though it is interesting how different the models do on some of these tasks - model performance is surprisingly task dependent \n\nWhat I propose is for the authors to investigate if their sentence embeddings are in fact noticeably different than the ones trained in the more conventional matter. They could evaluate on sentence embedding and probing tasks (like SentEval) and see how the two models compare. It would be interesting to see wha encoded information differs between the models.\n\nIm summary, I think this is an interesting experiment and it's nice to see that the MLP isn't doing a lot of heavy lifting (which also might be slightly counter to their hypothesis about the MLP containing a lot of entailment information). However, I find the transfer experiments unconvincing and the paper is short on analysis about when their model does better on transfer and when having an MLP helps, or how the learned sentence embeddings of the two models differ.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Bkgq7QUptr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an interesting approach towards learning NLI via parameter free operations over pairs of sentence embeddings. The authors propose entailment and contradiction operators that learn entailment and contradiction scores while training the parameters of the sentence encoders.\n\nThis is an interesting approach, however the experiments make me doubt the effectiveness of the proposed method. Admittedly, the authors do point out that at the cost of fewer training parameters, the proposed approach attains the same performance as NLI encoders with MLP based or attention based classifiers. However, the question becomes, how many fewer parameters are being learned to accept a performance that is in the same ball park but that does not exceed the SOTA.\n\nThe authors should have tried to an ablation type of approach in equation 1, to check if concatenation alone, element wise dot product alone or absolute difference alone or a combination of any two would work better with the scoring function.\n\nThis paper while taking a step in the right direction, seems a little premature for publication. That being said, the reported results my be of some value after all. It is hard to narrow down on the exact contributions of this paper.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method for learning sentence embeddings such that entailment and contradiction relationships between sentence pairs can be inferred by a simple parameter-free operation on the vectors for the two sentences.

Reviewers found the method and the results interesting, but in private discussion, couldn't reach a consensus on what (if any) substantial valuable contributions the paper had proven. The performance of the method isn't compellingly strong in absolute or relative terms, yielding doubts about the value of the method for entailment applications, and the reviewers didn't see a strong enough motivation for the line of work to justify publishing it as a tentative or exploratory effort at ICLR.
Decision: Reject