Question: On what benchmark datasets do they evaluate the method?
Evidence: We propose a method of stacking multiple long short-term memory (LSTM) layers for modeling sentences. In contrast to the conventional stacked LSTMs where only hidden states are fed as input to the next layer, our architecture accepts both hidden and memory cell states of the preceding layer and fuses information from the left and the lower context using the soft gating mechanism of LSTMs. Thus the proposed stacked LSTM architecture modulates the amount of information to be delivered not only in horizontal recurrence but also in vertical connections, from which useful features extracted from lower layers are effectively conveyed to upper layers. We dub this architecture Cell-aware Stacked LSTM (CAS-LSTM) and show from experiments that our models achieve state-of-the-art results on benchmark datasets for natural language inference, paraphrase detection, and sentiment classification.
For the evaluation of performance of the proposed method on the NLI task, SNLI BIBREF22 and MultiNLI BIBREF23 datasets are used. The objective of both datasets is to predict the relationship between a premise and a hypothesis sentence: entailment, contradiction, and neutral. SNLI and MultiNLI datasets are composed of about 570k and 430k premise-hypothesis pairs respectively.
GloVe pretrained word embeddings BIBREF49 are used and remain fixed during training. The dimension of encoder states ( INLINEFORM0 ) is set to 300 and a 1024D MLP with one or two hidden layers is used. We apply dropout BIBREF50 to the word embeddings and the MLP layers. The features used as input to the MLP classifier are extracted following Eq. EQREF28 .
Table TABREF32 and TABREF33 contain results of the models on SNLI and MultiNLI datasets. In SNLI, our best model achieves the new state-of-the-art accuracy of 87.0% with relatively fewer parameters. Similarly in MultiNLI, our models match the accuracy of state-of-the-art models in both in-domain (matched) and cross-domain (mismatched) test sets. Note that only the GloVe word vectors are used as word representations, as opposed to some models that introduce character-level features. It is also notable that our proposed architecture does not restrict the selection of pooling method; the performance could further be improved by replacing max-pooling with other advanced algorithms e.g. intra-sentence attention BIBREF39 and generalized pooling BIBREF19 .
We use Quora Question Pairs dataset BIBREF24 in evaluating the performance of our method on the PI task. The dataset consists of over 400k question pairs, and each pair is annotated with whether the two sentences are paraphrase of each other or not.
Similar to the NLI experiments, GloVe pretrained vectors, 300D encoders, and 1024D MLP are used. The number of CAS-LSTM layers is fixed to 2 in PI experiments. Two sentence vectors are aggregated using Eq. EQREF29 and fed as input to the MLP. The results on the Quora Question Pairs dataset are summarized in Table TABREF34 . Again we can see that our models outperform other models by large margin, achieving the new state of the art.
In evaluating sentiment classification performance, the Stanford Sentiment Treebank (SST) BIBREF25 is used. It consists of about 12,000 binary-parsed sentences where constituents (phrases) of each parse tree are annotated with a sentiment label (very positive, positive, neutral, negative, very negative). Following the convention of prior work, all phrases and their labels are used in training but only the sentence-level data are used in evaluation.
In evaluation we consider two settings, namely SST-2 and SST-5, the two differing only in their level of granularity with regard to labels. In SST-2, data samples annotated with `neutral' are ignored from training and evaluation. The two positive labels (very positive, positive) are considered as the same label, and similarly for the two negative labels. As a result 98,794/872/1,821 data samples are used in training/validation/test, and the task is considered as a binary classification problem. In SST-5, data are used as-is and thus the task is a 5-class classification problem. All 318,582/1,101/2,210 data samples for training/validation/test are used in the SST-5 setting.
We use 300D GloVe vectors, 2-layer 150D or 300D encoders, and a 300D MLP classifier for the models, however unlike previous experiments we tune the word embeddings during training. The results on SST are listed in Table TABREF35 . Our models achieve the new state-of-the-art accuracy on SST-2 and competitive accuracy on SST-5, without utilizing parse tree information.
We dub this architecture Cellaware Stacked LSTM (CAS-LSTM) and show from experiments that our models bring significant performance gain over the standard LSTMs on benchmark datasets for natural language inference, paraphrase detection, sentiment classification, and machine translation.****For the evaluation of performance of the proposed method on the NLI task, SNLI (Bowman et al., 2015) and MultiNLI (Williams et al., 2018b) datasets are used.****We use Quora Question Pairs dataset (Wang et al., 2017) in evaluating the performance of our method on the PI task.****In evaluating sentiment classification performance, the Stanford Sentiment Treebank (SST) (Socher et al., 2013) is used.
Answer: SNLI, MultiNLI, Quora Question Pairs, and SST.