Indeed, it is phrases and sentences, rather than individual words, that encode the human-like general world knowledge (or `common sense')  that is a critical missing part of most current language understanding systems. We address this issue with a systematic comparison of cutting-edge methods for learning distributed representations of sentences. We constrain our comparison to methods that do not require labelled data gathered for the purpose of training models, since such methods are more cost-effective and applicable across languages and domains. We also propose two new phrase or sentence representation learning objectives - Sequential Denoising Autoencoders (SDAEs) and FastSent, a sentence-level log-linear bag-of-words model. We compare all methods on two types of task - supervised and unsupervised evaluations - reflecting different ways in which representations are ultimately to be used. In the former setting, a classifier or regression model is applied to representations and trained with task-specific labelled data, while in the latter, representation spaces are directly queried using cosine distance. We observe notable differences in approaches depending on the nature of the evaluation metric.

Distributed Sentence Representations	To constrain the analysis, we compare neural language models that compute sentence representations from unlabelled, naturally-ocurring data, as with the predominant methods for word representations. Likewise, we do not focus on `bottom up' models where phrase or sentence representations are built from fixed mathematical operations on word vectors (although we do consider a canonical case - see CBOW below); these were already compared by milajevs2014evaluating. Most space is devoted to our novel approaches, and we refer the reader to the original papers for more details of existing models.

Existing Models Trained on Text	SkipThought Vectors For consecutive sentences INLINEFORM0 in some document, the SkipThought model  is trained to predict target sentences INLINEFORM1 and INLINEFORM2 given source sentence INLINEFORM3 . As with all sequence-to-sequence models, in training the source sentence is `encoded' by a Recurrent Neural Network (RNN) (with Gated Recurrent uUnits  ) and then `decoded' into the two target sentences in turn. Importantly, because RNNs employ a single set of update weights at each time-step, both the encoder and decoder are sensitive to the order of words in the source sentence. For each position in a target sentence INLINEFORM0 , the decoder computes a softmax distribution over the model's vocabulary. The cost of a training example is the sum of the negative log-likelihood of each correct word in the target sentences INLINEFORM1 and INLINEFORM2 . This cost is backpropagated to train the encoder (and decoder), which, when trained, can map sequences of words to a single vector. ParagraphVector le2014distributed proposed two log-linear models of sentence representation.

We also experiment with a variant (+AE) in which the encoded (source) representation must predict its own words as target in addition to those of adjacent sentences. Thus in FastSent+AE, ( EQREF12 ) becomes DISPLAYFORM0  At test time the trained model (very quickly) encodes unseen word sequences into distributed representations with INLINEFORM0 .

First, for each word INLINEFORM2 in INLINEFORM3 , INLINEFORM4 deletes INLINEFORM5 with (independent) probability INLINEFORM6 . Then, for each non-overlapping bigram INLINEFORM7 in INLINEFORM8 , INLINEFORM9 swaps INLINEFORM10 and INLINEFORM11 with probability INLINEFORM12 . We then train the same LSTM-based encoder-decoder architecture as NMT, but with the denoising objective to predict (as target) the original source sentence INLINEFORM13 given a corrupted version INLINEFORM14 (as source). The trained model can then encode novel word sequences into distributed representations. We call this model the Sequential Denoising Autoencoder (SDAE). Note that, unlike SkipThought, SDAEs can be trained on sets of sentences in arbitrary order. We label the case with no noise (i.e. INLINEFORM0 and INLINEFORM1 ) SAE. This setting matches the method applied to text classification tasks by dai2015semi. The `word dropout' effect when INLINEFORM2 has also been used as a regulariser for deep nets in supervised language tasks  , and for large INLINEFORM3 the objective is similar to word-level `debagging'  .

For the SDAE, we tuned INLINEFORM4 , INLINEFORM5 on the validation set (see Section SECREF23 ). We also tried a variant (+embs) in which words are represented by (fixed) pre-trained embeddings. FastSent The performance of SkipThought vectors shows that rich sentence semantics can be inferred from the content of adjacent sentences. The model could be said to exploit a type of sentence-level Distributional Hypothesis  ,  . Nevertheless, like many deep neural language models, SkipThought is very slow to train (see Table TABREF14 ). FastSent is a simple additive (log-linear) sentence model designed to exploit the same signal, but at much lower computational expense. Given a BOW representation of some sentence in context, the model simply predicts adjacent sentences (also represented as BOW) . More formally, FastSent learns a source INLINEFORM0 and target INLINEFORM1 embedding for each word in the model vocabulary. For a training example INLINEFORM2 of consecutive sentences, INLINEFORM3 is represented as the sum of its source embeddings INLINEFORM4 . The cost of the example is then simply: DISPLAYFORM0  where INLINEFORM0 is the softmax function.

Models Trained on Structured Resources	The following models rely on (freely-available) data that has more structure than raw text. DictRep hill2015learning trained neural language models to map dictionary definitions to pre-trained word embeddings of the words defined by those definitions. They experimented with BOW and RNN (with LSTM) encoding architectures and variants in which the input word embeddings were either learned or pre-trained (+embs.) to match the target word embeddings. We implement their models using the available code and training data. CaptionRep Using the same overall architecture, we trained (BOW and RNN) models to map captions in the COCO dataset  to pre-trained vector representations of images. The image representations were encoded by a deep convolutional network  trained on the ILSVRC 2014 object recognition task  . Multi-modal distributed representations can be encoded by feeding test sentences forward through the trained model. NMT We consider the sentence representations learned by neural MT models. These models have identical architecture to SkipThought, but are trained on sentence-aligned translated texts. We used a standard architecture  on all available En-Fr and En-De data from the 2015 Workshop on Statistical MT (WMT).

Learning Distributed Representations of Sentences from Unlabelled Data	Unsupervised methods for learning distributed representations of words are ubiquitous in today's NLP research, but far less is known about the best ways to learn distributed phrase or sentence representations from unlabelled data. This paper is a systematic comparison of models that learn such representations. We find that the optimal approach depends critically on the intended application. Deeper, more complex models are preferable for representations to be used in supervised systems, but shallow log-linear models work best for building representation spaces that can be decoded with simple spatial distance metrics. We also propose two new unsupervised representation-learning objectives designed to optimise the trade-off between training time, domain portability and performance.

Conclusion	Advances in deep learning algorithms, software and hardware mean that many architectures and objectives for learning distributed sentence representations from unlabelled data are now available to NLP researchers. We have presented the first (to our knowledge) systematic comparison of these methods. We showed notable variation in the performance of approaches across a range of evaluations. Among other conclusions, we found that the optimal approach depends critically on whether representations will be applied in supervised or unsupervised settings - in the latter case, fast, shallow BOW models can still achieve the best performance. Further, we proposed two new objectives, FastSent and Sequential Denoising Autoencoders, which perform particularly well on specific tasks (MSRP and SICK sentence relatedness respectively). If the application is unknown, however, the best all round choice may be DictRep: learning a mapping of pre-trained word embeddings from the word-phrase signal in dictionary definitions. While we have focused on models using naturally-occurring training data, in future work we will also consider supervised architectures (including convolutional, recursive and character-level models), potentially training them on multiple supervised tasks as an alternative way to induce the 'general knowledge' needed to give language technology the elusive human touch.

The DBOW model learns a vector INLINEFORM0 for every sentence INLINEFORM1 in the training corpus which, together with word embeddings INLINEFORM2 , define a softmax distribution optimised to predict words INLINEFORM3 given INLINEFORM4 . The INLINEFORM5 are shared across all sentences in the corpus. In the DM model, INLINEFORM6 -grams of consecutive words INLINEFORM7 are selected and INLINEFORM8 is combined with INLINEFORM9 to make a softmax prediction (parameterised by additional weights) of INLINEFORM10 . We used the Gensim implementation, treating each sentence in the training data as a `paragraph' as suggested by the authors. During training, both DM and DBOW models store representations for every sentence (as well as word) in the training corpus. Even on large servers it was therefore only possible to train models with representation size 200, and DM models whose combination operation was averaging (rather than concatenation). Bottom-Up Methods We train CBOW and SkipGram word embeddings  on the Books corpus, and compose by elementwise addition as proposed by mitchell2010composition.

Differences in resource requirements As shown in Table TABREF14 , different models require different resources to train and use. This can limit their possible applications. For instance, while it was easy to make an online demo for fast querying of near neighbours in the CBOW and FastSent spaces, it was not practical for other models owing to memory footprint, encoding time and representation dimension. The role of word order is unclear The average scores of models that are sensitive to word order (76.3) and of those that are not (76.6) are approximately the same across supervised evaluations. Across the unsupervised evaluations, however, BOW models score 0.55 on average compared with 0.42 for RNN-based (order sensitive) models. This seems at odds with the widely held view that word order plays an important role in determining the meaning of English sentences. One possibility is that order-critical sentences that cannot be disambiguated by a robust conceptual semantics (that could be encoded in distributed lexical representations) are in fact relatively rare. However, it is also plausible that current available evaluations do not adequately reflect order-dependent aspects of meaning (see below).

Novel Text-Based Models	We introduce two new approaches designed to address certain limitations with the existing models. Sequential (Denoising) Autoencoders The SkipThought objective requires training text with a coherent inter-sentence narrative, making it problematic to port to domains such as social media or artificial language generated from symbolic knowledge. To avoid this restriction, we experiment with a representation-learning objective based on denoising autoencoders (DAEs). In a DAE, high-dimensional input data is corrupted according to some noise function, and the model is trained to recover the original data from the corrupted version. As a result of this process, DAEs learn to represent the data in terms of features that explain its important factors of variation  . Transforming data into DAE representations (as a `pre-training' or initialisation step) gives more robust (supervised) classification performance in deep feedforward networks  . The original DAEs were feedforward nets applied to (image) data of fixed size. Here, we adapt the approach to variable-length sentences by means of a noise function INLINEFORM0 , determined by free parameters INLINEFORM1 .

This latter conjecture is supported by the comparatively strong performance of TFIDF BOW vectors, in which the effective lexical semantics are limited to simple relative frequencies. The evaluations have limitations The internal consistency (Chronbach's INLINEFORM0 ) of all evaluations considered together is INLINEFORM1 (just above `acceptable'). Table TABREF25 shows that consistency is far higher (`excellent') when considering the supervised or unsupervised tasks as independent cohorts. This indicates that, with respect to common characteristics of sentence representations, the supervised and unsupervised benchmarks do indeed prioritise different properties. It is also interesting that, by this metric, the properties measured by MSRP and image-caption relatedness are the furthest removed from other evaluations in their respective cohorts. While these consistency scores are a promising sign, they could also be symptomatic of a set of evaluations that are all limited in the same way. The inter-rater agreement is only reported for one of the 8 evaluations considered (MPQA, INLINEFORM0  ), and for MR, SUBJ and TREC, each item is only rated by one or two annotators to maximise coverage.

One notable difference is between (AE-style) models whose semantics come from within-sentence relationships (CBOW, SDAE, DictRep, ParagraphVec) and SkipThought/FastSent, which exploit the context around sentences. In the former case, nearby sentences generally have a high proportion of words in common, whereas for the latter it is the general concepts and/or function of the sentence that is similar, and word overlap is often minimal. Indeed, this may be a more important trait of FastSent than the marginal improvement on the SICK task. Readers can compare the CBOW and FastSent spaces at http://45.55.60.98/. Differences between supervised and unsupervised performance Many of the best performing models on the supervised evaluations do not perform well in the unsupervised setting. In the SkipThought, S(D)AE and NMT models, the cost is computed based on a non-linear decoding of the internal sentence representations, so, as also observed by  , the informative geometry of the representation space may not be reflected in a simple cosine distance. The log-linear models generally perform better in this unsupervised setting.

Introduction	Distributed representations - dense real-valued vectors that encode the semantics of linguistic units - are ubiquitous in today's NLP research. For single-words or word-like entities, there are established ways to acquire such representations from naturally occurring (unlabelled) training data based on comparatively task-agnostic objectives (such as predicting adjacent words). These methods are well understood empirically  and theoretically  . The best word representation spaces reflect consistently-observed aspects of human conceptual organisation  , and can be added as features to improve the performance of numerous language processing systems  . By contrast, there is comparatively little consensus on the best ways to learn distributed representations of phrases or sentences. With the advent of deeper language processing techniques, it is relatively common for models to represent phrases or sentences as continuous-valued vectors. Examples include machine translation  , image captioning  and dialogue systems  . While it has been observed informally that the internal sentence representations of such models can reflect semantic intuitions  , it is not known which architectures or objectives yield the `best' or most useful representations. Resolving this question could ultimately have a significant impact on language processing systems.

Evaluating Sentence Representations	In previous work, distributed representations of language were evaluated either by measuring the effect of adding representations as features in some classification task - supervised evaluation  ,  ,  - or by comparing with human relatedness judgements - unspervised evaluation  ,  ,  . The former setting reflects a scenario in which representations are used to inject general knowledge (sometimes considered as pre-training) into a supervised model. The latter pertains to applications in which the sentence representation space is used for direct comparisons, lookup or retrieval. Here, we apply and compare both evaluation paradigms.

Results	Performance of the models on the supervised evaluations (grouped according to the data required by their objective) is shown in Table TABREF20 . Overall, SkipThought vectors perform best on three of the six evaluations, the BOW DictRep model with pre-trained word embeddings performs best on two, and the SDAE on one. SDAEs perform notably well on the paraphrasing task, going beyond SkipThought by three percentage points and approaching state-of-the-art performance of models designed specifically for the task  . SDAE is also consistently better than SAE, which aligns with other findings that adding noise to AEs produces richer representations  . Results on the unsupervised evaluations are shown in Table TABREF21 . The same DictRep model performs best on four of the six STS categories (and overall) and is joint-top performer on SICK. Of the models trained on raw text, simply adding CBOW word vectors works best on STS. The best performing raw text model on SICK is FastSent, which achieves almost identical performance to C-PHRASE's state-of-the-art performance for a distributed model  .

Discussion	Many additional conclusions can be drawn from the results in Tables TABREF20 and TABREF21 . Different objectives yield different representations It may seem obvious, but the results confirm that different learning methods are preferable for different intended applications (and this variation appears greater than for word representations). For instance, it is perhaps unsurprising that SkipThought performs best on TREC because the labels in this dataset are determined by the language immediately following the represented question (i.e. the answer)  . Paraphrase detection, on the other hand, may be better served by a model that focused entirely on the content within a sentence, such as SDAEs. Similar variation can be observed in the unsupervised evaluations. For instance, the (multimodal) representations produced by the CaptionRep model do not perform particularly well apart from on the Image category of STS where they beat all other models, demonstrating a clear effect of the well-studied modality differences in representation learning  . The nearest neighbours in Table TABREF24 give a more concrete sense of the representation spaces.

Training and Model Selection	Unless stated above, all models were trained on the Toronto Books Corpus, which has the inter-sentential coherence required for SkipThought and FastSent. The corpus consists of 70m ordered sentences from over 7,000 books. Specifications of the models are shown in Table TABREF14 . The log-linear models (SkipGram, CBOW, ParagraphVec and FastSent) were trained for one epoch on one CPU core. The representation dimension INLINEFORM0 for these models was found after tuning INLINEFORM1 on the validation set. All other models were trained on one GPU. The S(D)AE models were trained for one epoch ( INLINEFORM3 days). The SkipThought model was trained for two weeks, covering just under one epoch. For CaptionRep and DictRep, performance was monitored on held-out training data and training was stopped after 24 hours after a plateau in cost. The NMT models were trained for 72 hours.

We also compare to C-PHRASE  , an approach that exploits a (supervised) parser to infer distributed semantic representations based on a syntactic parse of sentences. C-PHRASE achieves state-of-the-art results for distributed representations on several evaluations used in this study. Non-Distributed Baseline We implement a TFIDF BOW model in which the representation of sentence INLINEFORM0 encodes the count in INLINEFORM1 of a set of feature-words weighted by their tfidf in INLINEFORM2 , the corpus. The feature-words are the 200,000 most common words in INLINEFORM3 .

Further, it uses less than a third of the training text and does not require access to (supervised) syntactic representations for training. Together, the results of FastSent on the unsupervised evaluations and SkipThought on the supervised benchmarks provide strong support for the sentence-level distributional hypothesis: the context in which a sentence occurs provides valuable information about its semantics. Across both unsupervised and supervised evaluations, the BOW DictRep with pre-trained word embeddings exhibits by some margin the most consistent performance. Ths robust performance suggests that DictRep representations may be particularly valuable when the ultimate application is non-specific or unknown, and confirms that dictionary definitions (where available) can be a powerful resource for representation learning.

Unsupervised Evaluations	We also measure how well representation spaces reflect human intuitions of the semantic sentence relatedness, by computing the cosine distance between vectors for the two sentences in each test pair, and correlating these distances with gold-standard human judgements. The SICK dataset  consists of 10,000 pairs of sentences and relatedness judgements. The STS 2014 dataset  consists of 3,750 pairs and ratings from six linguistic domains. Example ratings are shown in Table TABREF15 . All available pairs are used for testing apart from the 500 SICK `trial' pairs, which are held-out for tuning hyperparameters (representation size of log-linear models, and noise parameters in SDAE). The optimal settings on this task are then applied to both supervised and unsupervised evaluations.

In particular, deeper or more complex models (which require greater time and resources to train) generally perform best in the supervised setting, whereas shallow log-linear models work best on unsupervised benchmarks. Specifically, SkipThought Vectors  perform best on the majority of supervised evaluations, but SDAEs are the top performer on paraphrase identification. In contrast, on the (unsupervised) SICK sentence relatedness benchmark, FastSent, a simple, log-linear variant of the SkipThought objective, performs better than all other models. Interestingly, the method that exhibits strongest performance across both supervised and unsupervised benchmarks is a bag-of-words model trained to compose word embeddings using dictionary definitions  . Taken together, these findings constitute valuable guidelines for the application of phrasal or sentential representation-learning to language understanding systems.

Table TABREF15 illustrates why this may be an issue for the unsupervised evaluations; the notion of sentential 'relatedness' seems very subjective. It should be emphasised, however, that the tasks considered in this study are all frequently used for evaluation, and, to our knowledge, there are no existing benchmarks that overcome these limitations.

Supervised Evaluations	Representations are applied to 6 sentence classification tasks: paraphrase identification (MSRP)  , movie review sentiment (MR)  , product reviews (CR)  , subjectivity classification (SUBJ)  , opinion polarity (MPQA)  and question type classification (TREC)  . We follow the procedure (and code) of kiros2015skip: a logistic regression classifier is trained on top of sentence representations, with 10-fold cross-validation used when a train-test split is not pre-defined.

Acknowledgments	This work was supported by a Google Faculty Award to AK and FH and a Google European Doctoral Fellowship to FH. Thanks also to Marek Rei, Tamara Polajnar, Laural Rimell, Jamie Ryan Kiros and Piotr Bojanowski for helpful discussion and comments.

Question: Which data sources do they use?
Answer: Toronto Books Corpus, STS 2014 dataset , SICK dataset