Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features

Abstract: The recent tremendous success of unsupervised word embeddings in a multitude of applications raises the obvious question if similar methods could be derived to improve embeddings (i.e. semantic representations) of word sequences as well. We present a simple but efficient unsupervised objective to train distributed representations of sentences. Our method outperforms the state-of-the-art unsupervised models on most benchmark tasks, highlighting the robustness of the produced general-purpose sentence embeddings.

 Improving unsupervised learning is of key importance for advancing machine learning methods, as to unlock access to almost unlimited amounts of data to be used as training resources. The majority of recent success stories of deep learning does not fall into this category but instead relied on supervised training (in particular in the vision domain). A very notable exception comes from the text and natural language processing domain, in the form of semantic word embeddings trained unsupervised BIBREF0 , BIBREF1 , BIBREF2 . Within only a few years from their invention, such word representations – which are based on a simple matrix factorization model as we formalize below – are now routinely trained on very large amounts of raw text data, and have become ubiquitous building blocks of a majority of current state-of-the-art NLP applications.
 While very useful semantic representations are available for words, it remains challenging to produce and learn such semantic embeddings for longer pieces of text, such as sentences, paragraphs or entire documents. Even more so, it remains a key goal to learn such general-purpose representations in an unsupervised way.
Currently, two contrary research trends have emerged in text representation learning: On one hand, a strong trend in deep-learning for NLP leads towards increasingly powerful and complex models, such as recurrent neural networks (RNNs), LSTMs, attention models and even Neural Turing Machine architectures. While extremely strong in expressiveness, the increased model complexity makes such models much slower to train on larger datasets. On the other end of the spectrum, simpler “shallow” models such as matrix factorizations (or bilinear models) can benefit from training on much larger sets of data, which can be a key advantage, especially in the unsupervised setting.
 Surprisingly, for constructing sentence embeddings, naively using averaged word vectors was shown to outperform LSTMs (see BIBREF3 for plain averaging, and BIBREF4 for weighted averaging). This example shows potential in exploiting the trade-off between model complexity and ability to process huge amounts of text using scalable algorithms, towards the simpler side. In view of this trade-off, our work here further advances unsupervised learning of sentence embeddings. Our proposed model can be seen as an extension of the C-BOW BIBREF0 , BIBREF1 training objective to train sentence instead of word embeddings. We demonstrate that the empirical performance of our resulting general-purpose sentence embeddings very significantly exceeds the state of the art, while keeping the model simplicity as well as training and inference complexity exactly as low as in averaging methods BIBREF3 , BIBREF4 , thereby also putting the work by BIBREF4 in perspective.
 Contributions. The main contributions in this work can be summarized as follows:
Our model is inspired by simple matrix factor models (bilinear models) such as recently very successfully used in unsupervised learning of word embeddings BIBREF0 , BIBREF1 , BIBREF2 , BIBREF5 as well as supervised of sentence classification BIBREF6 . More precisely, these models can all be formalized as an optimization problem of the form DISPLAYFORM0 
for two parameter matrices INLINEFORM0 and INLINEFORM1 , where INLINEFORM2 denotes the vocabulary. Here, the columns of the matrix INLINEFORM3 represent the learnt source word vectors whereas those of INLINEFORM4 represent the target word vectors. For a given sentence INLINEFORM5 , which can be of arbitrary length, the indicator vector INLINEFORM6 is a binary vector encoding INLINEFORM7 (bag of words encoding).
Fixed-length context windows INLINEFORM0 running over the corpus are used in word embedding methods as in C-BOW BIBREF0 , BIBREF1 and GloVe BIBREF2 . Here we have INLINEFORM1 and each cost function INLINEFORM2 only depends on a single row of its input, describing the observed target word for the given fixed-length context INLINEFORM3 . In contrast, for sentence embeddings which are the focus of our paper here, INLINEFORM4 will be entire sentences or documents (therefore variable length). This property is shared with the supervised FastText classifier BIBREF6 , which however uses soft-max with INLINEFORM5 being the number of class labels.
We propose a new unsupervised model, Sent2Vec, for learning universal sentence embeddings. Conceptually, the model can be interpreted as a natural extension of the word-contexts from C-BOW BIBREF0 , BIBREF1 to a larger sentence context, with the sentence words being specifically optimized towards additive combination over the sentence, by means of the unsupervised objective function.
Formally, we learn a source (or context) embedding INLINEFORM0 and target embedding INLINEFORM1 for each word INLINEFORM2 in the vocabulary, with embedding dimension INLINEFORM3 and INLINEFORM4 as in ( EQREF6 ). The sentence embedding is defined as the average of the source word embeddings of its constituent words, as in ( EQREF8 ). We augment this model furthermore by also learning source embeddings for not only unigrams but also n-grams present in each sentence, and averaging the n-gram embeddings along with the words, i.e., the sentence embedding INLINEFORM5 for INLINEFORM6 is modeled as DISPLAYFORM0 
 where INLINEFORM0 is the list of n-grams (including unigrams) present in sentence INLINEFORM1 . In order to predict a missing word from the context, our objective models the softmax output approximated by negative sampling following BIBREF0 . For the large number of output classes INLINEFORM2 to be predicted, negative sampling is known to significantly improve training efficiency, see also BIBREF7 . Given the binary logistic loss function INLINEFORM3 coupled with negative sampling, our unsupervised training objective is formulated as follows: INLINEFORM4 
 where INLINEFORM0 corresponds to the current sentence and INLINEFORM1 is the set of words sampled negatively for the word INLINEFORM2 . The negatives are sampled following a multinomial distribution where each word INLINEFORM5 is associated with a probability INLINEFORM6 , where INLINEFORM7 is the normalized frequency of INLINEFORM8 in the corpus.
To select the possible target unigrams (positives), we use subsampling as in BIBREF6 , BIBREF5 , each word INLINEFORM0 being discarded with probability INLINEFORM1 where INLINEFORM2 . Where INLINEFORM3 is the subsampling hyper-parameter. Subsampling prevents very frequent words of having too much influence in the learning as they would introduce strong biases in the prediction task. With positives subsampling and respecting the negative sampling distribution, the precise training objective function becomes DISPLAYFORM0 
In contrast to more complex neural network based models, one of the core advantages of the proposed technique is the low computational cost for both inference and training. Given a sentence INLINEFORM0 and a trained model, computing the sentence representation INLINEFORM1 only requires INLINEFORM2 floating point operations (or INLINEFORM3 to be precise for the n-gram case, see ( EQREF8 )), where INLINEFORM4 is the embedding dimension. The same holds for the cost of training with SGD on the objective ( EQREF10 ), per sentence seen in the training corpus. Due to the simplicity of the model, parallel training is straight-forward using parallelized or distributed SGD.
Also, in order to store higher-order n-grams efficiently, we use the standard hashing trick, see e.g. BIBREF8 , with the same hashing function as used in FastText BIBREF6 , BIBREF5 .
C-BOW BIBREF0 , BIBREF1 aims to predict a chosen target word given its fixed-size context window, the context being defined by the average of the vectors associated with the words at a distance less than the window size hyper-parameter INLINEFORM0 . If our system, when restricted to unigram features, can be seen as an extension of C-BOW where the context window includes the entire sentence, in practice there are few important differences as C-BOW uses important tricks to facilitate the learning of word embeddings. C-BOW first uses frequent word subsampling on the sentences, deciding to discard each token INLINEFORM1 with probability INLINEFORM2 or alike (small variations exist across implementations). Subsampling prevents the generation of n-grams features, and deprives the sentence of an important part of its syntactical features. It also shortens the distance between subsampled words, implicitly increasing the span of the context window. A second trick consists of using dynamic context windows: for each subsampled word INLINEFORM3 , the size of its associated context window is sampled uniformly between 1 and INLINEFORM4 . Using dynamic context windows is equivalent to weighing by the distance from the focus word INLINEFORM5 divided by the window size BIBREF9 . This makes the prediction task local, and go against our objective of creating sentence embeddings as we want to learn how to compose all n-gram features present in a sentence. In the results section, we report a significant improvement of our method over C-BOW.
Three different datasets have been used to train our models: the Toronto book corpus, Wikipedia sentences and tweets. The Wikipedia and Toronto books sentences have been tokenized using the Stanford NLP library BIBREF10 , while for tweets we used the NLTK tweets tokenizer BIBREF11 . For training, we select a sentence randomly from the dataset and then proceed to select all the possible target unigrams using subsampling. We update the weights using SGD with a linearly decaying learning rate.
Also, to prevent overfitting, for each sentence we use dropout on its list of n-grams INLINEFORM0 , where INLINEFORM1 is the set of all unigrams contained in sentence INLINEFORM2 . After empirically trying multiple dropout schemes, we find that dropping INLINEFORM3 n-grams ( INLINEFORM4 ) for each sentence is giving superior results compared to dropping each token with some fixed probability. This dropout mechanism would negatively impact shorter sentences. The regularization can be pushed further by applying L1 regularization to the word vectors. Encouraging sparsity in the embedding vectors is particularly beneficial for high dimension INLINEFORM5 . The additional soft thresholding in every SGD step adds negligible computational cost. See also Appendix SECREF8 . We train two models on each dataset, one with unigrams only and one with unigrams and bigrams. All training parameters for the models are provided in Table TABREF25 in the supplementary material. Our C++ implementation builds upon the FastText library BIBREF6 , BIBREF5 . We will make our code and pre-trained models available open-source.
We discuss existing models which have been proposed to construct sentence embeddings. While there is a large body of works in this direction – several among these using e.g. labelled datasets of paraphrase pairs to obtain sentence embeddings in a supervised manner BIBREF12 , BIBREF3 , BIBREF13 to learn sentence embeddings – we here focus on unsupervised, task-independent models. While some methods require ordered raw text i.e., a coherent corpus where the next sentence is a logical continuation of the previous sentence, others rely only on raw text i.e., an unordered collection of sentences. Finally, we also discuss alternative models built from structured data sources.
The ParagraphVector DBOW model BIBREF14 is a log-linear model which is trained to learn sentence as well as word embeddings and then use a softmax distribution to predict words contained in the sentence given the sentence vector representation. They also propose a different model ParagraphVector DM where they use n-grams of consecutive words along with the sentence vector representation to predict the next word.
 BIBREF15 also presented an early approach to obtain compositional embeddings from word vectors. They use different compositional techniques including static averaging or Fisher vectors of a multivariate Gaussian to obtain sentence embeddings from word2vec models.
 BIBREF16 propose a Sequential (Denoising) Autoencoder, S(D)AE. This model first introduces noise in the input data: Firstly each word is deleted with probability INLINEFORM0 , then for each non-overlapping bigram, words are swapped with probability INLINEFORM1 . The model then uses an LSTM-based architecture to retrieve the original sentence from the corrupted version. The model can then be used to encode new sentences into vector representations. In the case of INLINEFORM2 , the model simply becomes a Sequential Autoencoder. BIBREF16 also propose a variant (S(D)AE + embs.) in which the words are represented by fixed pre-trained word vector embeddings.
 BIBREF4 propose a model in which sentences are represented as a weighted average of fixed (pre-trained) word vectors, followed by post-processing step of subtracting the principal component. Using the generative model of BIBREF17 , words are generated conditioned on a sentence “discourse” vector INLINEFORM0 : INLINEFORM1 
 where INLINEFORM0 and INLINEFORM1 and INLINEFORM2 , INLINEFORM3 are scalars. INLINEFORM4 is the common discourse vector, representing a shared component among all discourses, mainly related to syntax. It allows the model to better generate syntactical features. The INLINEFORM5 term is here to enable the model to generate some frequent words even if their matching with the discourse vector INLINEFORM6 is low.
Therefore, this model tries to generate sentences as a mixture of three type of words: words matching the sentence discourse vector INLINEFORM0 , syntactical words matching INLINEFORM1 , and words with high INLINEFORM2 . BIBREF4 demonstrated that for this model, the MLE of INLINEFORM3 can be approximated by INLINEFORM4 , where INLINEFORM5 is a scalar. The sentence discourse vector can hence be obtained by subtracting INLINEFORM6 estimated by the first principal component of INLINEFORM7 's on a set of sentences. In other words, the sentence embeddings are obtained by a weighted average of the word vectors stripping away the syntax by subtracting the common discourse vector and down-weighting frequent tokens. They generate sentence embeddings from diverse pre-trained word embeddings among which are unsupervised word embeddings such as GloVe BIBREF2 as well as supervised word embeddings such as paragram-SL999 (PSL) BIBREF18 trained on the Paraphrase Database BIBREF19 .
In a very different line of work, C-PHRASE BIBREF20 relies on additional information from the syntactic parse tree of each sentence, which is incorporated into the C-BOW training objective.
 BIBREF21 show that single layer CNNs can be modeled using a tensor decomposition approach. While building on an unsupervised objective, the employed dictionary learning step for obtaining phrase templates is task-specific (for each use-case), not resulting in general-purpose embeddings.
The SkipThought model BIBREF22 combines sentence level models with recurrent neural networks. Given a sentence INLINEFORM0 from an ordered corpus, the model is trained to predict INLINEFORM1 and INLINEFORM2 .
FastSent BIBREF16 is a sentence-level log-linear bag-of-words model. Like SkipThought, it uses adjacent sentences as the prediction target and is trained in an unsupervised fashion. Using word sequences allows the model to improve over the earlier work of paragraph2vec BIBREF14 . BIBREF16 augment FastSent further by training it to predict the constituent words of the sentence as well. This model is named FastSent + AE in our comparisons.
Compared to our approach, Siamese C-BOW BIBREF23 shares the idea of learning to average word embeddings over a sentence. However, it relies on a Siamese neural network architecture to predict surrounding sentences, contrasting our simpler unsupervised objective.
Note that on the character sequence level instead of word sequences, FastText BIBREF5 uses the same conceptual model to obtain better word embeddings. This is most similar to our proposed model, with two key differences: Firstly, we predict from source word sequences to target words, as opposed to character sequences to target words, and secondly, our model is averaging the source embeddings instead of summing them.
DictRep BIBREF24 is trained to map dictionary definitions of the words to the pre-trained word embeddings of these words. They use two different architectures, namely BOW and RNN (LSTM) with the choice of learning the input word embeddings or using them pre-trained. A similar architecture is used by the CaptionRep variant, but here the task is the mapping of given image captions to a pre-trained vector representation of these images.
We use a standard set of supervised as well as unsupervised benchmark tasks from the literature to evaluate our trained models, following BIBREF16 . The breadth of tasks allows to fairly measure generalization to a wide area of different domains, testing the general-purpose quality (universality) of all competing sentence embeddings. For downstream supervised evaluations, sentence embeddings are combined with logistic regression to predict target labels. In the unsupervised evaluation for sentence similarity, correlation of the cosine similarity between two embeddings is compared to human annotators.
Downstream Supervised Evaluation. Sentence embeddings are evaluated for various supervised classification tasks as follows. We evaluate paraphrase identification (MSRP) BIBREF25 , classification of movie review sentiment (MR) BIBREF26 , product reviews (CR) BIBREF27 , subjectivity classification (SUBJ) BIBREF28 , opinion polarity (MPQA) BIBREF29 and question type classification (TREC) BIBREF30 . To classify, we use the code provided by BIBREF22 in the same manner as in BIBREF16 . For the MSRP dataset, containing pairs of sentences INLINEFORM0 with associated paraphrase label, we generate feature vectors by concatenating their Sent2Vec representations INLINEFORM1 with the component-wise product INLINEFORM2 . The predefined training split is used to tune the L2 penalty parameter using cross-validation and the accuracy and F1 scores are computed on the test set. For the remaining 5 datasets, Sent2Vec embeddings are inferred from input sentences and directly fed to a logistic regression classifier. Accuracy scores are obtained using 10-fold cross-validation for the MR, CR, SUBJ and MPQA datasets. For those datasets nested cross-validation is used to tune the L2 penalty. For the TREC dataset, as for the MRSP dataset, the L2 penalty is tuned on the predefined train split using 10-fold cross-validation, and the accuracy is computed on the test set.
Unsupervised Similarity Evaluation. We perform unsupervised evaluation of the learnt sentence embeddings using the sentence cosine similarity, on the STS 2014 BIBREF31 and SICK 2014 BIBREF32 datasets. These similarity scores are compared to the gold-standard human judgements using Pearson's INLINEFORM0 BIBREF33 and Spearman's INLINEFORM1 BIBREF34 correlation scores. The SICK dataset consists of about 10,000 sentence pairs along with relatedness scores of the pairs. The STS 2014 dataset contains 3,770 pairs, divided into six different categories on the basis of the origin of sentences/phrases, namely Twitter, headlines, news, forum, WordNet and images.
In Tables TABREF18 and TABREF19 , we compare our results with those obtained by BIBREF16 on different models. Table TABREF21 in the last column shows the dramatic improvement in training time of our models (and other C-BOW-inspired models) in contrast to neural network based models. All our Sent2Vec models are trained on a machine with 2x Intel Xeon E5 INLINEFORM0 2680v3, 12 cores @2.5GHz.
Along with the models discussed in Section SECREF3 , this also includes the sentence embedding baselines obtained by simple averaging of word embeddings over the sentence, in both the C-BOW and skip-gram variants. TF-IDF BOW is a representation consisting of the counts of the 200,000 most common feature-words, weighed by their TF-IDF frequencies. To ensure coherence, we only include unsupervised models in the main paper. Performance of supervised and semi-supervised models on these evaluations can be observed in Tables TABREF29 and TABREF30 in the supplementary material.
Downstream Supervised Evaluation Results. On running supervised evaluations and observing the results in Table TABREF18 , we find that on an average our models are second only to SkipThought vectors. Also, both our models achieve state of the art results on the CR task. We also observe that on half of the supervised tasks, our unigrams + bigram model is the best model after SkipThought. Our models are weaker on the MSRP task (which consists of the identification of labelled paraphrases) compared to state-of-the-art methods. However, we observe that the models which perform very strongly on this task end up faring very poorly on the other tasks, indicating a lack of generalizability. On rest of the tasks, our models perform extremely well. The SkipThought model is able to outperform our models on most of the tasks as it is trained to predict the previous and next sentences and a lot of tasks are able to make use of this contextual information missing in our Sent2Vec models. For example, the TREC task is a poor measure of how one predicts the content of the sentence (the question) but a good measure of how the next sentence in the sequence (the answer) is predicted.
Unsupervised Similarity Evaluation Results. In Table TABREF19 , we see that our Sent2Vec models are state-of-the-art on the majority of tasks when comparing to all the unsupervised models trained on the Toronto corpus, and clearly achieve the best averaged performance. Our Sent2Vec models also on average outperform or are at par with the C-PHRASE model, despite significantly lagging behind on the STS 2014 WordNet and News subtasks. This observation can be attributed to the fact that a big chunk of the data that the C-PHRASE model is trained on comes from English Wikipedia, helping it to perform well on datasets involving definition and news items. Also, C-PHRASE uses data three times the size of the Toronto book corpus. Interestingly, our model outperforms C-PHRASE when trained on Wikipedia, as shown in Table TABREF21 , despite the fact that we use no parse tree information. Official STS 2017 benchmark. In the official results of the most recent edition of the STS 2017 benchmark BIBREF35 , our model also significantly outperforms C-PHRASE, and in fact delivers the best unsupervised baseline method.
For the Siamese C-BOW model trained on the Toronto corpus, supervised evaluation as well as similarity evaluation results on the SICK 2014 dataset are unavailable.
Macro Average. To summarize our contributions on both supervised and unsupervised tasks, in Table TABREF21 we present the results in terms of the macro average over the averages of both supervised and unsupervised tasks along with the training times of the models. For unsupervised tasks, averages are taken over both Spearman and Pearson scores. The comparison includes the best performing unsupervised and semi-supervised methods described in Section SECREF3 . For models trained on the Toronto books dataset, we report a 3.8 INLINEFORM0 points improvement over the state of the art. Considering all supervised, semi-supervised methods and all datasets compared in BIBREF16 , we report a 2.2 INLINEFORM1 points improvement.
We also see a noticeable improvement in accuracy as we use larger datasets like Twitter and Wikipedia. We furthermore see that the Sent2Vec models are faster to train when compared to methods like SkipThought and DictRep, owing to the SGD optimizer allowing a high degree of parallelizability.
We can clearly see Sent2Vec outperforming other unsupervised and even semi-supervised methods. This can be attributed to the superior generalizability of our model across supervised and unsupervised tasks.
Comparison with BIBREF4 . We also compare our work with BIBREF4 who also use additive compositionality to obtain sentence embeddings. However, in contrast to our model, they use fixed, pre-trained word embeddings to build a weighted average of these embeddings using unigram probabilities. While we couldn't find pre-trained state of the art word embeddings trained on the Toronto books corpus, we evaluated their method using GloVe embeddings obtained from the larger Common Crawl Corpus, which is 42 times larger than our twitter corpus, greatly favoring their method over ours.
In Table TABREF22 , we report an experimental comparison to their model on unsupervised tasks. In the table, the suffix W indicates that their down-weighting scheme has been used, while the suffix R indicates the removal of the first principal component. They report values of INLINEFORM0 as giving the best results and used INLINEFORM1 for all their experiments. We observe that our results are competitive with the embeddings of BIBREF4 for purely unsupervised methods. It is important to note that the scores obtained from supervised task-specific PSL embeddings trained for the purpose of semantic similarity outperform our method on both SICK and average STS 2014, which is expected as our model is trained purely unsupervised.
In order to facilitate a more detailed comparison, we also evaluated the unsupervised Glove + WR embeddings on downstream supervised tasks and compared them to our twitter models. To use BIBREF4 's method in a supervised setup, we precomputed and stored the common discourse vector INLINEFORM0 using 2 million random Wikipedia sentences. On an average, our models outperform their unsupervised models by a significant margin, this despite the fact that they used GloVe embeddings trained on larger corpora than ours (42 times larger). Our models also outperform their semi-supervised PSL + WR model. This indicates our model learns a more precise weighing scheme than the static one proposed by BIBREF4 .
The effect of datasets and n-grams. Despite being trained on three very different datasets, all of our models generalize well to sometimes very specific domains. Models trained on Toronto Corpus are the state-of-the-art on the STS 2014 images dataset even beating the supervised CaptionRep model trained on images. We also see that addition of bigrams to our models doesn't help much when it comes to unsupervised evaluations but gives a significant boost-up in accuracy on supervised tasks. We attribute this phenomenon to the ability of bigrams models to capture some non-compositional features missed by unigrams models. Having a single representation for “not good" or “very bad" can boost the supervised model's ability to infer relevant features for the corresponding classifier. For semantic similarity tasks however, the relative uniqueness of bigrams results in pushing sentence representations further apart, which can explain the average drop of scores for bigrams models on those tasks.
On learning the importance and the direction of the word vectors. Our model – by learning how to generate and compose word vectors – has to learn both the direction of the word embeddings as well as their norm. Considering the norms of the used word vectors as by our averaging over the sentence, we observe an interesting distribution of the “importance” of each word. In Figure FIGREF24 we show the profile of the INLINEFORM0 -norm as a function of INLINEFORM1 for each INLINEFORM2 , and compare it to the static down-weighting mechanism of BIBREF4 . We can observe that our model is learning to down-weight frequent tokens by itself. It is also down-weighting rare tokens and the INLINEFORM3 profile seems to roughly follow Luhn's hypothesis BIBREF36 , a well known information retrieval paradigm, stating that mid-rank terms are the most significant to discriminate content.
In this paper, we introduce a novel, computationally efficient, unsupervised, C-BOW-inspired method to train and infer sentence embeddings. On supervised evaluations, our method, on an average, achieves better performance than all other unsupervised competitors with the exception of SkipThought. However, SkipThought vectors show a very poor performance on sentence similarity tasks while our model is state-of-the-art for these evaluations on average. Also, our model is generalizable, extremely fast to train, simple to understand and easily interpretable, showing the relevance of simple and well-grounded representation models in contrast to the models using deep architectures. Future work could focus on augmenting the model to exploit data with ordered sentences. Furthermore, we would like to investigate the model's ability to use pre-trained embeddings for downstream transfer learning tasks.
Optionally, our model can be additionally improved by adding an L1 regularizer term in the objective function, leading to slightly better generalization performance. Additionally, encouraging sparsity in the embedding vectors is beneficial for memory reasons, allowing higher embedding dimensions INLINEFORM0 .
We propose to apply L1 regularization individually to each word (and n-gram) vector (both source and target vectors). Formally, the training objective function ( EQREF10 ) then becomes DISPLAYFORM0 
 where INLINEFORM0 is the regularization parameter.
Now, in order to minimize a function of the form INLINEFORM0 where INLINEFORM1 is not differentiable over the domain, we can use the basic proximal-gradient scheme. In this iterative method, after doing a gradient descent step on INLINEFORM2 with learning rate INLINEFORM3 , we update INLINEFORM4 as DISPLAYFORM0 
where INLINEFORM0 is called the proximal function BIBREF37 of INLINEFORM1 with INLINEFORM2 being the proximal parameter and INLINEFORM3 is the value of INLINEFORM4 after a gradient (or SGD) step on INLINEFORM5 .
In our case, INLINEFORM0 and the corresponding proximal operator is given by DISPLAYFORM0 
 where INLINEFORM0 corresponds to element-wise product.
Similar to the proximal-gradient scheme, in our case we can optionally use the thresholding operator on the updated word and n-gram vectors after an SGD step. The soft thresholding parameter used for this update is INLINEFORM0 and INLINEFORM1 for the source and target vectors respectively where INLINEFORM2 is the current learning rate, INLINEFORM3 is the INLINEFORM4 regularization parameter and INLINEFORM5 is the sentence on which SGD is being run.
We observe that INLINEFORM0 regularization using the proximal step gives our models a small boost in performance. Also, applying the thresholding operator takes only INLINEFORM1 floating point operations for the updating the word vectors corresponding to the sentence and INLINEFORM2 for updating the target as well as the negative word vectors, where INLINEFORM3 is the number of negatives sampled and INLINEFORM4 is the embedding dimension. Thus, performing INLINEFORM5 regularization using soft-thresholding operator comes with a small computational overhead.
We set INLINEFORM0 to be 0.0005 for both the Wikipedia and the Toronto Book Corpus unigrams + bigrams models.

Figure and table captions:
Table 1: Comparison of the performance of different models on different supervised evaluation tasks. An underline indicates the best performance for the dataset. Top 3 performances in each data category are shown in bold. The average is calculated as the average of accuracy for each category (For MSRP, we take the accuracy). )
Table 2: Unsupervised Evaluation Tasks: Comparison of the performance of different models on Spearman/Pearson correlation measures. An underline indicates the best performance for the dataset. Top 3 performances in each data category are shown in bold. The average is calculated as the average of entries for each correlation measure.
Table 3: Best unsupervised and semi-supervised methods ranked by macro average along with their training times. ** indicates trained on GPU. * indicates trained on a single node using 30 threads. Training times for non-Sent2Vec models are due to Hill et al. (2016a). For CPU based competing methods, we were able to reproduce all published timings (+-10%) using our same hardware as for training Sent2Vec.
Table 4: Comparison of the performance of the unsupervised and semi-supervised sentence embeddings by (Arora et al., 2017) with our models. Unsupervised comparisons are in terms of Pearson’s correlation, while comparisons on supervised tasks are stating the average described in Table 1.
Figure 1: Left figure: the profile of the word vector L2norms as a function of log(fw) for each vocabulary word w, as learnt by our unigram model trained on Toronto books. Right figure: down-weighting scheme proposed by Arora et al. (2017): weight(w) = a a+fw .
Table 5: Training parameters for the Sent2Vec models
Table 6: Comparison of the performance of different Sent2Vec models with different semisupervised/supervised models on different downstream supervised evaluation tasks. An underline indicates the best performance for the dataset and Sent2Vec model performances are bold if they perform as well or better than all other non-Sent2Vec models, including those presented in Table 1.
Table 7: Unsupervised Evaluation: Comparison of the performance of different Sent2Vec models with semi-supervised/supervised models on Spearman/Pearson correlation measures. An underline indicates the best performance for the dataset and Sent2Vec model performances are bold if they perform as well or better than all other non-Sent2Vec models, including those presented in Table 2.
Table 8: Average sentence lengths for the datasets used in the comparison.

Questions about this paper:
What metric is used to measure performance?

Answers:
Accuracy and F1 score for supervised tasks, Pearson's and Spearman's correlation for unsupervised tasks