Title: DECODING DECODERS: FINDING OPTIMAL REPRESENTATION SPACES FOR UNSUPERVISED SIMILARITY TASKS

Abstract: Experimental evidence indicates that simple models outperform complex deep networks on many unsupervised similarity tasks. Introducing the concept of an optimal representation space, we provide a simple theoretical resolution to this apparent paradox. In addition, we present a straightforward procedure that, without any retraining or architectural modifications, allows deep recurrent models to perform equally well (and sometimes better) when compared to shallow models. To validate our analysis, we conduct a set of consistent empirical evaluations and introduce several new sentence embedding models in the process. Even though this work is presented within the context of natural language processing, the insights are readily applicable to other domains that rely on distributed representations for transfer tasks.

1. INTRODUCTION:

Distributed representations have played a pivotal role in the current success of machine learning. In contrast with the symbolic representations of classical AI, distributed representation spaces can encode rich notions of semantic similarity in their distance measures, allowing systems to generalise to novel inputs. Methods to learn these representations have gained significant traction, in particular for modelling words (Mikolov et al., 2013) . They have since been successfully applied to many other domains, including images (Girod et al., 2011; Razavian et al., 2014) and graphs (Kipf & Welling, 2016; Grover & Leskovec, 2016; Narayanan et al., 2017) .

Using unlabelled data to learn effective representations is at the forefront of modern machine learning research. The Natural Language Processing (NLP) community in particular, has invested significant efforts in the construction (Mikolov et al., 2013; Pennington et al., 2014; Bojanowski et al., 2016; Joulin et al., 2017) , evaluation (Baroni et al., 2014) and theoretical analysis (Levy & Goldberg, 2014) of distributed representations for words.

Recently, attention has shifted towards the unsupervised learning of representations for larger pieces of text, such as phrases (Yin & Schütze, 2015; Zhang et al., 2017) , sentences (Kalchbrenner et al., 2014; Kiros et al., 2015; Tai et al., 2015; Hill et al., 2016; Arora et al., 2017) , and entire paragraphs (Le & Mikolov, 2014) . Some of this work simply sums or averages constituent word vectors to obtain a sentence representation (Mitchell & Lapata, 2010; Milajevs et al., 2014; Wieting et al., 2015; Arora et al., 2017) , which is surprisingly effective but naturally cannot leverage any contextual information.

Another line of research has relied on a sentence-level distributional hypothesis (Polajnar et al., 2015) , originally applied to words (Harris, 1954) , which is an assumption that sentences which occur in similar contexts have a similar meaning. Such models often use an encoder-decoder architecture (Cho et al., 2014) to predict the adjacent sentences of any given sentence. Examples of such models include SkipThought (Kiros et al., 2015) , which uses Recurrent Neural Networks (RNNs) for its encoder and decoders, and FastSent (Hill et al., 2016) , which replaces the RNNs with simpler bagof-words (BOW) versions.

Models trained in an unsupervised manner on large text corpora are usually applied to supervised transfer tasks, where the representation for a sentence forms the input to a supervised classification problem, or to unsupervised similarity tasks, where the similarity (typically taken to be the cosine similarity) of two inputs is compared with corresponding human judgements of semantic similarity in order to inform some downstream process, such as information retrieval.

Interestingly, some researchers have observed that deep complex models like SkipThought tend to do well on supervised transfer tasks but relatively poorly on unsupervised similarity tasks, whereas for shallow log-linear models like FastSent the opposite is true (Hill et al., 2016; Conneau et al., 2017) . It has been highlighted that this should be addressed by analysing the geometry of the representation space (Almahairi et al., 2015; Schnabel et al., 2015; Hill et al., 2016) , however, to the best of our knowledge it has not been systematically attempted1 .

In this work we attempt to address the observed performance gap on unsupervised similarity tasks between representations produced by simple models and those produced by deep complex models. Our main contributions are as follows:

• We introduce the concept of an optimal representation space, in which the space has a similarity measure that is optimal with respect to the objective function.

• We show that models with log-linear decoders are usually evaluated in their optimal space, while recurrent models are not. This effectively explains the performance gap on unsupervised similarity tasks.

• We show that, when evaluated in their optimal space, recurrent models close that gap. We also provide a procedure for extracting this optimal space using the decoder hidden states.

• We validate our findings with a series of consistent empirical evaluations utilising a single publicly available codebase.

2. OPTIMAL REPRESENTATION SPACE:

We begin by considering a general problem of learning a conditional probability distribution P model (y | x) over the output symbols y ∈ Y given the input symbols x ∈ X .

Definition 1. A space H combined with a similarity measure ρ : H × H → R in which semantically close symbols s i , s j ∈ S have representations h i , h j ∈ H that are close in ρ is called a distributed representation space (Goodfellow et al., 2016) .

In general, a distributed representation of a symbol s is obtained via some function h s = f (s; θ f ), parametrised by weights θ f . Distributed representations of the input symbols are typically found as the layer activations of a Deep Neural Network (DNN). One can imagine running all possible x ∈ X through a DNN and using the activations h x of the n th layer as vectors in H x :

H x = h x = Activation (n) (x) | x ∈ X .

The distributed representation space of the output symbols H y can be obtained via some function h y = g(y; θ g ) that does not depend on the input symbol x, e.g. a row of the softmax projection matrix that corresponds to the output y.

In practice, although H obtained in such a manner with a reasonable vector similarity ρ (such as cosine or Euclidean distance) forms a distributed representation space, there is no a priori reason why an arbitrary choice of a similarity function would be appropriate given H and the model's objective. There is no analytic guarantee, for arbitrarily chosen H and ρ, that small changes in semantic similarity of symbols correspond to small changes in similarity ρ between their vector representations in H and vice versa. This motivates Definition 2.

Definition 2. A space H equipped with a similarity measure ρ such that log P model (y | x) ∝ ρ (h y , h x ) is called an optimal representation space.

In words, if a model has an optimal representation space, the conditional log-probability of an output symbol y given an input symbol x is proportional to the similarity ρ(h y , h x ) between their corresponding vector representations h y , h x ∈ H.

For example, consider the following standard classification model

P model (y | x) = exp (u y • DNN(x)) y exp (u y • DNN(x)) (1)

where u y is the y th row of the output projection matrix U.

If H x = {DNN(x) | x ∈ X } and H y = {u y | y ∈ Y}, then H = H x ∪ H y equipped with ρ(h 1 , h 2 ) = h 1 • h 2 (the dot product

) is an optimal representation space. Note that if the exponents of Equation (1) contained Euclidean distance, then we would find log P model (y | x) ∝ ||u y -DNN(x)|| 2 . The optimal representation space would then be equipped with Euclidean distance as its optimal distance measure ρ. This easily extends to any other distance measures desired to be induced on the optimal representation space.

Let us elaborate on why Definition 2 is a reasonable definition of an optimal space. Let x 1 , x 2 ∈ X be the input symbols and y 1 , y 2 ∈ Y their corresponding outputs. Using 1). It follows that any model that produces a probability distribution has an optimal representation space. Also note that the optimal space for the inputs does not necessarily have to come from the final layer before the softmax projection but instead can be constructed from any layer, as we now demonstrate.

Let n be the index of the final activation before the softmax projection and let k ∈ {1, . . . , n}. We split the network into three parts:

EQUATION

where G k contains first k layers, F n contains the remaining n -k layers and U is the softmax projection matrix. Let the space for inputs H x be defined as

H x = {G k (x) | x ∈ X }

and the space for outputs H y defined as

H y = {u y | y ∈ Y} .

Their union

H = H x ∪ H y equipped with ρ(h 1 , h 2 ) = J(h 1 ) • J (h 2 )

where

J(h) = F n (h) if h ∈ H x h otherwise

is again an optimal representation space. We will show a specific example where this holds in Section 3.3.

3. OPTIMAL SPACES FOR SENTENCE REPRESENTATIONS:

For the remainder of this paper, we focus on unsupervised models for learning distributed representations of sentences, an area of particular interest in NLP.

3.1. BACKGROUND:

Let S = (s 1 , s 2 , . . . , s N ) be a corpus of contiguous sentences where each sentence s i = w 1 si w 2 si . . . w τs i si consists of words from a pre-defined vocabulary V of size |V |.

We transform the corpus into a set of pairs D = {(s i , c i )} N i=1 , where s i ∈ S and c i is a context of s i . The context usually (but not necessarily) contains some number of surrounding sentences of s i , e.g. c i = (s i-1 , s i+1 ).

We are interested in modelling the probability of a context c given a sentence s. In general

EQUATION

One popular way to model P (c | s) for sentence-level data is suggested by the encoder-decoder framework. The encoder E produces a fixed-length vector representation h E s = E(s) for a sentence s and the decoder gives a context prediction ĉ = D(h E s ) from that representation. Due to a clear architectural separation between E and D, it is common to take h E s as a representation of a sentence s in the downstream tasks. Furthermore, since h E s is usually encoded as a vector, such representations are often compared via simple similarity measures, such as dot product or cosine similarity.

3.2. LOG-LINEAR DECODERS:

We first consider encoder-decoder architectures with a log-linear BOW decoder for the context. Let h i = E(s i ) be a sentence representation of s i produced by some encoder E. The nature of E is not important for our analysis; for concreteness, the reader can consider a model such as FastSent (Hill et al., 2016) , where E is a BOW (sum) encoder.

In the case of the log-linear BOW decoder, words are conditionally independent of the previously occurring sequence, thus Equation (3) becomes

P model (c i |s i ; θ) = w∈ci P model (w|s i ; θ) = w∈ci exp (u w • h i ) w ∈V exp (u w • h i ) = w∈ci exp (u w • h i ) |c i | w ∈V exp (u w • h i )

.

(4) where u w ∈ R d is the output word embedding for a word w and h i is the encoder output. (Biases are omitted for brevity.)

The objective is to maximise the model probability of contexts c i given sentences s i across the corpus D, which corresponds to finding the Maximum Likelihood Estimator (MLE) for the trainable parameters θ:

EQUATION

By switching to the negative log-likelihood and inserting the above expression, we arrive at the following optimisation problem:

EQUATION

we see that the objective in Equation ( 6) forces the sentence representation h i to be similar under dot product to its context representation c i , which is simply the sum of the output embeddings of the context words. Simultaneously, output embeddings of words that do not appear in the context of a sentence are forced to be dissimilar to its representation. Putting it differently, sentences that occur in related contexts are assigned representations that are similar under the dot product. Hence we see that the encoder output equipped with the dot product constitutes an optimal representation space as defined in Section 2.

3.3. RECURRENT SEQUENCE DECODERS:

Another common choice for the context decoder is an RNN decoder

EQUATION

where h i = E(s i ) is the encoder output. The specific structure of E is again not important for our analysis. (When E is also an RNN, this is similar to SkipThought (Kiros et al., 2015) .)

The time unrolled states of decoder are converted to probability distributions over the vocabulary, conditional on the sentence s i and all the previously occurring words. Equation (3) becomes

EQUATION

Similarly to Equation ( 6), MLE for the model parameters θ can be found as

EQUATION

)

Using ⊕ to denote vector concatenation, we note that

EQUATION

where the sentence representation h D i is now an ordered concatenation of the hidden states of the decoder and the context representation c i is an ordered concatenation of the output embeddings of 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10

Number of unroll steps Spearman correlation coefficient:

Figure 2 : Performance on the STS tasks depending on the number of unrolled hidden states of the decoders, using dot product as the similarity measure. The top row presents results for the RNN encoder and the bottom row for the BOW encoder. Red: Raw encoder output with BOW decoder. Green: Raw encoder output with RNN decoder. Blue: Unrolled RNN decoder output. Independent of the encoder architecture, unrolling even a single state of the decoder always outperforms the raw encoder output with RNN decoder, and almost always outperforms the raw encoder output with BOW decoder for some number of unrolls. the context words. Hence we can come to the same conclusion as in the log-linear case, except we have order-sensitive representations as opposed to unordered ones. As before, h D i is forced to be similar to the context c i under dot product, and is made dissimilar to sequences of u w that do not appear in the context.

The "transitivity" argument from Section 3.2 remains intact, except the length of decoder hidden state sequences might differ from sentence to sentence. To avoid this problem, we can formally treat them as infinite-dimensional vectors in 2 with only a finite number of initial components occupied by the sequence and the rest set to zero. Alternatively, we can agree on the maximum sequence length, which in practice can be determined from the training corpus.

Regardless, the above space of unrolled concatenated decoder states, equipped with dot product, is the optimal representation space for models with recurrent decoders. Consequently, this space could be a much better candidate for unsupervised similarity tasks.

We refer to the method of accessing the decoder states at every time step as unrolling the decoder, illustrated in Figure 1 . Note that accessing the decoder output does not require re-architecting or retraining the model, yet gives a potential performance boost on unsupervised similarity tasks almost for free. We will demonstrate the effectiveness of this technique empirically in Section 5.

4. EXPERIMENTAL SETUP:

We have seen in Section 2 that the optimal representation space for a given model depends on the choice of decoder architecture. To support this theory, we train several encoder-decoder architectures for sentences with the decoder types analysed in Section 3, and evaluate them on downstream tasks using both their optimal space and the standard space of the encoder output as the sentence representations.

Models and training.:

Each model has an encoder for the current sentence, and decoders for the previous and next sentences. As our analysis is independent of encoder type, we train and evaluate models with BOW and RNN encoders, two common choices in the literature for sentence representation learners (Hill et al., 2016; Kiros et al., 2015) . The BOW encoder is the sum of word vectors (Hill et al., 2016) . The RNN encoder and decoders are Gated Recurrent Units (GRUs) (Cho et al., 2014) .

Table 1 : Performance of different architectures and sentence representations on unsupervised similarity tasks using dot product as the similarity measure. On each task, the highest performing setup for each encoder type is highlighted in bold and the highest performing setup overall is underlined. All reported values indicate Pearson/Spearman correlation coefficients for the task. RNN encoder: Unrolling the RNN decoders using the concatenation of the decoder hidden states (RNN-concat) dramatically improves the performance across all tasks compared to using the raw encoder output (RNN-RNN), validating the theoretical justification presented in Section 3.3. BOW encoder: Unrolling the RNN decoders improves performance overall, however, the improvement is less drastic than that observed for the RNN encoder, which we discuss further in the main text. Using the notation ENC-DEC, we train RNN-RNN, RNN-BOW, BOW-BOW, and BOW-RNN models. For each encoder-decoder combination, we test several methods of extracting sentence representations to be used in the downstream tasks. First, we use the standard choice of the final output of the encoder as the sentence representation. In addition, for models that have RNN decoders, we unroll between 1 and 10 decoder hidden states. Specifically, when we unroll n decoder hidden states, we take the first n hidden states from each of the decoders and concatenate them in order to get the resulting sentence representation. We refer to these representations as *-RNN-concat.

Encoder:

All models are trained on the Toronto Books Corpus (Zhu et al., 2015) , a dataset of 70 million ordered sentences from over 7,000 books. The sentences are pre-processed such that tokens are lower case and splittable by space.

Evaluation tasks. We use the SentEval tool (Conneau et al., 2017) to benchmark sentence embeddings on both supervised and unsupervised transfer tasks. The supervised tasks in SentEval include paraphrase identification (MSRP) (Dolan et al., 2004) , movie review sentiment (MR) (Pang & Lee, 2005) , product review sentiment (CR), (Hu & Liu, 2004 )), subjectivity (SUBJ) (Pang & Lee, 2004) , opinion polarity (MPQA) (Wiebe et al., 2005) , and question type (TREC) (Voorhees, 2002; Roth & Li, 2003) . In addition, there are two supervised tasks on the SICK dataset, entailment and relatedness (denoted SICK-E and SICK-R) (Marelli et al., 2014) . For the supervised tasks, SentEval trains a logistic regression model with 10-fold cross-validation using the model's embeddings as features.

The unsupervised Semantic Textual Similarity (STS) tasks are STS12-16 (Cer et al., 2017; Agirre et al., 2012; 2013; 2014; Agirre, 2015; Agirre et al., 2016) , which are scored in the same way as SICK-R but without training a new supervised model; in other words, the embeddings are used to directly compute similarity. We use dot product to compute similarity as indicated by our analysis; results and discussion using cosine similarity, which is canonical in the literature, are presented in Appendix B. For more details on all tasks and the evaluation strategy, see Conneau et al. (2017) .

Implementation and hyperparameters. Our goal is to study how different decoder types affect the performance of sentence embeddings on various tasks. To this end, we use identical hyperparameters and architecture for each model (except encoder and decoder types), allowing for a fair headto-head comparison. Specifically, for RNN encoders and decoders we use a single layer GRU with layer normalisation (Ba et al., 2016) . All the weights (including word embeddings) are initialised uniformly over [-0.1, 0.1] and trained with Adam without weight decay or dropout (Kingma & Ba, 2014) . Sentence length is clipped or zero-padded to 30 tokens and end-of-sentence tokens are used throughout training and evaluation. Following Kiros et al. (2015) , we use a vocabulary size of 20k with vocabulary expansion, 620-dimensional word embeddings, and 2400 hidden units in all RNNs.

5. RESULTS:

Performance of the unrolled models on the STS tasks is presented in Figure 2 . We note that unrolling even a single state of the decoder always improves the performance over the raw encoder output with the RNN decoder, and nearly always does so for the BOW decoder for some number of unrolled hidden states.

Table 2 : Performance of different architectures and sentence representations on supervised transfer tasks. On each task, the highest performing setup for each encoder type is highlighted in bold and the highest performing setup overall is underlined. All reported values indicate test accuracy on the task, except for SICK-R where we report the Pearson correlation with human-provided scores. Note that the analysis in Section 3 is not readily applicable here, as instead of using a similarity measure in the representation space directly, the supervised transfer tasks train an entirely new model on top the chosen representation. We observe that the performance tends to peak around 2-3 hidden states and fall off afterwards.

In principle, one might expect the peak to be around the average sentence length of the corpus. A possible explanation of this behaviour is the "softmax drifting effect". As there is no context available at inference time, we generate the word embedding for the next time step using the softmax output from the previous time step (see Figure 1 ). Given that for any sentence, there is no single correct context, the probability distribution over the next words in that context will be multi-modal. This will flatten the softmax and produce inputs for the decoder that diverge from the inputs it expects (i.e. word vectors for the vocabulary). Further work is needed to understand this and other possible causes in detail.

Performance across unsupervised similarity tasks is presented in Table 1 and performance across supervised transfer tasks is presented in Table 2 . For the unrolled architectures, in these tables we report on the one that performs best on the STS tasks. When the encoder is an RNN, the supervised transfer results validate our claims in Section 3.3. The results are less conclusive when the encoder is a BOW. We believe this is caused by the simplicity of the BOW encoder forcing its outputs to obey the sentence-level distributional hypothesis irrespective of decoder type, resulting in multiple candidates for the optimal representation space, but this should be investigated with a detailed analysis in future work.

In addition, see Appendix A for a comparison with the original SkipThought results from the literature, and Appendix B for results using cosine similarity rather than dot product as the similarity measure in STS tasks, as is the canonical choice.

When we look at the performance on supervised transfer in Table 2 , combined with the similarity results in Table 1 , we see that the notion that models cannot be good at both supervised transfer and unsupervised similarity tasks needs refining; for example, RNN-RNN achieves strong performance on supervised transfer, while RNN-RNN-concat achieves strong performance on unsupervised similarity. In general, our results indicate that a single model may be able to perform well on different downstream tasks, provided that the representation spaces chosen for each task are allowed to differ.

Curiously, the unusual combination of a BOW encoder and concatenation of the RNN decoders leads to the best performance on most benchmarks, even slightly exceeding that of some supervised models on some tasks (Conneau et al., 2017) . This architecture may be worth investigating.

6. CONCLUSION:

In this work, we introduced the concept of an optimal representation space, where semantic similarity directly corresponds to distance in that space, in order to shed light on the performance gap between simple and complex architectures on downstream tasks. In particular, we studied the space of initial hidden states to BOW and RNN decoders (typically the outputs of some encoder) and how that space relates to the training objective of the model.

For BOW decoders, the optimal representation space is precisely the initial hidden state of the decoder equipped with dot product, whereas for RNN decoders it is not. Noting that it is precisely these spaces that have been used for BOW and RNN decoders has led us to a simple explanation for the observed performance gap between these architectures, namely that the former has been evaluated in its optimal representation space, whereas the latter has not.

Furthermore, we showed that any neural network that outputs a probability distribution has an optimal representation space. Since a RNN does produce a probability distribution, we analysed its objective function which motivated a procedure of unrolling the decoder. This simple method allowed us to extract representations that are provably optimal under dot product, without needing to retrain the model.

We then validated our claims by comparing the empirical performance of different architectures across transfer tasks. In general, we observed that unrolling even a single state of the decoder always outperforms the raw encoder output with RNN decoder, and almost always outperforms the raw encoder output with BOW decoder for some number of unrolls. This indicates different vector embeddings can be used for different downstream tasks depending on what type of representation space is most suitable, potentially yielding high performance on a variety of tasks from a single trained model.

Although our analysis of decoder architectures was restricted to BOW and RNN, others such as convolutional (Xu et al., 2016) and graph (Kipf & Welling, 2016) decoders are more appropriate for many tasks. Similarly, although we focus on Euclidean vector spaces, hyperbolic vector spaces (Nickel & Kiela, 2017) , complex-valued vector spaces (Trouillon et al., 2016) and spinor spaces (Kanjamapornkul et al., 2017) all have beneficial modelling properties. In each case, although an optimal representation space should exist, it is not clear if the intuitive space and similarity measure is the optimal one. However, there should at least exist a mapping from the intuitive choice of space to the optimal space using a transformation provided by the network itself, as we showed with the RNN decoder. Evaluating in this space should further improve performance of these models. We leave this for future work.

Ultimately, a good representation is one that makes a subsequent learning task easier. For unsupervised similarity tasks, this essentially reduces to how well the model separates objects in the chosen representation space, and how appropriately the similarity measure compares objects in that space. Our findings lead us to the following practical advice: i) Use a simple model architecture where the optimal representation space is clear by construction, or ii) use an arbitrarily complex model architecture and analyse the objective function to reveal, for a chosen vector representation, an appropriate similarity metric.

We hope that future work will utilise a careful understanding of what similarity means and how it is linked to the objective function, and that our analysis can be applied to help boost the performance of other complex models. A COMPARISON WITH SKIPTHOUGHT (Kiros et al., 2015; Ba et al., 2016) , compared against the RNN-RNN model used in our experimental setup. On each task, the highest performing model is highlighted in bold. For SICK-R, we report the Pearson correlation, and for STS14 we report the Pearson/Spearman correlation with human-provided scores. For all other tasks, reported values indicate test accuracy. † indicates results taken from Conneau et al. (2017) . ‡ indicates our results from running SentEval on the model downloaded from Ba et al. ( 2016)'s publicly available codebase (https://github.com/ryankiros/layer-norm). We attribute the discrepancies in performance to differences in experimental setup or implementation. However, we expect our unrolling procedure to also boost SkipThought's performance on unsupervised similarity tasks, as we show for RNN-RNN in our fair singlecodebase comparisons in the main text. As discussed in Section 3, the objective function is maximising the dot product between the BOW decoder/unrolled RNN-decoder and the context. However, as other researchers in the field and the STS tasks specifically use cosine similarity by default, we present the results using cosine similarity in Table 4 and the results for different numbers of unrolled hidden decoder states in Figure 3 .

Although the results in Table 4 are consistent with the dot product results in Table 1 , the overall performance across STS tasks is noticeably lower when dot product is used instead of cosine similarity to determine semantic similarity. Switching from using cosine similarity to dot product transitions from considering only angle between two vectors, to also considering their length. Empirical studies have indicated that the length of a word vector corresponds to how sure of its context the model that produces it is. This is related to how often the model has seen the word, and how many different contexts it appears in (for example, the word vectors for "January" and "February" have similar norms, however, the word vector for "May" is noticeably smaller) (Schakel & Wilson, 2015) .

using the cosine similarity between two vectors as the measure of their similarity. On each task, the highest performing setup for each encoder type is highlighted in bold and the highest performing setup overall is underlined. All reported values indicate Pearson/Spearman correlation coefficients for the task. RNN encoder: Using the raw encoder output (RNN-RNN) achieves the lowest performance across all tasks. Unrolling the RNN decoders dramatically improves the performance across all tasks compared to using the raw encoder RNN output, validating the theoretical justification presented in Section 3.3. BOW encoder: We do not observe the same uplift in performance from unrolling the RNN encoder compared to the encoder output. This is consistent with our findings when using dot product (see Table 1 ). A corollary is that longer sentences on average have shorter norms, since they contain more words which, in turn, have appeared in more contexts (Adi et al., 2017) . During training, the corpus can induce differences in norms in a way that strongly penalises sentences potentially containing multiple contexts, and consequently will disfavour these sentences as similar to other sentences under the dot product. This induces a noise that potentially renders the dot product a less useful to choose for STS tasks than cosine similarity, which is unaffected by this issue. 



C UNROLLING RNN DECODERS BY TAKING THE MEAN:

A practical downside of the unrolling procedure described in Section 3.3 is that concatenating hidden states of the decoder leads to very high dimensional vectors, which might be undesirable due to memory or other practical constraints. An alternative is to instead average the hidden states, which also corresponds to a representation space in which the training objective optimises the dot product as a measure of similarity between a sentence and its context. We refer to this model choice as *-RNN-mean.

Results on similarity and transfer tasks for BOW-RNN-mean and RNN-RNN-mean are presented in Tables 5 and 6 respectively, with results for the other models from Section 5 included for completeness. While the strong performance of RNN-RNN-mean relative to RNN-RNN is consistent with our theory, exploring why it is able to outperform RNN-concat experimentally on STS tasks is left to future work.

:

Although the models inWieting & Gimpel (2017) are trained on a supervised dataset of aligned sentences or paraphrases (and are therefore distinct from our unsupervised setting), it is interesting to note that they observe a similar discrepancy between the performance of RNNs and simply averaging word vectors.

Figures and Tables:

a and b are close under ρ, a reasonable model trained on a subset of (X , Y) will ensure that h x1 ρ ∼ h y1 and h x2 ρ ∼ h y2 . If x 1 and x 2 are semantically close and assuming semantically close input symbols have similar outputs, we also have that h x1 ρ ∼ h y2 and h x2 ρ ∼ h y1 . Therefore it follows that h x1 ρ ∼ h x2 (and h y1 ρ ∼ h y2 ). Putting it differently, semantic similarity of input and output symbols translates into closeness of their distributed representations under ρ, in a way that is consistent with the model. Note that any model P model (y | x) parametrised by a continuous function can be approximated by a function in the form of Equation (

Figure1: Unrolling a RNN decoder at inference time. The initial hidden state for the decoder is typically the encoder output, either the recurrent cell final state for a RNN encoder, or the sum of the input word embeddings for a BOW encoder. At the first time step, a learned <GO> token is presented as the input. In subsequent time steps, a probability-weighted sum over word vectors is used. The decoder is then unrolled for a fixed number of steps. The hidden states are then concatenated to produce the unrolled decoder embedding. In the models evaluated in Section 4, this process is performed for the RNN corresponding to the previous and next sentences. The sentence representation is then taken as the concatenation across both RNNs.Using dot ∼ to denote close under dot product, we find that if two sentences s i and s j have similar contexts, then h i dot ∼ c j and h j dot ∼ c i . The objective function in Equation (6) ensures that h i dot ∼ c i

Figure3: Performance on the STS tasks depending on the number of unrolled hidden states of the decoders, using cosine similarity as the similarity measure. The top row presents results for the RNN encoder and the bottom row for the BOW encoder. Red: Raw encoder output with BOW decoder. Green: Raw encoder output with RNN decoder. Blue: Unrolled RNN decoder output. For both RNN and BOW encoders, unrolling the decoder strictly outperforms *-RNN for almost every number of unroll steps, and perform nearly as well as or better than *-BOW.

Performance of the SkipThought model, with and without layer normalisation

Table content: <table/>

Performance of different architectures and sentence representations on unsupervised similarity tasks using dot product as the similarity measure. On each task, the highest performing setup for each encoder type is highlighted in bold and the highest performing setup overall is underlined. All reported values indicate Pearson/Spearman correlation coefficients for the task. RNN-concat 0.422/0.438 0.478/0.498 0.498/0.506 0.512/0.544 0.402/0.460

Table content: <table><tr><td>Encoder</td><td>Decoder</td><td>STS12</td><td>STS13</td><td>STS14</td><td>STS15</td><td>STS16</td></tr><tr><td/><td>BOW</td><td>0.286/0.384</td><td>0.381/0.426</td><td>0.365/0.406</td><td>0.262/0.392</td><td>0.260/0.328</td></tr><tr><td>RNN</td><td>RNN</td><td>0.267/0.286</td><td>0.371/0.365</td><td>0.357/0.348</td><td>0.379/0.379</td><td>0.313/0.303</td></tr><tr><td/><td>RNN-mean</td><td>0.330/0.361</td><td colspan="3">0.420/0.427 0.438/0.428 0.419/0.426</td><td>0.324/0.342</td></tr><tr><td/><td colspan="2">RNN-concat 0.335/0.359</td><td>0.411/0.415</td><td>0.413/0.403</td><td>0.414/0.419</td><td>0.326/0.347</td></tr><tr><td/><td>BOW</td><td>0.351/0.390</td><td>0.418/0.478</td><td>0.442/0.488</td><td>0.455/0.535</td><td>0.370/0.461</td></tr><tr><td>BOW</td><td>RNN</td><td>0.310/0.342</td><td>0.365/0.422</td><td>0.396/0.440</td><td>0.412/0.469</td><td>0.281/0.330</td></tr><tr><td/><td>RNN-mean</td><td>0.394/0.414</td><td>0.469/0.495</td><td>0.490/0.498</td><td>0.495/0.530</td><td>0.381/0.439</td></tr></table>

Performance of different architectures and sentence representations on supervised transfer tasks. On each task, the highest performing setup for each encoder type is highlighted in bold and the highest performing setup overall is underlined. All reported values indicate test accuracy on the task, except for SICK-R where we report the Pearson correlation with human-provided scores.

Table content: <table><tr><td>Encoder</td><td>Decoder</td><td>MR</td><td>CR</td><td colspan="2">MPQA SUBJ</td><td>SST</td><td colspan="4">TREC MRPC SICK-R SICK-E</td></tr><tr><td/><td>BOW</td><td>75.78</td><td>79.34</td><td>86.25</td><td>90.77</td><td>81.99</td><td>84.60</td><td>70.55</td><td>0.80</td><td>78.81</td></tr><tr><td>RNN</td><td>RNN</td><td colspan="2">77.06 81.77</td><td>88.59</td><td colspan="2">92.56 82.65</td><td>86.60</td><td>71.94</td><td>0.83</td><td>81.10</td></tr><tr><td/><td>RNN-mean</td><td>76.55</td><td>81.03</td><td>87.35</td><td>92.29</td><td>81.11</td><td>84.80</td><td>73.51</td><td>0.84</td><td>78.22</td></tr><tr><td/><td colspan="2">RNN-concat 76.20</td><td>82.07</td><td>85.96</td><td>91.80</td><td>80.83</td><td>87.20</td><td>71.59</td><td>0.82</td><td>80.35</td></tr><tr><td/><td>BOW</td><td>76.16</td><td>81.14</td><td>87.03</td><td>92.77</td><td>81.66</td><td>84.20</td><td>71.07</td><td>0.84</td><td>80.58</td></tr><tr><td>BOW</td><td>RNN</td><td colspan="2">76.05 82.07</td><td>85.80</td><td>92.13</td><td>80.83</td><td>87.20</td><td>72.99</td><td>0.82</td><td>78.87</td></tr><tr><td/><td>RNN-mean</td><td>75.85</td><td>81.30</td><td>85.54</td><td>90.80</td><td>80.12</td><td>84.00</td><td>71.13</td><td>0.81</td><td>77.76</td></tr><tr><td/><td>RNN-concat</td><td>77.27</td><td>82.04</td><td>88.74</td><td>92.88</td><td>81.82</td><td>89.60</td><td>73.68</td><td>0.85</td><td>82.26</td></tr></table>

Peer review (iclr2018):
 The authors provide some theoretical justification for why simple log-linear decoders perform better than RNN decoders for various unsupervised sentence similarity tasks. They also provide a simple method for improving the performance of RNN based models. Please find below my comments/questions/suggestions: 1) I found the theory to be a bit superficial and there is a clearly a gap between what is proven theoretically and demonstrated empirically. For example, as per the theoretical arguments presented in the paper, RNN-concat should do better than RNN-mean. However, the experiments suggest that RNN-mean always does better and in some cases significantly better (referring to Table 1). How does this empirical observation reconcile with the theory ? 2) The authors mention that the results for SkipThought represented in their paper are lower than those presented in the original SkipThought paper. They say that they elaborate on this in Appendix C but there isn*t much information provided. In particular, it would be good to mention the original numbers also (of course I can check the original paper but if the authors provide those numbers then it would ensure that there is no misunderstanding) 3) As mentioned in the previous point, the original SkipThought decoder seems to do better than the modified decoder used by the authors. It is not clear, how this can be justified under the theoretical framework presented by the author. I agree that this could be because in the original formulation (referring to equations in Appendix C) the encoder contributes more directly to the decoder. However, it is not clear how this causes it to be "closer" to the optimal space. Can this be proven ? 4) Can you elaborate a bit more on how the model is used at test time? Consider the STS17 benchmark and the following sentence pair from it? - The bird is bathing in the sink. - Birdie is washing itself in the water basin. How will you use RNN-mean and RNN-concat to find the similarity between these two sentences. 5) In continuation of the above question, how does the computation of similarity differ when you unroll for 3 steps v/s when you unroll for 5 steps 6) Based on the answers to (4) and (5) above I would like to seek further clarifications on Figure 1.