Title: DECODING DECODERS: FINDING OPTIMAL REPRESENTATION SPACES FOR UNSUPERVISED SIMILARITY TASKS

Abstract: Experimental evidence indicates that simple models outperform complex deep networks on many unsupervised similarity tasks. Introducing the concept of an optimal representation space, we provide a simple theoretical resolution to this apparent paradox. In addition, we present a straightforward procedure that, without any retraining or architectural modifications, allows deep recurrent models to perform equally well (and sometimes better) when compared to shallow models. To validate our analysis, we conduct a set of consistent empirical evaluations and introduce several new sentence embedding models in the process. Even though this work is presented within the context of natural language processing, the insights are readily applicable to other domains that rely on distributed representations for transfer tasks.

1. INTRODUCTION:

Distributed representations have played a pivotal role in the current success of machine learning. In contrast with the symbolic representations of classical AI, distributed representation spaces can encode rich notions of semantic similarity in their distance measures, allowing systems to generalise to novel inputs. Methods to learn these representations have gained significant traction, in particular for modelling words (Mikolov et al., 2013) . They have since been successfully applied to many other domains, including images (Girod et al., 2011; Razavian et al., 2014) and graphs (Kipf & Welling, 2016; Grover & Leskovec, 2016; Narayanan et al., 2017) .

Using unlabelled data to learn effective representations is at the forefront of modern machine learning research. The Natural Language Processing (NLP) community in particular, has invested significant efforts in the construction (Mikolov et al., 2013; Pennington et al., 2014; Bojanowski et al., 2016; Joulin et al., 2017) , evaluation (Baroni et al., 2014) and theoretical analysis (Levy & Goldberg, 2014) of distributed representations for words.

Recently, attention has shifted towards the unsupervised learning of representations for larger pieces of text, such as phrases (Yin & Schütze, 2015; Zhang et al., 2017) , sentences (Kalchbrenner et al., 2014; Kiros et al., 2015; Tai et al., 2015; Hill et al., 2016; Arora et al., 2017) , and entire paragraphs (Le & Mikolov, 2014) . Some of this work simply sums or averages constituent word vectors to obtain a sentence representation (Mitchell & Lapata, 2010; Milajevs et al., 2014; Wieting et al., 2015; Arora et al., 2017) , which is surprisingly effective but naturally cannot leverage any contextual information.

Another line of research has relied on a sentence-level distributional hypothesis (Polajnar et al., 2015) , originally applied to words (Harris, 1954) , which is an assumption that sentences which occur in similar contexts have a similar meaning. Such models often use an encoder-decoder architecture (Cho et al., 2014) to predict the adjacent sentences of any given sentence. Examples of such models include SkipThought (Kiros et al., 2015) , which uses Recurrent Neural Networks (RNNs) for its encoder and decoders, and FastSent (Hill et al., 2016) , which replaces the RNNs with simpler bagof-words (BOW) versions.

Models trained in an unsupervised manner on large text corpora are usually applied to supervised transfer tasks, where the representation for a sentence forms the input to a supervised classification problem, or to unsupervised similarity tasks, where the similarity (typically taken to be the cosine similarity) of two inputs is compared with corresponding human judgements of semantic similarity in order to inform some downstream process, such as information retrieval.

Interestingly, some researchers have observed that deep complex models like SkipThought tend to do well on supervised transfer tasks but relatively poorly on unsupervised similarity tasks, whereas for shallow log-linear models like FastSent the opposite is true (Hill et al., 2016; Conneau et al., 2017) . It has been highlighted that this should be addressed by analysing the geometry of the representation space (Almahairi et al., 2015; Schnabel et al., 2015; Hill et al., 2016) , however, to the best of our knowledge it has not been systematically attempted1 .

In this work we attempt to address the observed performance gap on unsupervised similarity tasks between representations produced by simple models and those produced by deep complex models. Our main contributions are as follows:

• We introduce the concept of an optimal representation space, in which the space has a similarity measure that is optimal with respect to the objective function.

• We show that models with log-linear decoders are usually evaluated in their optimal space, while recurrent models are not. This effectively explains the performance gap on unsupervised similarity tasks.

• We show that, when evaluated in their optimal space, recurrent models close that gap. We also provide a procedure for extracting this optimal space using the decoder hidden states.

• We validate our findings with a series of consistent empirical evaluations utilising a single publicly available codebase.

2. OPTIMAL REPRESENTATION SPACE:

We begin by considering a general problem of learning a conditional probability distribution P model (y | x) over the output symbols y ∈ Y given the input symbols x ∈ X .

Definition 1. A space H combined with a similarity measure ρ : H × H → R in which semantically close symbols s i , s j ∈ S have representations h i , h j ∈ H that are close in ρ is called a distributed representation space (Goodfellow et al., 2016) .

In general, a distributed representation of a symbol s is obtained via some function h s = f (s; θ f ), parametrised by weights θ f . Distributed representations of the input symbols are typically found as the layer activations of a Deep Neural Network (DNN). One can imagine running all possible x ∈ X through a DNN and using the activations h x of the n th layer as vectors in H x :

H x = h x = Activation (n) (x) | x ∈ X .

The distributed representation space of the output symbols H y can be obtained via some function h y = g(y; θ g ) that does not depend on the input symbol x, e.g. a row of the softmax projection matrix that corresponds to the output y.

In practice, although H obtained in such a manner with a reasonable vector similarity ρ (such as cosine or Euclidean distance) forms a distributed representation space, there is no a priori reason why an arbitrary choice of a similarity function would be appropriate given H and the model's objective. There is no analytic guarantee, for arbitrarily chosen H and ρ, that small changes in semantic similarity of symbols correspond to small changes in similarity ρ between their vector representations in H and vice versa. This motivates Definition 2.

Definition 2. A space H equipped with a similarity measure ρ such that log P model (y | x) ∝ ρ (h y , h x ) is called an optimal representation space.

In words, if a model has an optimal representation space, the conditional log-probability of an output symbol y given an input symbol x is proportional to the similarity ρ(h y , h x ) between their corresponding vector representations h y , h x ∈ H.

For example, consider the following standard classification model

P model (y | x) = exp (u y • DNN(x)) y exp (u y • DNN(x)) (1)

where u y is the y th row of the output projection matrix U.

If H x = {DNN(x) | x ∈ X } and H y = {u y | y ∈ Y}, then H = H x ∪ H y equipped with ρ(h 1 , h 2 ) = h 1 • h 2 (the dot product

) is an optimal representation space. Note that if the exponents of Equation (1) contained Euclidean distance, then we would find log P model (y | x) ∝ ||u y -DNN(x)|| 2 . The optimal representation space would then be equipped with Euclidean distance as its optimal distance measure ρ. This easily extends to any other distance measures desired to be induced on the optimal representation space.

Let us elaborate on why Definition 2 is a reasonable definition of an optimal space. Let x 1 , x 2 ∈ X be the input symbols and y 1 , y 2 ∈ Y their corresponding outputs. Using 1). It follows that any model that produces a probability distribution has an optimal representation space. Also note that the optimal space for the inputs does not necessarily have to come from the final layer before the softmax projection but instead can be constructed from any layer, as we now demonstrate.

Let n be the index of the final activation before the softmax projection and let k ∈ {1, . . . , n}. We split the network into three parts:

EQUATION

where G k contains first k layers, F n contains the remaining n -k layers and U is the softmax projection matrix. Let the space for inputs H x be defined as

H x = {G k (x) | x ∈ X }

and the space for outputs H y defined as

H y = {u y | y ∈ Y} .

Their union

H = H x ∪ H y equipped with ρ(h 1 , h 2 ) = J(h 1 ) • J (h 2 )

where

J(h) = F n (h) if h ∈ H x h otherwise

is again an optimal representation space. We will show a specific example where this holds in Section 3.3.

3. OPTIMAL SPACES FOR SENTENCE REPRESENTATIONS:

For the remainder of this paper, we focus on unsupervised models for learning distributed representations of sentences, an area of particular interest in NLP.

3.1. BACKGROUND:

Let S = (s 1 , s 2 , . . . , s N ) be a corpus of contiguous sentences where each sentence s i = w 1 si w 2 si . . . w τs i si consists of words from a pre-defined vocabulary V of size |V |.

We transform the corpus into a set of pairs D = {(s i , c i )} N i=1 , where s i ∈ S and c i is a context of s i . The context usually (but not necessarily) contains some number of surrounding sentences of s i , e.g. c i = (s i-1 , s i+1 ).

We are interested in modelling the probability of a context c given a sentence s. In general

EQUATION

One popular way to model P (c | s) for sentence-level data is suggested by the encoder-decoder framework. The encoder E produces a fixed-length vector representation h E s = E(s) for a sentence s and the decoder gives a context prediction ĉ = D(h E s ) from that representation. Due to a clear architectural separation between E and D, it is common to take h E s as a representation of a sentence s in the downstream tasks. Furthermore, since h E s is usually encoded as a vector, such representations are often compared via simple similarity measures, such as dot product or cosine similarity.

3.2. LOG-LINEAR DECODERS:

We first consider encoder-decoder architectures with a log-linear BOW decoder for the context. Let h i = E(s i ) be a sentence representation of s i produced by some encoder E. The nature of E is not important for our analysis; for concreteness, the reader can consider a model such as FastSent (Hill et al., 2016) , where E is a BOW (sum) encoder.

In the case of the log-linear BOW decoder, words are conditionally independent of the previously occurring sequence, thus Equation (3) becomes

P model (c i |s i ; θ) = w∈ci P model (w|s i ; θ) = w∈ci exp (u w • h i ) w ∈V exp (u w • h i ) = w∈ci exp (u w • h i ) |c i | w ∈V exp (u w • h i )

.

(4) where u w ∈ R d is the output word embedding for a word w and h i is the encoder output. (Biases are omitted for brevity.)

The objective is to maximise the model probability of contexts c i given sentences s i across the corpus D, which corresponds to finding the Maximum Likelihood Estimator (MLE) for the trainable parameters θ:

EQUATION

By switching to the negative log-likelihood and inserting the above expression, we arrive at the following optimisation problem:

EQUATION

we see that the objective in Equation ( 6) forces the sentence representation h i to be similar under dot product to its context representation c i , which is simply the sum of the output embeddings of the context words. Simultaneously, output embeddings of words that do not appear in the context of a sentence are forced to be dissimilar to its representation. Putting it differently, sentences that occur in related contexts are assigned representations that are similar under the dot product. Hence we see that the encoder output equipped with the dot product constitutes an optimal representation space as defined in Section 2.

3.3. RECURRENT SEQUENCE DECODERS:

Another common choice for the context decoder is an RNN decoder

EQUATION

where h i = E(s i ) is the encoder output. The specific structure of E is again not important for our analysis. (When E is also an RNN, this is similar to SkipThought (Kiros et al., 2015) .)

The time unrolled states of decoder are converted to probability distributions over the vocabulary, conditional on the sentence s i and all the previously occurring words. Equation (3) becomes

EQUATION

Similarly to Equation ( 6), MLE for the model parameters θ can be found as

EQUATION

)

Using ⊕ to denote vector concatenation, we note that

EQUATION

where the sentence representation h D i is now an ordered concatenation of the hidden states of the decoder and the context representation c i is an ordered concatenation of the output embeddings of 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10

Number of unroll steps Spearman correlation coefficient:

Figure 2 : Performance on the STS tasks depending on the number of unrolled hidden states of the decoders, using dot product as the similarity measure. The top row presents results for the RNN encoder and the bottom row for the BOW encoder. Red: Raw encoder output with BOW decoder. Green: Raw encoder output with RNN decoder. Blue: Unrolled RNN decoder output. Independent of the encoder architecture, unrolling even a single state of the decoder always outperforms the raw encoder output with RNN decoder, and almost always outperforms the raw encoder output with BOW decoder for some number of unrolls. the context words. Hence we can come to the same conclusion as in the log-linear case, except we have order-sensitive representations as opposed to unordered ones. As before, h D i is forced to be similar to the context c i under dot product, and is made dissimilar to sequences of u w that do not appear in the context.

The "transitivity" argument from Section 3.2 remains intact, except the length of decoder hidden state sequences might differ from sentence to sentence. To avoid this problem, we can formally treat them as infinite-dimensional vectors in 2 with only a finite number of initial components occupied by the sequence and the rest set to zero. Alternatively, we can agree on the maximum sequence length, which in practice can be determined from the training corpus.

Regardless, the above space of unrolled concatenated decoder states, equipped with dot product, is the optimal representation space for models with recurrent decoders. Consequently, this space could be a much better candidate for unsupervised similarity tasks.

We refer to the method of accessing the decoder states at every time step as unrolling the decoder, illustrated in Figure 1 . Note that accessing the decoder output does not require re-architecting or retraining the model, yet gives a potential performance boost on unsupervised similarity tasks almost for free. We will demonstrate the effectiveness of this technique empirically in Section 5.

4. EXPERIMENTAL SETUP:

We have seen in Section 2 that the optimal representation space for a given model depends on the choice of decoder architecture. To support this theory, we train several encoder-decoder architectures for sentences with the decoder types analysed in Section 3, and evaluate them on downstream tasks using both their optimal space and the standard space of the encoder output as the sentence representations.

Models and training.:

Each model has an encoder for the current sentence, and decoders for the previous and next sentences. As our analysis is independent of encoder type, we train and evaluate models with BOW and RNN encoders, two common choices in the literature for sentence representation learners (Hill et al., 2016; Kiros et al., 2015) . The BOW encoder is the sum of word vectors (Hill et al., 2016) . The RNN encoder and decoders are Gated Recurrent Units (GRUs) (Cho et al., 2014) .

Table 1 : Performance of different architectures and sentence representations on unsupervised similarity tasks using dot product as the similarity measure. On each task, the highest performing setup for each encoder type is highlighted in bold and the highest performing setup overall is underlined. All reported values indicate Pearson/Spearman correlation coefficients for the task. RNN encoder: Unrolling the RNN decoders using the concatenation of the decoder hidden states (RNN-concat) dramatically improves the performance across all tasks compared to using the raw encoder output (RNN-RNN), validating the theoretical justification presented in Section 3.3. BOW encoder: Unrolling the RNN decoders improves performance overall, however, the improvement is less drastic than that observed for the RNN encoder, which we discuss further in the main text. Using the notation ENC-DEC, we train RNN-RNN, RNN-BOW, BOW-BOW, and BOW-RNN models. For each encoder-decoder combination, we test several methods of extracting sentence representations to be used in the downstream tasks. First, we use the standard choice of the final output of the encoder as the sentence representation. In addition, for models that have RNN decoders, we unroll between 1 and 10 decoder hidden states. Specifically, when we unroll n decoder hidden states, we take the first n hidden states from each of the decoders and concatenate them in order to get the resulting sentence representation. We refer to these representations as *-RNN-concat.

Encoder:

All models are trained on the Toronto Books Corpus (Zhu et al., 2015) , a dataset of 70 million ordered sentences from over 7,000 books. The sentences are pre-processed such that tokens are lower case and splittable by space.

Evaluation tasks. We use the SentEval tool (Conneau et al., 2017) to benchmark sentence embeddings on both supervised and unsupervised transfer tasks. The supervised tasks in SentEval include paraphrase identification (MSRP) (Dolan et al., 2004) , movie review sentiment (MR) (Pang & Lee, 2005) , product review sentiment (CR), (Hu & Liu, 2004 )), subjectivity (SUBJ) (Pang & Lee, 2004) , opinion polarity (MPQA) (Wiebe et al., 2005) , and question type (TREC) (Voorhees, 2002; Roth & Li, 2003) . In addition, there are two supervised tasks on the SICK dataset, entailment and relatedness (denoted SICK-E and SICK-R) (Marelli et al., 2014) . For the supervised tasks, SentEval trains a logistic regression model with 10-fold cross-validation using the model's embeddings as features.

The unsupervised Semantic Textual Similarity (STS) tasks are STS12-16 (Cer et al., 2017; Agirre et al., 2012; 2013; 2014; Agirre, 2015; Agirre et al., 2016) , which are scored in the same way as SICK-R but without training a new supervised model; in other words, the embeddings are used to directly compute similarity. We use dot product to compute similarity as indicated by our analysis; results and discussion using cosine similarity, which is canonical in the literature, are presented in Appendix B. For more details on all tasks and the evaluation strategy, see Conneau et al. (2017) .

Implementation and hyperparameters. Our goal is to study how different decoder types affect the performance of sentence embeddings on various tasks. To this end, we use identical hyperparameters and architecture for each model (except encoder and decoder types), allowing for a fair headto-head comparison. Specifically, for RNN encoders and decoders we use a single layer GRU with layer normalisation (Ba et al., 2016) . All the weights (including word embeddings) are initialised uniformly over [-0.1, 0.1] and trained with Adam without weight decay or dropout (Kingma & Ba, 2014) . Sentence length is clipped or zero-padded to 30 tokens and end-of-sentence tokens are used throughout training and evaluation. Following Kiros et al. (2015) , we use a vocabulary size of 20k with vocabulary expansion, 620-dimensional word embeddings, and 2400 hidden units in all RNNs.

5. RESULTS:

Performance of the unrolled models on the STS tasks is presented in Figure 2 . We note that unrolling even a single state of the decoder always improves the performance over the raw encoder output with the RNN decoder, and nearly always does so for the BOW decoder for some number of unrolled hidden states.

Table 2 : Performance of different architectures and sentence representations on supervised transfer tasks. On each task, the highest performing setup for each encoder type is highlighted in bold and the highest performing setup overall is underlined. All reported values indicate test accuracy on the task, except for SICK-R where we report the Pearson correlation with human-provided scores. Note that the analysis in Section 3 is not readily applicable here, as instead of using a similarity measure in the representation space directly, the supervised transfer tasks train an entirely new model on top the chosen representation. We observe that the performance tends to peak around 2-3 hidden states and fall off afterwards.

In principle, one might expect the peak to be around the average sentence length of the corpus. A possible explanation of this behaviour is the "softmax drifting effect". As there is no context available at inference time, we generate the word embedding for the next time step using the softmax output from the previous time step (see Figure 1 ). Given that for any sentence, there is no single correct context, the probability distribution over the next words in that context will be multi-modal. This will flatten the softmax and produce inputs for the decoder that diverge from the inputs it expects (i.e. word vectors for the vocabulary). Further work is needed to understand this and other possible causes in detail.

Performance across unsupervised similarity tasks is presented in Table 1 and performance across supervised transfer tasks is presented in Table 2 . For the unrolled architectures, in these tables we report on the one that performs best on the STS tasks. When the encoder is an RNN, the supervised transfer results validate our claims in Section 3.3. The results are less conclusive when the encoder is a BOW. We believe this is caused by the simplicity of the BOW encoder forcing its outputs to obey the sentence-level distributional hypothesis irrespective of decoder type, resulting in multiple candidates for the optimal representation space, but this should be investigated with a detailed analysis in future work.

In addition, see Appendix A for a comparison with the original SkipThought results from the literature, and Appendix B for results using cosine similarity rather than dot product as the similarity measure in STS tasks, as is the canonical choice.

When we look at the performance on supervised transfer in Table 2 , combined with the similarity results in Table 1 , we see that the notion that models cannot be good at both supervised transfer and unsupervised similarity tasks needs refining; for example, RNN-RNN achieves strong performance on supervised transfer, while RNN-RNN-concat achieves strong performance on unsupervised similarity. In general, our results indicate that a single model may be able to perform well on different downstream tasks, provided that the representation spaces chosen for each task are allowed to differ.

Curiously, the unusual combination of a BOW encoder and concatenation of the RNN decoders leads to the best performance on most benchmarks, even slightly exceeding that of some supervised models on some tasks (Conneau et al., 2017) . This architecture may be worth investigating.

6. CONCLUSION:

In this work, we introduced the concept of an optimal representation space, where semantic similarity directly corresponds to distance in that space, in order to shed light on the performance gap between simple and complex architectures on downstream tasks. In particular, we studied the space of initial hidden states to BOW and RNN decoders (typically the outputs of some encoder) and how that space relates to the training objective of the model.

For BOW decoders, the optimal representation space is precisely the initial hidden state of the decoder equipped with dot product, whereas for RNN decoders it is not. Noting that it is precisely these spaces that have been used for BOW and RNN decoders has led us to a simple explanation for the observed performance gap between these architectures, namely that the former has been evaluated in its optimal representation space, whereas the latter has not.

Furthermore, we showed that any neural network that outputs a probability distribution has an optimal representation space. Since a RNN does produce a probability distribution, we analysed its objective function which motivated a procedure of unrolling the decoder. This simple method allowed us to extract representations that are provably optimal under dot product, without needing to retrain the model.

We then validated our claims by comparing the empirical performance of different architectures across transfer tasks. In general, we observed that unrolling even a single state of the decoder always outperforms the raw encoder output with RNN decoder, and almost always outperforms the raw encoder output with BOW decoder for some number of unrolls. This indicates different vector embeddings can be used for different downstream tasks depending on what type of representation space is most suitable, potentially yielding high performance on a variety of tasks from a single trained model.

Although our analysis of decoder architectures was restricted to BOW and RNN, others such as convolutional (Xu et al., 2016) and graph (Kipf & Welling, 2016) decoders are more appropriate for many tasks. Similarly, although we focus on Euclidean vector spaces, hyperbolic vector spaces (Nickel & Kiela, 2017) , complex-valued vector spaces (Trouillon et al., 2016) and spinor spaces (Kanjamapornkul et al., 2017) all have beneficial modelling properties. In each case, although an optimal representation space should exist, it is not clear if the intuitive space and similarity measure is the optimal one. However, there should at least exist a mapping from the intuitive choice of space to the optimal space using a transformation provided by the network itself, as we showed with the RNN decoder. Evaluating in this space should further improve performance of these models. We leave this for future work.

Ultimately, a good representation is one that makes a subsequent learning task easier. For unsupervised similarity tasks, this essentially reduces to how well the model separates objects in the chosen representation space, and how appropriately the similarity measure compares objects in that space. Our findings lead us to the following practical advice: i) Use a simple model architecture where the optimal representation space is clear by construction, or ii) use an arbitrarily complex model architecture and analyse the objective function to reveal, for a chosen vector representation, an appropriate similarity metric.

We hope that future work will utilise a careful understanding of what similarity means and how it is linked to the objective function, and that our analysis can be applied to help boost the performance of other complex models. A COMPARISON WITH SKIPTHOUGHT (Kiros et al., 2015; Ba et al., 2016) , compared against the RNN-RNN model used in our experimental setup. On each task, the highest performing model is highlighted in bold. For SICK-R, we report the Pearson correlation, and for STS14 we report the Pearson/Spearman correlation with human-provided scores. For all other tasks, reported values indicate test accuracy. † indicates results taken from Conneau et al. (2017) . ‡ indicates our results from running SentEval on the model downloaded from Ba et al. ( 2016)'s publicly available codebase (https://github.com/ryankiros/layer-norm). We attribute the discrepancies in performance to differences in experimental setup or implementation. However, we expect our unrolling procedure to also boost SkipThought's performance on unsupervised similarity tasks, as we show for RNN-RNN in our fair singlecodebase comparisons in the main text. As discussed in Section 3, the objective function is maximising the dot product between the BOW decoder/unrolled RNN-decoder and the context. However, as other researchers in the field and the STS tasks specifically use cosine similarity by default, we present the results using cosine similarity in Table 4 and the results for different numbers of unrolled hidden decoder states in Figure 3 .

Although the results in Table 4 are consistent with the dot product results in Table 1 , the overall performance across STS tasks is noticeably lower when dot product is used instead of cosine similarity to determine semantic similarity. Switching from using cosine similarity to dot product transitions from considering only angle between two vectors, to also considering their length. Empirical studies have indicated that the length of a word vector corresponds to how sure of its context the model that produces it is. This is related to how often the model has seen the word, and how many different contexts it appears in (for example, the word vectors for "January" and "February" have similar norms, however, the word vector for "May" is noticeably smaller) (Schakel & Wilson, 2015) .

using the cosine similarity between two vectors as the measure of their similarity. On each task, the highest performing setup for each encoder type is highlighted in bold and the highest performing setup overall is underlined. All reported values indicate Pearson/Spearman correlation coefficients for the task. RNN encoder: Using the raw encoder output (RNN-RNN) achieves the lowest performance across all tasks. Unrolling the RNN decoders dramatically improves the performance across all tasks compared to using the raw encoder RNN output, validating the theoretical justification presented in Section 3.3. BOW encoder: We do not observe the same uplift in performance from unrolling the RNN encoder compared to the encoder output. This is consistent with our findings when using dot product (see Table 1 ). A corollary is that longer sentences on average have shorter norms, since they contain more words which, in turn, have appeared in more contexts (Adi et al., 2017) . During training, the corpus can induce differences in norms in a way that strongly penalises sentences potentially containing multiple contexts, and consequently will disfavour these sentences as similar to other sentences under the dot product. This induces a noise that potentially renders the dot product a less useful to choose for STS tasks than cosine similarity, which is unaffected by this issue. 



C UNROLLING RNN DECODERS BY TAKING THE MEAN:

A practical downside of the unrolling procedure described in Section 3.3 is that concatenating hidden states of the decoder leads to very high dimensional vectors, which might be undesirable due to memory or other practical constraints. An alternative is to instead average the hidden states, which also corresponds to a representation space in which the training objective optimises the dot product as a measure of similarity between a sentence and its context. We refer to this model choice as *-RNN-mean.

Results on similarity and transfer tasks for BOW-RNN-mean and RNN-RNN-mean are presented in Tables 5 and 6 respectively, with results for the other models from Section 5 included for completeness. While the strong performance of RNN-RNN-mean relative to RNN-RNN is consistent with our theory, exploring why it is able to outperform RNN-concat experimentally on STS tasks is left to future work.

:

Although the models inWieting & Gimpel (2017) are trained on a supervised dataset of aligned sentences or paraphrases (and are therefore distinct from our unsupervised setting), it is interesting to note that they observe a similar discrepancy between the performance of RNNs and simply averaging word vectors.

Figures and Tables:

a and b are close under ρ, a reasonable model trained on a subset of (X , Y) will ensure that h x1 ρ ∼ h y1 and h x2 ρ ∼ h y2 . If x 1 and x 2 are semantically close and assuming semantically close input symbols have similar outputs, we also have that h x1 ρ ∼ h y2 and h x2 ρ ∼ h y1 . Therefore it follows that h x1 ρ ∼ h x2 (and h y1 ρ ∼ h y2 ). Putting it differently, semantic similarity of input and output symbols translates into closeness of their distributed representations under ρ, in a way that is consistent with the model. Note that any model P model (y | x) parametrised by a continuous function can be approximated by a function in the form of Equation (

Figure1: Unrolling a RNN decoder at inference time. The initial hidden state for the decoder is typically the encoder output, either the recurrent cell final state for a RNN encoder, or the sum of the input word embeddings for a BOW encoder. At the first time step, a learned <GO> token is presented as the input. In subsequent time steps, a probability-weighted sum over word vectors is used. The decoder is then unrolled for a fixed number of steps. The hidden states are then concatenated to produce the unrolled decoder embedding. In the models evaluated in Section 4, this process is performed for the RNN corresponding to the previous and next sentences. The sentence representation is then taken as the concatenation across both RNNs.Using dot ∼ to denote close under dot product, we find that if two sentences s i and s j have similar contexts, then h i dot ∼ c j and h j dot ∼ c i . The objective function in Equation (6) ensures that h i dot ∼ c i

Figure3: Performance on the STS tasks depending on the number of unrolled hidden states of the decoders, using cosine similarity as the similarity measure. The top row presents results for the RNN encoder and the bottom row for the BOW encoder. Red: Raw encoder output with BOW decoder. Green: Raw encoder output with RNN decoder. Blue: Unrolled RNN decoder output. For both RNN and BOW encoders, unrolling the decoder strictly outperforms *-RNN for almost every number of unroll steps, and perform nearly as well as or better than *-BOW.

Performance of the SkipThought model, with and without layer normalisation

Table content: <table/>

Performance of different architectures and sentence representations on unsupervised similarity tasks using dot product as the similarity measure. On each task, the highest performing setup for each encoder type is highlighted in bold and the highest performing setup overall is underlined. All reported values indicate Pearson/Spearman correlation coefficients for the task. RNN-concat 0.422/0.438 0.478/0.498 0.498/0.506 0.512/0.544 0.402/0.460

Table content: <table><tr><td>Encoder</td><td>Decoder</td><td>STS12</td><td>STS13</td><td>STS14</td><td>STS15</td><td>STS16</td></tr><tr><td/><td>BOW</td><td>0.286/0.384</td><td>0.381/0.426</td><td>0.365/0.406</td><td>0.262/0.392</td><td>0.260/0.328</td></tr><tr><td>RNN</td><td>RNN</td><td>0.267/0.286</td><td>0.371/0.365</td><td>0.357/0.348</td><td>0.379/0.379</td><td>0.313/0.303</td></tr><tr><td/><td>RNN-mean</td><td>0.330/0.361</td><td colspan="3">0.420/0.427 0.438/0.428 0.419/0.426</td><td>0.324/0.342</td></tr><tr><td/><td colspan="2">RNN-concat 0.335/0.359</td><td>0.411/0.415</td><td>0.413/0.403</td><td>0.414/0.419</td><td>0.326/0.347</td></tr><tr><td/><td>BOW</td><td>0.351/0.390</td><td>0.418/0.478</td><td>0.442/0.488</td><td>0.455/0.535</td><td>0.370/0.461</td></tr><tr><td>BOW</td><td>RNN</td><td>0.310/0.342</td><td>0.365/0.422</td><td>0.396/0.440</td><td>0.412/0.469</td><td>0.281/0.330</td></tr><tr><td/><td>RNN-mean</td><td>0.394/0.414</td><td>0.469/0.495</td><td>0.490/0.498</td><td>0.495/0.530</td><td>0.381/0.439</td></tr></table>

Performance of different architectures and sentence representations on supervised transfer tasks. On each task, the highest performing setup for each encoder type is highlighted in bold and the highest performing setup overall is underlined. All reported values indicate test accuracy on the task, except for SICK-R where we report the Pearson correlation with human-provided scores.

Table content: <table><tr><td>Encoder</td><td>Decoder</td><td>MR</td><td>CR</td><td colspan="2">MPQA SUBJ</td><td>SST</td><td colspan="4">TREC MRPC SICK-R SICK-E</td></tr><tr><td/><td>BOW</td><td>75.78</td><td>79.34</td><td>86.25</td><td>90.77</td><td>81.99</td><td>84.60</td><td>70.55</td><td>0.80</td><td>78.81</td></tr><tr><td>RNN</td><td>RNN</td><td colspan="2">77.06 81.77</td><td>88.59</td><td colspan="2">92.56 82.65</td><td>86.60</td><td>71.94</td><td>0.83</td><td>81.10</td></tr><tr><td/><td>RNN-mean</td><td>76.55</td><td>81.03</td><td>87.35</td><td>92.29</td><td>81.11</td><td>84.80</td><td>73.51</td><td>0.84</td><td>78.22</td></tr><tr><td/><td colspan="2">RNN-concat 76.20</td><td>82.07</td><td>85.96</td><td>91.80</td><td>80.83</td><td>87.20</td><td>71.59</td><td>0.82</td><td>80.35</td></tr><tr><td/><td>BOW</td><td>76.16</td><td>81.14</td><td>87.03</td><td>92.77</td><td>81.66</td><td>84.20</td><td>71.07</td><td>0.84</td><td>80.58</td></tr><tr><td>BOW</td><td>RNN</td><td colspan="2">76.05 82.07</td><td>85.80</td><td>92.13</td><td>80.83</td><td>87.20</td><td>72.99</td><td>0.82</td><td>78.87</td></tr><tr><td/><td>RNN-mean</td><td>75.85</td><td>81.30</td><td>85.54</td><td>90.80</td><td>80.12</td><td>84.00</td><td>71.13</td><td>0.81</td><td>77.76</td></tr><tr><td/><td>RNN-concat</td><td>77.27</td><td>82.04</td><td>88.74</td><td>92.88</td><td>81.82</td><td>89.60</td><td>73.68</td><td>0.85</td><td>82.26</td></tr></table>

Peer review (iclr2018):
 ------ updates to review: --------- I think the paper is much improved. It is much more clear and the experiments are more focused and more closely connected to the earlier content in the paper. Thanks to the authors for trying to address all of my concerns. I now better understand in what sense the representation space is optimal. I had been thinking (or perhaps "hoping" is a better word) that the term "optimal" implied maximal in terms of some quantifiable measure, but it*s more of an empirical "optimality". This makes the paper an empirical paper based on a reasonable and sensible intuition, rather than a theoretical result. This was a little disappointing to me, but I do still think the paper is marginally above the acceptance threshold and have increased my score accordingly. ------ original review below: -------- This paper is about rethinking how to use encoder-decoder architectures for representation learning when the training objective contains a similarity between the decoder output and the encoding of something else. For example, for the skip-thought RNN encoder-decoder that encodes a sentence and decodes neighboring sentences: rather than use the final encoder hidden state as the representation of the sentence, the paper uses some function of the decoder, since the training objective is to maximize each dot product between a decoder hidden state and the embedding of a context word. If dot product (or cosine similarity) is going to be used as the similarity function for the representation, then it makes more sense, the paper argues, to use the decoder hidden state(s) as the representation of the input sentence. The paper considers both averaging and concatenating hidden states. One difficulty here is that the neighboring sentences are typically not available in downstream tasks, so the paper runs the decoder to produce a predicted sentence one word-at-a-time, using the predicted words as inputs to the decoder RNNs. Then those decoder RNN hidden states are used via averaging or concatenation as the representation of a sentence in downstream tasks. This paper is a source of contributions, but I think in its current form it is not yet ready for publication. Pros: I think it makes sense to pay attention to the training objective when deciding how to use the model for downstream tasks. I like the empirical investigation of combining RNN and BOW encoders and decoders. The experimental results show that a single encoder-decoder model can be trained and then two different functions of it can be used at test time for different kinds of tasks (RNN-RNN for supervised transfer and RNN-RNN-mean for unsupervised transfer). I think this is an interesting result. Cons: I have several concerns. The first relate to the theoretical arguments and their empirical support. Regarding the theoretical arguments: First, the paper discusses the notion of an "optimal representation space" and describes the argument as theoretical, but I don*t see much of a theoretical argument here. As far as I can tell, the paper does not formally define its terms or define in what sense the representation space is "optimal". I can only find heuristic statements like those in the paragraph in Sec 3.2 that begins "These observations...". What exactly is meant formally by statements like "any model where the decoder is log-linear with respect to the encoder" or "that distance is optimal with respect to the model’s objective"? It seems like the paper may want to start with formal definitions of an encoder and a decoder, then define what is meant by a "decoder that is log-linear with respect to the encoder", and define what it means for a distance to be optimal with respect to a training objective. That seems necessary in order to provide the foundation to make any theoretical statement about choices for encoders, decoders, and training objectives. I am still not exactly sure what that theoretical statement might look like, but maybe defining the terms would help the authors get started in heading toward the goal of defining a statement to prove. Second, the paper*s theoretical story seems to diverge almost immediately from the choices used in the model and experimental procedure. For example, in Sec. 3.2, it is stated that cosine similarity "is the appropriate similarity measure in the case of log-linear decoders." But the associated footnote (footnote 2) seems to admit a contradiction here by noting that actually the appropriate similarity measure is dot product: "Evidently, the correct measure is actually the dot product." This is a bit confusing. It also raises a question: If cosine similarity will be used later for computing similarity, then why not try using cosine similarity in place of dot product in the model? That is, replace "u_w .cdot h_i" in Eq. (2) with "cos(u_w, h_i)". If the paper*s story is correct (and if I understand the ideas correctly), training with cosine similarity should work better than training with dot product, because the similarity function used during training is more similar to that used in testing. This seems like a natural experiment to try. Other natural experiments would be to vary both the similarity function used in the model during training and the similarity function used at test time. The authors* claims could be validated if the optimal choices always use the same choice for the training and test-time similarity functions. That is, if Euclidean distance is used during training, then will Euclidean distance be the best choice at test time? Another example of the divergence lies in the use of the skip-thought decoder on downstream tasks. Since the decoder hidden states depend on neighboring sentences and these are considered to be unavailable at test time, the paper "unrolls" the decoder for several steps by using it to predict words which are then used as inputs on the next time step. To me, this is a potentially very significant difference between training and testing. Since much of the paper is about reconciling training and testing conditions in terms of the representation space and similarity function, this difference feels like a divergence from the theoretical story. It is only briefly mentioned at the end of Sec. 3.3 and then discussed again later in the experiments section. I think this should be described in more detail in Section 3.3 because it is an important note about how the model will be used in practice. It would be nice to be able to quantify the impact (of unrolling the decoder with predicted words) by, for example, using the decoder on a downstream evaluation dataset that has neighboring sentences in it. Then the actual neighboring sentences can be used as inputs to the decoder when it is unrolled, which would be closer to the training conditions and we could empirically see the difference. Perhaps there is an evaluation dataset with ordered sentences so that the authors could empirically compare using real vs predicted inputs to the decoder on a downstream task? The above experiments might help to better connect the experiments section with the theoretical arguments. Other concerns, including more specific points, are below: Sec. 2: When describing inferior performance of RNN-based models on unsupervised sentence similarity tasks, the paper states: "While this shortcoming of SkipThought and RNN-based models in general has been pointed out, to the best of our knowledge, it has never been systematically addressed in the literature before." The authors may want to check Wieting & Gimpel (2017) (and its related work) which investigates the inferiority of LSTMs compared to word averaging for unsupervised sentence similarity tasks. They found that averaging the encoder hidden states can work better than using the final encoder hidden state; the authors may want to try that as well. Sec. 3.2: When describing FastSent, the paper includes "Due to the model*s simplicity, it is particularly fast to train and evaluate, yet has shown state-of-the-art performance in unsupervised similarity tasks (Hill et al., 2015)." I don*t think it makes much sense to cite the SimLex-999 paper in this context, as that is a word similarity task and that paper does not include any results of FastSent. Maybe the Hill et al (2016) FastSent citation was meant instead? But in that case, I don*t think it is quite accurate to make the claim that FastSent is SOTA on unsupervised similarity tasks. In the original FastSent paper (Hill et al., 2016), FastSent is not as good as CPHRASE or "DictRep BOW+embs" on average across the unsupervised sentence similarity evaluations. FastSent is also not as good as sent2vec from Pagliardini et al (2017) or charagram-phrase from Wieting et al. (2016). Sec. 3.3: In describing skip-thought, the paper states: "While computationally complex, it is currently the state-of-the-art model for supervised transfer tasks (Hill et al., 2016)." I don*t think it is accurate to state that skip-thought is still state-of-the-art for supervised transfer tasks, in light of recent work (Conneau et al., 2017; Gan et al., 2017). Sec. 3.3: When discussing averaging the decoder hidden states, the paper states: "Intuitively, this corresponds to destroying the word order information the decoder has learned." I*m not sure this strong language can be justified here. Is there any evidence to suggest that averaging the decoder hidden states will destroy word order information? The hidden states may be representing word order information in a way that is robust to averaging, i.e., in a way such that the average of the hidden states can still lead to the reconstruction of the word order. Sec. 4: What does it mean to use an RNN encoder and a BOW decoder? This seems to be a strongly-performing setting and competitive with RNN-mean, but I don*t know exactly what this means. Minor things: Sec. 3.1: When defining v_w, it would be helpful to make explicit that it*s in .mathbb{R}^d. Sec. 4: For TREC question type classification, I think the correct citation should be Li & Roth (2002) instead of Vorhees (2002). Sec. 5: I think there*s a typo in the following sentence: "Our results show that, for example, the raw encoder output for SkipThought (RNN-RNN) achieves strong performance on supervised transfer, whilst its mean decoder output (RNN-mean) achieves strong performance on supervised transfer." I think "unsupervised" was meant in the latter mention. References: Conneau, A., Kiela, D., Schwenk, H., Barrault, L., & Bordes, A. (2017). Supervised Learning of Universal Sentence Representations from Natural Language Inference Data. EMNLP. Gan, Z., Pu, Y., Henao, R., Li, C., He, X., & Carin, L. (2017). Learning generic sentence representations using convolutional neural networks. EMNLP. Li, X., & Roth, D. (2002). Learning question classifiers. COLING. Pagliardini, M., Gupta, P., & Jaggi, M. (2017). Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features. arXiv preprint arXiv:1703.02507. Wieting, J., Bansal, M., Gimpel, K., & Livescu, K. (2016). Charagram: Embedding words and sentences via character n-grams. EMNLP. Wieting, J., & Gimpel, K. (2017). Revisiting Recurrent Networks for Paraphrastic Sentence Embeddings. ACL.