This paper is about rethinking how to use encoder-decoder architectures for representation learning when the training objective contains a similarity between the decoder output and the encoding of something else. For example, for the skip-thought RNN encoder-decoder that encodes a sentence and decodes neighboring sentences: rather than use the final encoder hidden state as the representation of the sentence, the paper uses some function of the decoder, since the training objective is to maximize each dot product between a decoder hidden state and the embedding of a context word. If dot product (or cosine similarity) is going to be used as the similarity function for the representation, then it makes more sense, the paper argues, to use the decoder hidden state(s) as the representation of the input sentence. The paper considers both averaging and concatenating hidden states. One difficulty here is that the neighboring sentences are typically not available in downstream tasks, so the paper runs the decoder to produce a predicted sentence one word-at-a-time, using the predicted words as inputs to the decoder RNNs. Then those decoder RNN hidden states are used via averaging or concatenation as the representation of a sentence in downstream tasks. This paper is a source of contributions, but I think in its current form it is not yet ready for publication. Pros: I think it makes sense to pay attention to the training objective when deciding how to use the model for downstream tasks. I like the empirical investigation of combining RNN and BOW encoders and decoders. The experimental results show that a single encoder-decoder model can be trained and then two different functions of it can be used at test time for different kinds of tasks (RNN-RNN for supervised transfer and RNN-RNN-mean for unsupervised transfer). I think this is an interesting result. Cons: I have several concerns. The first relate to the theoretical arguments and their empirical support. Regarding the theoretical arguments: First, the paper discusses the notion of an "optimal representation space" and describes the argument as theoretical, but I don't see much of a theoretical argument here. As far as I can tell, the paper does not formally define its terms or define in what sense the representation space is "optimal". I can only find heuristic statements like those in the paragraph in Sec 3.2 that begins "These observations...". What exactly is meant formally by statements like "any model where the decoder is log-linear with respect to the encoder" or "that distance is optimal with respect to the model’s objective"? It seems like the paper may want to start with formal definitions of an encoder and a decoder, then define what is meant by a "decoder that is log-linear with respect to the encoder", and define what it means for a distance to be optimal with respect to a training objective. That seems necessary in order to provide the foundation to make any theoretical statement about choices for encoders, decoders, and training objectives. I am still not exactly sure what that theoretical statement might look like, but maybe defining the terms would help the authors get started in heading toward the goal of defining a statement to prove. Second, the paper's theoretical story seems to diverge almost immediately from the choices used in the model and experimental procedure. For example, in Sec. 3.2, it is stated that cosine similarity "is the appropriate similarity measure in the case of log-linear decoders." But the associated footnote (footnote 2) seems to admit a contradiction here by noting that actually the appropriate similarity measure is dot product: "Evidently, the correct measure is actually the dot product." This is a bit confusing. It also raises a question: If cosine similarity will be used later for computing similarity, then why not try using cosine similarity in place of dot product in the model? That is, replace "u_w \cdot h_i" in Eq. (2) with "cos(u_w, h_i)". If the paper's story is correct (and if I understand the ideas correctly), training with cosine similarity should work better than training with dot product, because the similarity function used during training is more similar to that used in testing. This seems like a natural experiment to try. Other natural experiments would be to vary both the similarity function used in the model during training and the similarity function used at test time. The authors' claims could be validated if the optimal choices always use the same choice for the training and test-time similarity functions. That is, if Euclidean distance is used during training, then will Euclidean distance be the best choice at test time? Another example of the divergence lies in the use of the skip-thought decoder on downstream tasks. Since the decoder hidden states depend on neighboring sentences and these are considered to be unavailable at test time, the paper "unrolls" the decoder for several steps by using it to predict words which are then used as inputs on the next time step. To me, this is a potentially very significant difference between training and testing. Since much of the paper is about reconciling training and testing conditions in terms of the representation space and similarity function, this difference feels like a divergence from the theoretical story. It is only briefly mentioned at the end of Sec. 3.3 and then discussed again later in the experiments section. I think this should be described in more detail in Section 3.3 because it is an important note about how the model will be used in practice. It would be nice to be able to quantify the impact (of unrolling the decoder with predicted words) by, for example, using the decoder on a downstream evaluation dataset that has neighboring sentences in it. Then the actual neighboring sentences can be used as inputs to the decoder when it is unrolled, which would be closer to the training conditions and we could empirically see the difference. Perhaps there is an evaluation dataset with ordered sentences so that the authors could empirically compare using real vs predicted inputs to the decoder on a downstream task? The above experiments might help to better connect the experiments section with the theoretical arguments. Other concerns, including more specific points, are below: Sec. 2: When describing inferior performance of RNN-based models on unsupervised sentence similarity tasks, the paper states: "While this shortcoming of SkipThought and RNN-based models in general has been pointed out, to the best of our knowledge, it has never been systematically addressed in the literature before." The authors may want to check Wieting & Gimpel (2017) (and its related work) which investigates the inferiority of LSTMs compared to word averaging for unsupervised sentence similarity tasks. They found that averaging the encoder hidden states can work better than using the final encoder hidden state; the authors may want to try that as well. Sec. 3.2: When describing FastSent, the paper includes "Due to the model's simplicity, it is particularly fast to train and evaluate, yet has shown state-of-the-art performance in unsupervised similarity tasks (Hill et al., 2015)." I don't think it makes much sense to cite the SimLex-999 paper in this context, as that is a word similarity task and that paper does not include any results of FastSent. Maybe the Hill et al (2016) FastSent citation was meant instead? But in that case, I don't think it is quite accurate to make the claim that FastSent is SOTA on unsupervised similarity tasks. In the original FastSent paper (Hill et al., 2016), FastSent is not as good as CPHRASE or "DictRep BOW+embs" on average across the unsupervised sentence similarity evaluations. FastSent is also not as good as sent2vec from Pagliardini et al (2017) or charagram-phrase from Wieting et al. (2016). Sec. 3.3:In describing skip-thought, the paper states: "While computationally complex, it is currently the state-of-the-art model for supervised transfer tasks (Hill et al., 2016)." I don't think it is accurate to state that skip-thought is still state-of-the-art for supervised transfer tasks, in light of recent work (Conneau et al., 2017; Gan et al., 2017). Sec. 3.3:When discussing averaging the decoder hidden states, the paper states: "Intuitively, this corresponds to destroying the word order information the decoder has learned." I'm not sure this strong language can be justified here. Is there any evidence to suggest that averaging the decoder hidden states will destroy word order information? The hidden states may be representing word order information in a way that is robust to averaging, i.e., in a way such that the average of the hidden states can still lead to the reconstruction of the word order. Sec. 4: What does it mean to use an RNN encoder and a BOW decoder? This seems to be a strongly-performing setting and competitive with RNN-mean, but I don't know exactly what this means. Minor things:Sec. 3.1:When defining v_w, it would be helpful to make explicit that it's in \mathbb{R}^d. Sec. 4: For TREC question type classification, I think the correct citation should be Li & Roth (2002) instead of Vorhees (2002). Sec. 5:I think there's a typo in the following sentence: "Our results show that, for example, the raw encoder output for SkipThought (RNN-RNN) achieves strong performance on supervised transfer, whilst its mean decoder output (RNN-mean) achieves strong performance on supervised transfer." I think "unsupervised" was meant in the latter mention. References: Conneau, A., Kiela, D., Schwenk, H., Barrault, L., & Bordes, A. (2017). Supervised Learning of Universal Sentence Representations from Natural Language Inference Data. EMNLP. Gan, Z., Pu, Y., Henao, R., Li, C., He, X., & Carin, L. (2017). Learning generic sentence representations using convolutional neural networks. EMNLP. Li, X., & Roth, D. (2002). Learning question classifiers. COLING. Pagliardini, M., Gupta, P., & Jaggi, M. (2018). Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features. arXiv preprint arXiv:1703.02507. Wieting, J., Bansal, M., Gimpel, K., & Livescu, K. (2016). Charagram: Embedding words and sentences via character n-grams. EMNLP. Wieting, J., & Gimpel, K. (2017). Revisiting Recurrent Networks for Paraphrastic Sentence Embeddings. ACL.