Title: EXPLORING ASYMMETRIC ENCODER-DECODER STRUCTURE FOR CONTEXT-BASED SENTENCE REPRESENTATION LEARNING

Abstract: Context information plays an important role in human language understanding, and it is also useful for machines to learn vector representations of language. In this paper, we explore an asymmetric encoder-decoder structure for unsupervised context-based sentence representation learning. As a result, we build an encoderdecoder architecture with an RNN encoder and a CNN decoder, and we show that neither an autoregressive decoder nor an RNN decoder is required. We further combine a suite of effective designs to significantly improve model efficiency while also achieving better performance. Our model is trained on two different large unlabeled corpora, and in both cases transferability is evaluated on a set of downstream language understanding tasks. We empirically show that our model is simple and fast while producing rich sentence representations that excel in downstream tasks. Recently, Tang et al. (2017) proposed the Skip-thought Neighbor model, which only decodes the next

1. INTRODUCTION:

Learning distributed representations of sentences is an important and hard topic in both the deep learning and natural language processing communities, since it requires machines to encode a sentence with rich language content into a fixed-dimension vector filled with continuous values. We are interested in learning to build a distributed sentence encoder in an unsupervised fashion by exploiting the structure and relationship in a large unlabeled corpus. Since humans interpret sentences by composing from the meanings of the words, we decompose the task of learning a sentence encoder into two essential components: learning distributed word representations, and learning how to compose a sentence representation from the representations of words in the given sentence.

Numerous studies in human language processing have claimed that the context in which words and sentences are understood plays an important role in human language understanding (Altmann & Mirkovic, 2009; Binder & Desai, 2011) . The idea of learning from the context information (Turney & Pantel, 2010) was recently successfully applied to vector representation learning for words in Mikolov et al. (2013) ; Pennington et al. (2014) . Collobert et al. (2011) proposed a unified framework for learning language representation from the unlabeled data, and it is able to generalize to various NLP tasks. Inspired by the prior work on incorporating context information into representation learning, Kiros et al. (2015) proposed the Skipthought model, which is an encoder-decoder model for unsupervised sentence representation learning. The paper exploits the semantic similarity within a tuple of adjacent sentences as supervision, and successfully built a generic, distributed sentence encoder. Rather than applying the conventional autoencoder model, the skip-thought model tries to reconstruct the surrounding 2 sentences instead of the input sentence. The learned sentence representation encoder outperforms previous unsupervised pretrained models on the evaluation tasks with no finetuning, and the results are comparable to the models which were trained directly on the datasets in a supervised fashion.

The usage of 2 independent decoders in Skip-thought model matches our intuition that, given the current sentence, inferring the previous sentence and inferring the next one should be different. sentence, and the performance on the downstream tasks is similar to that of their implementation of the Skip-thought model.

In this paper, we follow the idea in the Skip-thought Neighbor model, which exploits the subsequent context information for learning representation, and aim to bring asymmetry into structure design as well. Our proposed model has an asymmetric encoder-decoder structure, which keeps an RNN as the encoder and has a CNN as the decoder, and will be referred to as an "RNN-CNN" model. The key components of our model design can be summarized as:

1. a bidirectional RNN encodes the input sentence, and a CNN decodes all words in the paired target sequence at once, which speeds up the training process;

2. the supervision for training comes from inferring the next contiguous words given the current sentence, which helps the model to learn from the context in an unsupervised fashion;

3. the mean+max pooling captures complex interactions among words, which augments the transferability of the proposed model; 4. tying word embeddings in the encoder with the word prediction layer in the decoder constrains the input and output space to be the same, which also reduces number of parameters and regularizes the model.

We demonstrate the transferability of our model by evaluation on various downstream tasks, and the performance shows that our model improves both results and training efficiency.

2. RNN-CNN MODEL:

Our model is highly asymmetric in terms of both training pairs and model structure. Specifically, our model has an RNN as the encoder, and a CNN as the decoder. During training, the encoder takes the ith sentence s i as input, and then generates a fixed-dimension vector z i as the sentence representation; the decoder is applied to reconstruct the next sentence or the subsequent few contiguous words t i . The difference of the generated sequence and the target sequence is measured by cross-entropy loss. An illustration is in Figure 1 . (For simplicity, we omit the subscript i in the section.)

Encoder: The encoder is a bi-directional Gated Recurrent Unit (GRU) (Chung et al., 2014) . We experimented with both Long-short Term Memory (LSTM, Hochreiter & Schmidhuber (1997) ) and GRU. Since LSTM didn't give us significant performance boost, and generally GRU runs faster than LSTM, in our experiments, we stick to using GRU in the encoder. Suppose that a sentence s contains M words, which are w 1 , w 2 , ..., w M , and they are transformed by an embedding matrix E to word vectors. The bi-directional GRU will take one word vector at a time, and run in both forward and backward direction; both sets of hidden states are concatenated to form the hidden state matrix

H = [h 1 , h 2 , ..., h M ] ∈ R D×M , where d is the dimension of the representations h m = ← - h m ; -→ h m

(∀m ∈ {1, 2, ..., M }).

Representation: We aim to provide a model with faster training speed with better transferability than existing algorithms, thus we choose to apply a parameter-free composition function, which is a concatenation of the outputs from a global mean pooling over time and a global max pooling over time, on the computed sequence of hidden states. The composition function can be represented as

EQUATION

where max H d• is the max operation on the d-th row of the matrix H, which outputs a scalar. Thus the representation z has a dimension of 2d.

Decoder: The decoder is a 3-layer CNN to reconstruct the paired target sequence t, which needs to expand z from length 1 to the length of t. Intuitively, the decoder could be a stack of deconvolution layers. For fast training speed, we optimized the architecture to make it plausible to use fullyconnected layers and convolution layers in the decoder, since generally, convolution layers run faster than deconvolution layers in modern deep learning frameworks.

Suppose that the target sequence t has N words, the first layer of deconvolution will expand z, which could be considered as a sequence with length 1, into a feature map with length N . It can be easily implemented as a concatenation of outputs from N linear transformations in parallel. Then the second and third layer are 1D-convolution layers with kernel size 3 and 1, respectively. The output feature map

V = [v 1 , v 2 , ..., v N ],

where v ∈ R e , and e is dimension of the word vectors.

Note that our decoder is not an autoregressive model, and it brings us high training efficiency. We will discuss the reason of choosing this decoder which we call a predict-all-words CNN decoder.

Objective: A softmax layer is applied after the decoder to produce a probability distribution over words at each position, softmax(Ev n ), and the training objective is to minimize the sum of the negative log-likelihood over all positions in the target sequence t:

EQUATION

The loss function L is summed over all sentences in the training corpus.

3. ARCHITECTURE DESIGN:

We follow the idea of an encoder-decoder model with using the context information for learning sentence representations in an unsupervised fashion. Since the decoder won't be used after training, and the quality of the generated sequences is not our main focus, it is important to study the design of the decoder. Generally, a fast training algorithm is preferred, thus proposing a new decoder with high training efficiency and also strong transferability is crucial for an encoder-decoder model.

3.1. CNN AS THE DECODER:

Our design of the decoder is basically a 3-layer ConvNet, and it predicts all words in the next sequence all at once. In contrast, existing work, such as Skip-thought Kiros et al. (2015) , and CNN-LSTM Gan et al. (2017) ,use autoregressive RNNs as the decoders.

An autoregressive model is good at generating sequences with high quality, such as language and speech. However, an autoregressive decoder seems to be unnecessary in an encoder-decoder model for learning sentence representations, since it won't be used after training, and it runs quite slow during training. Therefore, we conducted experiments to test the necessity of using an autoregressive decoder in learning sentence representations, and we had 2 findings.

Finding I: It is not necessary to input the correct words into an autoregressive decoder in terms of learning good sentence representations. The models here all have a bi-directional GRU as the encoder (dimensionality 300 in each direction). The default way of producing the representation is a concatenation of outputs from a global mean-pooling and a global max-pooling, while "•-Max" refers to the model with only global maxpooling. Bold numbers are the best results among all presented models. We found that 1) inputting correct words to an autoregressive decoder is not necessary; 2) predict-all-words decoders work roughly the same as autoregressive decoders; 3) mean+max pooling provides stronger transferability than the max-pooling alone does. The table supports our choice of the predict-all-words CNN decoder and the way of producing vector representations from the bi-directional RNN encoder.

The experimental design was inspired by Bengio et al. (2015) . The model we designed for the experiment has a bi-directional GRU as the encoder, and an autoregressive decoder, including both RNN and CNN. We started by analyzing the effect of different sampling strategies of the input words on learning an auto-regressive decoder.

We compared 3 autoregressive decoding settings: 1) using ground-truth words (Baseline), 2) using previously predicted words (Always Sampling), and 3) using uniformly sampled words from the dictionary (Uniform Sampling). The 3 decoding settings were named by Bengio et al. (2015) . The results are presented in the Table 1 .

Generally, the three different decoding settings didn't make much of a difference in terms of the performance on selected downstream tasks, with RNN or CNN as the decoder. The results tell us that, in terms of learning good sentence representations, the autoregressive decoder doesn't require the correct ground-truth words as the inputs.

Finding II: The model with an autoregressive decoder works roughly the same as the model with a predict-all-words decoder.

With Finding I, we noticed that the correct ground-truth input words to the autoregressive decoder is not necessary in terms of learning sentence representations. Therefore, it makes sense to test whether we need an autoregressive model at all.

In our model, the CNN decoder predicts all words at once during training, which is different from autoregressive decoders, and we call it a predict-all-words CNN decoder. We want to compare the performance of the predict-all-words decoders and that of the autoregressive decoders separate from the RNN/CNN distinction, thus we designed a predict-all-words CNN decoder and RNN decoder.

The predict-all-words CNN decoder is described in Section 2, which is a stack of 3 convolutional layers, and all words are predicted once at the output of the decoder. The predict-all-words RNN decoder is built based on our CNN decoder. To keep the number of parameters roughly the same, we replaced the last 2 convolutional layers with a bidirectional GRU.

The results are also presented in the Table 1 . The performance of the predict-all-words RNN decoder does not significantly differ from that of any one of the autoregressive RNN decoders, and the same observation was observed in CNN decoders.

These two findings actually support our choice of using a predict-all-words CNN as the decoder, and it brings the model higher training efficiency and strong transferability.

3.2. MEAN+MAX POOLING:

Since the encoder is a bi-directional RNN in our model, we have multiple ways to select/compute on the generated hidden states to produce a sentence representation. In Skip-thought (Kiros et al., 2015) and SDAE (Hill et al., 2016) , only the hidden state at the last time step produced by the RNN encoder is regarded as the vector representation for a given sentence, which may not be the most expressive vector for representing the input sentence.

We followed the idea proposed in Chen et al. (2016) . They built a model for supervised SNLI task (Bowman et al., 2015) that concatenates the outputs from a global mean pooling and a global max pooling to serve as a sentence representation, and showed a performance boost on the SNLI dataset. Also, Conneau et al. (2017) found that the model with global max pooling function has stronger transferability than the model with a global mean pooling function after supervised training on SNLI.

In our proposed RNN-CNN model, we empirically show that the mean+max pooling provides stronger transferability than the max pooling does, and the results are presented in Table 1 . The concatenation of a mean-pooling and a max pooling function is actually a parameter-free composition function, and the computation load is negligible compared to heavy matrix multiplications. Also, the non-linearity of the max pooling function augments the mean pooling function for building a representation that captures a more complex composition of the syntactic information.

3.3. TYING WORD EMBEDDINGS AND WORD PREDICTION LAYER:

We choose to share the parameters in the word embedding layer in RNN encoder and the word prediction layer in CNN decoder. The tying was proposed in both Press & Wolf (2017) and Inan et al. (2016) , and it generally helps to learn a better language model. In our model, the tying also drastically reduces the number of parameters, which could prevent overfitting.

Furthermore, we initialize the word embeddings with pretrained word vectors, such as word2vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014) , since it has been shown that these pretrained word vectors can serve as good initialization for deep learning models, and more likely lead to better results than random samples from a uniform distribution.

3.4. STUDY OF THE HYPERPARAMETERS IN OUR MODEL DESIGN:

We studied hyperparameters in our model design based on 3 out of 10 downstream tasks, including SICK-r, SICK-E (Marelli et al., 2014) , and STS14 (Agirre et al., 2014) . The first model we created, which is reported in Section 2, is a decent design, and the following variations didn't give us much performance change except small improvements with increasing the dimensionality of the encoder. However, we think it is worth mentioning the effect of hyperparameters in our model design. We present the Table in the supplementary material and we summarize it as follows:

1. Decoding the next sentence worked similarly as decoding the subsequent contiguous words.

2. Decoding subsequent 30 words, which was adopted from the Skip-thought training code1 , gave us a reasonable good performance. More words for decoding didn't give us a significant performance gain, while it took longer to train.

3. Adding more layers into the decoder and enlarging the dimension of the convolutional layers indeed sightly improved the performance on the 3 downstream tasks, but as training efficiency is one of our main concerns, we decided it wasn't worth sacrificing training efficiency for the minor performance improvement.

4. Increasing the dimensionality of the RNN encoder improved the model performance, and the additional training time brought by it was less than that by adding more layers and enlarging the dimension of the convolutional layers in the CNN decoder. We reported results from both smallest and largest models in Table 2 .

4. EXPERIMENT SETTINGS:

The large corpus we used for unsupervised training is the BookCorpus dataset Zhu et al. (2015) , which contains 74 million sentences from 7000 books in total. For stable training, we use ADAM (Kingma & Ba, 2014) algorithm for optimization, and gradient clipping (Pascanu et al., 2013) when the norm of gradient exceeds a certain value. Since we didn't find any significant difference between word2vec and GloVe as initialization in terms of the performance, we stick to using the word vectors from word2vec to initialize the word embedding layer in our models.

The vocabulary for unsupervised training contains the top 20k most frequent words in BookCorpus. In order to generalize the model trained with a relatively small, fixed vocabulary to the much larger set of all possible English words, Kiros et al. (2015) proposed a word expansion method that learns a linear projection from the pretrained word embeddings word2vec to the learned RNN word embeddings. Thus, the model benefits from the generalization ability of the pretrained word embeddings.

The downstream tasks for evaluation include semantic relatedness (SICK) (Marelli et al., 2014) , paraphrase detection (MSRP) (Dolan et al., 2004) , question-type classification (TREC) (Li & Roth, 2002) , and 5 benchmark sentiment and subjective datasets, which includes movie review sentiment (MR, SST) (Pang & Lee, 2005; Socher et al., 2013) , customer product reviews (CR) (Hu & Liu, 2004) , subjectivity/objectivity classification (SUBJ) (Pang & Lee, 2004) , opinion polarity (MPQA) (Wiebe et al., 2005) , and semantic textual similarity (STS14) (Agirre et al., 2014) . After unsupervised training on the BookCorpus dataset, we fix the parameters in the encoder, and apply it as a sentence representation extractor on the 10 tasks.

In order to compare the effect of different corpora, we also trained 2 models on Amazon Book Review dataset (without ratings) which is the largest subset of the Amazon Review dataset (McAuley et al., 2015) with 142 million sentences after tokenization, about twice as large as BookCorpus.

Both training and evaluation of our models were conducted in PyTorch2 , and we used SentEval3 provided by Conneau et al. (2017) to evaluate the transferability of models with different settings. All the models were trained for the same number of iterations with the same batch size, and the performance was measured at the end of training for each of the models.

5. RELATED WORK AND COMPARISON:

Table 2 presented the results on 10 evaluation tasks of our proposed RNN-CNN models, and related work. "small RNN-CNN" refers to the model with the dimension of representation as 1200, and "large RNN-CNN" refers to that as 4800. The results of our model on SNLI can be found in Table 3 .

Our work was inspired by analyzing the Skip-thought model (Kiros et al., 2015) . Skip-thought model successfully applied this form of learning from the context information into unsupervised representation learning for sentences, in which the model learns to encode the current sentence and decode the surrounding 2 sentences, and then, Ba et al. (2016) augmented the LSTM with proposed layer-normalization (Skip-thought+LN), which improved the skip-thought model generally on all downstream tasks. Instead of applying RNNs in the model, Hill et al. (2016) proposed the FastSent model which only learns source and target word embeddings, and it is a generalization of CBOW (Mikolov et al., 2013) to sentence-level learning, and the composition function over word embeddings is a summation operation. Later on, Gan et al. (2017) applied a CNN as the encoder, which is called the CNN-LSTM model. The proposed composition model follows the idea of encoding the current sentence and predicting itself and the next sentence; the proposed hierarchical model leverages the context information from both sentence-level and paragraph-level, while learning to encode the current sentence and predict the next one, the model has another RNN to process the sentence representation one at a time at paragraph-level.

computational efficient and have clear intuition. However, the performance on the downstream tasks is still worse than encoder-decoder models.

Proposed by Radford et al. (2017) , BYTE m-LSTM model uses a multiplicative LSTM unit (Krause et al., 2016) to learn a language model on Amazon Review data McAuley et al. (2015) . The model works reasonably well on the downstream tasks, since the RNNs are able to produce a distributed representation for the given left-context information, such as a sentence or a document. In our experiment, we also trained our RNN-CNN model on the Amazon Book review, which is the largest subset of the Amazon review dataset, and indeed, we had a performance gain on all single-sentence classification tasks. The performance gain in our experiment and also in BYTE m-LSTM was brought by the matching between the corpus domain and the domain of downstream tasks, and it raises 2 questions 1) which corpus is good for learning sentence representations, and 2) whether the downstream tasks are comprehensive to cover sufficient aspects of a sentence.

Previously mentioned models are learned from ordered sentences, but unordered sentences can also be used for learning representations of sentences. Supervised transfer learning is also promising when we are able to get large enough labeled data. Conneau et al. (2017) applied a bi-directional LSTM as the sentence encoder with multiple fully-connected layers to deal with both SNLI (Bowman et al., 2015) , and MultiNLI (Williams et al., 2017) . The trained model demonstrates a very impressive transferability on all downstream tasks, including both supervised and unsupervised. The direct and discriminative training signal pushes the RNN encoder to focus on the semantics of a given sentence, which learns to a boost in performance, and beats all other methods. Our RNN-CNN model trained on Amazon Book Review data has better results on supervised classification tasks than BiLSTM-Max does, while the performance of ours on semantic relatedness tasks is inferior to BiLSTM-Max. We argue that labeling a large amount of training data is time-consuming and costly; unsupervised learning could potentially provide a great initial point for human labeling making it less costly and more efficient.

6. CONCLUSION:

Inspired by learning to exploit the contextual information present in adjacent sentences, we proposed an asymmetric encoder-decoder model with a suite of techniques for improving context-based unsupervised sentence representation learning. Since we believe that a simple model will be faster in training and easier to analyze, we opt to use simple techniques in our proposed model, including 1) an RNN as the encoder, and a predict-all-words CNN as the decoder, 2) learning by inferring next contiguous words, 3) mean+max pooling, and 4) tying word vectors with word prediction. With thorough discussion and extensive evaluation, we justify our decision making for each component in our RNN-CNN model. In terms of the performance and the efficiency of training, we justify that our model is a fast and simple algorithm for learning generic sentence representations from unlabeled corpora. Further research will focus on how to maximize the utility of the context information, and how to design simple architectures to best make use of it.

:

https://github.com/ryankiros/skip-thoughts/blob/master/training/train.py

http://pytorch.org/

https://github.com/facebookresearch/SentEval

Figures and Tables:

Figure 1: Our proposed model is composed of an RNN encoder, and a CNN decoder. During training, a batch of sentences are sent to the model, and the RNN encoder computes a vector representation for each of sentences; then the CNN decoder needs to reconstruct the paired target sequence, which contains 30 contiguous words right after the input sentence, given the vector representation. 300 is the dimension of word vectors. D is the dimension of sentence representation, and it varies along with the change of the RNN encoder size. (Better view in color.)



Table content: <table><tr><td>Decoder</td><td colspan="2">SICK-r SICK-E</td><td>STS14</td><td>MSRP (Acc/F1)</td><td colspan="2">SST TREC</td></tr><tr><td/><td/><td colspan="3">auto-regressive RNN as decoder</td><td/><td/></tr><tr><td>Baseline</td><td>0.8530</td><td>82.6</td><td>0.51/0.50</td><td>74.1 / 81.7</td><td>82.5</td><td>88.2</td></tr><tr><td>Always Sampling</td><td>0.8576</td><td>83.2</td><td>0.55/0.53</td><td>74.7 / 81.3</td><td>80.6</td><td>87.0</td></tr><tr><td>Uniform Sampling</td><td>0.8559</td><td>82.9</td><td>0.54/0.53</td><td>74.0 / 81.8</td><td>81.0</td><td>87.4</td></tr><tr><td/><td/><td colspan="3">auto-regressive CNN as decoder</td><td/><td/></tr><tr><td>Baseline</td><td>0.8510</td><td>82.8</td><td>0.49/0.48</td><td>74.7 / 82.8</td><td>81.4</td><td>82.6</td></tr><tr><td>Always Sampling</td><td>0.8535</td><td>83.3</td><td>0.53/0.52</td><td>75.0 / 81.7</td><td>81.4</td><td>87.6</td></tr><tr><td>Uniform Sampling</td><td>0.8568</td><td>83.4</td><td>0.56/0.54</td><td>74.7 / 81.4</td><td>83.0</td><td>88.4</td></tr><tr><td/><td colspan="4">predict-all-words RNN as decoder</td><td/><td/></tr><tr><td>RNN</td><td>0.8508</td><td>82.8</td><td>0.58/0.55</td><td>74.2 / 82.8</td><td>81.6</td><td>88.8</td></tr><tr><td/><td colspan="4">predict-all-words CNN as decoder</td><td/><td/></tr><tr><td>CNN</td><td>0.8530</td><td>82.6</td><td>0.58/0.56</td><td>75.6 / 82.9</td><td>82.8</td><td>89.2</td></tr><tr><td>CNN-Max</td><td>0.8465</td><td>82.6</td><td>0.50/0.47</td><td>73.3 / 81.5</td><td>79.1</td><td>82.2</td></tr><tr><td/><td/><td colspan="3">Double-sized RNN Encoder</td><td/><td/></tr><tr><td>CNN</td><td>0.8631</td><td>83.9</td><td>0.58/0.55</td><td>74.7 / 83.1</td><td>83.4</td><td>90.2</td></tr><tr><td>CNN-Max</td><td>0.8485</td><td>83.2</td><td>0.47/0.44</td><td>72.9 / 80.8</td><td>82.2</td><td>86.6</td></tr></table>

ParagraphVec (Le & Mikolov, 2014) learns a fixed-dimension vector for each sentence by predicting the words within the given sentence. However, after training, the representation for a new sentence is hard to derive, since it requires optimizing the sentence representation towards an objective. SDAE(Hill et al., 2016) learns the sentence representations with a denoising auto-encoder model. The noise was added in the encoder by replacing words with a fixed token, and swapping two words, both with a specific probability. Our proposed RNN-CNN model trains faster than SDAE does, since the CNN decoder runs faster than the RNN decoder in SDAE, and since we utilized the sentence-level continuity as a supervision which SDAE doesn't, our model largely performs better than SDAE.

Table content: <table><tr><td>Model</td><td>SNLI (Acc %)</td></tr><tr><td colspan="2">Unsupervised Transfer Learning</td></tr><tr><td>Skip-thought (Vendrov et al.)</td><td>81.5</td></tr><tr><td>large RNN-CNN BookCorpus</td><td>81.7</td></tr><tr><td>large RNN-CNN Amazon</td><td>81.5</td></tr><tr><td colspan="2">Supervised Training</td></tr><tr><td>ESIM (Chen et al.)</td><td>86.7</td></tr><tr><td>DIIN (Gong et al.)</td><td>88.9</td></tr></table>

We implemented the same classifier as mentioned inVendrov et al. (2015) on top of the features computed by our model. Our proposed RNN-CNN model gets similar result on SNLI as Skip-thought, but with much less training time.

Table content: <table/>

Peer review (iclr2018):
 -- updates to review: -- Thanks for trying to respond to my comments. I find the new results very interesting and fill in some empirical gaps that I think were worth investigating. I*m now more confident that this paper is worth publishing and I increased my rating from 6 to 7. I admit that this is a pretty NLP-specific paper, but to the extent that ICLR has core NLP papers (I think it does have some), I think the paper is a reasonable fit for ICLR. It might feel more at home at a *ACL conference though. -- original review is below: -- This paper is about modifications to the skip-thought framework for learning sentence embeddings. The results show performance comparable to or better than skip-thought while decreasing training time. I think the overall approach makes sense: use an RNN encoder because we know it works well, but improve training efficiency by changing the decoder to a combination of feed-forward and convolutional layers. I think it may be the case that this works well because the decoder is not auto-regressive but merely predicts each word independently. This is possible because the decoder will not be used after training. So all the words can be predicted all at once with a fixed maximum sentence length. In typical encoder-decoder applications, the decoder is used at test time to get predictions, so it is natural to make it auto-regressive. But in this case, the decoder is thrown away after training, so it makes more sense to make the decoder non-auto-regressive. I think this point should be made in the paper. Also, I think it*s worth noting that an RNN decoder could be used in a non-auto-regressive architecture as well. That is, the sentence encoding could be mapped to a sequence of length 30 as is done with the CNN decoder currently; then a (multi-layer) BiLSTM could be run over that sequence, and then a softmax classifier can be attached to each hidden vector to predict the word at that position. It would be interesting to compare that BiLSTM decoder with the proposed CNN decoder and also to compare it to a skip-thought-style auto-regressive RNN decoder. This would let us understand whether the benefit is coming more from the non-auto-regressive nature of the decoder or from the CNN vs RNN differences. That is, it would make sense to factor the decision of decoder design along multiple axes. One axis could be auto-regressive vs predict-all-words. Another axis could be using a CNN over the sequence of word positions or an RNN over the sequence of word positions. For auto-regressive models, another axis could be train using previous ground-truth word vs train using previous predicted word. Skip-thought corresponds to an auto-regressive RNN (using the previous ground-truth word IIRC). The proposed decoder is a predict-all-words CNN. It would be natural to also experiment with an auto-regressive CNN and a predict-all-words RNN (like what I described in the paragraph above). The paper is choosing a single point in the space and referring to it as a "CNN decoder" whereas there are many possible architectures that can be described this way and I think it would strengthen the paper to increase the precision in discussing the architecture and possible alternatives. Overall, I think the architectural choices and results are strong enough to merit publication. Adding any of the above empirical comparisons would further strengthen the paper. However, I did have quibbles with some of the exposition and some of the claims made throughout the paper. They are detailed below: Sec. 2: In the "Decoder" paragraph: please add more details about how the words are predicted. Are there final softmax layers that provide distributions over output words? I couldn*t find this detail in the paper. What loss is minimized during training? Is it the sum of log losses over all words being predicted? Sec. 3: Section 3 does not add much to the paper. The motivations there are mostly suggestive rather than evidence-based. Section 3 could be condensed by about 80% or so without losing much information. Overall, the paper has more than 10 pages of content, and the use of 2 extra pages beyond the desired submission length of 8 should be better justified. I would recommend adding a few more details to Section 2 and removing most of Section 3. I*ll mention below some problematic passages in Section 3 that should be removed. Sec. 3.2: "...this same constraint (if using RNN as the decoder) could be an inappropriate constraint in the decoding process." What is the justification or evidence for this claim? I think the claim should be supported by an argument or some evidence or else should be removed. If the authors intend the subsequent paragraphs to justify the claim, then see my next comments. Sec. 3.2: "The existence of the ground-truth current word embedding potentially decreases the tendency for the decoder to exploit other information from the sentence representation." But this is not necessarily an inherent limitation of RNN decoders since it could be addressed by using the embedding of the previously-predicted word rather than the ground-truth word. This is a standard technique in sequence-to-sequence learning; cf. scheduled sampling (Bengio et al., 2015). Sec. 3.2: "Although the word order information is implicitly encoded in the CNN decoder, it is not emphasized as it is explicitly in the RNN decoder. The CNN decoder cares about the quality of generated sequences globally instead of the quality of the next generated word. Relaxing the emphasis on the next word, may help the CNN decoder model to explore the contribution of context in a larger space." Again, I don*t see any evidence or justification for these arguments. Also see my discussion above about decoder variations; these are not properties of RNNs vs CNNs but rather properties of auto-regressive vs predict-all-words decoders. Sec. 5.2-5.3: There are a few high-level decisions being tuned on the test sets for some of the tasks, e.g., the length of target sequences in Section 5.2 and the number of layers and channel size in Section 5.3. Sec. 5.4: When trying to explain why an RNN encoder works better than a CNN encoder, the paper includes the following: "We stated above that, in our belief, explicit usage of the word order information will augment the transferability of the encoder, and constrain the search space of the parameters in the encoder. The results match our belief." I don*t think these beliefs are concrete enough to be upheld or contradicted. Both encoders explicitly use word order information. Can you provide some formal or theoretical statement about how the two encoders treat word order differently? I fear that it*s only impressions and suppositions that lead to this difference, rather than necessarily something formal. Sec. 5.4: In Table 1, it is unclear why the "future predictor" model is the one selected to be reported from Gan et al (2017). Gan et al has many settings and the "future predictor" setting is the worst. An explanation is needed for this choice. Sec. 6.1: In the "BYTE m-LSTM" paragraph: "Our large RNN-CNN model trained on Amazon Book Review (the largest subset of Amazon Review) performs on par with BYTE m-LSTM model, and ours works better than theirs on semantic relatedness and entailment tasks." I*m not sure this "on par" assessment is warranted by the results in Table 2. BYTE m-LSTM is better on MR by 1.6 points and better on CR by 4.7 points. The authors* method is better on SUBJ by 0.7 and better on MPQA by 0.5. So on sentiment tasks, BYTE m-LSTM is clearly better, and on the other tasks the RNN-CNN is typically better, especially on SICK-r. More minor things are below: Sec. 1: The paper contains this: "The idea of learning from the context information was first successfully applied to vector representation learning for words in Mikolov et al. (2013b)" I don*t think this is accurate. When restricting attention to neural network methods, it would be more correct to give credit to Collobert et al. (2011). But moving beyond neural methods, there were decades of previous work in using context information (counts of context words) to produce vector representations of words. typo: "which d reduces" --> "which reduces" Sec. 2: The notation in the text doesn*t match that in Figure 1: w_i^1 vs. w_1 and h_i^1 vs h_1. Instead of writing "non-parametric composition function", describe it as "parameter-free". "Non-parametric" means that the number of parameters grows with the data, not that there are no parameters. In the "Representation" paragraph: how do you compute a max over vectors? Is it a separate max for each dimension? This is not clear from the notation used. Sec. 3.1: inappropriate word choice: the use of "great" in "a great and efficient encoding model" Sec. 3.2: inappropriate word choice: the use of "unveiled" in "is still to be unveiled" Sec. 3.4: Tying input and output embeddings can be justified with a single sentence and the relevant citations (which are present here). There is no need for speculation about what may be going on, e.g.: "the model learns to explore the non-linear compositionality of the input words and the uncertain contribution of the target words in the same space". Sec. 4: I think STS14 should be defined and cited where the other tasks are described. Sec. 5.3: typo in Figure 2 caption: "and and" Sec. 6.1: In the "Skip-thought" paragraph: inappropriate word choice: "kindly" The description that says "we cut off a branch for decoding" is not clear to me. What is a "branch for decoding" in this context? Please modify it to make it more clear. References: Bengio S, Vinyals O, Jaitly N, Shazeer N. Scheduled sampling for sequence prediction with recurrent neural networks. NIPS 2015. Collobert R, Weston J, Bottou L, Karlen M, Kavukcuoglu K, Kuksa P. Natural language processing (almost) from scratch. Journal of Machine Learning Research 2011.