Title: EXPLORING ASYMMETRIC ENCODER-DECODER STRUCTURE FOR CONTEXT-BASED SENTENCE REPRESENTATION LEARNING

Abstract: Context information plays an important role in human language understanding, and it is also useful for machines to learn vector representations of language. In this paper, we explore an asymmetric encoder-decoder structure for unsupervised context-based sentence representation learning. As a result, we build an encoderdecoder architecture with an RNN encoder and a CNN decoder, and we show that neither an autoregressive decoder nor an RNN decoder is required. We further combine a suite of effective designs to significantly improve model efficiency while also achieving better performance. Our model is trained on two different large unlabeled corpora, and in both cases transferability is evaluated on a set of downstream language understanding tasks. We empirically show that our model is simple and fast while producing rich sentence representations that excel in downstream tasks. Recently, Tang et al. (2017) proposed the Skip-thought Neighbor model, which only decodes the next

1. INTRODUCTION:

Learning distributed representations of sentences is an important and hard topic in both the deep learning and natural language processing communities, since it requires machines to encode a sentence with rich language content into a fixed-dimension vector filled with continuous values. We are interested in learning to build a distributed sentence encoder in an unsupervised fashion by exploiting the structure and relationship in a large unlabeled corpus. Since humans interpret sentences by composing from the meanings of the words, we decompose the task of learning a sentence encoder into two essential components: learning distributed word representations, and learning how to compose a sentence representation from the representations of words in the given sentence.

Numerous studies in human language processing have claimed that the context in which words and sentences are understood plays an important role in human language understanding (Altmann & Mirkovic, 2009; Binder & Desai, 2011) . The idea of learning from the context information (Turney & Pantel, 2010) was recently successfully applied to vector representation learning for words in Mikolov et al. (2013) ; Pennington et al. (2014) . Collobert et al. (2011) proposed a unified framework for learning language representation from the unlabeled data, and it is able to generalize to various NLP tasks. Inspired by the prior work on incorporating context information into representation learning, Kiros et al. (2015) proposed the Skipthought model, which is an encoder-decoder model for unsupervised sentence representation learning. The paper exploits the semantic similarity within a tuple of adjacent sentences as supervision, and successfully built a generic, distributed sentence encoder. Rather than applying the conventional autoencoder model, the skip-thought model tries to reconstruct the surrounding 2 sentences instead of the input sentence. The learned sentence representation encoder outperforms previous unsupervised pretrained models on the evaluation tasks with no finetuning, and the results are comparable to the models which were trained directly on the datasets in a supervised fashion.

The usage of 2 independent decoders in Skip-thought model matches our intuition that, given the current sentence, inferring the previous sentence and inferring the next one should be different. sentence, and the performance on the downstream tasks is similar to that of their implementation of the Skip-thought model.

In this paper, we follow the idea in the Skip-thought Neighbor model, which exploits the subsequent context information for learning representation, and aim to bring asymmetry into structure design as well. Our proposed model has an asymmetric encoder-decoder structure, which keeps an RNN as the encoder and has a CNN as the decoder, and will be referred to as an "RNN-CNN" model. The key components of our model design can be summarized as:

1. a bidirectional RNN encodes the input sentence, and a CNN decodes all words in the paired target sequence at once, which speeds up the training process;

2. the supervision for training comes from inferring the next contiguous words given the current sentence, which helps the model to learn from the context in an unsupervised fashion;

3. the mean+max pooling captures complex interactions among words, which augments the transferability of the proposed model; 4. tying word embeddings in the encoder with the word prediction layer in the decoder constrains the input and output space to be the same, which also reduces number of parameters and regularizes the model.

We demonstrate the transferability of our model by evaluation on various downstream tasks, and the performance shows that our model improves both results and training efficiency.

2. RNN-CNN MODEL:

Our model is highly asymmetric in terms of both training pairs and model structure. Specifically, our model has an RNN as the encoder, and a CNN as the decoder. During training, the encoder takes the ith sentence s i as input, and then generates a fixed-dimension vector z i as the sentence representation; the decoder is applied to reconstruct the next sentence or the subsequent few contiguous words t i . The difference of the generated sequence and the target sequence is measured by cross-entropy loss. An illustration is in Figure 1 . (For simplicity, we omit the subscript i in the section.)

Encoder: The encoder is a bi-directional Gated Recurrent Unit (GRU) (Chung et al., 2014) . We experimented with both Long-short Term Memory (LSTM, Hochreiter & Schmidhuber (1997) ) and GRU. Since LSTM didn't give us significant performance boost, and generally GRU runs faster than LSTM, in our experiments, we stick to using GRU in the encoder. Suppose that a sentence s contains M words, which are w 1 , w 2 , ..., w M , and they are transformed by an embedding matrix E to word vectors. The bi-directional GRU will take one word vector at a time, and run in both forward and backward direction; both sets of hidden states are concatenated to form the hidden state matrix

H = [h 1 , h 2 , ..., h M ] ∈ R D×M , where d is the dimension of the representations h m = ← - h m ; -→ h m

(∀m ∈ {1, 2, ..., M }).

Representation: We aim to provide a model with faster training speed with better transferability than existing algorithms, thus we choose to apply a parameter-free composition function, which is a concatenation of the outputs from a global mean pooling over time and a global max pooling over time, on the computed sequence of hidden states. The composition function can be represented as

EQUATION

where max H d• is the max operation on the d-th row of the matrix H, which outputs a scalar. Thus the representation z has a dimension of 2d.

Decoder: The decoder is a 3-layer CNN to reconstruct the paired target sequence t, which needs to expand z from length 1 to the length of t. Intuitively, the decoder could be a stack of deconvolution layers. For fast training speed, we optimized the architecture to make it plausible to use fullyconnected layers and convolution layers in the decoder, since generally, convolution layers run faster than deconvolution layers in modern deep learning frameworks.

Suppose that the target sequence t has N words, the first layer of deconvolution will expand z, which could be considered as a sequence with length 1, into a feature map with length N . It can be easily implemented as a concatenation of outputs from N linear transformations in parallel. Then the second and third layer are 1D-convolution layers with kernel size 3 and 1, respectively. The output feature map

V = [v 1 , v 2 , ..., v N ],

where v ∈ R e , and e is dimension of the word vectors.

Note that our decoder is not an autoregressive model, and it brings us high training efficiency. We will discuss the reason of choosing this decoder which we call a predict-all-words CNN decoder.

Objective: A softmax layer is applied after the decoder to produce a probability distribution over words at each position, softmax(Ev n ), and the training objective is to minimize the sum of the negative log-likelihood over all positions in the target sequence t:

EQUATION

The loss function L is summed over all sentences in the training corpus.

3. ARCHITECTURE DESIGN:

We follow the idea of an encoder-decoder model with using the context information for learning sentence representations in an unsupervised fashion. Since the decoder won't be used after training, and the quality of the generated sequences is not our main focus, it is important to study the design of the decoder. Generally, a fast training algorithm is preferred, thus proposing a new decoder with high training efficiency and also strong transferability is crucial for an encoder-decoder model.

3.1. CNN AS THE DECODER:

Our design of the decoder is basically a 3-layer ConvNet, and it predicts all words in the next sequence all at once. In contrast, existing work, such as Skip-thought Kiros et al. (2015) , and CNN-LSTM Gan et al. (2017) ,use autoregressive RNNs as the decoders.

An autoregressive model is good at generating sequences with high quality, such as language and speech. However, an autoregressive decoder seems to be unnecessary in an encoder-decoder model for learning sentence representations, since it won't be used after training, and it runs quite slow during training. Therefore, we conducted experiments to test the necessity of using an autoregressive decoder in learning sentence representations, and we had 2 findings.

Finding I: It is not necessary to input the correct words into an autoregressive decoder in terms of learning good sentence representations. The models here all have a bi-directional GRU as the encoder (dimensionality 300 in each direction). The default way of producing the representation is a concatenation of outputs from a global mean-pooling and a global max-pooling, while "•-Max" refers to the model with only global maxpooling. Bold numbers are the best results among all presented models. We found that 1) inputting correct words to an autoregressive decoder is not necessary; 2) predict-all-words decoders work roughly the same as autoregressive decoders; 3) mean+max pooling provides stronger transferability than the max-pooling alone does. The table supports our choice of the predict-all-words CNN decoder and the way of producing vector representations from the bi-directional RNN encoder.

The experimental design was inspired by Bengio et al. (2015) . The model we designed for the experiment has a bi-directional GRU as the encoder, and an autoregressive decoder, including both RNN and CNN. We started by analyzing the effect of different sampling strategies of the input words on learning an auto-regressive decoder.

We compared 3 autoregressive decoding settings: 1) using ground-truth words (Baseline), 2) using previously predicted words (Always Sampling), and 3) using uniformly sampled words from the dictionary (Uniform Sampling). The 3 decoding settings were named by Bengio et al. (2015) . The results are presented in the Table 1 .

Generally, the three different decoding settings didn't make much of a difference in terms of the performance on selected downstream tasks, with RNN or CNN as the decoder. The results tell us that, in terms of learning good sentence representations, the autoregressive decoder doesn't require the correct ground-truth words as the inputs.

Finding II: The model with an autoregressive decoder works roughly the same as the model with a predict-all-words decoder.

With Finding I, we noticed that the correct ground-truth input words to the autoregressive decoder is not necessary in terms of learning sentence representations. Therefore, it makes sense to test whether we need an autoregressive model at all.

In our model, the CNN decoder predicts all words at once during training, which is different from autoregressive decoders, and we call it a predict-all-words CNN decoder. We want to compare the performance of the predict-all-words decoders and that of the autoregressive decoders separate from the RNN/CNN distinction, thus we designed a predict-all-words CNN decoder and RNN decoder.

The predict-all-words CNN decoder is described in Section 2, which is a stack of 3 convolutional layers, and all words are predicted once at the output of the decoder. The predict-all-words RNN decoder is built based on our CNN decoder. To keep the number of parameters roughly the same, we replaced the last 2 convolutional layers with a bidirectional GRU.

The results are also presented in the Table 1 . The performance of the predict-all-words RNN decoder does not significantly differ from that of any one of the autoregressive RNN decoders, and the same observation was observed in CNN decoders.

These two findings actually support our choice of using a predict-all-words CNN as the decoder, and it brings the model higher training efficiency and strong transferability.

3.2. MEAN+MAX POOLING:

Since the encoder is a bi-directional RNN in our model, we have multiple ways to select/compute on the generated hidden states to produce a sentence representation. In Skip-thought (Kiros et al., 2015) and SDAE (Hill et al., 2016) , only the hidden state at the last time step produced by the RNN encoder is regarded as the vector representation for a given sentence, which may not be the most expressive vector for representing the input sentence.

We followed the idea proposed in Chen et al. (2016) . They built a model for supervised SNLI task (Bowman et al., 2015) that concatenates the outputs from a global mean pooling and a global max pooling to serve as a sentence representation, and showed a performance boost on the SNLI dataset. Also, Conneau et al. (2017) found that the model with global max pooling function has stronger transferability than the model with a global mean pooling function after supervised training on SNLI.

In our proposed RNN-CNN model, we empirically show that the mean+max pooling provides stronger transferability than the max pooling does, and the results are presented in Table 1 . The concatenation of a mean-pooling and a max pooling function is actually a parameter-free composition function, and the computation load is negligible compared to heavy matrix multiplications. Also, the non-linearity of the max pooling function augments the mean pooling function for building a representation that captures a more complex composition of the syntactic information.

3.3. TYING WORD EMBEDDINGS AND WORD PREDICTION LAYER:

We choose to share the parameters in the word embedding layer in RNN encoder and the word prediction layer in CNN decoder. The tying was proposed in both Press & Wolf (2017) and Inan et al. (2016) , and it generally helps to learn a better language model. In our model, the tying also drastically reduces the number of parameters, which could prevent overfitting.

Furthermore, we initialize the word embeddings with pretrained word vectors, such as word2vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014) , since it has been shown that these pretrained word vectors can serve as good initialization for deep learning models, and more likely lead to better results than random samples from a uniform distribution.

3.4. STUDY OF THE HYPERPARAMETERS IN OUR MODEL DESIGN:

We studied hyperparameters in our model design based on 3 out of 10 downstream tasks, including SICK-r, SICK-E (Marelli et al., 2014) , and STS14 (Agirre et al., 2014) . The first model we created, which is reported in Section 2, is a decent design, and the following variations didn't give us much performance change except small improvements with increasing the dimensionality of the encoder. However, we think it is worth mentioning the effect of hyperparameters in our model design. We present the Table in the supplementary material and we summarize it as follows:

1. Decoding the next sentence worked similarly as decoding the subsequent contiguous words.

2. Decoding subsequent 30 words, which was adopted from the Skip-thought training code1 , gave us a reasonable good performance. More words for decoding didn't give us a significant performance gain, while it took longer to train.

3. Adding more layers into the decoder and enlarging the dimension of the convolutional layers indeed sightly improved the performance on the 3 downstream tasks, but as training efficiency is one of our main concerns, we decided it wasn't worth sacrificing training efficiency for the minor performance improvement.

4. Increasing the dimensionality of the RNN encoder improved the model performance, and the additional training time brought by it was less than that by adding more layers and enlarging the dimension of the convolutional layers in the CNN decoder. We reported results from both smallest and largest models in Table 2 .

4. EXPERIMENT SETTINGS:

The large corpus we used for unsupervised training is the BookCorpus dataset Zhu et al. (2015) , which contains 74 million sentences from 7000 books in total. For stable training, we use ADAM (Kingma & Ba, 2014) algorithm for optimization, and gradient clipping (Pascanu et al., 2013) when the norm of gradient exceeds a certain value. Since we didn't find any significant difference between word2vec and GloVe as initialization in terms of the performance, we stick to using the word vectors from word2vec to initialize the word embedding layer in our models.

The vocabulary for unsupervised training contains the top 20k most frequent words in BookCorpus. In order to generalize the model trained with a relatively small, fixed vocabulary to the much larger set of all possible English words, Kiros et al. (2015) proposed a word expansion method that learns a linear projection from the pretrained word embeddings word2vec to the learned RNN word embeddings. Thus, the model benefits from the generalization ability of the pretrained word embeddings.

The downstream tasks for evaluation include semantic relatedness (SICK) (Marelli et al., 2014) , paraphrase detection (MSRP) (Dolan et al., 2004) , question-type classification (TREC) (Li & Roth, 2002) , and 5 benchmark sentiment and subjective datasets, which includes movie review sentiment (MR, SST) (Pang & Lee, 2005; Socher et al., 2013) , customer product reviews (CR) (Hu & Liu, 2004) , subjectivity/objectivity classification (SUBJ) (Pang & Lee, 2004) , opinion polarity (MPQA) (Wiebe et al., 2005) , and semantic textual similarity (STS14) (Agirre et al., 2014) . After unsupervised training on the BookCorpus dataset, we fix the parameters in the encoder, and apply it as a sentence representation extractor on the 10 tasks.

In order to compare the effect of different corpora, we also trained 2 models on Amazon Book Review dataset (without ratings) which is the largest subset of the Amazon Review dataset (McAuley et al., 2015) with 142 million sentences after tokenization, about twice as large as BookCorpus.

Both training and evaluation of our models were conducted in PyTorch2 , and we used SentEval3 provided by Conneau et al. (2017) to evaluate the transferability of models with different settings. All the models were trained for the same number of iterations with the same batch size, and the performance was measured at the end of training for each of the models.

5. RELATED WORK AND COMPARISON:

Table 2 presented the results on 10 evaluation tasks of our proposed RNN-CNN models, and related work. "small RNN-CNN" refers to the model with the dimension of representation as 1200, and "large RNN-CNN" refers to that as 4800. The results of our model on SNLI can be found in Table 3 .

Our work was inspired by analyzing the Skip-thought model (Kiros et al., 2015) . Skip-thought model successfully applied this form of learning from the context information into unsupervised representation learning for sentences, in which the model learns to encode the current sentence and decode the surrounding 2 sentences, and then, Ba et al. (2016) augmented the LSTM with proposed layer-normalization (Skip-thought+LN), which improved the skip-thought model generally on all downstream tasks. Instead of applying RNNs in the model, Hill et al. (2016) proposed the FastSent model which only learns source and target word embeddings, and it is a generalization of CBOW (Mikolov et al., 2013) to sentence-level learning, and the composition function over word embeddings is a summation operation. Later on, Gan et al. (2017) applied a CNN as the encoder, which is called the CNN-LSTM model. The proposed composition model follows the idea of encoding the current sentence and predicting itself and the next sentence; the proposed hierarchical model leverages the context information from both sentence-level and paragraph-level, while learning to encode the current sentence and predict the next one, the model has another RNN to process the sentence representation one at a time at paragraph-level.

computational efficient and have clear intuition. However, the performance on the downstream tasks is still worse than encoder-decoder models.

Proposed by Radford et al. (2017) , BYTE m-LSTM model uses a multiplicative LSTM unit (Krause et al., 2016) to learn a language model on Amazon Review data McAuley et al. (2015) . The model works reasonably well on the downstream tasks, since the RNNs are able to produce a distributed representation for the given left-context information, such as a sentence or a document. In our experiment, we also trained our RNN-CNN model on the Amazon Book review, which is the largest subset of the Amazon review dataset, and indeed, we had a performance gain on all single-sentence classification tasks. The performance gain in our experiment and also in BYTE m-LSTM was brought by the matching between the corpus domain and the domain of downstream tasks, and it raises 2 questions 1) which corpus is good for learning sentence representations, and 2) whether the downstream tasks are comprehensive to cover sufficient aspects of a sentence.

Previously mentioned models are learned from ordered sentences, but unordered sentences can also be used for learning representations of sentences. Supervised transfer learning is also promising when we are able to get large enough labeled data. Conneau et al. (2017) applied a bi-directional LSTM as the sentence encoder with multiple fully-connected layers to deal with both SNLI (Bowman et al., 2015) , and MultiNLI (Williams et al., 2017) . The trained model demonstrates a very impressive transferability on all downstream tasks, including both supervised and unsupervised. The direct and discriminative training signal pushes the RNN encoder to focus on the semantics of a given sentence, which learns to a boost in performance, and beats all other methods. Our RNN-CNN model trained on Amazon Book Review data has better results on supervised classification tasks than BiLSTM-Max does, while the performance of ours on semantic relatedness tasks is inferior to BiLSTM-Max. We argue that labeling a large amount of training data is time-consuming and costly; unsupervised learning could potentially provide a great initial point for human labeling making it less costly and more efficient.

6. CONCLUSION:

Inspired by learning to exploit the contextual information present in adjacent sentences, we proposed an asymmetric encoder-decoder model with a suite of techniques for improving context-based unsupervised sentence representation learning. Since we believe that a simple model will be faster in training and easier to analyze, we opt to use simple techniques in our proposed model, including 1) an RNN as the encoder, and a predict-all-words CNN as the decoder, 2) learning by inferring next contiguous words, 3) mean+max pooling, and 4) tying word vectors with word prediction. With thorough discussion and extensive evaluation, we justify our decision making for each component in our RNN-CNN model. In terms of the performance and the efficiency of training, we justify that our model is a fast and simple algorithm for learning generic sentence representations from unlabeled corpora. Further research will focus on how to maximize the utility of the context information, and how to design simple architectures to best make use of it.

:

https://github.com/ryankiros/skip-thoughts/blob/master/training/train.py

http://pytorch.org/

https://github.com/facebookresearch/SentEval

Figures and Tables:

Figure 1: Our proposed model is composed of an RNN encoder, and a CNN decoder. During training, a batch of sentences are sent to the model, and the RNN encoder computes a vector representation for each of sentences; then the CNN decoder needs to reconstruct the paired target sequence, which contains 30 contiguous words right after the input sentence, given the vector representation. 300 is the dimension of word vectors. D is the dimension of sentence representation, and it varies along with the change of the RNN encoder size. (Better view in color.)



Table content: <table><tr><td>Decoder</td><td colspan="2">SICK-r SICK-E</td><td>STS14</td><td>MSRP (Acc/F1)</td><td colspan="2">SST TREC</td></tr><tr><td/><td/><td colspan="3">auto-regressive RNN as decoder</td><td/><td/></tr><tr><td>Baseline</td><td>0.8530</td><td>82.6</td><td>0.51/0.50</td><td>74.1 / 81.7</td><td>82.5</td><td>88.2</td></tr><tr><td>Always Sampling</td><td>0.8576</td><td>83.2</td><td>0.55/0.53</td><td>74.7 / 81.3</td><td>80.6</td><td>87.0</td></tr><tr><td>Uniform Sampling</td><td>0.8559</td><td>82.9</td><td>0.54/0.53</td><td>74.0 / 81.8</td><td>81.0</td><td>87.4</td></tr><tr><td/><td/><td colspan="3">auto-regressive CNN as decoder</td><td/><td/></tr><tr><td>Baseline</td><td>0.8510</td><td>82.8</td><td>0.49/0.48</td><td>74.7 / 82.8</td><td>81.4</td><td>82.6</td></tr><tr><td>Always Sampling</td><td>0.8535</td><td>83.3</td><td>0.53/0.52</td><td>75.0 / 81.7</td><td>81.4</td><td>87.6</td></tr><tr><td>Uniform Sampling</td><td>0.8568</td><td>83.4</td><td>0.56/0.54</td><td>74.7 / 81.4</td><td>83.0</td><td>88.4</td></tr><tr><td/><td colspan="4">predict-all-words RNN as decoder</td><td/><td/></tr><tr><td>RNN</td><td>0.8508</td><td>82.8</td><td>0.58/0.55</td><td>74.2 / 82.8</td><td>81.6</td><td>88.8</td></tr><tr><td/><td colspan="4">predict-all-words CNN as decoder</td><td/><td/></tr><tr><td>CNN</td><td>0.8530</td><td>82.6</td><td>0.58/0.56</td><td>75.6 / 82.9</td><td>82.8</td><td>89.2</td></tr><tr><td>CNN-Max</td><td>0.8465</td><td>82.6</td><td>0.50/0.47</td><td>73.3 / 81.5</td><td>79.1</td><td>82.2</td></tr><tr><td/><td/><td colspan="3">Double-sized RNN Encoder</td><td/><td/></tr><tr><td>CNN</td><td>0.8631</td><td>83.9</td><td>0.58/0.55</td><td>74.7 / 83.1</td><td>83.4</td><td>90.2</td></tr><tr><td>CNN-Max</td><td>0.8485</td><td>83.2</td><td>0.47/0.44</td><td>72.9 / 80.8</td><td>82.2</td><td>86.6</td></tr></table>

ParagraphVec (Le & Mikolov, 2014) learns a fixed-dimension vector for each sentence by predicting the words within the given sentence. However, after training, the representation for a new sentence is hard to derive, since it requires optimizing the sentence representation towards an objective. SDAE(Hill et al., 2016) learns the sentence representations with a denoising auto-encoder model. The noise was added in the encoder by replacing words with a fixed token, and swapping two words, both with a specific probability. Our proposed RNN-CNN model trains faster than SDAE does, since the CNN decoder runs faster than the RNN decoder in SDAE, and since we utilized the sentence-level continuity as a supervision which SDAE doesn't, our model largely performs better than SDAE.

Table content: <table><tr><td>Model</td><td>SNLI (Acc %)</td></tr><tr><td colspan="2">Unsupervised Transfer Learning</td></tr><tr><td>Skip-thought (Vendrov et al.)</td><td>81.5</td></tr><tr><td>large RNN-CNN BookCorpus</td><td>81.7</td></tr><tr><td>large RNN-CNN Amazon</td><td>81.5</td></tr><tr><td colspan="2">Supervised Training</td></tr><tr><td>ESIM (Chen et al.)</td><td>86.7</td></tr><tr><td>DIIN (Gong et al.)</td><td>88.9</td></tr></table>

We implemented the same classifier as mentioned inVendrov et al. (2015) on top of the features computed by our model. Our proposed RNN-CNN model gets similar result on SNLI as Skip-thought, but with much less training time.

Table content: <table/>

Peer review (iclr2018):
 Update: I*m going to change my review to a 6 to acknowledge the substantial improvements you*ve made—I no longer fear that there are major errors in the paper, but this paper is still solidly borderline, and I*m not completely convinced that any new claim is true. The evidence presented for the main claim—that you can get by without an autoregressive decoder when pretraining encoders—is somewhat persuasive, but isn*t as unequivocal as I*d hope, and even if the claim is true, it is arguably too limited a claim for an ICLR main conference paper. As R1 says, a *ACL short paper would be more appropriate. The writing is also still unclear in places. ---- This paper presents a new RNN encoder–CNN decoder hybrid design for use in pretraining reusable sentence encoders on Kiros*s SkipThought objective. The task is interesting and important, and the results are generally good: The new model outperforms SkipThought, and all other prior models for training sentence encoders on unlabeled data. However, some of the design choices seem a bit odd, and I have a large number of minor concerns about the paper. I*d like to see the authors* replies and the other reviews before I can confidently endorse this paper as correct. Non-autoregressive decoding with a CNN strikes me as a somewhat ill-posed problem, even for in this case where you don*t actually use the decoder in the final application of your model. At each position, you*re training your model to predict a distribution over all words that could appear at the beginning/tenth position/twentieth position in sentences on some topic. I*d appreciate some more discussion of why this should or shouldn*t hurt performance. I*d be less concerned about this if the results supporting the use of the CNN decoder were a bit more conclusive: while they are better on average across your smaller experiments, your largest experiment (2400D) shows them roughly tied. Your paper opens with the line "Context information plays an important role in human language understanding." This sounds like it*s making an empirical claim that your paper doesn*t support, but it*s so vague that it*s hard to tell exactly what that claim is. Please clarify this or remove it. This sentence is quite inaccurate: "The idea of learning from the context information was first successfully applied to vector representation learning for words in Mikolov et al. (2013b) and learning from the occurrence of words also succeeded in Pennington et al. (2014)." Turney and Pantel 2010 ( https://www.jair.org/media/2934/live-2934-4846-jair.pdf ) offer a survey of the substantial prior work that existed at that time. The "Neighborhood Hypothesis" is given quite a lot of importance, given that it*s a fairly small empirical effect without any corresponding theory. The fact that it*s emphasized so heavily makes me suspect that I can guess the author of the paper. I*d tone down that part of your framing. I would appreciate some more analysis of which of the non-central tricks that you describe in section 3 help. For example, max pooling seems reasonable, but you report yourself that mean pooling generally works much better in prior work. Without an explicit experiment, it*s not clear why you*d add a mean pooling component. It seems misleading to claim that your CNN is modeled after AdaSent, as that model uses a number of layers that varies with the length of the sentence (and differs from yours in a few other less-important ways). Please correct or clarify. The use of “†” in table to denote models that predict the next sentence in a sequence doesn*t make sense. It should apply to all of your models if I understand correctly. Please clarify. You could do a better job at table placement and formatting. Table 3 is in the wrong section, for example. You write that: "Our proposed RNN-CNN model gets similar result on SNLI as Skip-thought, but with much less training time." This seems to be based on a comparison between your model run on your hardware and their model run on their (presumably older) hardware, and possibly also with their older version of CuDNN. If that*s right, you should tone down this claim or offer some more evidence.