Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: Hi, when extrapolation based augmentation is carried out, do the neighboring context vectors belong to different classes (this is not clear from the paper, it seems they belong to the same class)?  If they are in the same class extrapolation based augmentation feels more risky as it could push a class into a confusable class (e.g. digit 1 into digit 7)?
review_title: Is extrapolation based augmentation risky?

final_score: rating: null
review_content: The proposed method is interesting and the paper is well presented. Just some thoughts on the augmentation methods. Besides adding random noise, interpolation and extrapolation, one other way of improving model robustness is to randomly corrupt the representation. Wondering how that works within the proposed framework. 
review_title: Random corruption

final_score: rating: 7: Good paper, accept
review_content: The concept of data augmentation in the embedding space is very interesting. The method is well presented and also justified on different tasks such as spoken digits and image recognition etc.

One comments of the comparison is the use of a simple 2-layer MLP as the baseline model throughout all the tasks. It's not clear whether the gains maintain when a more complex baseline model is used. 

Another comment is that the augmented context vectors are used for classification, just wondering how does it compare to using the reconstructed inputs. And furthermore, as in Table 4, both input and feature space extrapolation improves the performance, whether these two are complementary or not? 
review_title: null

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this paper authors propose a novel data augmentation scheme where instead of augmenting the input data, they augment intermediate feature representations.  Sequence auto-encoder based features are considered, and random perturbation, feature interpolation, and extrapolation based augmentation are evaluated. On three sequence classification tasks and on MNIST and CIFAR-10, it is shown that augmentation in feature space, specifically extrapolation based augmentation, results in good accuracy gains w.r.t. authors baseline.

My main questions and suggestions for further strengthening the paper are:

a) The proposed data augmentation approach is applied to a learnt auto-encoder based feature space termed ‘context vector’ in the paper.  The context vectors are then augmented and used as input to train classification models. Have the authors considered applying their feature space augmentation idea directly to the classification model during training, and applying it to potentially many layers of the model?  Also, have the authors considered convolutional neural network (CNN) architectures as well for feature space augmentation?  CNNs are now the state-of-the-art in many image and sequence classification task, it would be very valuable to see the impact of the proposed approach in that model.

b) When interpolation or extrapolation based augmentation was being applied, did the authors also consider utilizing nearby samples from competing classes as well?  Especially in case of extrapolation based augmentation it will be interesting to check if the extrapolated features are closer to competing classes than original ones.

c) With random interpolation or nearest neighbor interpolation based augmentation the accuracy seems to degrade pretty consistently.  This is counter-intuitive.  Do the authors have explanation for why the accuracy degraded with interpolation based augmentation?

d) The results on MNIST and CIFAR-10 are inconclusive.  For instance the error rate on CIFAR-10 is well below 10% these days, so I think it is hard to draw conclusions based on error rates above 30%.  For MNIST it is surprising to see that data augmentation in the input space substantially degrades the accuracy (1.093% -> 1.477%).  As mentioned above, I think this will require extending the feature space augmentation idea to CNN based models.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: TDLR: The authors present a regularization method wherein they add noise to some representation space. The paper mainly applies the technique w/ sequence autoencoders (Dai et al., 2015) without the usage of attention (i.e., only using the context vector). Experimental results show improvement from author's baseline on some toy tasks.

=== Augmentation ===
The augmentation process is simple enough, take the seq2seq context vector and add noise/interpolate/extrapolate to it (Section 3.2). This reviewer is very curious whether this process will also work in non seq2seq applications. 

This reviewer would have liked to see comparison with dropout on the context vector.

=== Experiments ===
Since the authors are experimenting w/ seq2seq architectures, its a little bit disappointing they didn't compare it w/ Machine Translation (MT), where there are many published papers to compare to.

The authors did compare their method on several toy datasets (that are less commonly used in DL literature) and MNIST/CIFAR. The authors show improvement over their own baselines on several toy datasets. The improvement on MNIST/CIFAR over the author's baseline seems marginal at best. The author also didn't cite/compare to the baseline published by Dai et al., 2015 for CIFAR -- here they have a much better LSTM baseline of 25% for CIFAR which beats the author's baseline of 32.35% and the author's method of 31.93%.

The experiments would be much more convincing if they did it on seq2seq+MT on say EN-FR or EN-DE. There is almost no excuse why the experiments wasn't run on the MT task, given this is the first application of seq2seq was born from. Even if not MT, then at least the sentiment analysis tasks (IMDB/Rotten Tomatoes) of the Dai et al., 2015 paper which this paper is so heavily based on for the sequence autoencoder.

=== References ===
Something is wrong w/ your references latex setting? Seems like a lot of the conference/journal names are omitted. Additionally, you should update many cites to use the conference/journal name rather than just "arxiv".

Listen, attend and spell (should be Listen, Attend and Spell: A Neural Network for Large Vocabulary Conversational Speech Recognition) -> ICASSP
if citing ICASSP paper above, should also cite Bahandau paper "End-to-End Attention-based Large Vocabulary Speech Recognition" which was published in parallel (also in ICASSP).

Adam: A method for stochastic optimization -> ICLR
Auto-encoding variational bayes -> ICLR
Addressing the rare word problem in neural machine translation -> ACL
Pixel recurrent neural networks -> ICML
A neural conversational model -> ICML Workshop

review_title: review

Meta-review: This paper proposes to regularize neural networks by adding synthetic data created by interpolating or extrapolating in an abstract feature space, learning by an autoencoder.
 
 The main idea is sensible, and clearly presented and motivated. Overall this paper is a good contribution. However, the idea seems unlikely to have much impact for two reasons:
  - It's unclear when we should expect this method to help vs hurt
  - Relatedly, the method has a number of hyperparameters that it's unclear how to set except by cross-validation.
 
 We also want to remark that other regularization methods effectively already do closely related things. Dropout, gradient noise, and Bayesian methods, for instance, effectively produce 'synthetic data' in a similar way when the high-level weights of the network are perturbed.
Decision: Invite to Workshop Track