Title: Dataset Augmentation in Feature Space
Authors: Terrance DeVries
Graham W. Taylor
Venue: ICLR 2017
Keywords: Unsupervised Learning

Abstract: Dataset augmentation, the practice of applying a wide array of domain-specific transformations to synthetically expand a training set, is a standard tool in supervised learning. While effective in tasks such as visual recognition, the set of transformations must be carefully designed, implemented, and tested for every new domain, limiting its re-use and generality. In this paper, we adopt a simpler, domain-agnostic approach to dataset augmentation. We start with existing data points and apply simple transformations such as adding noise, interpolating, or extrapolating between them. Our main insight is to perform the transformation not in input space, but in a learned feature space. A re-kindling of interest in unsupervised representation learning makes this technique timely and more effective. It is a simple proposal, but to-date one that has not been tested empirically. Working in the space of context vectors generated by sequence-to-sequence models, we demonstrate a technique that is effective for both static and sequential data.


TL;DR: We argue for domain-agnostic data augmentation in feature space by applying simple transformations to seq2seq context vectors.

Review discussion:
The proposed method is interesting and the paper is well presented. Just some thoughts on the augmentation methods.
Besides adding random noise, interpolation and extrapolation, one other way of improving model robustness is to randomly corrupt the representation. Wondering how that works within the proposed framework.
Thanks for your interest and positive feedback. We view random noise as a type of “random corruption”. If we have correctly understood your question, perhaps you are referring to a more structured type of corruption? Yes, this should work within our framework but if the corruption process is domain-specific then we lose one of the advantages of our method, which is that it requires no domain knowledge.
Hi, when extrapolation based augmentation is carried out, do the neighboring context vectors belong to different classes (this is not clear from the paper, it seems they belong to the same class)?  If they are in the same class extrapolation based augmentation feels more risky as it could push a class into a confusable class (e.g. digit 1 into digit 7)?
When extrapolation based augmentation is carried out it is between neighbouring samples from the same class. The amount of extrapolation can be controlled by setting the lambda parameter in Equation 3. For large values of lambda there would definitely be the possibility of generating a sample that appears to belong to a different class, but for smaller values of lambda this risk is much reduced. We found that using a lambda value of 0.5 consistently yielded good results; generated samples were different enough from the original samples so as to contribute helpful diversity to the training set, but still similar enough that they did not appear to come from different classes.
In this paper authors propose a novel data augmentation scheme where instead of augmenting the input data, they augment intermediate feature representations.  Sequence auto-encoder based features are considered, and random perturbation, feature interpolation, and extrapolation based augmentation are evaluated. On three sequence classification tasks and on MNIST and CIFAR-10, it is shown that augmentation in feature space, specifically extrapolation based augmentation, results in good accuracy gains w.r.t. authors baseline.
My main questions and suggestions for further strengthening the paper are:
a) The proposed data augmentation approach is applied to a learnt auto-encoder based feature space termed ‘context vector’ in the paper.  The context vectors are then augmented and used as input to train classification models. Have the authors considered applying their feature space augmentation idea directly to the classification model during training, and applying it to potentially many layers of the model?
Also, have the authors considered convolutional neural network (CNN) architectures as well for feature space augmentation?  CNNs are now the state-of-the-art in many image and sequence classification task, it would be very valuable to see the impact of the proposed approach in that model.
b) When interpolation or extrapolation based augmentation was being applied, did the authors also consider utilizing nearby samples from competing classes as well?  Especially in case of extrapolation based augmentation it will be interesting to check if the extrapolated features are closer to competing classes than original ones.
c) With random interpolation or nearest neighbor interpolation based augmentation the accuracy seems to degrade pretty consistently.  This is counter-intuitive.  Do the authors have explanation for why the accuracy degraded with interpolation based augmentation?
d) The results on MNIST and CIFAR-10 are inconclusive.
For instance the error rate on CIFAR-10 is well below 10% these days, so I think it is hard to draw conclusions based on error rates above 30%.  For MNIST it is surprising to see that data augmentation in the input space substantially degrades the accuracy (1.093% -> 1.477%).  As mentioned above, I think this will require extending the feature space augmentation idea to CNN based models.
Thank you for your valuable suggestions for improving the paper.
a) This is an interesting suggestion. In the case where noise is the transformation, this would simply amount to noise injection in the hidden layer (which we know works well as a regularizer).
However, doing interpolation or extrapolation amounts to something that, to the best of our knowledge, hasn't been reported. For this paper one of our aims was to keep the encoding architecture and transformation in feature space consistent, however, we are very interested in exploring this direction for the next paper.
b) Since the reviews were submitted we conducted some tests on utilizing samples from competing classes for extrapolation and interpolation, however we found that both approaches resulted in worse performance compared to the baseline.
c) We ran some additional synthetic data experiments where we could control the complexity of the class boundaries. We found that extrapolation helped only in the case where there were complex class boundaries, not when the boundaries were simple (e.g. linearly separable, or one class encircling another).
However, interpolation did help in these simple cases. Our best explanation at present is that interpolation tends to tighten class boundaries and unnecessarily increase confidence, leading to overfitting. In essence, it may cause the model to ignore informative extremities that can describe a complex decision boundary and produce an unnecessarily smooth decision boundary. High-dimensional, real datasets will typically have complex decision boundaries, and this is the case where we found extrapolation to shine. We have added a discussion on this matter to the conclusion section in the most recent revision of the paper.
d) Please see our response to Reviewer 1 detailing the use of Wide ResNets.
The concept of data augmentation in the embedding space is very interesting. The method is well presented and also justified on different tasks such as spoken digits and image recognition etc.
One comments of the comparison is the use of a simple 2-layer MLP as the baseline model throughout all the tasks. It's not clear whether the gains maintain when a more complex baseline model is used.
Another comment is that the augmented context vectors are used for classification, just wondering how does it compare to using the reconstructed inputs. And furthermore, as in Table 4, both input and feature space extrapolation improves the performance, whether these two are complementary or not?
Thank you for your positive feedback on the paper and your constructive suggestions.
With respect to the more complex baseline, since the reviews were received, we experimented with a more state-of-the-art architecture (Wide ResNets by Zagoruyko and Komodakis, 2016) for the CIFAR-10 task. In all of our tests we held constant the  Wide Residual Network architecture (same as the one used in the Wide ResNets paper) and changed only the data. We conducted tests on the following 6 setups:
Test 1 – 32x32 input with no data augmentation (8.79% test error)
Test 2 - 24x24 center crops with no data augmentation (11.21% test error)
Test 3 - 24x24 center crops, reconstructed from context vectors of the original images, with no data augmentation (18.68% test error)
Test 4 - 24x24 center crops + extrapolation (14.11% test error)
Test 5 - 24x24 with simple data augmentation (shifting and horizontal flipping) (7.33% test error)
Test 6 - 24x24 with simple data augmentation + extrapolation (8.55% test error)
Overall performance improved greatly compared to our current sequence autoencoder + MLP results. In the Test 4 and 6 described above, we used reconstructions of the extrapolated feature vectors (in addition to the original images) to train the classifier.
Although visual inspection of the reconstructions looked to us like valid training cases (i.e. objects maintained their class identity and we saw no visible artifacts) training with reconstructed extrapolated context vectors actually worsened performance. We also trained a model on only reconstructions of the original examples mapped to context vectors (i.e. no extrapolation) for Test 3. This test performed considerably worse compared to training on the original examples themselves (Test 2).
Therefore we believe that, at least for CIFAR-10, there is significant loss of fidelity of the examples (original or extrapolated) when mapping to and from context vectors. We are currently investigating alternative methods that could be used to reduce the amount of error introduced during the reconstruction process,  such as static encoder-decoder architectures or performing extrapolation within the feature space of the classifier itself.
We’re still in the process of running these experiments, but will add the Wide ResNet results to the final version.
TDLR: The authors present a regularization method wherein they add noise to some representation space. The paper mainly applies the technique w/ sequence autoencoders (Dai et al., 2015) without the usage of attention (i.e., only using the context vector). Experimental results show improvement from author's baseline on some toy tasks.
=== Augmentation ===
The augmentation process is simple enough, take the seq2seq context vector and add noise/interpolate/extrapolate to it (Section 3.2). This reviewer is very curious whether this process will also work in non seq2seq applications.
This reviewer would have liked to see comparison with dropout on the context vector.
=== Experiments ===
Since the authors are experimenting w/ seq2seq architectures, its a little bit disappointing they didn't compare it w/ Machine Translation (MT), where there are many published papers to compare to.
The authors did compare their method on several toy datasets (that are less commonly used in DL literature) and MNIST/CIFAR. The authors show improvement over their own baselines on several toy datasets. The improvement on MNIST/CIFAR over the author's baseline seems marginal at best. The author also didn't cite/compare to the baseline published by Dai et al., 2015 for CIFAR -- here they have a much better LSTM baseline of 25% for CIFAR which beats the author's baseline of 32.35% and the author's method of 31.93%.
The experiments would be much more convincing if they did it on seq2seq+MT on say EN-FR or EN-DE. There is almost no excuse why the experiments wasn't run on the MT task, given this is the first application of seq2seq was born from.
Even if not MT, then at least the sentiment analysis tasks (IMDB/Rotten Tomatoes) of the Dai et al., 2015 paper which this paper is so heavily based on for the sequence autoencoder.
=== References ===
Something is wrong w/ your references latex setting? Seems like a lot of the conference/journal names are omitted.
Additionally, you should update many cites to use the conference/journal name rather than just "arxiv".
Listen, attend and spell (should be Listen, Attend and Spell: A Neural Network for Large Vocabulary Conversational Speech Recognition) -> ICASSP
if citing ICASSP paper above, should also cite Bahandau paper "End-to-End Attention-based Large Vocabulary Speech Recognition" which was published in parallel (also in ICASSP).
Adam: A method for stochastic optimization -> ICLR
Auto-encoding variational bayes -> ICLR
Addressing the rare word problem in neural machine translation -> ACL
Pixel recurrent neural networks -> ICML
A neural conversational model -> ICML Workshop
Thanks for your feedback. We wanted to point out a few simplifications in Reviewer 3’s TLDR statement which we thought unfairly represented the work:
- When R3 said “they add noise to some representation space”, this incorrectly represents what we did. We experimented with noise injection, interpolation, and extrapolation in feature space and actually found that adding noise did not work well compared to extrapolation.
- When R3 said “Experimental results show improvement from author’s baseline on some toy tasks”, there are two errors here.
First, our Arabic Digits, Australian Sign Language Signs, and UCFKinect tests all include results that were not our own baselines; they were the best results reported by other groups that we could find on these datasets.
Second, while we completely agree that Arabic Digits, AUSLAN, UCFKinect, MNIST, and CIFAR-10 are not large-scale datasets, calling them toy seems unnecessarily punitive. Our aim in choosing these datasets was breadth of modalities.
- We’re not sure why it was necessary to qualify “without the usage of attention” in the TLDR statement as it does not seem relevant to what we are exploring.
We agree that it would be interesting to explore this approach on Machine Translation (or other text-based applications).
However, neither of the authors have experience in this domain and, to our understanding, most existing approaches require significant infrastructure (in terms of datasets and computational resources). We weren't able to introduce such a pipeline in the time frame allotted by the review period, but we will certainly consider it in future work.
Dropout on the context vector can be seen as a specific case of our method; it’s a (severe) kind of noise applied to the feature-mapped inputs. We performed some tests by applying dropout to the context vectors and found that it neither increased nor decreased performance significantly.
We acknowledge the discrepancy between the baseline published by Dai et al. (26% error vs. our 32% error) and while we believe that we have exactly replicated their approach, we cannot explain the discrepancy. We reached out to these authors requesting more detail but have yet to receive a response. We have since tried more sophisticated classification (Wide ResNets), the details of which are in the response to Reviewer 1.
Thanks for catching the problem with the references; there was a serious error which we corrected. We have updated the Arxiv papers that were subsequently published. Thanks for taking the time to identify these.
Dear Authors,
Thank you for your rebuttal.
RE: Datasets
As mentioned in my original review, the problem I have with [Arabic Digits, AUSLAN, UCFKinect] datasets are that they are not commonly used datasets in DL literature. Even for the reported results, SOTA is not always achieved. For Arabic Digits (Table 1), the method does not show SOTA, 0.74 (author's method) vs. 0.69 (Hammami et al) (lower is better). For UCFKinect (Table 3), similarly 96.41 (author's method) vs. 98.9 (Beh et al) (higher is better). Only AUSLAN showed SOTA results (this reviewer is also not familiar w/ the AUSLAN dataset).
Additionally, these tasks are not necessarily suitable for the sequence-autoencoder (i.e., see Dai et al., for more relevant sequence autoencoder tasks). Side: might be wise to be consistent in reporting numbers to stick with errors (lower is better) or accuracy (higher is better).
RE: "without the usage of attention"
I mentioned that in my TLDR since many (not all) modern seq2seq applications/papers use attention. Since experiments were conducted w/o attention; it is unclear whether the method would extend to other models of seq2seq+attention.
RE: Machine Translation
As repeated in the original review, if the author's showed results on MT, and showed improvements over other published numbers, this reviewer would be much more convinced w/ the method. An alternative (as mentioned in original review) is some of Dai et al. text classification tasks which also uses sequence autoencoders (i.e., there is a lot of published literature with good baselines). The text classification tasks that Dai et al. is used should be good comparison points as well, since this paper is heavily relies on the sequence autoencoder that Dai et al. introduced.
RE: discrepancy between the baseline
It is unfortunate that the CIFAR baseline could not be replicated from Dai et al. If the author's showed convincing results on CIFAR, this reviewer would be much more inclined to improve the score even without the said MT/text experiments mentioned above.
However, the discrepancy is too large between the author's baseline and Dai et al.'s baseline -- and even the improvement shown from the author's baseline is not very convincing (32.3 vs. 31.9; Dai et al., achieved 26% error -- lower is better).
This reviewer will maintain their score at 4.
In our previous experiments with the Wide ResNet architecture (see our response to Reviewer1's review for details) we tested to see whether using reconstructed inputs, rather than context vectors, still yielded improved model performance. In these tests we encoded CIFAR-10 images with the encoder portion of a sequence autoencoder, extrapolated between the resulting context vectors, and then projected the context vectors back into input space with the decoder to reconstruct augmented images. The newly generated images were then added to the original dataset. We found that this data augmentation approach resulted in worse classification performance compared to the baseline model.
However, we also observed that training a model on only reconstructed images (encoded then decoded, with no extrapolation) yielded results much worse than the equivalent test trained on unmodified data. We hypothesized that the decrease in performance from the tests with extrapolation may possibly be attributed to the decoder's reconstruction error rather than the effects of extrapolation.
To isolate the effects of extrapolation we conducted several more experiments, this time using reconstructed samples for all tests so that reconstruction error affected all tests equally. Any difference between baseline and extrapolation tests could now be contributed solely to the effects of extrapolation. The results from our tests are shown below:
Test 1 - 24x24 reconstructions, center crop: 18.75 +/- 0.24 (% test error)
Test 2 - 24x24 reconstructions, center crop + extrapolation: 17.72 +/- 0.45 (% test error)
Test 3 - 24x24 reconstructions, simple data augmentation (shift + mirror): 13.55 +/- 0.15 (% test error)
Test 4 - 24x24 reconstructions, simple data augmentation + extrapolation: 11.99 +/- 0.11 (% test error)
In these tests we observed a consistent reduction in classification error when feature space extrapolation is applied: about 1% reduction in test error when applied to 24x24 center crops of the original images, and a 1.5% reduction in test error when coupled with input space data augmentation. We believe that these results demonstrate the potential benefits of the feature space extrapolation technique, which persist even when applied to complex model architectures such as Wide ResNet. As previously mentioned we are currently investigating methods which may reduce the reconstruction error introduced during the decoding process.

Meta-review:
role: PC
data: comment: This paper proposes to regularize neural networks by adding synthetic data created by interpolating or extrapolating in an abstract feature space, learning by an autoencoder.
 
 The main idea is sensible, and clearly presented and motivated. Overall this paper is a good contribution. However, the idea seems unlikely to have much impact for two reasons:
  - It's unclear when we should expect this method to help vs hurt
  - Relatedly, the method has a number of hyperparameters that it's unclear how to set except by cross-validation.
 
 We also want to remark that other regularization methods effectively already do closely related things. Dropout, gradient noise, and Bayesian methods, for instance, effectively produce 'synthetic data' in a similar way when the high-level weights of the network are perturbed.

Decision: Invite to Workshop Track