Title: Data Augmentation by Pairing Samples for Images Classification
Authors: Hiroshi Inoue
Venue: ICLR 2018
Keywords: Data augmentation
Image classification

Abstract: Data augmentation is a widely used technique in many machine learning tasks, such as image classification, to virtually enlarge the training dataset size and avoid overfitting. Traditional data augmentation techniques for image classification tasks create new samples from the original training data by, for example, flipping, distorting, adding a small amount of noise to, or cropping a patch from an original image. In this paper, we introduce a simple but surprisingly effective data augmentation technique for image classification tasks. With our technique, named SamplePairing, we synthesize a new sample from one image by overlaying another image randomly chosen from the training data (i.e., taking an average of two images for each pixel). By using two images randomly selected from the training set, we can generate N^2 new samples from N training samples. This simple data augmentation technique significantly improved classification accuracy for all the tested datasets; for example, the top-1 error rate was reduced from 33.5% to 29.0% for the ILSVRC 2012 dataset with GoogLeNet and from 8.22% to 6.93% in the CIFAR-10 dataset. We also show that our SamplePairing technique largely improved accuracy when the number of samples in the training set was very small. Therefore, our technique is more valuable for tasks with a limited amount of training data, such as medical imaging tasks.



Review discussion:
I found there is another submission discussing a quite similar technique.
mixup: Beyond Empirical Risk Minimization
https://openreview.net/forum?id=r1Ddp1-Rb&noteId=r1Ddp1-Rb
The paper proposes a new data augmentation technique based on picking random image pairs and producing
a new average image which is associated with the label of one of the two original samples. The experiments show
that this strategy allows to reduce the risk of overfitting especially in the case of a limited amount of training
samples or in experimental settings with a small number of categories.
+ The paper is easy to read: the method and the experiments are explained clearly.
- the method is presented as a heuristic technique.
1) The training process has some specific steps with the Sample Pairing intermittently disabled.
The number of epochs with enabled or disabled Sample Pairing changes depending on the dataset.
How much si the method robust/sensitive to variations on these choices?
2) There is no specific analysis on the results besides showing the validation and training errors: would it
be possible to see the results per class? Would the confusion matrices reveal something more about the
effect of the method?  Does Sample Pairing help to differentiate similar categories even if they are mixed
at trainign time?
3)  Would it be possible to better control the importance of each sample label rather
than always choosing one of the two as ground truth?
The paper misses an in-depth analysis of the proposed practical strategy.
Thank you so much for your comments.
Please refer the updates in above response on three points you mentioned in the comment.
I like to specially thank the advice on confusion matrix. I have never investigated it.
On average, SamplePairing gave improvements in classification of similar classes (e.g. two animals or two vehicles) or different classes (e.g. animal and vehicle).
But I am doing further investigation on the characteristics of SamplePairing using confusion matrices.
The paper investigates a method of data augmentation for image classification, where two images from the training set are averaged together as input, but the label from only one image is used as a target.  Since this scheme is asymmetric and uses quite unrealistic input images, a training scheme is used where the technique is only enabled in the middle of training (not very beginning or end), and in an alternating on-off fashion.  This improves classification performance nicely on a variety of datasets.
This is a simple technique, and the paper is concise and to the point.
However, I would have liked to see a few additional comparisons.
First, this augmentation technique seems to have two components:  One is the mixing of inputs, but another is the effective dropping of labels from one of the two images in the pair.  Which of these are more important, and can they be separated?  What if some of the images' labels are changed at random, for half the images in a minibatch, for example?  This would have the effect of random label changes, but without the input mixing.
Likewise, what if both labels in the pair are used as targets (with 0.5 assigned to each in the softmax target)?  This would mix the images, but keep targets intact.
Second, the bottom of p.3 says that multiple training procedures were evaluated, but I'd be interested to see the results of some of these.
In particular, is it important to alternate enabling and disabling SamplePairing, or does it also work to mix samples with and without it in each minibatch (e.g. 3/4 of the minibatch with pairing augmentation, and 1/4 without it)?
I liked the experiment mixing images from within a restricted training set composed of a subset of the CIFAR images, compared to mixing these images with CIFAR training set images outside the restricted sample (p.5 and Fig 5).  This suggests to me, however, that it's possible the label manipulations may play an important role.  Or, is an explanation why this performs not as well that the network will train these mixing images to random targets (that of the training image in the pair), and never see this example again, whereas by using the training set alone, the mixing image is likely to be repeated with its correct label?  Some more discussion on this would be nice.
Overall, I think this is an interesting technique that appears to achieve nice results.  It could be investigated deeper at some key points.
Thank you so much for your comments.
Please refer the updates 1) and 2) in the above response.
I am currently implementing SamplePairing in a sub-minibatch granularity.
So far, I do not see the significant differences by using smaller granularity of enabling/disabling SamplePairing, e.g. disabling for one mini batch after enabling for four mini batches instead of disabling two epochs after enabling eight epochs.
But I am going to add the data with different granularity including the sub-minibatch granularity.
The paper reports that averaging pairs of training images improves image classification generalization in many datasets.
This is quite interesting. The paper is also straightforward to read and clear, which is positive.
Overall i think the finding is of sufficient interest for acceptance.
The paper would benefit from adding some speculation on reasons why this phenomenon occurs.
There are a couple of choices that would benefit from more explanation / analysis:  a) averaging, then forcing the classifier to pick one of the two classes present; why not pick both? b) the choice of hard-switching between sample pairing and regular training - it would be interesting if sample-pairing as an augmentation meshed better with other augmentations implementation-wise, so that it could be easier to integrate in other frameworks.
Thank you so much for your comments.
Please refer the updates 1) and 2) in above response on two points you mentioned in the comment (using two labels and switching between SamplePairing and regular training).
I am adding more experiments on the second point (switching), e.g. using different granularity.  I hope I can add more discussion on this point.
First of all, we greatly thank the reviewers for their valuable comments.
Also, we like to thank who made effort to reproduce our results.
I updated the submission based on the comments from reviewers.
The major updates are:
1) I added discussion on mixup, which is proposed in another submission (https://openreview.net/forum?id=r1Ddp1-Rb&;noteId=r1Ddp1-Rb), in related work.
Although mixup does blending two samples as we do in this paper, mixup also blends labels from both samples while we pick only one.
There is a blog post by Ferenc Huszár (http://www.inference.vc/mixup-data-dependent-data-augmentation/), which points out that using label from one sample will give the same results by reformulating the loss function of mixup.
I also tested using both labels in our SamplePairing and it did not give significant difference as show in Figure 7 (in Appendix).
2) In this paper, we intermittently disable SamplePairing in 20% of the epochs. I added Figure 6 on how this ratio affects the final results to answer the reviewers' questions. By intermittently disabling SamplePairing, we can get small improvements compared to the case without disabling SamplePairing.
But this improvement is minor compared to the improvements by SamplePairing itself;
hence the training with SamplePairing is not so sensitive to this (potentially workload dependent) tuning parameter.
3) I added confusion matrices with and without SamplePairing to show how samples in each class are predicted in Figure 8 (in Appendix).

Meta-review:
role: PC
data: comment: The paper proposes a data augmentation technique for image classification which consists in averaging two input images and using the label of one of them. The method is shown to outperform the baseline on the image classification task, the but evaluation doesn’t extend beyond that (to other tasks or alternative augmentation mechanisms); theoretical justification is also lacking.

Decision: Reject