USING OBJECT-FOCUSED IMAGES AS AN IMAGE AUGMENTATION TECHNIQUE TO IMPROVE THE ACCURACY OF IMAGE-CLASSIFICATION MODELS WHEN VERY LIMITED DATA SETS ARE AVAILABLE
Authors: Ahmad Melhem Hammoud
Ahmad Rabih Ghandour
Venue:  (2021)
Keywords: Machine Learning
Computer Vision
Data Augmentation
Background Removal

Abstract: Today, many of the machine learning models are extremely data hungry. On the other hand, the accuracy of the algorithms used is very often affected by the amount of the training data available, which is, unfortunately, rarely abundant. Fortunately, image augmentation is one of the very powerful techniques that can be used by computer-vision engineers to expand their existing image data sets. This paper presents an innovative way for creating a variation of existing images and introduces the idea of using an Object-Focused Image (OFI). This is when an image includes only the labeled object and everything else is made transparent. The objective of OFI method is to expand the existing image data set and hence improve the accuracy of the model used to classify images. This paper also elaborates on the OFI approach and compares the accuracy of five different models with the same network design and settings but with different content of the training data set. The experiments presented in this paper show that using OFIs along with the original images can lead to an increase in the validation accuracy of the used model. In fact, when the OFI technique is used, the number of the images supplied nearly doubles.

Under review as conference paper at ICLR 2021
USING OBJECT-FOCUSED IMAGES AS AN IMAGE A UUMENNAAIONN TECHNIQUE TO IMPROVE THE CCURA CY OF IMAGE-CLA SSLSSIIIIATION MODELS W HEN V ERY LIMITED DATA SETS A RE A VAILABLE
A nonymous authors
P Paper under double-bling review
ABSTRACT
Today, many the machine learning models are extremely data hungry, On the other hand, the accuracy of the algorithms used is very often affected by the amount of the training data available, which is, unfortunately, rarely abundant. Fortunately, image augmentation is one of the very powerful techniques that can be used by computer-visse
engineers to expand their existing image data sets. This paper presents an innovative way for creating a variation of existing images and introduces the idea of using an Object- Focused Image (OFI). This is when an image includes only the labeled object and ev everythine thing else made transparent. The objective of OFI method is to expand the existing image data set and hence improve the accuracy of the model used to classify images. This paper also elaborates on the OFI approach and compares the accuracy of five different models with the same network design and settings but with different content of the training data set. The experiments presented in this paper show that using OIIs along with the original images can lead to an increase in the validation accuracy of the used model. In fact, when the OFI technique is used, the number of the images supplied nearly doubles.
1 INTRODUCTION
Nowadays, Convolutional Neural Networks (CNNs) are among the most common tools used for image classification. For machine learning (ML) problems such as image classification, the size of the training image set is very important to build sighaaccuracy classifiers. As the popularity of CNNs grows, so does the interest in data augmentation. A Ithough augmented data sets are sometimes artificial, they are very similar to the original data sets. Thus, augmentation can make a network learn more useful representations because of the increase of training data.
In this paper, the researchers propose a new method produce new images from existing ones and therefore augment data. Several experiments were conducted to validate that the model could benefit further from using the mixed data set of old and new images.
This paper compares five models with the same design and settings. The only difference is the set of supplied images. The first model uses the original 2,000 images of dogs and cats. The second and the third use the OFI version of those 2,000 images. The fourth and the fifth use all the images: the original images as well as the OFI images. There are two methods to obtain the OFIs. The automated method uses an A pplication Programming Interface (A to remove the background of the image and leave only the labeled object (a cat or a dog) in the foreground. The OFIs produced by the first method are called automatic OFls. The other method is executed manually: every image is edited by a human expert who will remove the background or any additional object in the image. The OFIs produced by this second method are called manual OFIs. The manual method is more accurate than the automated AI method and might lead to better results. The only difference between the second and the third models is the fact that the second model uses the automatic method w while the third model uses the manual method. This is also the difference between the fourth and the fifth models.
In this paper, the five models are tested to answer the following questions:

---

Under review as conference paper at ICLR 2021
Will the model have a better validation accuracy when only the original set of images is used?
W ill it have better validation accuracy when only the set of OFIs is used?
Will we have better validation accuracy when both sets are used?
Are models with manual OFIs more accurate than those which use automatic OFIs?
2 RELATED WORK
In deep learning, augmentation is a commonly used practice since the 80's and early 90's (Simard et al., 1992). It is considered a critical component of many ML models (Ciresan al., 2010; Krizhevsky et al., 2012; LeCun et al., 2015).
Augmentation is of paramount importance in ex treme cases where only few training examples are available (V (Vinyals et al., 2016). In fact, it has been considered an ex extremely important element of numerous successful modern models. A mong these models are: the AlexNet model (Krizhevsky et al., 2012), All-CNN model (Springenbery et al., 2014), and the ResNet model (He et al., 2016).
In some implementations, experts were able to rely on data augmentation, apply it heavily, and achieve successful results (Wu u et al., 2015). This is not the case with computer vision only. Data augmentation was effective with other domains also like text categorization (Lu et al., 2006), speech recogmition (Jaitly & Hinton, 2013), and music source separation (Uhlich et al., 2017).
When given sufficient data, deep neural networks were providing exceptional results They have been studied, proven, and demonstrated in many domains (Gu et al., 2015), including: image classification (Krizhevsky et al., 2012; Huang et al., 2016), natural language processing (Gu et al., 2015), reinforcement learning (Mnih et al., 2015; Foerster et al., 2016; Silver et al., 2016; Gu et al., 2016; V an Hasselt et al., 2016), machine translation (Wu et al., 2016), synthesis (W angg et al., 2017), and many more. In all these experiments and implementations, used datasets have been significantly large.
A ugmentation is vital technique, not only for cases where datasets are small but also for any size of dataset. Indeed, even those models trained on enormous datasets such as Imagenet (Deng et al., 2009) could benefit from data augmentation.
Without data augmentation, deep neural networks may suffer from the lack of generalization (Perez & W Wang, 2017) and adversarial vulnerability (Zhang et al., 2017). A Ithough CNNs have led to substantial achievements in image- processing tasks and image classification (Zeiler and Fergus, 2014; Sermanet et al., 2014), CNNs with abundant parameters might over fit. This is due to the fact that they leam very detailed features of supplied training images that do not generalize to images unseen during training (Zeiler and Fergus, 2014; Zintgraf et al., 2017). Data augmentation has been considered a solution to this overfitting problem (Krizhevsky et al., 2012; He et al., 2016; DeV ries and Taylor, 2017).
Nowadays, data augmentation techniques are gaining continuous attention (DeV ries and Taylor, 2017; Zhong et al., 2017; Zhang et al., 2017 7). When it comes to image recognition, data augmentation is one of the fundamental building blocks for almost all state-of- of-the--rt results (Ciresan et al., 2010; Dosovitskiy et al., 2016; Graham, 2014; Sajjadi et al., 2016). Nonetheless, because augmentation strategies may have a large impact on the model performance, they require extensive selection and tuning (Ratner et al., 2017).
In fact, data augmentation experts proposed cutout (DeV ries and Taylor, 2017). At every training step, cutout randomly masks out a square region in an image. It is an extension of dropout.
Random erasing is similar to cutout (Zhong et al. 2017), as it masks out a subregion in an image but with the following three differences: i) It randomly chooses whether or not to mask out; ii) It uses a random size; and iii) It uses random aspect ratio of the masked region.
On the other hand, Mixup a- bleends two images to form new image (Zhang et al., 2017). Mixup also behaves like class label smoothing. It mixes the class labels of two images with the ratio a a (Szegedy al., 2016).
Other researchers proposed techniques such as andom-cropppi and horizontal- -flitpilll (K rizhevsky et al., 2012).

---

Under review conference paper at ICLR 2021
These useful augmentation methods have been applied to modern deep CNNs. They were proven to be very efficient and demonstrated the added value of data augmentation.
fter ResNet was proposed (He et al., 2016), new network architectures have been proposed as well (Zagoruy ko and Komodakis, 2016; Han et al., 2017). Pixel-dropping however, injects noise into the image (Sietsma and Dow, 1991).
In short, there are numerous data augmentation techniques. Figure summarizes them.
Each line with shape in Figure 1 tells which data augmentation method is used by the corresponding meta- leatmlg scheme. For instance, meta-leaming using mixing images is covered in smart augmentation (Perez W Wang, 2017).
Kernel Filters
Geometric
Transformations
Basic Image
Neural Augmentation
Random Erasing
Manipulations
Color Space
Transformations
Image Data Augmentation
Meta Learning
AutoAugment
Mixing Images
Adversarial Training
Deep Learning Approaches
Neural Style Transfer
Smart Augmentation
GAN Data Augmentation
Figure 1 A taxonomy of image data augmentations covered
3 THE OFI TECHNIQUE
Computer-viste vision engineers use images to train their models. These images include the object that needs to be correctly classified. There is, nevertheless, small problem with these images: they include other objects that we do not want to train the model on.
If we want to train a model to classify images of dogs and cats, we should always ask ourselves the following questions:
W hy should we train the model on something that might not be seen later?
Wh hy, for example, should train the model on cat with ball behind but when the training ends, we'll II ask the model to classify an image without a ball? Once the model trained, it will classify an image of a cat with a tree, a fence, or a couch behind it, so why train the model on a cat with a ball, while real-life images might include millions of other things?
It is a good practice to try to train model using images that include the labeled object only and nothing else. This what the OFI technique attempts do. For every image the training set, another image is generated where the labeled object is kept untouched and everything in the background is made transparent as shown in Figure 2.

---

Under review conference paper at ICLR 2021
When such an image supplied as a part of the training set, the model instructed to focus only on the labeled object. There will be other objects for the network to be trained on.
Here, the scenario can be seen exactly as training baby on what a cat When that baby sees only cat in the image, s/he would not get confused which object is the cat. Otherwise, some questions need be answered:
W hat if there were many balls in the many images we have?
W Wuul our model get little confused by those additional objects and try be trained on?
W ould our model keep changing its parameters and weights based on the additional objects in the image?
Figure
Same image with and without the background
THE EXPERIMENTS
4.1 USED MODELS
With all the above questions in mind, several experiments need to be conducted to validate the OFI technique. The tested models are listed in Table 1. The goal behind checking all these models to determine which model would produce the best results in terms of validation accuracy.
Table 1: Description of Models Used
MODEL
MODEL NAME
CONTENT OF THE TRAINING FOLDER
This model has been trained using 1,000 images for cats and another 1,000 images for dogs.
These images include the labeled objects among other objects in the background, as it is done in the image to the left in Figure 2.
Original
They are taken from the famous Kaggle Dogs vs Cats dataset. The training folder includes 2,000 images while the validation folder has 1,000 images only.
This model includes a copy the images used model #1, but backgrounds are erased using online A PI called remove.bg.
The process background removal is completely automated. All images in the training folder will be sent to the online A PI, in turn. The returned OFI image is then stored in the training folder.
OFI A Automated
Once all images are processed, the training can start. All the images will then include one object only, just like the image to the right Figure 2.

---

Under review conference paper at ICLR 2021
MODEL
MODEL NAME
CONTENT OF THE TRAINING FOLDER
This model is similar to the previous (model 2), but instead of using an AII to edit the images and remove additional objects, human photo editor will edit all the images manually.
3
OFI Manual
The question is: will this slow process make the accuracy of this model higher than the accuracy of the previous automated model?
This model includes those images used in model #1 and those used in model #2. Will using both types of images result in better model?
Mixed
Automated
This model includes those images used model #1 and those used in model #3. Which mix will result in a more accurate model: using original images with the manual OFIs or with the automated OFIs?
Mixed Manual
4.2 NETWORK DESIGN
To test the above models, Tensorflow 2.3.0 has been used. The following is the design of the neural network:
model = tf..keas..odels.. Mel.esssmodesssSSeenttaall
ke.se layers. Convvers.onn22(32,, (3,3), activation='relu, input shape=(150, 150, 3)),
2), d.k.ersslayers.Coov22666 (3,3), activation=rellu) tf. Maaiyerss.MaxPoooing22(22..... (.,.ees,laerss.Com2222222,, (3,3), activation=reluu),
tf. leeaasllaers wtw.slaayrs..aaoooiggg2222 MaxPoollngg2((2,22) f..keraslayerssCom2222222222 (3,3), activation=relu))
kf.keas.. layers. e...eas.ayess..aaoolliiiii MaXPooling2D((222) f.keras.layers.Dnoooutt....
ke. Flatttn) Seeeras.ayerssDenneee activation=trelu",
l Dense(1, activation=sigmoidd)
The following settings are used for compilation:
modee.compielooss-bbaayyycooo httpiiiaryyccsssstoopooooo/ (ptimizee-RMSppp((111114)) "metrics=''accuracy"
Two image generators are used to rescale the images and do other image augmentation operations as follows:
train datagen = Imagemmtaa
rescale=1./255, rotation range=40, width shift range=0.2, height shift range=0.2, shear range=0.2, zoom range=0.2, horizontal flip=True, fill mode='nearest") test datagen InnaggDDateeeerrroorrrsseeee train Beeilinnneeerreeeee train datagen.flow from directory(
train dir, target size=(150, 150), batch size=20, class mode=bbinary)) validation generator ralidation_generatoor- test, datagen.flow from directory( validation dir, target (15e= 150), batch size=20, class mode=/binary))
Finally, this is how the model is trained: history model.fit(
train generator, steps per eee__ppcc=000 epochs=150,
validation data=validation generator, validation steps=50, verbose=2)
All the above settings apply to all the models in Table 1. The only difference between model and another is the content of the training folder. Accordingly, the steps per epoch might be changed based on the number of training images.

---

Under review as conference paper at ICLR 2021
4.3 RESULTS
All the models in Table were tested and the results are presented in Table 2.
The top row shows the traditional scenario, where the original images are used without any kind of editing. A fter 150 epochs, the validation accuracy is 80%. Will this accuracy be higher when the OFIs are used?
The second row in Table 2 includes the result of using OFIs only without using the original ones. These images are automatically generated by calling an PI just before the training starts The training accuracy is 88% while the validation accuracy is 66%. Why is there such a significant difference? This is because the model is trained on an object that exists in the image alone. It never expected to see a dog next to a tree, for example. It always expects to see dog alone or a cat alone. In this case, the training data set is not similar to the validation data set. This why we have a difference of around 22% between training and validation accuracy.
The third row in Table 2 shows the results of using OFIs that were manually produced. When a human photo editor keeps one object in the image and removes all other objects, the image tends to be more accurate. The PI might sometimes crop part of the labeled object leave a part of another object. Thus, it is expected to produce a better result from training the model using manually edited images. A gain, these images alone are misleading because the training data set not similar to the validation data set. This is why there is gap between training and validation in both accuracy and loss curves.
The fourth row in Table 2 represents the results after using the original 2,000 images along with the OFIs. This mix seems be successful. enables the model to receive training on two types of images. This is the same as giving more weight to the labeled object itself because it will be seen twice. The first time, the object will appear alone in the image and the second time, it will show along with other objects (or vice versa).
The difference between the last two rows is that the model in the fourth row uses an A to generate the OFIs, while the model in the fifth row uses OFIs that are manually generated by a human photo editor. The last model has higher accuracy 88% instead of 84% because uses images edited by an expert, not by an A PI that might crop a part of the labeled object or leave a part of another object. A balance needs to be achieved between the pain needed to manually edit thousands of images and the gain of 4% in terms of accuracy.
quick comparison between the first and the last rows of T Table 2 would also lead us to the two following observations:
First, the OFI technique is just another successful approach that can be used augment data and almost double the size of the image set. Indeed, some images cannot be automatically processed.
Second, using the OFI technique might be able to enhance the accuracy of the model from 80% (as in the top row of T a'alle 2) to 88% (as in the last row).
CONCLUSIONS
W When very limited data sets are available, sets of images can be augmented through the use of a mixed model that includes two sets of images (original images and OF1s). As shown in Table 2, the last model is 8% more accurate than the first one. It is recommended that developers of artificial intelligence (AI) engines include the OFI technique in their packages enable users use this feature on the fly the same way TensorFlow users, for example, use ImageDataGenenaan to augment data.
Table 2: Results of the different models

---

Under review conference paper at ICLR 2021
MODEL NAME
TRAINING/VALIDATIN
RESULTS
ACCURACY
LOSS
train loss:
0.3335
train acc.:
0.8575
Original
val loss:
0.5364
val acc.:
0.8030
train loss:
0.2749
OFI Auto.
train acc.:
0.8833
val loss:
1.0507
val acc.:
0.6690
Sumarg Liw
train loss:
0.2713
OFI Manual
train acc.:
0.8908
val loss:
2.2211
val acc.:
0.6400
0.03
To Yo Marne
train loss:
0.2495
Mixed Auto.
train acc.:
0.8988
val loss: 0.5773
val acc.:
0.8400
Tarthi Low
0.5
0.5
train loss:
0.2375
Mixed Manual
train acc.:
0.9033
val loss:
0.3053
0.0
val acc.:
0.8860
055
03
20
300 120 300
120
110
FUTURE WORK
Two issues need to be tackled. The first is that an error might happen during the automated process when removing background. This happens when the AI cannot identify the background. Our tests show that other A can be used when the first one fails. D During this research, the API (remove.bg) failed in removing the background for only 6% of images. The second issue happens when the labeled object (a dog, for example) is in the background. When the background removed, the dog will be also removed, as shown Figure 3. The question, then, would be: can we detect such specific scenario? A Although this is not very common, it needs to be handled properly.

---

Under review conference paper at ICLR 2021
Figure When the background is removed, the dog vanishes (source: doolee-do--oumaand .om)
REFERENCES
Dan Claudiu Ciresan, Ueli Meier, Luca Maria Gambardella, and Jurgen Schmidhuber. Deep big simple neural nets excel on handwritten digit recogmition. arXiv preprint arXiv:1003.033 2010.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei, Imagenet: large-scale hierarchical image database, In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pp. 248 255. IEEE, 2009.
Terracce DeV ries and Graham Taylor. Improved regularization of convolutional neural networks with cutout. irX preprint arX 1v: 0et.177..0555555 2017.
A Alexey Dosovitskiy, Philipp Fischer, Jost Tobias Springenbery, Martin Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with exemplar convolutional neural networks. Transactions on Pattem A nalysis and Machine Intelligence, 38(9):1734 1747, 2016.
Jakob Foerster, Y annniss M A ssael, Nando de Freitas, and Shimon W Whiteson. Leamiigg to communicate with deep multi-agent reinforcement learning. In dvances in Neural Information Processing Systems, pp. 2137-2145, 2016.
Benjamin Graham, Fractional max- max--oooingg arX preprint arX 1v:1422,6071, 2014.
Jiuxiang Gu, Zhenhua W Wang, Jason Kuen, Lianyang Ma, Amir Shahroudy, Bing Shuai, Ting Liu, Xingxing W ang, and Gang W ang. Recent A dvances in Convolutional Neural Networks. dec 2015.
URL http://arxiv. http://arxiv.org//bs//tt wwt.:os///////../////. 151.007108.
Shixiang Gu, Timothy Lillicrap, Ilya Sutskever, and Sergey Levine. Continuous deep q- q-leamnig with model-basee acceleration. In Interational Conference on Machine Leaming, pp. 1829928338, 2016.
Dongyoon Han, Jiwhan Kim, and Junmo Kim. Deep Pyramidal Residual Networks. In Proc. of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CV (CVPR2017) pages 6315, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual leaming for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Gao Huang, Zhuang Liu, Kilian Weinberger, and Laurens van der Maaten, Densely connected convolutional networks. arXiv preprint arX 1v:1608 100.0699..,,, 2016.
Navdeep Jaitly and Geoffrey Hinton. Vocal tract length perturbation (vtlp) improves speech recognition. In Proc. ICML W Workshop on Deep L Leaming for Audio, Speech and Language, pp. 625. 660, 2013.

---

Under review conference paper at ICLR 2021
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pp. 1097- 1105, 2012.
ann LeCun, oshuua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521((5553::336 444, 2015.
X Xinghua Lu, Bin Zheng, A Atuyy V Velivelli, and ChengX iang Zhai. Enhancing text categorization with semantic- semantic-enriched representation and training data augmentation. Journal of the A American Medical Informatics A ssociation, 13(5):526-533) 2006.
V Volodymyr Mnih, K Koray Kavukcuoglu, David Silver, A Andeii A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Geory Ostrovski, et al. Human-level control through deep reinforcement leaminn. Nature, 518(7540):529 533, 2015.
Luis Perez and Jason W ang. The effectiveness of data augmentation in image classification using deep learning. ir preprint ir:
iv: (tV.:1712.0422.. 2017.
Alexander] Ratner, Henry Ehrenberg, Zeshan Hussain, Jared Dunnmon, and Christopher Re. Leaming to compose domain-specific transformations for data augmentation. In Neural Information Processing Systems, 2017.
Mehdi Sajjadi, Mehran avanmardi, and Tolga Tasdizen. Regularization with stochastic transformations and perturbations for deep semi-superviin learning, In Neural Information Processing Sy stems, 2016.
Pierre Sermanet, David Eigen, Xiang Zhang, Michael Mathieu, Rob Fergus, and Y ann LeCun. OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks. In Proc. of International Conference on Learning Representations (ICLR2014), pages 1-16, 2014.
Jocelyn Sietsma and Robert.J.F. Dow. Creating artificial neural networks that generalize. Neural Networks, 4(1):67-79, 1991.
David Silver, Aja Huang, Chris J Maddison, rthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis A Atonogloo,, V Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484-49) 2016.
Patrice Simard, Bernard Victorri, ann LeCun, and John Denker. Tangent prop-a formalism for specifying selected invariances in adaptive network. In A dvances in neural information processing systems, pp. 895-903, 1992.
Jost Tobias Springenberg, A lexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. In International Conference Learning Representations, ICLR, arXiv:1412.6008 2014.
Christian Szegedy, Vincent V Vanhoucke, Sergey Ioffe, Jon Shlens, and Z Wojna. Rethinking the Inception Architecture for Computer V Vision. In Proc. of the IEEE Computer Society Conference on Computer V isson and Pattern Recognition (CVPR2016) pages 2818-2826, 2016.
Stefan Uhlich, Marcello Porcu, Franck Giron, Michael Enenkl, Thomas Kemp, Naoya Takahashi, and Yuki Mitsufuji. Improving music source separation based deep neural networks through data augmentation and network blending. Submitted to ICASSP, 2017.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement leaminng with double qleaming.
In A pp. 2094-2100, 2016.
Oriol Vinyals, Charles Blundell, Tim Lillicrap, koray kavukcuoglu, and Daan W Vierstra. Matching networks for one shot leaming. dvances Neural Information Processing Systems 29, pp. 3630-3638, 2016.

---

Under review conference paper at ICLR 2021
Y uxuan W ang, R. Skeny- Skeny--yan, Daisy Stanton, Y onghui Wu, Ron J. W Weiss, Navdeep JaceepJaai Jaitly, Zongheng Y ang, ing Xiao, Zhifeng Chen, Samy Bengio, Quoc V. Le, Y annis A giomyrgiannaais, Rob Clark, and Rif A. Saurous. Tacotron: A fully end-to end text: to speech synthesis model. CoRR, abs/1703.10135, 2017. URL http://axxiv.orr/aas///0..
Ren Wu, Shengen Y an, Yi Shan, Qingqing Dang, and Gang Sun. Deep image: Scaling up image recognition, irX preprint arXiv:1501.02877 7(8), 2015.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, W Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, al. Google's neural machine translation system: Bridging the gap between human and machine translation. arX preprint arX iv:1609.088144 2016.
Seryey Zagonuyko and Nikos Komodakis. Wide Residual Networks. Proc. of the British Machine Vision Conference (BMV C2016), pages 87.1 87.12, 2016.
Matthew D. Zeiler and Rob Fergus. Visualizing and Understanding Convolutional Networks. In Proc. European Conference Computer Vision (ECCV2014), pages 818 833, 2014.
Hongyi Zhang, Moustapha Cisse, Y amn N Dauphin, and David Lopez- Pazz mixup: Beyond empirical risk minimization. arXiv preprint arX iv:1710.09412, 2017.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Y ang. Random Erasing Data Augmentation. arX iv, pages 1-10, 2017.
Luisa M Zintgraf, Taco S Cohen, Tameem del, and Max Welling. Visualizing Deep Neural Network Decisions: Prediction Difference A nalysis. Proc. of International Conference on Learning Representations (ICLR2017), pages 1-12, 2017.
10


Reviews:
[{"review_id": "y2nIK3z8pBo", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an image data set augmentation approach (4 variants+ original data set variant) in order to cater to the lack of sufficient data for learning purposes by increasing available image data sets through creation of  variations of existing images. \nThe variations are all about extracting recognizable objects in images (Object-Focused Images or  OFIs), manually and automatically,  and combination with original data set images.\n\nQuestions:\n- “In fact, when the OFI technique is used, the number of the images supplied nearly doubles.”\n\t- Does not this depend on the type of images in the data set: multi-object images, no discernible object sets (textures, backgrounds etc.)?\n        - Image given as example in Figure 2 is quite simplistic as the cat figure is clearly easily isolated. You Should state the type of application your model is intended for as you use for such a simplistic application.\n\n- In 3. THE OFI TECHNIQUE: \n\t- You state “It is a good practice to try to train a model using images that include the labeled object only and nothing else.”. \n        \t- Does not this go against building models that will use real life images and therefore generalize well? Except may be in controlled setting like manufacturing lines.\n\n- In 4. THE EXPERIMENTS /4.1 USED MODELS:\n\t\t- You conduct 5 different tests (or models as you call them): Original images, Automatic OFIs, Manual OFIs, Original + Automatic OFIs and Original + Manual OFIs. \n\t \t- Did not you try: Original + Automatic OFIs + Manual OFIs? Why not, you had the data.\n\nComments:\n-\tIn  1 INTRODUCTION, you state: \n\t \t- \" benefit further from using the mixed data set of old and new images”. Rather original data set images and derived/augmented images.\n\t \t- \"The automated method uses an Application Programming Interface (API) to remove the background of the image and leave only the labeled object (a cat or a dog) in the foreground.” An API is an interface to remotely run an algorithm/call a method, not the actual method. \n\t \t- For consistency, since you deemed each experiment as a model, asking “Will the model have a better validation accuracy when only the original set of images is used?” is not consistent. Rather “will the model with only the original set of images …”\n\t \t- This paragraph seems out of place ” After ResNet was proposed (He et al., 2016), new network architectures have been proposed as well (Zagoruyko and Komodakis, 2016; Han et al., 2017).”.\n\n- In 3, THE OFI TECHNIQUE: \n\t- Am perplexed by these questions:\n\t \t- Would our model get a little confused by those additional objects and try to be trained on? \n\t \t- Would our model keep changing its parameters and weights based on the additional objects in the image? \n- In 4. THE EXPERIMENTS /4.1 USED MODELS:\n\t- “When a human photo editor keeps one object in the image and removes all other objects, the image tends to be more accurate.” Not sure, this phrasing is”accurate”.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "tn_cPiMCTTD", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an object focused image based technique for CNN training, in which, for every image in the training set, another image is generated where the labeled object is kept untouched and everything in the background is rubbed out. A simple comparison is made among five scenarios on using separately or combined the original and OFI, illustrating that in the examined experiment somewhat better accuracy is obtained when combining them. \n\nThe presented research cannot be considered novel. Extracting regions of interest in images and using them for analysis, or extracting points of interest (e.g., facial landmarks) and using them together, or in addition, to the original images (e.g., faces) has been a frequently used approach to enhance object classification or regression tasks. The presented comparison has some interest, but it lacks even the required experimental validation, by comparing this data augmentation to any other competing approach - described in Figure 1.\n\nIn particular:\n\nSection 2 on related work makes reference to a large number of techniques used for data augmentation, which are summarized in Figure 1. Nevertheless, none of these techniques is used to compare to the OFI based method.\n\nThe results presented in Section 4 are of low significance. Testing in scenarios 2 and 3 would be meaningful if the statistics were similar to training; thus, obtaining lower accuracy was expected. Similarly, when comparing scenarios 1 and 4, with the latter containing augmented data, it was expected to see a better performance in the latter case.\n\nVarious presentation and language errors should be corrected (missing references, poor language, etc). ", "extra_scores": {}, "extra_text": {}}, {"review_id": "hMHlXEpe4Je", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes to augment object focused image to improve image classification. Essentially, this work tries to remove background from original image using an existing algorithm and human editor, and train an image classification model using different combinations of original images and background-removed image. Five simple models are trained and their performance is compared to show that using background-removed images together with original images can help improve accuracy on validation set on a simple dataset. \n\nOverall, this paper has serious flaws and is far from the level of ICLR. It is more like a course project report than a scientific paper. It lacks rigorous experiment design and result analysis, and the conclusion is not surprising or insightful. A clear rejection.\n\nDetailed comments:\n\nTechnical:\n- Although it makes sense to augment. input images to improve model performance, the design of the proposed work is less convincing. When talking about data augmentation, there usually won't be human editor/annotator involved and all the operations should be performed on-the-fly algorithmically. However, in this work, the background removal for original image is done either by a human editor or a third-party tool, which is not scalable at all. \n- The technical part is also significantly limited. The background removal assumes that there is only one dominant object related to the label in an image. This simplifies the real problem too much. As the paper shows in Figure 3, this poses a serious problem when there are more than one object and the object of interest is not dominant. I do not think the proposed method can be applied to more complex scenarios at all. \n- The paper states \"When that baby sees only a cat in the image, s/he would not get confused which object is the cat.\" However, this may not be true. Actually we haven't really understood how exactly human's perception/neural system works.\n- The performance of the model trained on images using the third-party background removal tool heavily depends on the quality of the tool. Since there is no evaluation on the tool, we do not know how the quality of background removal contributes to model performance improvement/degradation. \n\nExperiments:\n- The proposed model is only evaluated on one simple dataset containing only cats and dogs. It is more a sanity check/toy example than a realistic evaluation setting since there is just binary classification. Larger-scale datasets with multiple classes should be used.\n- The model evaluated is a simple convolutional network, and not a standard model; i.e., ResNet, VGG or other widely used model, so it is difficult to say whether the conclusion drawn from this model can by applied to other models and datasets. \n- Why not mix human edited images and images modified by the tool and train a model on all the images to see how it works? ", "extra_scores": {}, "extra_text": {}}, {"review_id": "7T0lOfy7Kmr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: Authors propose an augmentation technique for image classification. The augmented image is obtained by segmenting the salient object and masking the background. Therefore the technique gives one additional augmented image per each training image. The authors show an improved performance when using this augmentation on binary classification task (cats vs. dogs) using a dataset of 2000 images.\n\nI find the paper in its current form lacking in many aspects, mainly: questionable novelty, bad presentation, minimal experiments.\n\nConcerning novelty, the authors themselves remark: \"It is a good practice to train a model using images that include the labeled object only\". The section on related work is very superficial ignoring the achievements thanks to synthetic images. The approaches such as the following, used usually in the context of object detection, should be considered prior art and at least properly discussed, e.g.:\n\n- Gupta et al, \"Synthetic Data for Text Localisation in Natural Images\", CVPR, 2016.\n- Dwibedi et al, \"Cut, Paste and Learn: Surprisingly Easy Synthesis for Instance Detection\", ICCV, 2017.\n- Georgakis et al, \"Synthesizing Training Data for Object Detection in Indoor Scenes\", RSS, 2017\n- Dvonik et al, \"Modeling Visual Context is Key to Augmenting Object Detection Datasets\", ECCV, 2018\n- Tripathi et al, \"Learning to Generate Synthetic Data via Compositing\", CVPR, 2019\n\nThere are many places in the manuscript where the sense of the text is not clear or the text appears out of context, e.g.: \"After ResNet was proposed (He et al., 2016), new network architectures have been proposed as well (Zagoruyko and Komodakis, 2016; Han et al., 2017). Pixel-dropping, however, injects noise into the image (Sietsma and Dow, 1991).\" The text seems in general unfinished and outdated at the same time (no citations of prior art newer than 2017).\n\nConcerning experiments. The authors speak about lack of data but instead of evaluating on some few-shot datasets (e.g., mini-Imagenet or omniglot), they evaluate on a dataset with two classes and 1000 training images per class. The experiments are limited to evaluating a a model trained on the original unaugmented dataset, augmented using automatic segmentation, and augmented using manual segmentation. There is no comparison to other methods, not even the ones cited in the paper. Given obtaining segmentation can be costly I would expect the authors to test whether it is necessary at all and whether cutting the object to the bounding box is not enough. I would also expect a discussion of when can background removal actually hurt the performance (e.g., because it is predictive of the target class).", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces an augmentation technique that, given an image with a detected object, keeps the object and removes the background.

The reviewers expressed numerous valid concerns about the paper's novelty, the setting (assumption that there's a single object), the scalability of the approach and the experimental setup, including the baselines used.

The authors have not addressed these concerns.
Decision: Reject