Self-Pretraining for Small Datasets by Exploiting Patch Information
Authors: Zhang Chunyang
Venue:  (2021)
Keywords: Learning with Small Datasets
Self-Pretraining

Abstract:   Deep learning tasks with small datasets are often tackled by pretraining models with large datasets on relevent tasks. Although pretraining methods mitigate the problem of overfitting, it can be difficult to find appropriate pretrained models sometimes. In this paper, we proposed a self-pretraininng method by exploiting patch information in the dataset itself without pretraining on other datasets. Our experiments show that the self-pretraining method leads to better performance than training from scratch both in the condition of not using other data.

Under review as a conference paper at ICLR 2021
SELF-PRETRAINIY FOR SMALL DATASETS BY EX- PLOITING PATCH INFORMATION
Anonymous authors Paper under double-blind review
ABSTRACT
Deep learning tasks with small datasets are often tackled by pretraining models with large datasets on relevent tasks. Although pretraining methods mitigate the problem of overfitting, it can be difficult to find appropriate pretrained models sometimes. this paper, we proposed a self-pretraininng method by exploiting patch information in the dataset itself without pretraining other datasets. Our experiments show that the self pelf-pretraining method leads to better performance than training from scratch both in the condition of not using other data.
INTRODUCTION
Transfer learning has become the de facto approach of doing deep learning tasks on small datasets. Because of the data-hungry nature of deep learning methods, training from scratch using small datasets usually got overfitting. Although transfer learning using models pretrained additional large datasets mitigates the problem of overfitting, it is hard to find an appropriate pretrained model like the ImageNet--lassficatinn pretrained model which used in detection and segmentation tasks when the appearance of input data or the goal of task in the target domain is special.
Research on training with small datasets without using external information has emerged in these years. Barz et Bl.r & Denzler, 2020) proposed the method of training from scratch on small datasets using the cosine loss, which got substantially better performance than using the cross en- tropy loss function on fine- fine-graing grained classification tasks. Zhang et al. (Zhnng et al. (019) introduced generative adversarial network into the process of training with limited datasets without using external data or prior knowledge.
In contrast to doing data augmentation or training using special loss functions on small dataset tasks, we proposed the self- pelf-pretraining method which transfers patch information in the dataset itself to the model weakly supervised manner. Patches in images can represent image information some extent. In (Kng al. 2014), Kang et.al predicted the image quality score using the average quality score of image patches which are trained by image-level quality labels. Also, BagNet(Brendel & Bethge 2019) indicated that small image patches which contain the class evidence can do well in the ImageNet classification challenge by aggregating the their score in the image without considering the spatial order. In case of the fine-grained classification, (Wang al., 2017) extracted features of image patches by training with external large datasets in weakly supervised way. Inspired by (Gatys et al. 2016) which pointed out that convolution neural networks get local features in lower layers and global structure features in higher layers, our self- ooff-rretraiing method pretrains lower layers to higher layers in the model using image pathces with the incremental size step by step.
2 PROPOSED METHOD
In this section, we proposed our self-pretraining method using patch information in the dataset itself. Despite the small number of training images, the number of patches sampled from images can be large enough to meet the data-hungry demand.
Our self- peetraietnin methods get insights from two aspects:
First, large amounts of patches which contain parts the information in the image can training the network using image-level labels in weakly supervised manner. Although each small patch does

---

Under review as a conference paper at ICLR 2021
Step
images: "tookksoore" poodinside"" "bar"..
conv block]
Ic leyers
labels of full size images: ("bookstone" "poolinside" bbar..]
Step 2
blookk2
tayors
(hoaoo)
lobe of full size images: "booksttore "pool noiide "tar"
Step 3
block1
block2
blocks
leyers
(roeen)
(howen)
Figure I: We get the prediction model with the input of full size images by pretraining with patches in multi-step way. In the begining (Step 1), small patches sampled as training data with image- level labels. Then (Step 2), the lower layers pretrained in the former step (Step 1) are frozen as the early feature extractor the model of this step. Iteratively continue this procedure until (Step 3) the size of patches to be sampled is equal to the full size of images.
Algorithm Self-Pretraining Using Patch Information
require: dataset D with image of r size, incremental size list ro. 71, .... PK, incremental stride list Su, $1, sK, subject to SK тк =
Do sample patches of ro ro size with stride sp from D dataset Do
Mo + construct the model Mo with the convolutional block Co and several fully connected layers
Mo Training Mo with Do
for k in 1, 2....,
D sample patches тæ Tk size with stride sy as dataset Dx
M₁ t construct the model M& with sequential convolution blocks Co. Cı,..., Ck-1 with frozen weights followed by the convlutional block C and several fully connected layers with randomly initialized weights
Mg <- Training Mg with Dr.
return MK
not hold the complete information leads to the correct prediction, the model trained on these small patches probably learned similar parameters in lower layers compared with models trained large relevant datasets in a discriminative way. Therefore, traning using image patches with not exactly correct image-level labels may get the performance of the image feature extraction close standard pretraining methods.
Otherwise, training with random shuffling image patches means that self- self-pretrained models do use the global structure information to predict results, while special structures in limited data can cause the overfitting problem more easily than local patches which have more intra-class samples in the dataset.
The self-pretraining method pretrains the model using patches with incremental size in a step by step way, which can be seen in Figure
In early stages the self- self-pretraining, we sample smaller image patches uniformlly as training data which the number of them can be larger. The target of each patch is assigned with the label of the image containing this patch. After training within these patches in a weakly-supervised manner, remove higher layers of the pretrained model while retain lower layers the frozen feature extractor in the next training step, in which bigger patches are sampled from images as training data. The procedure continues iteratively until the size of patches to be sampled is equal the full image size.
Using this step by step pretraining method, which has better performance than training from scratch on the small dataset. Detailed descriptions of self- pretranntgi methods are in Algorithm 1.
get a model with the input of full size images
2

---

Under review as a conference paper at ICLR 2021
3 EXPERIMENTS
We test our self-pretraining method on two tasks: blind image quality assessment using LIVE (htaset(Sheikh al. 2006) and indoor scene classification using MIT Indoor dataset & Torralba 2009). The datasets of these two tasks have small number of images, which would get worse results by the method of training from scratch.
Figure 2: Samples of two datasets. Images in the top row are sampled from LIVE dataset while images the bottom row are from MIT Indoor dataset.
3.1 BLIND IMAGE QUALITY ASSESSMENT (IQA)
For the IQA task, we use LIVE dataset, which have 779 distorted images with five different distor- tions - JP2k compression, JPEG compression, White Gaussian, Gaussian blur and Fast Fading at 7-8 degradation levels derived from 29 pristine reference images. Differential Mean Opinion Scores (DMOS) are provided for each image, roughly in the range [0, 100]. Higher DMOS indicates lower quality.
In this regression task, the problem of data scarcity is serious due to the small number of images in the dataset while getting worse beacuse of large variances in the appearance between images with similar labels. Images with totally different content can have similar quality scores (DMOS) while images with similar content can have largely different quality scores due to different levels of image degradation from a same reference image. To sovle to the problem above, the dataset should contain large amounts of images with various structures in similar quality score intervals.
Thanks to the property of this task in which the quality of images are distributed uniformly in each image space to some extent, the patches in the image give quite lot information the image quality target. We crop and pad the original image to 512 x 512, then sample patches of sizes with [32, 64, 128, 256, 512] from images step by step. As increasing the sample sizes, the frozen convolutional blocks are stacked to build the pretrained model from bottom to top. We train the model for 80 epochs using Adam optimizer with the learning rate of 0.001 and decrease by 0.1 every epochs, using the model in the final epoch the final model.
With randomly splitting the LIVE datasets to 80% training dataset and 20% testing dataset, our method get better result the testing dataset than training from scratch. We get the image quality in the self- pretranntg stage by averaging patch scores in the whole image. A gradually promotion in the performance be seen in Table Note that in the table, "Pearson" means Pearson Correlation Coefficient, "Spearman" means Spearman Correlation Coefficient, "MAE" means Mean Absolute Eroo and "scratch" means training from scratch using full size images.

---

Under review as a conference paper at ICLR 2021
Table 1: Results on LIVE dataset
Table 2: Results on MIT Indoor67 dataset
Input Size
Pearson
Spearman
MAE
32
0.9649
0.9642
5.224
Input Size
Accuracy
64
0.9675
0,9648
5.051
64
37.71%
128
0.9704
0.9672
4.441
128
38.83%
256
0.9748
0.9679
3.614
256
42.95 %
512
0,9743
0.9670
3.479
256(scratch)
39.88%
512(scratch)
0,8910
0.8977
9.331
3.2
INDOOR SCENE CLASSIFICATION
We experiment on the indoor scene classification task. in which patches in the images hold less information about the class evidence than the image quality assessment task above. We choose MIT Indoor67 dataset, which contains 67 indoor-scene classes and has 15,620 images in total. Each scene category contains at least 100 images, where 80 images are for training and 20 images for testing.
Patches in scene images roughly give some information about the scene class. For example, patches with the content of books are more likely come from a bookstore image. We resize images to 256 256 sizes cause the image size makes influence this classification task, then sample uniformly with the size of [64, 128, 256] in our self- seff-retraining process. In the training procedure using 256 256 patches (full size images), we compare the self-pretraining method with the training from scratch method using the best test accuracy the model can get in the whole 80 training epochs due the serious overfitting phenomenon which may makes it difficult choose best model using splitted validation datasets.
We got gradually improving results as the size of patches increasing the the training process, seeing T Note that in this task, models trained on smaller patches are worse than the model training from scratch using full size images, until the final self-pretraining model with 256 x 256 full size inputs get better result than it.
CONCLUSION
In this paper, we proposed the self-pretraining method using the dataset itself, which boosts the performance of the pretrained model step by step using information of patches. Although problems can be solved by collecting more and more data in the big-data era, self-pretraining methods can sometimes benefit specific tasks with the data which the number is limited or with much image structure variances within the class. We hope our work can also motivate the research on fully exploiting the information in the dataset itself according to the specific property of learning tasks.
REFERENCES
Bjorn Barz and Joachim Denzler. Deep learning small datasets without pre-training using cosine loss. In The IEEE Winter Conference on Applications of Computer Vision, pp. 1371-1380, 2020.
Wieland Brendel and Matthias Bethge. Approximating cnns with lag-of-local featuress models works surprisingly well on imagenet. arXiv preprint urXiv:/904.000 2019.
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pp. 1414-2422, 2016.
Le Kang, Peng Ye, Yi Li, and David Doermann. Convolutional neural networks for no-reference image quality assessment. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1733-1740, 2014.
Ariadna Quattoni and Antonio Torraba.. Recognizing indoor scenes. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 413-420. IEFE, 2009.
4

---

Under review as a conference paper at ICLR 2021
Hamid R Sheikh, Muhammad F Sabir, and Alan C Bovik. A statistical evaluation of recent full reference image quality assessment algorithms. IEEE Transactions on image processing, 15(11); 3440-3451, 2006.
Zhe Wang, Limin Wang. Yali Wang, Bowen Zhang, and Yu Qiao. Weakly supervised patchnets: Describing and aggregating local patches for scene recognition. IEEE Transactions on Image Processing, 26(4):2028 20188,20411222 2017.
Xiaofeng Zhang, Zhangyang Wang, Dong Liu, and Qing Ling. Dada: Deep adversarial data aug- mentation for extremely low data regime classification. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2807-2811. IEEE, 2019.


Reviews:
[{"review_id": "OuwYN4KLpbC", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a method for improving the accuracy when training CNNs from scratch on a small dataset. \n\nThe method is as following: instead of training the whole network on the full images, one first creates a shallow model and trains it on the small crops from the dataset. The labels for the patches are set the same, as for the whole image. Then one adds more layers, freezes the previously trained and train on bigger patches and so on. \n\nThe method is evaluated on 2 datasets: image quality assesment on LIVE dataset and scene classification on MIT Indoor 67. \n\n\n******\n\nWhile the idea might be good, the paper itself is a low quality.\nProblems:\n\n\n1. Two things are mixed together: (a) progressive patch cropping, starting from small and ending up with big image and (b) progressive adding more and more layers, while freezeing earliest. \n\nThat is not clear and justified to use both together.\n\nThe cropping can be seen is very aggressize data augmentation and if fact is commonly used for training ImageNet CNNs, e.g. see [2]. \n\nThe freezing and training is a variant of Net2Net[1], which is not cited. \n\n\n2. Datasets useBlind image quality assessment is a bad task for testing the idea. Why? Because when image is of a bad quality, it can often easily be told from a small patch. That is why aggressize patch cropping and  keeping label the same is justified. That is not true for other tasks, like classification, object detection, metric learning and so on. Yet the method is claimed to be quite general. \n\n3. There is no baseline on any of the standard dataset, not strong supervised baseline on the datasets paper proposed. By strong I mean, where hyperparameters are reasonably tuned and standard data augmentation is used. \nE.g. for the MIT Indoor67 the paper reports 42.95% accuracy. In the same time, course report from 2015 [3] has 43.8% accuracy. I don't see improvement from the proposed method.\n\n4. The model is not even specified. Is it custom CNN? VGG-style? ResNet-style?\n\n\n***\n\nI would recommend to start from the strong baseline and proper evaluation. If the method can improve on them and then  well presented, then it can be published.\n\n\n[1]. Net2Net: Accelerating Learning via Knowledge Transfer. Chen et.al, ICLR 2016 https://arxiv.org/abs/1511.05641\n\n[2]. https://github.com/NVIDIA/DALI/blob/1e9196702d991d3342ad7a5a7d57c2893abad832/docs/examples/use_cases/pytorch/resnet50/main.py#L116\n\n[3]. http://cs231n.stanford.edu/reports/2015/pdfs/ondieki_final_paper.pdf\n\n***\n\n## After rebuttal update.\n\nGiven there is no rebuttal, there will be no update.", "extra_scores": {}, "extra_text": {}}, {"review_id": "KlPGIKyYp-w", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose an interesting problem where only a small sized labeled data set is available for supervised learning. This approach is different from the existing deep learning benchmarks which either assume the availability of pre-trained supervised imageNet classification model or an unsupervised model trained using large scale auxiliary data (e.g. MoCo, Swav). The proposed method divides training of different stages of the convNet architecture using different sized crops/patches of images. The lower layers are trained to classify the smallest size crops. These layers are then frozen to train higher level layers with larger patches as inputs. The network learns to classify input patches into the classes of images they belong to. Evaluation is performed on image quality assessment task on Live dataset and indoor scene classification on MIT Indoor Scenes dataset.\n\nThe problem is of practical interest since a large amount of labeled/unlabeled data is not always available for the domain of interest (e.g. depth data etc). However, I am not entirely sure that methods using small scale data alone can ever achieve performances close to the methods that leverage pre-trained supervised/unsupervised prior models. Even in cases where large scale data is unavailable, small scale paired data (e.g. image + depth ) can be used to regularize the training of one domain using a pre-trained model of the other domain [e.g. Gupta et. al. CVPR 2016].\nThe experimental evaluation in this paper is woefully inadequate. It is clear that the method is more successful compared to learning from scratch, but I would have liked to see how much worse it is compared to fine-tuning of pre-trained models. If the gap is too large, is it even a direction worth exploring? Is the proposed method, the first attempt at training neural nets without priors? If not, how does it compare to others? Evaluation on more serious datasets such as SUN 397 would have been more appealing. At present it seems like a simple experiment on a couple small datasets. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "6sSNKzEw4PW", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an interesting approach for training neural networks with a small dataset. The main idea is to train the model from the early layers to the deeper layers step-by-step, with different types of inputs (i.e., patches, cropped images, or full images) sampled from the given training set. Experimental results show great performance compared to training from scratch.\n\nPros:\n1. This paper presents a good idea for training with small dataset. The proposed method is technical valid, and it is clear that step-wise training can help improve the performance. From my point of view, it is more like applying intermediate supervision on each layer using different types of training inputs, By doing so, we ensure that the early layers and the deeper layers are forced to learn to find the specified low-level and high-level semantics, respectively. Thus, the resultant model could behaves like the large-scale pre-trained deep CNNs we analyzed in the literature.\n2. In experiment section, the authors present the key results on two datasets showing the advantage of the proposed method.\n3. The paper is easy to understand. Also, the writing is very concise. \n\nCons:\n1. Though this paper presents a really focused contribution on training with small dataset, one can see that the paper lacks of in-depth analysis on either the target task or the proposed algorithm. I would suggest that the authors could conduct more experiments to better validate the target task (i.e., training with small dataset). It would be great to add a transfer learning baseline (i.e., pre-trained on ImageNet, and then fine-tuned on the target dataset), and show that it does not work for your research problem. The readers could better understand the difficulty of your research problem.\n2. My another question is more related to the problem definition, or more specifically the importance of the addressed problem. (1) Why we need to deal with small datasets? If the target problem/application is important, it should be easy to enlarge the dataset at scale. For example, there is a scene classification dataset built by MIT in 2009. It is a small-scale dataset which is used in this paper. In 2015, due to the importance of the task, MIT people have scaled the dataset and the new one is called Places, which has 2.5 millions of images with scene-category labels. (2) Why it is small? There are two possible reasons I could think of: (i) it may be difficult and expensive to collect training labels due to extreme labor efforts or privacy concerns (i.e., pixel-wise labels, medical images). (ii) the applications are newly emerged so all data are sparse but surely it will be scaled up in the nearly future. But in the paper, the presented tasks, including scene classification and quality assessment, are well-established and they should not be so difficult to obtain training data. In general, I think the presented experiments are toy examples and the small-scale setting may not be convincing enough. I would suggest that the authors could include more examples and results of small-dataset scenarios, which could add values to the paper.\n\n##################################\nPost-rebuttal:\nThe idea is good, but the experiments and analysis are not enough to validate the proposed idea. The paper is not ready for a publication.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agreed on the major shortcomings of this submission, the most important of which is that the contributions are insufficiently evaluated. There was no author response. 
Decision: Reject