Federated Learning Based Data Distillation and Augmentation for Improving
the Performance of MobileNet-V2
Brijraj Singh
Indian Insitute of Technology, Roorkee
Roorkee, Uttarakhand, India

Durga Toshniwal
Indian Insitute of Technology, Roorkee
Roorkee, Uttarakhand, Indiabsingh1@cs.iitr.ac.in

durgatoshniwal@gmail.com

Abstract
Overfitting is one of the most notorious and commonlyoccurring problems of deep neural networks (DNN), whichmostly arises because of having large gape between themodel’s capacity and data availability. Increasing the training data to cope up with the model’s capacity is one ofthe popular regularization methods to solve the problem.
Additional training data can be generated using data augmentation method on available data. But there exists nomethod to guide about which samples from the available
training data should be selected for augmentation purpose.
Hence, randomly selecting samples is the convention. In
this work, we have proposed a federated learning-based
data distillation method, which provides the most appropriate samples whose augmentation improves the performanceof MobileNet-V2 better than that of randomly selected samples.

1. Introduction
DNNs are known as data-hungry networks because of
demanding huge data in order to learn the patterns with decent generalization performance. If DNNs are applied withlimited data, it leads to the problem of overfitting. Hence,the predicting power of DNN is found restricted in the applications which lack in heuristic data. Artificially generating the data and consequently satisfying the high capacity
DNN model is one of the regularization method used forbetter generalization performance. There have been coupleof work on artificial data generation and regularization ([1],[2], [3], [4], [5], [6], [7], [8], [9]). The artificial data generation using data distillation technique is the key conceptof this work. Term data distillation is used for segregating
the data samples in two categories: (a). Clear data points(b). Crucial data points. Data samples that reside close
to the class decision boundaries are referred as crucial data
points and data points that reside away from the class de-cision boundaries are known as clear data points. The paper proposes a novel method of data distillation by exploiting the concept of federated learning performed using tinymodels. The dataset is divided among several disjoint andthe definite number of data chunks. The model trained oneach data chunk is referred as tiny model. Each tiny model
is supposed to have the loosely fitted hyperplane built using partial observations of dataset (for instance one-tenth of
total data). Since tiny models see only the partial dataset
therefore, they are expected to be weak towards crucial data
points. Based on this idea all the tiny models are exposed
to each sample of training data and score vector (of length10) is calculated to identify the clean, crucial data points
by setting a threshold on the score. The work focuses on
crucial data points as these are the points which actuallycontribute in making the fine decision boundaries. Hence,
if crucial data points are understood well by the model thenit will help in improving the model’s performance. Hence,the present work generates the augmented data from the crucial data points and include them in the actual dataset. This
inclusion aspires to satisfy the model’s requirement of minimum data samples in order to build the most appropriate
decision boundary without any loss of generality.
The novelty of the work can be highlighted as:(1). This paper proposes a novel data distillation method
for segregating the data in two parts based on their role inbuilding the model. (2). In case of training the deep neuralnetwork with the restricted amount of data, the proposed
method helps in taking the decision about which points
should be used for data augmentation.

2. Proposed Method:
The proposed method can be categorized in following
subsections and explained in Algorithm 1.
Knowledge distribution This step uses a hyperparameter~T r among kk and distributes the whole training dataset [D]~~~number of disjoint data chunks [D]T r 1 , [D]T r 2 ...[D]T r k.
Then a model is trained corresponding to each data chunk~Algorithm 1 Distillation based Augmentation([D],M N etV 2 : M, K : Datachunks Count,θ : Threshold)

3. Results and Conclusion~ = M1 , M2 , ...Mk in a way that each modelsuch that M
sees only the samples belonging to the same datachunk.
Pruning This process takes into account all the tiny modelsof last phase and they are all exposed with entire training
data. Each training sample is given to all the trained models
for prediction. This gives a matrix with rows corresponding to samples of training data, where each row is a vectorof size k and k is the number of prediction results. The
matrix of the predicted value is than compared against thetrue label and the corresponding score is calculated. The
score achieved by each sample uses threshold (θ) to decide
whether the data sample is a clear sample or crucial sample.
Augmentation All the data samples which are categorized
as crucial samples are the points which reside close to thedecision boundary. In order to provide the general understanding about these points to the model, the points aretransformed with rotation, shifting hence, new samples aregenerated.
Assimilation The data samples generated in the previous
step by augmentation are appended with the training data.
The method aspires to provide more general training by understanding each crucial data point and adjusting the decision boundaries accordingly.

References

The proposed work uses a novel method for data distillation and showcases its performance on four benchmark~ T R ], [D~ T E ] = [D]~datasets (Table: 1). The method focuses on the qualita1: [D~ T R ])tive aspect of data points, hence only specific data points
2: l = length([D(crucial) are selected for augmentation purpose. To judge
3: for i = 0 to k do~
the quality of each data sample, a distillation based pruning4:
data set[i] = (DT R [l/k ∗ i : l/k ∗ (i + 1)])
method is used for filtering out the points, which contribute5:
Mi = T rain(M, data set[i])~ T R .data)
less effectively in the formation of the decision boundary.
6:
resultsi = Mi (DThe results show improvements after adding the distilled,
7: end for~ = results0 ||results1 ||...||resultsk−1
augmented samples in the dataset. This work utilizes two
8: Res~ T R .labels, Res)
~augmenting techniques, 1. Shifting sh : 0.2 2. Rotation
9: scores~= F inding Scores(D~ T R .data, D~ T R .labels) α : 45◦ . The values of these two parameters are randomly
10: P runed data = P runing(scores,~θ, Dtaken as sh ∈ (0, 0.2), θ ∈ (0, 45)◦ . The proposed method
11: P A Data = Augmentation(P runed data, Angle, Shif t)~ T R ] || P A Data
generates one augment image against each selected crucial
12: N ew T rain Data = [D~ T R ]||[N ew T rain~ Data])
image. In this work, total number of data chunks and cor13: Mnew = T rain(M, [D~ T R ])responding trainable models are fixed as 10 and pruning
14: Morig = T rain(M, [D~ T E ])threshold (θ) is considered as 9.
15: Aorig = Morig ([D~ T E ])Resources used Intel Xeon CPU, 11 GB frame buffer GPU.
16: Anew = Mnew ([DAcknowledgement Thanks to Vishal Keshav, Sharan Ku17: return (Anew , Aorig )mar Allur and SNAP team from SRIB, India.

Augmentation performance
Data Size
Acc (in %)(in %)
Random Proposed
MethodMethodCIFAR1090.49
158.75
91.86
92.21
CIFAR100
65.42
194.2
68.15
69.26
CALTECH101
62.67
177.75
65.7
67.41
CALTECH256
35.8
194.7
40.8
41.77
Table 1. Augmented results in random and proposed method.
DatasetOriginalAcc(in %)[1] Y. Gal and Z. Ghahramani. A theoretically grounded application of dropout in recurrent neural networks. In Advancesin neural information processing systems, pages 1019–1027,
2016.
[2] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learningfor image recognition. In Proceedings of the IEEE conferenceon computer vision and pattern recognition, pages 770–778,
2016.
[3] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097–1105, 2012.
[4] M. Kulkarni, K. Patil, and S. Karande. Knowledge distillation using unlabeled mismatched images. arXiv preprint
arXiv:1703.07131, 2017.
[5] I. Radosavovic, P. Dollár, R. Girshick, G. Gkioxari, and K. He.
Data distillation: Towards omni-supervised learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4119–4128, 2018.
[6] H. Scudder. Probability of error of some adaptive patternrecognition machines. IEEE Transactions on Information
Theory, 11(3):363–371, 1965.
[7] T. Simon, H. Joo, I. Matthews, and Y. Sheikh. Hand keypoint
detection in single images using multiview bootstrapping. In
Proceedings of the IEEE Conference on Computer Vision andPattern Recognition, pages 1145–1153, 2017.
[8] X. Wang, A. Shrivastava, and A. Gupta. A-fast-rcnn: Hard
positive generation via adversary for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2606–2615, 2017.
[9] L. Xie, J. Wang, Z. Wei, M. Wang, and Q. Tian. Disturblabel: Regularizing cnn on the loss layer. In Proceedings of theIEEE Conference on Computer Vision and Pattern Recognition, pages 4753–4762, 2016.