# Multisource Mobile Transfer Learning Algorithm Based on Dynamic Model Compression

Authors: Peng Gao; Jingmei Li; Changhong Ding
Journal: Security and Communication Networks
 (2022)

---

## Abstract

With the development of the Internet of Things, the application of computer vision on mobile phones is becoming more and more extensive and people have higher and higher requirements for the timeliness of the recognition results returned and the processing capabilities of the mobile phone for image recognition. However, the processing capability and storage capability of the user terminal equipment cannot meet the needs of identifying and storing a large number of pictures, and the data transmission process will cause high energy consumption of the terminal equipment. At the same time, multisource deep transfer learning has outstanding performance in computer vision and image classification. However, due to the huge amount of calculation of the deep network model, it is impossible to use the existing excellent network model to realize image recognition and classification on the mobile terminal. In order to solve the abovementioned problems, we propose a multisource mobile transfer learning algorithm based on dynamic model compression, this algorithm considers the realization of multisource transfer learning computing in the case of multiple mobile device computing source domains, and the method also guarantees data privacy and security for each device (origin domain). Meanwhile, extensive experiments show that our method can achieve remarkable results in popular image classification datasets.


---

## Body

## 1. Introduction

Machine learning can achieve good results in computer vision, often based on the following assumptions: there are enough data samples in the training dataset to train a high-precision classifier and training data and test data come from the same feature space and the same distribution. However, obtaining sufficient labeled data for practical applications is often expensive and time-consuming. In this case, transfer learning [1] is a promising approach. It transfers knowledge from the tagged source domain to the target domain. At the same time, the emergence of convolutional neural networks has also accelerated the technical level of transfer learning models. Transfer learning generally assumes that training and testing data come from similar but different distributions [2]. For example, images of objects taken at different angles, backgrounds and lighting may result in different edge or conditional distributions. By observation, existing transfer learning methods mainly focus on distribution adaptation to alleviate the domain gap through distribution adaptation. For example, several unsupervised transfer learning methods [3–5] incorporate maximum mean difference loss into neural networks to reduce domain differences; other models introduce different learning modes to align source and target domains, including aligning second-order correlations [6, 7]. In practical applications, we are often faced with multiple source domains. Common multisource transfer learning [8] maps the data of these two domains into a common feature space, which describes the invariant features of the source and target domains by minimizing the difference in domain distribution [5, 9–12]. With the development of deep learning, many scholars have proposed transfer learning models based on deep learning [13–17].At present, the global Internet of Things has entered the third wave of development. The mobile terminal generates a large amount of data. In the traditional computing architecture, the data needs to be centrally transmitted to the cloud for processing, which undoubtedly increases the network load and data transmission time. Therefore, mobile computing came into being. Processing deep learning on mobile devices has natural advantages over cloud computing. The entire workflow does not need to upload data to the cloud, which can run offline while avoiding the privacy risks caused by data transmission and off-site storage in the cloud computing process. Although deep learning models have excellent performance, the convolutional layers in CNNs consume a lot of computational and energy resources, posing severe challenges to devices. AlexNet [19], VGG [20], GoogleNet [21], and ResNet [22] are all excellent algorithm models of ILSVRC, but these models usually have tens of millions of parameters and require hundreds of megabytes or even 1G of memory. For multisource learning, multiple boxing neural networks are usually required to work together, so deploying multisource deep learning models on mobile devices is a very serious challenge, regardless of the amount of computation and memory overhead. GoogleNet and ResNet also have to face this problem.Deploying multisource transfer learning on mobile devices faces two challenges: (1) How to effectively use mobile devices to deploy transfer learning models to mobile device side. (2) The model and calculation volume of multisource transfer learning are often very large and how to use the advantages of the cloud and mobile terminals to achieve the purpose of optimizing computing.In this paper, a novel multisource mobile transfer learning algorithm based on dynamic model compression is proposed by combining the advantages of mobile computing, convolutional neural networks, and multisource transfer learning.(Multi-source Mobile Transfer Learning Algorithm Based on Dynamic Model Compression, MMTLDMC). MMTLDMC first performs BN pruning on the convolutional neural network. The classification loss and MMD loss is then computed on the device side (source domain). The classifier alignment loss is then calculated server-side. Then, the parameters are updated by minimizing the objective function. Finally, the model results are obtained, and the data classification is completed.Compared with previous work, the contributions of this work are as follows:(1)
Different from the previous multisource transfer learning algorithms, we consider the needs of mobile computing and propose a new multi-source transfer learning algorithm MMTLDMC algorithm, which combines the advantages of both to accelerate the multisource transfer learning model.(2)
Use data on a device as a source domain. For samples on multiple devices, calculations are only performed on the device side, and the server and device side only synchronize parameters and models, taking into account the security of device data.(3)
Experiments on real datasets show that the proposed algorithm outperforms or at least comparable to state-of-the-art benchmark algorithms in classification accuracy. At the same time, the speed is much better than the existing transfer learning algorithm.The rest of the paper is organized as follows: Section2 reviews related work on convolutional neural network pruning, mobile computing, and multisource transfer learning; In Section 3, a multisource mobile transfer learning algorithm based on dynamic model compression is proposed; Section 4 verifies the effectiveness of the algorithm on SVHN, USPS, MINIST, Office-31, Caltech-256, and DomainNet; Section 5 summarizes the main work of this paper.


## 2. Brief Review of Related Work

### 2.1. Convolutional Neural Network Pruning and Mobile Computing

The powerful feature sampling performance of convolutional neural networks comes from a large number of parameters and a complex multilayer structure. Modern convolutional neural networks are also deepening and widening the classical structure to improve the accuracy. For example, VGG [20] increases the number of layers from 8 to 19 on the basis of AlexNet [21]; GoogleNet [22] is increased to 22 layers, adopts inception structure design, and uses average pooling to replace the full connection layer. ResNet [23] only learns the relationship between the residual and the input through the residual identity mapping, and builds a model that is easier to optimize. The depth has also developed from 152 layers of the standard to thousands of layers. The traditional pruning method is divided into three steps: baseline training, pruning, and fine tuning. In terms of pruning granularity, it can be divided into two categories: unstructured pruning [24] and structured pruning. Unstructured pruning means directly pruning individual weights. The weight matrix formed is sparse, which is not easy to achieve the compression and acceleration effect of general and easy deployment. Structured pruning refers to pruning for convolution core [25], channel [26, 27] or layer [28]. In the direction of convolution kernel pruning, literature [29] improved the pruning strategy based on convolution kernel weight ranking, and adopted the sum of the absolute values of the regular term L1 of convolution kernel as the pruning weight. Layer pruning needs to cut the complete layer with a poor flexibility and high precision risk. It is generally used to cut the deep network structure [30]. Channel pruning is one of the most studied and widely used structural pruning methods. Reference [31] proposed a channel pruning method for BN layer for model compression. By using the weight of BN layer to evaluate the score of input channels, the author sets a threshold to filter out the channels with low scores. When connecting, the neurons of these channels with too small scores do not participate in the connection, and then prune layer by layer. The pruning technology of convolutional neural network reduces the actual deployment of the model in the mobile terminal. Figure 1 shows the pruning model of BN channel factor.Figure 1
Pruning model of the BN channel factor.


### 2.2. Multisource Transfer Learning

Multisource transfer learning as a research direction of transfer learning has very important practical values. In the process of real life and practical application, there are often multiple source domains. Although each source domain has a different similarity with the target, these source domains can still be used for knowledge transfer. Moreover, multisource transfer learning contains more knowledge, which can make the effect of the model better. At the same time, transfer learning also has a theoretical basis. Crammer [28] first proposed the expected loss boundary condition of multisource transfer learning. Later, Mansour [29] proved that the distribution weighted combination rule can reduce the instantaneous function between the source domain and the target domain. Ben-David [30] gave two learning boundaries of minimizing empirical risk by introducing the distance between the target domain and the source domain.In recent years, a lot of work has been centered on multisource transfer learning and deep learning. Zhuang et al. [31] proposed the Deep Cocktail Network (DCTN), which uses a single domain discriminator and a classifier for each source domain and target domain. The domain discriminator is used to align the feature distribution, and the classifier outputs the predicted probability distribution. Based on the output of the domain discriminator, DCTN designed a method of voting by multiple classifiers. Carmmer et al. [32] proposed a moment matching multisource domain adaptation (M3SDA) method, which not only considers the alignment between the source domain and the target domain but also aligns the feature distribution of different source domains. Zhu proposed a framework named aligning domain-specific distribution and classifier for cross-domain classification from multiple sources (MFSAN) [33]. However, the current deep multisource transfer learning algorithms often only consider marginal probability distribution or consider the marginal probability distribution and the conditional probability distribution separately. In this paper, multisource transfer learning is based on balanced distribution adaptation, which considers the joint probability distribution to improve the accuracy of the algorithm. However, the multisource deep transfer learning algorithm is not combined with mobile computing because of its large amount of computation. This paper proposes a multisource mobile transfer learning algorithm based on dynamic model compression, which aims to train the multisource transfer model on the mobile terminal.


### 2.3. Problem

In mobile computing or edge computing, we can take the data collected by each device as a source domain, but for privacy and security reasons, the data of each device cannot be completely uploaded to the server for model construction. According to this restriction, we redefine multisource transfer learning. In multisource transfer learning, there areN source domains(clients), and their labeled sample data can be represented as XCsi=xjsi,yjsi,ejsij=1Ni, where xjsij=1Ni represents the i-th sample data in the j-th source domain, and yjsij=1Ni represents the i-th source domain in the j-th source domain, ejsij=1Ni∈0,1 represents whether the j-th sample tag in the i-th source domain can be synchronized to the server, 0 indicates that it cannot be synchronized, and 1 indicates that it can be synchronized. The joint probability distribution of N different domain can be expressed as Psix,yi=1Ni , where the marginal probability can be expressed as Psixi=1Ni, and the conditional probability can be expressed as Psiy|xi=1Ni. Similarly, we give the definition of the target domain, the sample of the target domain can be expressed as Xt=xjtj=1Nt, and the probability distribution can be expressed as Ptx,y. This paper mainly considers the problem that the source domain data cannot be shared with the server, but the target domain data can be shared with the server which means XCsi=xjsi,yjsi,0j=1Ni, Xt=xjt,1j=1Nt.In recent years, some papers have defined the objective function of multisource deep transfer learning. They first map all domains to the same target space, then use the common domain invariant representation in the common feature space for learning all domains. Zhu [33] gave a definition of the loss function:(1)minF,C∑i=1NΕx∼XsiJCiHiFxjsi,yjsi+λ∑i=1ND∧HiFXsi,FXt+γLdisc.The first term represents the loss of the classification function, the general classification loss is the cross-entropy loss, and the second term represents the statistical measurement of the source domain and the target domain. Nowadays, the commonly used metrics are MMD [34, 35] loss, CORAL loss [5], and confusion loss [14, 15]. The third item, Zhu defines it as a specific difference loss. At present, there is no relevant method to consider the combination of multisource transfer learning and mobile computing. Most of these algorithms have high requirements for GPU and CPU of computing devices and cannot be calculated on the mobile end. At the same time, due to the concept of user privacy, it is very unfriendly to synchronize user data to the cloud. Therefore, how to realize the calculation of a large-scale model on the mobile terminal, such as multisource migration learning, is our concern.In order to solve the above problems, we first map multiple source domains and target domains into the same subspace, and then prune the BN channel of the convolutional neural network in the space to reduce the amount of computation at the mobile end. Then, in order to ensure data security, we calculate the classification loss and MMD loss at the mobile terminal. Then, the calculated loss is synchronized to the server, and the server calculates the loss of the alignment classifier. Finally, we optimize the parameters by minimizing the loss function to obtain the optimal solution.


## 2.1. Convolutional Neural Network Pruning and Mobile Computing

The powerful feature sampling performance of convolutional neural networks comes from a large number of parameters and a complex multilayer structure. Modern convolutional neural networks are also deepening and widening the classical structure to improve the accuracy. For example, VGG [20] increases the number of layers from 8 to 19 on the basis of AlexNet [21]; GoogleNet [22] is increased to 22 layers, adopts inception structure design, and uses average pooling to replace the full connection layer. ResNet [23] only learns the relationship between the residual and the input through the residual identity mapping, and builds a model that is easier to optimize. The depth has also developed from 152 layers of the standard to thousands of layers. The traditional pruning method is divided into three steps: baseline training, pruning, and fine tuning. In terms of pruning granularity, it can be divided into two categories: unstructured pruning [24] and structured pruning. Unstructured pruning means directly pruning individual weights. The weight matrix formed is sparse, which is not easy to achieve the compression and acceleration effect of general and easy deployment. Structured pruning refers to pruning for convolution core [25], channel [26, 27] or layer [28]. In the direction of convolution kernel pruning, literature [29] improved the pruning strategy based on convolution kernel weight ranking, and adopted the sum of the absolute values of the regular term L1 of convolution kernel as the pruning weight. Layer pruning needs to cut the complete layer with a poor flexibility and high precision risk. It is generally used to cut the deep network structure [30]. Channel pruning is one of the most studied and widely used structural pruning methods. Reference [31] proposed a channel pruning method for BN layer for model compression. By using the weight of BN layer to evaluate the score of input channels, the author sets a threshold to filter out the channels with low scores. When connecting, the neurons of these channels with too small scores do not participate in the connection, and then prune layer by layer. The pruning technology of convolutional neural network reduces the actual deployment of the model in the mobile terminal. Figure 1 shows the pruning model of BN channel factor.Figure 1
Pruning model of the BN channel factor.


## 2.2. Multisource Transfer Learning

Multisource transfer learning as a research direction of transfer learning has very important practical values. In the process of real life and practical application, there are often multiple source domains. Although each source domain has a different similarity with the target, these source domains can still be used for knowledge transfer. Moreover, multisource transfer learning contains more knowledge, which can make the effect of the model better. At the same time, transfer learning also has a theoretical basis. Crammer [28] first proposed the expected loss boundary condition of multisource transfer learning. Later, Mansour [29] proved that the distribution weighted combination rule can reduce the instantaneous function between the source domain and the target domain. Ben-David [30] gave two learning boundaries of minimizing empirical risk by introducing the distance between the target domain and the source domain.In recent years, a lot of work has been centered on multisource transfer learning and deep learning. Zhuang et al. [31] proposed the Deep Cocktail Network (DCTN), which uses a single domain discriminator and a classifier for each source domain and target domain. The domain discriminator is used to align the feature distribution, and the classifier outputs the predicted probability distribution. Based on the output of the domain discriminator, DCTN designed a method of voting by multiple classifiers. Carmmer et al. [32] proposed a moment matching multisource domain adaptation (M3SDA) method, which not only considers the alignment between the source domain and the target domain but also aligns the feature distribution of different source domains. Zhu proposed a framework named aligning domain-specific distribution and classifier for cross-domain classification from multiple sources (MFSAN) [33]. However, the current deep multisource transfer learning algorithms often only consider marginal probability distribution or consider the marginal probability distribution and the conditional probability distribution separately. In this paper, multisource transfer learning is based on balanced distribution adaptation, which considers the joint probability distribution to improve the accuracy of the algorithm. However, the multisource deep transfer learning algorithm is not combined with mobile computing because of its large amount of computation. This paper proposes a multisource mobile transfer learning algorithm based on dynamic model compression, which aims to train the multisource transfer model on the mobile terminal.


## 2.3. Problem

In mobile computing or edge computing, we can take the data collected by each device as a source domain, but for privacy and security reasons, the data of each device cannot be completely uploaded to the server for model construction. According to this restriction, we redefine multisource transfer learning. In multisource transfer learning, there areN source domains(clients), and their labeled sample data can be represented as XCsi=xjsi,yjsi,ejsij=1Ni, where xjsij=1Ni represents the i-th sample data in the j-th source domain, and yjsij=1Ni represents the i-th source domain in the j-th source domain, ejsij=1Ni∈0,1 represents whether the j-th sample tag in the i-th source domain can be synchronized to the server, 0 indicates that it cannot be synchronized, and 1 indicates that it can be synchronized. The joint probability distribution of N different domain can be expressed as Psix,yi=1Ni , where the marginal probability can be expressed as Psixi=1Ni, and the conditional probability can be expressed as Psiy|xi=1Ni. Similarly, we give the definition of the target domain, the sample of the target domain can be expressed as Xt=xjtj=1Nt, and the probability distribution can be expressed as Ptx,y. This paper mainly considers the problem that the source domain data cannot be shared with the server, but the target domain data can be shared with the server which means XCsi=xjsi,yjsi,0j=1Ni, Xt=xjt,1j=1Nt.In recent years, some papers have defined the objective function of multisource deep transfer learning. They first map all domains to the same target space, then use the common domain invariant representation in the common feature space for learning all domains. Zhu [33] gave a definition of the loss function:(1)minF,C∑i=1NΕx∼XsiJCiHiFxjsi,yjsi+λ∑i=1ND∧HiFXsi,FXt+γLdisc.The first term represents the loss of the classification function, the general classification loss is the cross-entropy loss, and the second term represents the statistical measurement of the source domain and the target domain. Nowadays, the commonly used metrics are MMD [34, 35] loss, CORAL loss [5], and confusion loss [14, 15]. The third item, Zhu defines it as a specific difference loss. At present, there is no relevant method to consider the combination of multisource transfer learning and mobile computing. Most of these algorithms have high requirements for GPU and CPU of computing devices and cannot be calculated on the mobile end. At the same time, due to the concept of user privacy, it is very unfriendly to synchronize user data to the cloud. Therefore, how to realize the calculation of a large-scale model on the mobile terminal, such as multisource migration learning, is our concern.In order to solve the above problems, we first map multiple source domains and target domains into the same subspace, and then prune the BN channel of the convolutional neural network in the space to reduce the amount of computation at the mobile end. Then, in order to ensure data security, we calculate the classification loss and MMD loss at the mobile terminal. Then, the calculated loss is synchronized to the server, and the server calculates the loss of the alignment classifier. Finally, we optimize the parameters by minimizing the loss function to obtain the optimal solution.


## 3. Multisource Mobile Transfer Learning Algorithm Based on Dynamic Model Compression

In order to solve the impact of mobile computing with memory and CPU on the existing multisource transfer learning algorithms. In this chapter, we introduce the multisource mobile transfer learning algorithm based on dynamic model compression. We use the model compression strategy proposed in literature [31] to compress the deep learning model.


### 3.1. Algorithm Structure

Our algorithm structure consists of two parts. The first part is the server side, including the calculation of minimization loss function and parameter update; The second part is the client (source domain), which mainly calculates the loss function and updates the parameters according to the results of server, as shown in Figure2.Figure 2
Framework of multisource mobile transfer learning algorithm based on dynamic model compression (MMTLDMC). Our model consists of two parts: (i) sever: calculating loss function and updating parameters and (ii) client: calculating MMD loss and Classification loss.


#### 3.1.1. Preparation-BN Channel Pruning

We extract the source domain features into the same feature space in the form of shared parameters. Before extraction, we prune the BN channel of the feature extraction network according to the previously trained migration learning model. The purpose here is to reduce the parameters of the mobile terminal network model. We prune according to the literature [31], and the main method is to directly use the gamma parameters of BN layer for pruning evaluation.The loss function after introducing the channel factor of BN layer is as follows:(2)Lbn==∑x,ylfx,W,y+λ∑γ∈Γgγ,where x,y represents training data and labels, and W represents parameters. The first half is CNN’s original loss function, and the second half is the introduced penalty term. λ is the sparsity factor used to balance the formula term.The mean and variance of BN activation values are calculated as follows:(3)μ=1m∑i=1mxi,σ2=1m∑i=1mxi−μ2.Then, the calculation process of BN layer output can be expressed as(4)z^=zin−μσ2+ε,zout=γz^+β.Among themλ And β Are BN layer linear transformation parameters that can be trained. We refer to the method given in reference [31] to pretrain and fine tune the existing RESNET network. The initial convolution neural network parameters are formed.Client: In order to ensure the data security of the client (source domain), we separate the loss computation of the multisource transfer learning objective function proposed in [17]. Put the classification loss and MMD loss into the client side to calculate. A set of images were given: xjs from source domain XCs=xjs,yjs and a set of images xt from target domain xjt. The features of these specific fields are mapped to the same feature space through a common feature extractor. Some are specifically expressed as source domain mapping feature fxjs, target domain mapping characteristics fxt. Therefore, we can get N feature extractor h⋅ corresponding to specific source domains xjs,yjs. We use the pruned convolutional neural network as our classifier, we define Ci as the classifier of N source domains. According to experience, our classification loss is crossing entropy loss, and the loss function is J⋅,⋅.Server: From Figure2, we can see that the server needs to calculate the alignment loss of domain-specific classifiers proposed in the literature [17]. At the same time, on server side, we need to calculate the minimization objective function and the nonshared parameters of the neural network in each client.


#### 3.1.2. Objective Function

According to Figure2, we define the final objective function of the algorithm as(5)L=Lcls+λLmmd+γLdisc,L=Lclssever+λLmmdseverb+γLdiscsever=∑Lclient−i+γLdiscsever.Classification lossLcls-the loss caused by a specific domain classifier, according to Figure 2, we can see that the variable xj in source domain(client) i undergoes a three-step transformation, first get FxjClient−i through the public feature extractor, then get HiFxjClient−i through domain-specific feature extractor, and finally get CiHiFxjClient−i through the CNN classification after pruning. The final loss of the i-th client classification is:(6)LclsClient−i=Εx∼XClient−iJCiHiFxjClient−i,yjClient−i,Lclssever=∑i=1NLclsClient−i=∑i=1NΕx∼XClient−iJCiHiFxjClient−i,yjClient−i.MMD lossLmmd-specific domain classification loss, maximum mean discrepancy (MMD) is a commonly used method to estimate the difference of distribution measurement. It is a commonly used two sample test method in statistics. From two probability distributions p and q, first assume p=q, and then we can decide to accept or reject this hypothesis according to the results calculated or observed by MMD. Generally speaking, we can measure the difference between the two distributions according to the value of MMD.(7)dk2p,q≔Εx∼pϕx−Εx∼qϕxH2,where H is the reproducing kernel Hilbert space (RKHS) endowed with a characteristic kernel k. Here, ϕ· denotes some feature map to map the original samples to RKHS and the kernel k means kxi,xj=ϕxi,ϕxj,where ⋅,⋅ represents inner product of vectors. The main theoretical result is that p=q if and only if dk2p,q=0 . In practice, an estimate of the MMD compress the square distance between the empirical kernel embedding is obtained, that is, the MMD loss of the client.(8)D∧Hp,q=1ns∑xa∈Xsϕxa−1nt∑xb∈XtϕxbH2.Equation (8) defines the difference estimation between single source domain and target domain. Therefore, we can give the MMD loss on the server side.(9)Lmmdserver=1N∑i=1ND^BiFXsi,FXt.Classifier alignment lossLdisc-The target samples near the class boundary are more likely to be misclassified by the classifiers learned from the source samples. The classifier is trained in different source domains, so there may be differences in the prediction of target samples, especially near the class boundary. Intuitively, the same target samples predicted by different classifiers should get the same prediction. Therefore, we implement the classifier alignment strategy between domains on the server. Inspired by reference [17], we define the server side and the classifier alignment strategy as follows:(10)Ldiscserver=2N×N−1∑j=1N∑j=1NΕx∼XtCclient−iHclient−iFxk−Cclient−jHclient−jFxk.The final objective function is(11)Lclient−i=Lclsclient−i+λLmmdclient−iLsever=Lclssever+λLmmdsever+γLdiscsever=∑i=1NLclient−i+γLdiscsever=∑i=1NLclsclient−i+λN∑i=1NLmmdclient−i+γLdiscsever.In summary, the specific process steps of MMTLDMC algorithm are shown in Algorithm1θ0G, θℒ is the model parameter.Algorithm 1: MMTLDMC algorithm training MMTLDMC steps.
Sever:(1)
initializeθ0G with the BN pruning model(2)
for each roundr=1,2,3,…,dο(3)
for each clienti=1,2,3,…,dο(4)
ℒiClient←clientℒck+ℒmmd(5)
end for(6)
calculateℒdisc(7)
ℒsever←min∑ℒclient+λℒdisc(8)
Updateθ0G, θℒi by minimizing ℒsever(9
)end forClient: Run on client(1)
Updateθℒi by sever(2)
for each local epoch do(3)
CalculateLclient=Lcls+Lmmd(4)
end for(5)
returnℒclient


## 3.1. Algorithm Structure

Our algorithm structure consists of two parts. The first part is the server side, including the calculation of minimization loss function and parameter update; The second part is the client (source domain), which mainly calculates the loss function and updates the parameters according to the results of server, as shown in Figure2.Figure 2
Framework of multisource mobile transfer learning algorithm based on dynamic model compression (MMTLDMC). Our model consists of two parts: (i) sever: calculating loss function and updating parameters and (ii) client: calculating MMD loss and Classification loss.


### 3.1.1. Preparation-BN Channel Pruning

We extract the source domain features into the same feature space in the form of shared parameters. Before extraction, we prune the BN channel of the feature extraction network according to the previously trained migration learning model. The purpose here is to reduce the parameters of the mobile terminal network model. We prune according to the literature [31], and the main method is to directly use the gamma parameters of BN layer for pruning evaluation.The loss function after introducing the channel factor of BN layer is as follows:(2)Lbn==∑x,ylfx,W,y+λ∑γ∈Γgγ,where x,y represents training data and labels, and W represents parameters. The first half is CNN’s original loss function, and the second half is the introduced penalty term. λ is the sparsity factor used to balance the formula term.The mean and variance of BN activation values are calculated as follows:(3)μ=1m∑i=1mxi,σ2=1m∑i=1mxi−μ2.Then, the calculation process of BN layer output can be expressed as(4)z^=zin−μσ2+ε,zout=γz^+β.Among themλ And β Are BN layer linear transformation parameters that can be trained. We refer to the method given in reference [31] to pretrain and fine tune the existing RESNET network. The initial convolution neural network parameters are formed.Client: In order to ensure the data security of the client (source domain), we separate the loss computation of the multisource transfer learning objective function proposed in [17]. Put the classification loss and MMD loss into the client side to calculate. A set of images were given: xjs from source domain XCs=xjs,yjs and a set of images xt from target domain xjt. The features of these specific fields are mapped to the same feature space through a common feature extractor. Some are specifically expressed as source domain mapping feature fxjs, target domain mapping characteristics fxt. Therefore, we can get N feature extractor h⋅ corresponding to specific source domains xjs,yjs. We use the pruned convolutional neural network as our classifier, we define Ci as the classifier of N source domains. According to experience, our classification loss is crossing entropy loss, and the loss function is J⋅,⋅.Server: From Figure2, we can see that the server needs to calculate the alignment loss of domain-specific classifiers proposed in the literature [17]. At the same time, on server side, we need to calculate the minimization objective function and the nonshared parameters of the neural network in each client.


### 3.1.2. Objective Function

According to Figure2, we define the final objective function of the algorithm as(5)L=Lcls+λLmmd+γLdisc,L=Lclssever+λLmmdseverb+γLdiscsever=∑Lclient−i+γLdiscsever.Classification lossLcls-the loss caused by a specific domain classifier, according to Figure 2, we can see that the variable xj in source domain(client) i undergoes a three-step transformation, first get FxjClient−i through the public feature extractor, then get HiFxjClient−i through domain-specific feature extractor, and finally get CiHiFxjClient−i through the CNN classification after pruning. The final loss of the i-th client classification is:(6)LclsClient−i=Εx∼XClient−iJCiHiFxjClient−i,yjClient−i,Lclssever=∑i=1NLclsClient−i=∑i=1NΕx∼XClient−iJCiHiFxjClient−i,yjClient−i.MMD lossLmmd-specific domain classification loss, maximum mean discrepancy (MMD) is a commonly used method to estimate the difference of distribution measurement. It is a commonly used two sample test method in statistics. From two probability distributions p and q, first assume p=q, and then we can decide to accept or reject this hypothesis according to the results calculated or observed by MMD. Generally speaking, we can measure the difference between the two distributions according to the value of MMD.(7)dk2p,q≔Εx∼pϕx−Εx∼qϕxH2,where H is the reproducing kernel Hilbert space (RKHS) endowed with a characteristic kernel k. Here, ϕ· denotes some feature map to map the original samples to RKHS and the kernel k means kxi,xj=ϕxi,ϕxj,where ⋅,⋅ represents inner product of vectors. The main theoretical result is that p=q if and only if dk2p,q=0 . In practice, an estimate of the MMD compress the square distance between the empirical kernel embedding is obtained, that is, the MMD loss of the client.(8)D∧Hp,q=1ns∑xa∈Xsϕxa−1nt∑xb∈XtϕxbH2.Equation (8) defines the difference estimation between single source domain and target domain. Therefore, we can give the MMD loss on the server side.(9)Lmmdserver=1N∑i=1ND^BiFXsi,FXt.Classifier alignment lossLdisc-The target samples near the class boundary are more likely to be misclassified by the classifiers learned from the source samples. The classifier is trained in different source domains, so there may be differences in the prediction of target samples, especially near the class boundary. Intuitively, the same target samples predicted by different classifiers should get the same prediction. Therefore, we implement the classifier alignment strategy between domains on the server. Inspired by reference [17], we define the server side and the classifier alignment strategy as follows:(10)Ldiscserver=2N×N−1∑j=1N∑j=1NΕx∼XtCclient−iHclient−iFxk−Cclient−jHclient−jFxk.The final objective function is(11)Lclient−i=Lclsclient−i+λLmmdclient−iLsever=Lclssever+λLmmdsever+γLdiscsever=∑i=1NLclient−i+γLdiscsever=∑i=1NLclsclient−i+λN∑i=1NLmmdclient−i+γLdiscsever.In summary, the specific process steps of MMTLDMC algorithm are shown in Algorithm1θ0G, θℒ is the model parameter.Algorithm 1: MMTLDMC algorithm training MMTLDMC steps.
Sever:(1)
initializeθ0G with the BN pruning model(2)
for each roundr=1,2,3,…,dο(3)
for each clienti=1,2,3,…,dο(4)
ℒiClient←clientℒck+ℒmmd(5)
end for(6)
calculateℒdisc(7)
ℒsever←min∑ℒclient+λℒdisc(8)
Updateθ0G, θℒi by minimizing ℒsever(9
)end forClient: Run on client(1)
Updateθℒi by sever(2)
for each local epoch do(3)
CalculateLclient=Lcls+Lmmd(4)
end for(5)
returnℒclient


## 3.1.1. Preparation-BN Channel Pruning

We extract the source domain features into the same feature space in the form of shared parameters. Before extraction, we prune the BN channel of the feature extraction network according to the previously trained migration learning model. The purpose here is to reduce the parameters of the mobile terminal network model. We prune according to the literature [31], and the main method is to directly use the gamma parameters of BN layer for pruning evaluation.The loss function after introducing the channel factor of BN layer is as follows:(2)Lbn==∑x,ylfx,W,y+λ∑γ∈Γgγ,where x,y represents training data and labels, and W represents parameters. The first half is CNN’s original loss function, and the second half is the introduced penalty term. λ is the sparsity factor used to balance the formula term.The mean and variance of BN activation values are calculated as follows:(3)μ=1m∑i=1mxi,σ2=1m∑i=1mxi−μ2.Then, the calculation process of BN layer output can be expressed as(4)z^=zin−μσ2+ε,zout=γz^+β.Among themλ And β Are BN layer linear transformation parameters that can be trained. We refer to the method given in reference [31] to pretrain and fine tune the existing RESNET network. The initial convolution neural network parameters are formed.Client: In order to ensure the data security of the client (source domain), we separate the loss computation of the multisource transfer learning objective function proposed in [17]. Put the classification loss and MMD loss into the client side to calculate. A set of images were given: xjs from source domain XCs=xjs,yjs and a set of images xt from target domain xjt. The features of these specific fields are mapped to the same feature space through a common feature extractor. Some are specifically expressed as source domain mapping feature fxjs, target domain mapping characteristics fxt. Therefore, we can get N feature extractor h⋅ corresponding to specific source domains xjs,yjs. We use the pruned convolutional neural network as our classifier, we define Ci as the classifier of N source domains. According to experience, our classification loss is crossing entropy loss, and the loss function is J⋅,⋅.Server: From Figure2, we can see that the server needs to calculate the alignment loss of domain-specific classifiers proposed in the literature [17]. At the same time, on server side, we need to calculate the minimization objective function and the nonshared parameters of the neural network in each client.


## 3.1.2. Objective Function

According to Figure2, we define the final objective function of the algorithm as(5)L=Lcls+λLmmd+γLdisc,L=Lclssever+λLmmdseverb+γLdiscsever=∑Lclient−i+γLdiscsever.Classification lossLcls-the loss caused by a specific domain classifier, according to Figure 2, we can see that the variable xj in source domain(client) i undergoes a three-step transformation, first get FxjClient−i through the public feature extractor, then get HiFxjClient−i through domain-specific feature extractor, and finally get CiHiFxjClient−i through the CNN classification after pruning. The final loss of the i-th client classification is:(6)LclsClient−i=Εx∼XClient−iJCiHiFxjClient−i,yjClient−i,Lclssever=∑i=1NLclsClient−i=∑i=1NΕx∼XClient−iJCiHiFxjClient−i,yjClient−i.MMD lossLmmd-specific domain classification loss, maximum mean discrepancy (MMD) is a commonly used method to estimate the difference of distribution measurement. It is a commonly used two sample test method in statistics. From two probability distributions p and q, first assume p=q, and then we can decide to accept or reject this hypothesis according to the results calculated or observed by MMD. Generally speaking, we can measure the difference between the two distributions according to the value of MMD.(7)dk2p,q≔Εx∼pϕx−Εx∼qϕxH2,where H is the reproducing kernel Hilbert space (RKHS) endowed with a characteristic kernel k. Here, ϕ· denotes some feature map to map the original samples to RKHS and the kernel k means kxi,xj=ϕxi,ϕxj,where ⋅,⋅ represents inner product of vectors. The main theoretical result is that p=q if and only if dk2p,q=0 . In practice, an estimate of the MMD compress the square distance between the empirical kernel embedding is obtained, that is, the MMD loss of the client.(8)D∧Hp,q=1ns∑xa∈Xsϕxa−1nt∑xb∈XtϕxbH2.Equation (8) defines the difference estimation between single source domain and target domain. Therefore, we can give the MMD loss on the server side.(9)Lmmdserver=1N∑i=1ND^BiFXsi,FXt.Classifier alignment lossLdisc-The target samples near the class boundary are more likely to be misclassified by the classifiers learned from the source samples. The classifier is trained in different source domains, so there may be differences in the prediction of target samples, especially near the class boundary. Intuitively, the same target samples predicted by different classifiers should get the same prediction. Therefore, we implement the classifier alignment strategy between domains on the server. Inspired by reference [17], we define the server side and the classifier alignment strategy as follows:(10)Ldiscserver=2N×N−1∑j=1N∑j=1NΕx∼XtCclient−iHclient−iFxk−Cclient−jHclient−jFxk.The final objective function is(11)Lclient−i=Lclsclient−i+λLmmdclient−iLsever=Lclssever+λLmmdsever+γLdiscsever=∑i=1NLclient−i+γLdiscsever=∑i=1NLclsclient−i+λN∑i=1NLmmdclient−i+γLdiscsever.In summary, the specific process steps of MMTLDMC algorithm are shown in Algorithm1θ0G, θℒ is the model parameter.Algorithm 1: MMTLDMC algorithm training MMTLDMC steps.
Sever:(1)
initializeθ0G with the BN pruning model(2)
for each roundr=1,2,3,…,dο(3)
for each clienti=1,2,3,…,dο(4)
ℒiClient←clientℒck+ℒmmd(5)
end for(6)
calculateℒdisc(7)
ℒsever←min∑ℒclient+λℒdisc(8)
Updateθ0G, θℒi by minimizing ℒsever(9
)end forClient: Run on client(1)
Updateθℒi by sever(2)
for each local epoch do(3)
CalculateLclient=Lcls+Lmmd(4)
end for(5)
returnℒclient


## 4. Experimental Results

In order to test the effectiveness and generalization of the MMTLDMC algorithm, we test it on two types of image datasets which are shown in Tables1 and 2. The first type is a digital classification dataset including SVHN[36] dataset, USPS[37] dataset, and MNIST[38] dataset; the second category is image classification dataset including Office31[39] dataset and Caltech [40] dataset; the third category is network dataset named DomainNet [16].Table 1
Description of digital datasets and image datasets.
DatasetDigital datasetImage datasetNameUSPSMNISTSVHNOffice31CaltechAmazonWebcamDSLRShortUMSAWDCTable 2
Description of network datasets.
DatasetNetwork dataset (DomainNet)NameClipartInfographPaintingQuickdrawRealSketchShortclpinfpntqdrrelsktThe experiment will compare the multisource transfer learning algorithms DCTN, MFSAN, and MultiSource TrAdaBoost and give the results of the experiment under different pruning rates. For the fairness of the experiments, a 5-fold cross-validation strategy was selected for all experiments, and the experimental results of repeating this strategy twice were used as the final comparison results. In the experiment, we use the average classification accuracy [41] and recall rate of each algorithm after running 10 times as the evaluation criteria. The recall rate reflects how many positive examples in the sample are predicted correctly. The form of expression of classification accuracy and recall are defined as follows:Classification accuracy:Accuracy=x:x∈X∧fx=yx/x:x∈XRecall rate:R=FP/TP+FN×100%Among them, TP represents the number of positive samples that are correctly classified as positive, FP represents the number of negative samples that are incorrectly classified as positive, TN represents the number of negative samples that are correctly classified as negative, and FN represents the number of positive samples that are incorrectly classified as negative.X represents the target domain number test dataset,fx is the sample x-class label predicted by the classifier, and yx is the reality-class label of the sample x.


### 4.1. Digital Classification Dataset

#### 4.1.1. Dataset Introduction

Both the USPS dataset and the MNIST dataset contain handwritten digits 0″–″9″, the former is composed of 9298 16 × 16 images, and the latter is composed of 70,000 28 × 28 images. Street View House Number (SVHN) comes from Google. Each picture contains a group of Arabic numerals’ 0–9′, which contains 73257 digits and the image pixel is 32 × 32. Figure3 shows examples of USPS, MNIST, and SVHN. We can see that the distributions of USPS and MNIST are different, but they contribute the same feature space. SVHN datasets are different from their distribution and feature space. We extract 9000 images from MNIST and SVHN as two domains. Because USPS has only 9298 pictures, we regard the whole dataset as a domain. Due to the limitation of the mobile terminal, 3000 images are extracted from MNIS, SVHN, and USPS as a domain, respectively.Figure 3
Example of USPS MNIST and SVHN pictures.


#### 4.1.2. Experimental Data

In this part, we compare some multisource transfer learning algorithms such as DCTN and M3SDA MultiSource TrAdaBoost with our algorithm MMTLDMC.It can be seen from Table3 that in the three cross-domain tasks, the MMTLDMC algorithm has an accuracy of 79.87%, 97.68%, and 95.49% when the pruning rate is 90%, which is higher than the comparison algorithms DCTN, MultiSource TrAdaBoost, and M3SDA. Compared with the data without pruning, in the tasks U.M- > S, U.S- > M, M.S- > U, the accuracy of our algorithm only drops by 1.36%, 1.45%, and 1.46%. However, when the pruning rate of the MMTLDMC algorithm is 95%, the accuracy dropped sharply. Because our algorithm runs on the mobile terminal, the pruning of 30% is not considered for the algorithm MMTLDMC.Table 3
Pruning rate, accuracy (%), and decrease point of the algorithm on the digit dataset.
AlgorithmsU.M- > SU.S- > MM.S- > UPruning (%)Accuracy (%)Decrease PointPruning (%)Accuracy (%)Decrease PointPruning (%)Accuracy (%)Decrease PointDCTN078.02097.58093.643077.880.143097.470.113093.510.135077.70.325097.390.195093.390.257077.60.427097.270.317093.180.469076.491.539096.141.449092.311.339549.5628.469569.2228.369564.5229.12MultiSource TrAdaBoost078.83096.41093.823078.720.113096.280.133093.50.325078.620.215096.20.215093.440.387078.260.577096.150.267093.290.539077.41.439095.051.369092.411.419548.5730.269566.4329.989563.7130.11M3SDA079.28099.05095.933079.210.073098.990.063095.820.115079.130.155098.960.095095.70.237079.050.237098.890.167095.610.329077.721.569097.371.529094.271.669549.9229.369569.5929.469566.9528.98MMTLDMC081.23099.13096.95XXXXXXXXX5081.10.135099.070.065096.790.167081.070.167098.990.147096.690.269079.871.369097.681.459095.491.469552.2428.999569.9229.219566.6830.01


### 4.2. Image Classification Dataset

#### 4.2.1. Dataset Introduction

The Office-31 dataset is a commonly used standard transfer learning dataset. It contains 4652 sample pictures collected from different areas named Amazon (A), Webcam (W), and DSLR (D), these pictures can be divided into 31 categories. Among them, Amazon’s samples are fromhttps://amazon.com, and the samples in Webcam and DSLR are obtained through web cameras and digital SLR cameras in different environments. Caltech-256 [40] is a standard database for object recognition. The database has 30607 images and 256 categories. In these experiments, we used the dataset as office-31+Caltech published by Gong [39] et al., as shown in Figure 4. Specifically, we have four domains, C(caltech-256), A (Amazon), W (webcam), and D (DSLR). We randomly select three domains as the source domain and the remaining one as the target domain, that is, (A, C, D- > W), (A, C, W- > D), (A, D, W- > C), (C, D, W- > A).Figure 4
Example of Office31 and Caltech-256 pictures.


#### 4.2.2. Experimental Data

In this section, we compare MMTLDMC with multisource transfer learning algorithms such as DCTN, M3SDA, and MultiSource TrAdaBoost.It can be seen from Table4 that among the four cross-domain tasks, when the pruning rate of the MMTLDMC algorithm is 90%, the accuracy rates are 90.77%, 98.48%, 98.52%, and 94.13%, which are higher than the comparative algorithms DCTN and MultiSource TrAdaBoost. At the same time, under the tasks of A.W.C- > D and W.D.C- > A, the accuracy only drops by 0.98%. However, when the pruning rate of the MMTLDMC algorithm is 95%, the accuracy dropped sharply. Because our algorithm runs on the mobile terminal, the pruning of 30% is not considered for the algorithm MMTLDMC.Table 4
Pruning rate, accuracy (%), and decrease point of the algorithm on the image dataset.
AlgorithmsA.W.D- > CA.W.C- > DA.D.C- > WW.D.C- > APruning(%)Accuracy(%)Decrease pointPruning (%)Accuracy (%)Decrease pointPruning (%)Accuracy (%)Decrease pointPruning (%)Accuracy (%)Decrease pointDCTN090.48098.7098.31091.223090.320.163098.650.053098.280.033091.10.125090.190.295098.620.085098.210.15090.970.257090.070.417098.490.217098.050.267090.790.439089.261.229097.691.019096.751.569089.771.459565.8524.639568.4830.229568.0630.259561.5929.63MultiSource TrAdaBoost090.32098.42097.89090.623090.070.253098.30.123097.810.083090.410.215090.010.315098.230.195097.770.125090.280.347089.870.457098.080.347097.690.27090.10.529088.112.219097.21.229096.411.489088.661.969562.0128.319571.6826.749569.5828.319562.727.56M3SDA092.55099.29099.06094.083092.370.183099.20.093099.030.033093.970.115092.350.25099.150.145098.990.075093.890.197092.320.237099.110.187098.930.137093.820.269091.291.269098.061.239097.511.559092.072.019565.2227.339573.9225.379568.9330.139564.8429.24MMTLDMC091.88099.46099.52095.11XXXXXXXXXXXX5091.750.135099.40.065099.490.035095.050.067091.670.217099.330.137099.470.057095.010.19090.771.119098.480.989098.521.259094.130.989567.6724.219575.1424.329573.2126.319567.1527.96


### 4.3. Effect of the Pruning Rate on Computation Time

In order to demonstrate the advantages of the influence of our algorithm category, we choose the network dataset DomainNet proposed by the literature [14] (as shown in Figure 5). We randomly sample 20 classes from each domain, 3000 data as our training data.Figure 5
Example of DomainNet pictures.


#### 4.3.1. Experimental Data

(1) Effect of the Pruning Rate on Computation Time and Iterations. As can be seen from Figure 6, the algorithm MMTLDMC is under the dataset DomainNet:a) The model calculation time will gradually decrease with the increase of the pruning rate. (b) The computational accuracy of the model will gradually decrease with the increase of the pruning rate. (c) From the figure, we can clearly see that when the pruning rate is 94%, the calculation accuracy will drop significantly. (d) When the pruning rate reaches 90%, the accuracy generally only drops by 2–4%. (e) When the pruning rate reaches 80%, the model calculation time will stabilize.Figure 6
Comparison on DomainNet.(2) Influence of Iterations. Figure 7 shows the effect of the number of iterations on the accuracy. (a) When the number of iterations exceeds 1000, the accuracy of the algorithm tends to be stable. (b) At the same time, MMTLDMC has better results.Figure 7
Effect of iteration times on accuracy.


## 4.1. Digital Classification Dataset

### 4.1.1. Dataset Introduction

Both the USPS dataset and the MNIST dataset contain handwritten digits 0″–″9″, the former is composed of 9298 16 × 16 images, and the latter is composed of 70,000 28 × 28 images. Street View House Number (SVHN) comes from Google. Each picture contains a group of Arabic numerals’ 0–9′, which contains 73257 digits and the image pixel is 32 × 32. Figure3 shows examples of USPS, MNIST, and SVHN. We can see that the distributions of USPS and MNIST are different, but they contribute the same feature space. SVHN datasets are different from their distribution and feature space. We extract 9000 images from MNIST and SVHN as two domains. Because USPS has only 9298 pictures, we regard the whole dataset as a domain. Due to the limitation of the mobile terminal, 3000 images are extracted from MNIS, SVHN, and USPS as a domain, respectively.Figure 3
Example of USPS MNIST and SVHN pictures.


### 4.1.2. Experimental Data

In this part, we compare some multisource transfer learning algorithms such as DCTN and M3SDA MultiSource TrAdaBoost with our algorithm MMTLDMC.It can be seen from Table3 that in the three cross-domain tasks, the MMTLDMC algorithm has an accuracy of 79.87%, 97.68%, and 95.49% when the pruning rate is 90%, which is higher than the comparison algorithms DCTN, MultiSource TrAdaBoost, and M3SDA. Compared with the data without pruning, in the tasks U.M- > S, U.S- > M, M.S- > U, the accuracy of our algorithm only drops by 1.36%, 1.45%, and 1.46%. However, when the pruning rate of the MMTLDMC algorithm is 95%, the accuracy dropped sharply. Because our algorithm runs on the mobile terminal, the pruning of 30% is not considered for the algorithm MMTLDMC.Table 3
Pruning rate, accuracy (%), and decrease point of the algorithm on the digit dataset.
AlgorithmsU.M- > SU.S- > MM.S- > UPruning (%)Accuracy (%)Decrease PointPruning (%)Accuracy (%)Decrease PointPruning (%)Accuracy (%)Decrease PointDCTN078.02097.58093.643077.880.143097.470.113093.510.135077.70.325097.390.195093.390.257077.60.427097.270.317093.180.469076.491.539096.141.449092.311.339549.5628.469569.2228.369564.5229.12MultiSource TrAdaBoost078.83096.41093.823078.720.113096.280.133093.50.325078.620.215096.20.215093.440.387078.260.577096.150.267093.290.539077.41.439095.051.369092.411.419548.5730.269566.4329.989563.7130.11M3SDA079.28099.05095.933079.210.073098.990.063095.820.115079.130.155098.960.095095.70.237079.050.237098.890.167095.610.329077.721.569097.371.529094.271.669549.9229.369569.5929.469566.9528.98MMTLDMC081.23099.13096.95XXXXXXXXX5081.10.135099.070.065096.790.167081.070.167098.990.147096.690.269079.871.369097.681.459095.491.469552.2428.999569.9229.219566.6830.01


## 4.1.1. Dataset Introduction

Both the USPS dataset and the MNIST dataset contain handwritten digits 0″–″9″, the former is composed of 9298 16 × 16 images, and the latter is composed of 70,000 28 × 28 images. Street View House Number (SVHN) comes from Google. Each picture contains a group of Arabic numerals’ 0–9′, which contains 73257 digits and the image pixel is 32 × 32. Figure3 shows examples of USPS, MNIST, and SVHN. We can see that the distributions of USPS and MNIST are different, but they contribute the same feature space. SVHN datasets are different from their distribution and feature space. We extract 9000 images from MNIST and SVHN as two domains. Because USPS has only 9298 pictures, we regard the whole dataset as a domain. Due to the limitation of the mobile terminal, 3000 images are extracted from MNIS, SVHN, and USPS as a domain, respectively.Figure 3
Example of USPS MNIST and SVHN pictures.


## 4.1.2. Experimental Data

In this part, we compare some multisource transfer learning algorithms such as DCTN and M3SDA MultiSource TrAdaBoost with our algorithm MMTLDMC.It can be seen from Table3 that in the three cross-domain tasks, the MMTLDMC algorithm has an accuracy of 79.87%, 97.68%, and 95.49% when the pruning rate is 90%, which is higher than the comparison algorithms DCTN, MultiSource TrAdaBoost, and M3SDA. Compared with the data without pruning, in the tasks U.M- > S, U.S- > M, M.S- > U, the accuracy of our algorithm only drops by 1.36%, 1.45%, and 1.46%. However, when the pruning rate of the MMTLDMC algorithm is 95%, the accuracy dropped sharply. Because our algorithm runs on the mobile terminal, the pruning of 30% is not considered for the algorithm MMTLDMC.Table 3
Pruning rate, accuracy (%), and decrease point of the algorithm on the digit dataset.
AlgorithmsU.M- > SU.S- > MM.S- > UPruning (%)Accuracy (%)Decrease PointPruning (%)Accuracy (%)Decrease PointPruning (%)Accuracy (%)Decrease PointDCTN078.02097.58093.643077.880.143097.470.113093.510.135077.70.325097.390.195093.390.257077.60.427097.270.317093.180.469076.491.539096.141.449092.311.339549.5628.469569.2228.369564.5229.12MultiSource TrAdaBoost078.83096.41093.823078.720.113096.280.133093.50.325078.620.215096.20.215093.440.387078.260.577096.150.267093.290.539077.41.439095.051.369092.411.419548.5730.269566.4329.989563.7130.11M3SDA079.28099.05095.933079.210.073098.990.063095.820.115079.130.155098.960.095095.70.237079.050.237098.890.167095.610.329077.721.569097.371.529094.271.669549.9229.369569.5929.469566.9528.98MMTLDMC081.23099.13096.95XXXXXXXXX5081.10.135099.070.065096.790.167081.070.167098.990.147096.690.269079.871.369097.681.459095.491.469552.2428.999569.9229.219566.6830.01


## 4.2. Image Classification Dataset

### 4.2.1. Dataset Introduction

The Office-31 dataset is a commonly used standard transfer learning dataset. It contains 4652 sample pictures collected from different areas named Amazon (A), Webcam (W), and DSLR (D), these pictures can be divided into 31 categories. Among them, Amazon’s samples are fromhttps://amazon.com, and the samples in Webcam and DSLR are obtained through web cameras and digital SLR cameras in different environments. Caltech-256 [40] is a standard database for object recognition. The database has 30607 images and 256 categories. In these experiments, we used the dataset as office-31+Caltech published by Gong [39] et al., as shown in Figure 4. Specifically, we have four domains, C(caltech-256), A (Amazon), W (webcam), and D (DSLR). We randomly select three domains as the source domain and the remaining one as the target domain, that is, (A, C, D- > W), (A, C, W- > D), (A, D, W- > C), (C, D, W- > A).Figure 4
Example of Office31 and Caltech-256 pictures.


### 4.2.2. Experimental Data

In this section, we compare MMTLDMC with multisource transfer learning algorithms such as DCTN, M3SDA, and MultiSource TrAdaBoost.It can be seen from Table4 that among the four cross-domain tasks, when the pruning rate of the MMTLDMC algorithm is 90%, the accuracy rates are 90.77%, 98.48%, 98.52%, and 94.13%, which are higher than the comparative algorithms DCTN and MultiSource TrAdaBoost. At the same time, under the tasks of A.W.C- > D and W.D.C- > A, the accuracy only drops by 0.98%. However, when the pruning rate of the MMTLDMC algorithm is 95%, the accuracy dropped sharply. Because our algorithm runs on the mobile terminal, the pruning of 30% is not considered for the algorithm MMTLDMC.Table 4
Pruning rate, accuracy (%), and decrease point of the algorithm on the image dataset.
AlgorithmsA.W.D- > CA.W.C- > DA.D.C- > WW.D.C- > APruning(%)Accuracy(%)Decrease pointPruning (%)Accuracy (%)Decrease pointPruning (%)Accuracy (%)Decrease pointPruning (%)Accuracy (%)Decrease pointDCTN090.48098.7098.31091.223090.320.163098.650.053098.280.033091.10.125090.190.295098.620.085098.210.15090.970.257090.070.417098.490.217098.050.267090.790.439089.261.229097.691.019096.751.569089.771.459565.8524.639568.4830.229568.0630.259561.5929.63MultiSource TrAdaBoost090.32098.42097.89090.623090.070.253098.30.123097.810.083090.410.215090.010.315098.230.195097.770.125090.280.347089.870.457098.080.347097.690.27090.10.529088.112.219097.21.229096.411.489088.661.969562.0128.319571.6826.749569.5828.319562.727.56M3SDA092.55099.29099.06094.083092.370.183099.20.093099.030.033093.970.115092.350.25099.150.145098.990.075093.890.197092.320.237099.110.187098.930.137093.820.269091.291.269098.061.239097.511.559092.072.019565.2227.339573.9225.379568.9330.139564.8429.24MMTLDMC091.88099.46099.52095.11XXXXXXXXXXXX5091.750.135099.40.065099.490.035095.050.067091.670.217099.330.137099.470.057095.010.19090.771.119098.480.989098.521.259094.130.989567.6724.219575.1424.329573.2126.319567.1527.96


## 4.2.1. Dataset Introduction

The Office-31 dataset is a commonly used standard transfer learning dataset. It contains 4652 sample pictures collected from different areas named Amazon (A), Webcam (W), and DSLR (D), these pictures can be divided into 31 categories. Among them, Amazon’s samples are fromhttps://amazon.com, and the samples in Webcam and DSLR are obtained through web cameras and digital SLR cameras in different environments. Caltech-256 [40] is a standard database for object recognition. The database has 30607 images and 256 categories. In these experiments, we used the dataset as office-31+Caltech published by Gong [39] et al., as shown in Figure 4. Specifically, we have four domains, C(caltech-256), A (Amazon), W (webcam), and D (DSLR). We randomly select three domains as the source domain and the remaining one as the target domain, that is, (A, C, D- > W), (A, C, W- > D), (A, D, W- > C), (C, D, W- > A).Figure 4
Example of Office31 and Caltech-256 pictures.


## 4.2.2. Experimental Data

In this section, we compare MMTLDMC with multisource transfer learning algorithms such as DCTN, M3SDA, and MultiSource TrAdaBoost.It can be seen from Table4 that among the four cross-domain tasks, when the pruning rate of the MMTLDMC algorithm is 90%, the accuracy rates are 90.77%, 98.48%, 98.52%, and 94.13%, which are higher than the comparative algorithms DCTN and MultiSource TrAdaBoost. At the same time, under the tasks of A.W.C- > D and W.D.C- > A, the accuracy only drops by 0.98%. However, when the pruning rate of the MMTLDMC algorithm is 95%, the accuracy dropped sharply. Because our algorithm runs on the mobile terminal, the pruning of 30% is not considered for the algorithm MMTLDMC.Table 4
Pruning rate, accuracy (%), and decrease point of the algorithm on the image dataset.
AlgorithmsA.W.D- > CA.W.C- > DA.D.C- > WW.D.C- > APruning(%)Accuracy(%)Decrease pointPruning (%)Accuracy (%)Decrease pointPruning (%)Accuracy (%)Decrease pointPruning (%)Accuracy (%)Decrease pointDCTN090.48098.7098.31091.223090.320.163098.650.053098.280.033091.10.125090.190.295098.620.085098.210.15090.970.257090.070.417098.490.217098.050.267090.790.439089.261.229097.691.019096.751.569089.771.459565.8524.639568.4830.229568.0630.259561.5929.63MultiSource TrAdaBoost090.32098.42097.89090.623090.070.253098.30.123097.810.083090.410.215090.010.315098.230.195097.770.125090.280.347089.870.457098.080.347097.690.27090.10.529088.112.219097.21.229096.411.489088.661.969562.0128.319571.6826.749569.5828.319562.727.56M3SDA092.55099.29099.06094.083092.370.183099.20.093099.030.033093.970.115092.350.25099.150.145098.990.075093.890.197092.320.237099.110.187098.930.137093.820.269091.291.269098.061.239097.511.559092.072.019565.2227.339573.9225.379568.9330.139564.8429.24MMTLDMC091.88099.46099.52095.11XXXXXXXXXXXX5091.750.135099.40.065099.490.035095.050.067091.670.217099.330.137099.470.057095.010.19090.771.119098.480.989098.521.259094.130.989567.6724.219575.1424.329573.2126.319567.1527.96


## 4.3. Effect of the Pruning Rate on Computation Time

In order to demonstrate the advantages of the influence of our algorithm category, we choose the network dataset DomainNet proposed by the literature [14] (as shown in Figure 5). We randomly sample 20 classes from each domain, 3000 data as our training data.Figure 5
Example of DomainNet pictures.


### 4.3.1. Experimental Data

(1) Effect of the Pruning Rate on Computation Time and Iterations. As can be seen from Figure 6, the algorithm MMTLDMC is under the dataset DomainNet:a) The model calculation time will gradually decrease with the increase of the pruning rate. (b) The computational accuracy of the model will gradually decrease with the increase of the pruning rate. (c) From the figure, we can clearly see that when the pruning rate is 94%, the calculation accuracy will drop significantly. (d) When the pruning rate reaches 90%, the accuracy generally only drops by 2–4%. (e) When the pruning rate reaches 80%, the model calculation time will stabilize.Figure 6
Comparison on DomainNet.(2) Influence of Iterations. Figure 7 shows the effect of the number of iterations on the accuracy. (a) When the number of iterations exceeds 1000, the accuracy of the algorithm tends to be stable. (b) At the same time, MMTLDMC has better results.Figure 7
Effect of iteration times on accuracy.


## 4.3.1. Experimental Data

(1) Effect of the Pruning Rate on Computation Time and Iterations. As can be seen from Figure 6, the algorithm MMTLDMC is under the dataset DomainNet:a) The model calculation time will gradually decrease with the increase of the pruning rate. (b) The computational accuracy of the model will gradually decrease with the increase of the pruning rate. (c) From the figure, we can clearly see that when the pruning rate is 94%, the calculation accuracy will drop significantly. (d) When the pruning rate reaches 90%, the accuracy generally only drops by 2–4%. (e) When the pruning rate reaches 80%, the model calculation time will stabilize.Figure 6
Comparison on DomainNet.(2) Influence of Iterations. Figure 7 shows the effect of the number of iterations on the accuracy. (a) When the number of iterations exceeds 1000, the accuracy of the algorithm tends to be stable. (b) At the same time, MMTLDMC has better results.Figure 7
Effect of iteration times on accuracy.


## 5. Conclusion

In this paper, aiming at the problem of multisource transfer learning for mobile computing, a multisource mobile transfer learning algorithm based on dynamic model compression is proposed. This method combines the advantages of mobile computing and multisource learning to complete the image classification on the mobile terminal. This method first performs BN pruning on the pretrained network, and then calculates the classification loss and MMD loss of the multisource transfer model on the mobile side, and calculates the losses of different classifiers on the server side. Finally, the parameters of the minimized objective function are synchronized to the client. The experimental results on the SVHN, USPS, MNIST, Office31, Caltech, and DomainNet show that MMTLDMC outperforms the benchmark algorithms in both classification accuracy and training efficiency. Although the experimental results show that the MMTLDMC algorithm is better than the benchmark algorithm, further research is still needed in the following aspects: through data encryption, partial data sharing, and dynamic adjustment of the BN pruning strategy on the server side to optimize the model classification accuracy; continuing to reduce the number of parameters to achieve faster client computing.



---

*Source: 3234078-2022-03-16.xml*