# Learning Transferable Convolutional Proxy by SMI-Based Matching Technique

Authors: Wei Jin; Nan Jia
Journal: Shock and Vibration
 (2020)

---

## Abstract

Domain-transfer learning is a machine learning task to explore a source domain data set to help the learning problem in a target domain. Usually, the source domain has sufficient labeled data, while the target domain does not. In this paper, we propose a novel domain-transfer convolutional model by mapping a target domain data sample to a proxy in the source domain and applying a source domain model to the proxy for the purpose of prediction. In our framework, we firstly represent both source and target domains to feature vectors by two convolutional neural networks and then construct a proxy for each target domain sample in the source domain space. The proxy is supposed to be matched to the corresponding target domain sample convolutional representation vector well. To measure the matching quality, we proposed to maximize their squared-loss mutual information (SMI) between the proxy and target domain samples. We further develop a novel neural SMI estimator based on a parametric density ratio estimation function. Moreover, we also propose to minimize the classification error of both source domain samples and target domain proxies. The classification responses are also smoothened by manifolds of both the source domain and proxy space. By minimizing an objective function of SMI, classification error, and manifold regularization, we learn the convolutional networks of both source and target domains. In this way, the proxy of a target domain sample can be matched to the source domain data and thus benefits from the rich supervision information of the source domain. We design an iterative algorithm to update the parameters alternately and test it over benchmark data sets of abnormal behavior detection in video, Amazon product reviews sentiment analysis, etc.


---

## Body

## 1. Introduction

### 1.1. Background

With the rapid development in Internet technologies, more and more people are using the Internet and generating large-scale sets of behavior data [1, 2]. Nowadays, machine learning technologies are widely used for the automatic annotation of the data to extract insights into the purpose of decision-making [3, 4]. To learn the machine learning models for this purpose, we need sufficient data and the corresponding labels so that the model can learn from the data-label pairs to build the mapping from the data to the labels. Moreover, the size of such data-label pairs should be large enough to cover most of the patterns of the data. However, the labeling of the data is sometimes expensive. When the data are generated, it is usually not labeled automatically, and thus, the label is missing. To fill this gap, human label work is required to provide the label. In many cases, the labeling of data is time-consuming and costly. As a result, we can only label a limited amount of data and train the model with a large amount of unlabeled data with a small amount of labeled data [5, 6]. The lack of labeled data is a bottleneck for the machine learning tasks over big data. To solve this problem, various methods are developed, among which the most popular ones are the semisupervised learning and transfer learning [7, 8].Remark 1.
The research method of this paper is a new method of transfer learning, and the application objects include many different applications. Their close relations are explained as follows.
Transfer learning aims to borrow the sufficiently labeled data from another domain to help the learning of the model in a domain where the data are not sufficient. This problem is defined over a source domain with sufficient labeled data and a target domain with insufficient labeled data, but a large set of unlabeled data. The source domain and target domain have the same data space and label space. However, the data distribution of the two domains is different; thus, it is not suitable to directly merge them to train a model for the target domain. Due to the mismatch of the data distributions of source and target domains, it is important to transfer the knowledge from the source domain to the target domain. Transfer learning algorithms are designed for this purpose. For example, in the problem of video anomaly detection, the problem is to detect the anomalies from the frames of the video, and the anomalies include cycle, skater, truck, car, wheelchair, and baby cart. In the real world, the data of video are usually collected from subway stations, communities, shopping centers, etc., which are treated as the target domain. Moreover, these data are usually not labeled at all, or only a small part is labeled. Meanwhile, we can use a fully labeled data set collected and annotated by the University of California and San Diego (UCSD) [9, 10] as the source domain data to help the training of the target domain model. However, the video of UCSD is usually captured on the campus of the university, which is different from the target domain’s locations, such as shopping centers and subway stations. Thus, the data of source and target domains do not match perfectly, and the mismatching of the two domain’s video requires transferring the model trained over the source domain to the target domain to adapt the target domain data. Otherwise, the model trained over the source domain cannot fit the target domain.
Another example is the Amazon product review sentiment analysis [11, 12]. This task is to detect the sentiment (positive/negative) from the text content of a review of a product. For different product types, the buyers may have different review styles. Some product types have sufficient labeled data, while other types are in lack of labeled data. Thus, we want to use the well-labeled product reviews as a source domain to help the learning of sentiment detectors for the limited-labeled reviews of other products. Again, the domain transfer is needed to adopt the model from the source domain product to the target domain product.
In the transfer learning models, the deep convolutional neural network (CNN) [13–17] is the most popular base models. It is composed of multiple convolutional layers and max-pooling layers. The convolutional layers use a filter bank to extract the location features by sliding over the input data, such as image or sentence. This model can extract both local and hierarchical features from data and thus can perform automatic engineering effectively. In this paper, we study the problem of transfer learning over a source domain and a target domain. We proposed a novel transfer learning method based on a convolutional neural network (CNN) [13] and use the squared-loss mutual information (SMI) [18] to measure the matching between the source and the target domain.


### 1.2. Existing Works

In traditional methods of transfer learning, to match the source and target domains, various methods are applied, such as mutual information minimization (MIM) [19], Hilbert Schmidt independence criterion (HSIC) maximization [20], and maximum mean discrepancy (MMD) minimization [21, 22]. Moreover, to train the transfer learning models, the most popular base model is the deep CNN model, which is also adapted in our paper. In this paper, we briefly introduce some state-of-the-arts relevant to our work of CNN-based transfer learning.Long et al. [21, 23] proposed a deep CNN model for transfer learning. This model is a two-branch network. The first four convolutional layers are shared by both source and target domains, and the last three full-connection layers are domain-specific. However, in the last three layers, the output features are forced to be in the same spaces by the MMD minimization. The MMD minimization measures mismatch in the features of each layer of two domain domains by the squared ℓ2 distance of the means of two domains.Zhang et al. [14] proposed a CNN model learning method for transfer learning to present both data features and attributes. The structure of this network is composed of a shared attribute-embedding CNN across domains, a domain-independent CNN, and a domain-dependent CNN for the embedding of the original data of each data sample. The outputs of these three CNN models are concatenated as the overall features of one sample and used to predict the class label. The attribute-embedding CNN is imposed to approximate the transformed attributes accurately, and the domain-independent CNN outputs are supposed to minimize the MMD of two domains.Wang et al. [19] developed a novel CNN model learning method for both source and target domains. The CNN outputs are imposed to be independent of the domain; i.e., from the CNN model output of a data sample, we should not be able to tell which domain it is from. To measure the independence of the CNN outputs and the domain indicator, the mutual information is applied. The CNN models are learned by minimizing the mutual information between the CNN outputs and the domain indicators to obtain the domain-independent CNN representations.Geand and Yu [24] designed a learning method of deep CNN model for transfer learning. This method is based on the source-target selective joint fine-tuning. This method is not to use all the data samples of the source domain but only use a subset of the training samples of the source domain. The selected subset is similar to the target domain samples at the low-level characteristics. To this end, the algorithm constructs the descriptors from the responses of the filter bank over the training samples. These descriptors are used to search for the subset of training samples used for the learning problem.


### 1.3. Our Contributions

In this paper, we propose a novel CNN-based deep learning framework for transfer learning. Our work is motivated by a phenomenon: the mismatch between the source domain and target domain is usually at the entire data set level but at the data sample level. Moreover, it is also difficult to find the one-to-one matching between the samples of the source and target domains. Thus, we propose to construct a proxy of each target domain in the source domain. This proxy is constructed by a linear combination of the source domain samples. We also argue that the CNN model is a powerful tool to extract the features of both source and target domains and an optimal choice for the proxy construction. Thus, we first extract the convolutional features from both source and target samples and then apply the proxy construction. We hope the constructed proxy of each target domain can match the original target sample convolutional representation as well as possible. In this paper, we propose to use an information theory measure to measure the quality of matching between the proxy and convolutional features of target domain samples. This measure is the squared-loss mutual information (SMI) [25, 26], which measures the Pearson divergence of the joint probability of proxies and convolutional features and the product of the independent profanities of proxies and convolutional features. We further release the calculation of the SMI by firstly designing a parametric density ratio estimator of proxy and convolutional feature of target domain samples and then approximate it as the squared matching error of the true density ratio against the density ratio estimator.To learn the proxy construction coefficients, we model the learning problem as a minimization problem. In this minimization problem, we minimize the SMI, the classification loss of labeled samples, and the classification response entropy of the unlabeled samples of both domains. Moreover, the manifold regularization and the model complexity are also considered in the minimization problem. The contributions of this paper are listed as follows:(1)
We build a novel transfer learning schema, which firstly represents the source and target domains by CNN models and then constructs proxies for the target samples from the source domain. The construction is guild by the SMI minimization. In this way, the matching of the two domains is directly applied at the proxy level, while the CNN representations can still keep the domain characteristics. A novel SMI approximation is proposed as the squared matching error between the density ratio function and its parametric estimator.(2)
We model the problem as a minimization problem with SMI, classification loss, classification entropy, manifold regularization, and model complexity regularization. This minimization is a joint learning framework to optimize the CNN model parameters and proxy construction coefficients simultaneously.(3)
We develop an iterative algorithm to solve the minimization problem based on the alternating direction method of multipliers (ADMM) [27]. Moreover, the parameter of the density ratio estimator of the SMI is also updated in this algorithm. With this algorithm, the SMI parameter is also automatically approximated together with the other variables of the objective.


### 1.4. Paper Organization

This paper is organized as follows: in Section2, we introduce the proposed transfer learning method based on CNN and proxy learning; in Section 3, we evaluate the proposed method experimentally over some benchmark data sets; in Section 4, we give the conclusion of this paper and some future works.


## 1.1. Background

With the rapid development in Internet technologies, more and more people are using the Internet and generating large-scale sets of behavior data [1, 2]. Nowadays, machine learning technologies are widely used for the automatic annotation of the data to extract insights into the purpose of decision-making [3, 4]. To learn the machine learning models for this purpose, we need sufficient data and the corresponding labels so that the model can learn from the data-label pairs to build the mapping from the data to the labels. Moreover, the size of such data-label pairs should be large enough to cover most of the patterns of the data. However, the labeling of the data is sometimes expensive. When the data are generated, it is usually not labeled automatically, and thus, the label is missing. To fill this gap, human label work is required to provide the label. In many cases, the labeling of data is time-consuming and costly. As a result, we can only label a limited amount of data and train the model with a large amount of unlabeled data with a small amount of labeled data [5, 6]. The lack of labeled data is a bottleneck for the machine learning tasks over big data. To solve this problem, various methods are developed, among which the most popular ones are the semisupervised learning and transfer learning [7, 8].Remark 1.
The research method of this paper is a new method of transfer learning, and the application objects include many different applications. Their close relations are explained as follows.
Transfer learning aims to borrow the sufficiently labeled data from another domain to help the learning of the model in a domain where the data are not sufficient. This problem is defined over a source domain with sufficient labeled data and a target domain with insufficient labeled data, but a large set of unlabeled data. The source domain and target domain have the same data space and label space. However, the data distribution of the two domains is different; thus, it is not suitable to directly merge them to train a model for the target domain. Due to the mismatch of the data distributions of source and target domains, it is important to transfer the knowledge from the source domain to the target domain. Transfer learning algorithms are designed for this purpose. For example, in the problem of video anomaly detection, the problem is to detect the anomalies from the frames of the video, and the anomalies include cycle, skater, truck, car, wheelchair, and baby cart. In the real world, the data of video are usually collected from subway stations, communities, shopping centers, etc., which are treated as the target domain. Moreover, these data are usually not labeled at all, or only a small part is labeled. Meanwhile, we can use a fully labeled data set collected and annotated by the University of California and San Diego (UCSD) [9, 10] as the source domain data to help the training of the target domain model. However, the video of UCSD is usually captured on the campus of the university, which is different from the target domain’s locations, such as shopping centers and subway stations. Thus, the data of source and target domains do not match perfectly, and the mismatching of the two domain’s video requires transferring the model trained over the source domain to the target domain to adapt the target domain data. Otherwise, the model trained over the source domain cannot fit the target domain.
Another example is the Amazon product review sentiment analysis [11, 12]. This task is to detect the sentiment (positive/negative) from the text content of a review of a product. For different product types, the buyers may have different review styles. Some product types have sufficient labeled data, while other types are in lack of labeled data. Thus, we want to use the well-labeled product reviews as a source domain to help the learning of sentiment detectors for the limited-labeled reviews of other products. Again, the domain transfer is needed to adopt the model from the source domain product to the target domain product.
In the transfer learning models, the deep convolutional neural network (CNN) [13–17] is the most popular base models. It is composed of multiple convolutional layers and max-pooling layers. The convolutional layers use a filter bank to extract the location features by sliding over the input data, such as image or sentence. This model can extract both local and hierarchical features from data and thus can perform automatic engineering effectively. In this paper, we study the problem of transfer learning over a source domain and a target domain. We proposed a novel transfer learning method based on a convolutional neural network (CNN) [13] and use the squared-loss mutual information (SMI) [18] to measure the matching between the source and the target domain.


## 1.2. Existing Works

In traditional methods of transfer learning, to match the source and target domains, various methods are applied, such as mutual information minimization (MIM) [19], Hilbert Schmidt independence criterion (HSIC) maximization [20], and maximum mean discrepancy (MMD) minimization [21, 22]. Moreover, to train the transfer learning models, the most popular base model is the deep CNN model, which is also adapted in our paper. In this paper, we briefly introduce some state-of-the-arts relevant to our work of CNN-based transfer learning.Long et al. [21, 23] proposed a deep CNN model for transfer learning. This model is a two-branch network. The first four convolutional layers are shared by both source and target domains, and the last three full-connection layers are domain-specific. However, in the last three layers, the output features are forced to be in the same spaces by the MMD minimization. The MMD minimization measures mismatch in the features of each layer of two domain domains by the squared ℓ2 distance of the means of two domains.Zhang et al. [14] proposed a CNN model learning method for transfer learning to present both data features and attributes. The structure of this network is composed of a shared attribute-embedding CNN across domains, a domain-independent CNN, and a domain-dependent CNN for the embedding of the original data of each data sample. The outputs of these three CNN models are concatenated as the overall features of one sample and used to predict the class label. The attribute-embedding CNN is imposed to approximate the transformed attributes accurately, and the domain-independent CNN outputs are supposed to minimize the MMD of two domains.Wang et al. [19] developed a novel CNN model learning method for both source and target domains. The CNN outputs are imposed to be independent of the domain; i.e., from the CNN model output of a data sample, we should not be able to tell which domain it is from. To measure the independence of the CNN outputs and the domain indicator, the mutual information is applied. The CNN models are learned by minimizing the mutual information between the CNN outputs and the domain indicators to obtain the domain-independent CNN representations.Geand and Yu [24] designed a learning method of deep CNN model for transfer learning. This method is based on the source-target selective joint fine-tuning. This method is not to use all the data samples of the source domain but only use a subset of the training samples of the source domain. The selected subset is similar to the target domain samples at the low-level characteristics. To this end, the algorithm constructs the descriptors from the responses of the filter bank over the training samples. These descriptors are used to search for the subset of training samples used for the learning problem.


## 1.3. Our Contributions

In this paper, we propose a novel CNN-based deep learning framework for transfer learning. Our work is motivated by a phenomenon: the mismatch between the source domain and target domain is usually at the entire data set level but at the data sample level. Moreover, it is also difficult to find the one-to-one matching between the samples of the source and target domains. Thus, we propose to construct a proxy of each target domain in the source domain. This proxy is constructed by a linear combination of the source domain samples. We also argue that the CNN model is a powerful tool to extract the features of both source and target domains and an optimal choice for the proxy construction. Thus, we first extract the convolutional features from both source and target samples and then apply the proxy construction. We hope the constructed proxy of each target domain can match the original target sample convolutional representation as well as possible. In this paper, we propose to use an information theory measure to measure the quality of matching between the proxy and convolutional features of target domain samples. This measure is the squared-loss mutual information (SMI) [25, 26], which measures the Pearson divergence of the joint probability of proxies and convolutional features and the product of the independent profanities of proxies and convolutional features. We further release the calculation of the SMI by firstly designing a parametric density ratio estimator of proxy and convolutional feature of target domain samples and then approximate it as the squared matching error of the true density ratio against the density ratio estimator.To learn the proxy construction coefficients, we model the learning problem as a minimization problem. In this minimization problem, we minimize the SMI, the classification loss of labeled samples, and the classification response entropy of the unlabeled samples of both domains. Moreover, the manifold regularization and the model complexity are also considered in the minimization problem. The contributions of this paper are listed as follows:(1)
We build a novel transfer learning schema, which firstly represents the source and target domains by CNN models and then constructs proxies for the target samples from the source domain. The construction is guild by the SMI minimization. In this way, the matching of the two domains is directly applied at the proxy level, while the CNN representations can still keep the domain characteristics. A novel SMI approximation is proposed as the squared matching error between the density ratio function and its parametric estimator.(2)
We model the problem as a minimization problem with SMI, classification loss, classification entropy, manifold regularization, and model complexity regularization. This minimization is a joint learning framework to optimize the CNN model parameters and proxy construction coefficients simultaneously.(3)
We develop an iterative algorithm to solve the minimization problem based on the alternating direction method of multipliers (ADMM) [27]. Moreover, the parameter of the density ratio estimator of the SMI is also updated in this algorithm. With this algorithm, the SMI parameter is also automatically approximated together with the other variables of the objective.


## 1.4. Paper Organization

This paper is organized as follows: in Section2, we introduce the proposed transfer learning method based on CNN and proxy learning; in Section 3, we evaluate the proposed method experimentally over some benchmark data sets; in Section 4, we give the conclusion of this paper and some future works.


## 2. Proposed Method

In this section, we will propose the novel transfer convolutional neural network learning method. Firstly, we will build the objective function for the learning of the source and target domain convolutional network, and then, we will discuss how to minimize this objective function and finally develop an iterative algorithm based on the optimization results.


### 2.1. Objective Function

Suppose we have a training set of two domains. In the target domain, we haven training samples x1t,…,xnt, where xit is the i-th training sample of the target domain. To represent a target domain sample xt, we use a deep CNN model, f, to convert it to a dt-dimensional vector:(1)f=fxt;Θ∈Rdt.Meanwhile, we have a set of source domain training setx1s,…,xms, which has m samples and xjs is the j-th source domain training sample. Similar to the target domain, we use another deep CNN model, g, to convert a source domain sample, xs to a ds-dimensional vector:(2)g=gxs;Φ∈Rds.To learn the network parameters Θ and Φ, we discuss the following problems.


#### 2.1.1. Squared-Loss Mutual Information

Due to the mismatch of the source and target domains, we propose to leverage the two domains by constructing a proxy for each target domain sample in the source domain space. To this send, for a target domain sample,xti, we denote its proxy zi∈Rds, as a ds-dimensional vector. Moreover, we impose that it can be reconstructed by a learning combination of the convolutional representations of source domain samples:(3)zi=∑j=1mαijgj,s.t.∑j=1mαij=1,1≥αij≥0,where gj=gxjs;Φ is the convolutional vector of the j-th source domain sample and αij is the nonnegative weight of the j-th source domain sample for the construction of the i-th proxy.To encourage the matching of the source and target domains, we argue to measure their quality of matching by the density ration of the proxies and convolutional vectors of the target domain samplesz and f. For this purpose, we firstly measure the density ratio of z and f, denoted as rf,z. According to the definition of density ratio [18],(4)rf,z=pf,zpfpz,where pf,z is the joint probability of f and z, pf is the probability of f, and pz is the probability of z. Since it is difficult to directly estimate the density ratio, we propose to learn a parametric density ratio estimator function:(5)r^f,z;ϕ=hϕ⊤fzf−z,where pf,z is the joint probability of f and z, pf is the probability of f, and pz is the probability of z. Since it is difficult to directly estimate the density ratio, we propose to learn a parametric density ratio estimator function. fzf−z is the concatenation of the f, z, and their element-wise difference vector, ϕ∈ℝ3ds is the parameter vector, and hx=1/1+exp−x is the Sigmoid activation function. To learn ϕ, we minimize the SMI between rf,z and r^f,z;ϕ, defined as follows:(6)SMIϕ=∫f∫zrf,z−r^f,z;ϕ2pfpzdfdz=∫f∫zrf,z2−2rf,zr^f,z;ϕ+r^f,z;ϕ2pfpzdfdz=∫f∫zrf,z2pfpzdfdz−2∫f∫zrf,zr^f,z;ϕpfpzdfdz+∫f∫zr^f,z;ϕ2pfpzdfdz=∫f∫zrf,z2pfpzdfdz−2∫f∫zpf,zr^f,z;ϕdfdz+∫f∫zr^f,z;ϕ2pfpzdfdz.To estimate the SMI given the training samples of target domain and their proxies, we collect the set of convolutional vectors and corresponding proxiesf1,z1,…,fn,zn. We assume the distributions of f, z, and f,z are uniform distributions, which lead the probability functions as(7)pf=1n,pz=1n,pf,z=1n.With probabilities in (7) and the empirical approximation, we rewrite the second and third terms of (6) as follows:(8)∫f∫zpf,zrf,z;ϕdfdz=1n∫f∫zrf,z;ϕdfdz=1n∑i=1nrfi,zi;ϕ,∫f∫zr^f,z;ϕ2pfpzdfdz=1n2∑i,i′=1nr^fi,zi;ϕr^fi′,zi′;ϕ.By substituting (8) to (6), we have the approximated SMI as(9)SMIϕ=C−2n∑i=1nrfi,zi;ϕ+1n2∑i,i′=1nrfi,zi;ϕrfi′,zi′;ϕ,where C=∫f∫zrf,z2pfpzdfdz is a constant. To obtain a good quality estimator, we propose to minimize the squared-loss mutual information learned from the parameter of r^f,z;ϕ:(10)ϕ∗=argminϕSMIϕ.With this optimalϕ∗, we have to use the SMI as a measure of the matching quality between f and z. Using this measure as a term of loss for the purpose of learning the convolutional representations f and proxy z, we organize these parameters as matrices as F=f1,…,fn and Z=z1,…,zn. With these variables and the SMI as the matching measure, we seek to learn them to maximize the SMI. In other words, we minimize a loss function derived from SMI, defined as ℳF,Z;ϕ∗:(11)ℳF,Z;ϕ∗=2n∑i=1nr^fi,zi;ϕ∗−1n2∑i,i′=1nr^fi,zi;ϕ∗r^fi′,zi′;ϕ∗=−2n∑i=1nhϕ∗⊤fizifi−zi+1n2∑i,i′=1nhϕ∗⊤fizifi−zihϕ∗⊤fi′zi′fi′−zi′.We rewrite(12)ϕ∗⊤fizifi−zi=ϕ1⊤fi+ϕ2⊤zi+ϕ3⊤fi−zi=ϕ1+ϕ3⊤fi+ϕ2−ϕ3⊤zi,where ϕ∗=ϕ1ϕ2ϕ3 accordingly.(13)ℳF,Z;ϕ∗=−2n∑i=1nhϕ1+ϕ3⊤fi+ϕ2−ϕ3⊤zi+1n2∑i,i′=1nhϕ1+ϕ3⊤fi+ϕ2−ϕ3⊤zihϕ1+ϕ3⊤fi′+ϕ2−ϕ3⊤zi′.The squared-loss mutual information regularization problem is modelled as(14)minF,ZℳF,Z;ϕ∗.By minimizing this objective, we seek to learn the convolutional representation vectors of both source and target domain samples, target domain sample proxies, and other parameters to match the two domains as much as possible. Please note that, in this problem, we are not matching the source and target domain samples directly but seek to match target domain samples and the proxies of target domain samples in the source domain space.


#### 2.1.2. Semisupervised Classification

In both the source and target domains, there are both labeled and unlabeled data samples. To learn from them, we use the semisupervised method. We discuss the semisupervised learning problems in both source and target domains as follows.Source Domain. To approximate the class label vector, ys∈1,0c, of a source domain sample, xs, from its convolutional feature vector, g, we design a linear classification function:(15)ys=W⊤g,where W∈ℝds×c is the parameter vector of the classifier and y^s∈ℝc is the classification response. To learn the classifier parameter W, we utilize both the labeled and unlabeled samples of the source domain. We denote the set of the labeled samples of source domain as ℒs and the set of unlabeled as Us. For the samples in ℒs, we minimize their classification errors measured by the squared loss:(16)minW,GO1W,G=∑j:xjs∈Lsyjs−W⊤gj22,where yjs is the class label vector of the xj and G=g1,…,gm. For the samples in ℒs, even we do not have the available labels to constrain the classification responses in y^s, and we still can regularize the learning of the parameter by its entropy. For a sample xjs, we hope the uncertainty of its classification responses in y^js is as small as possible. The uncertainty is measured by its entropy:(17)Hyjs=−∑k=1cyjkslogyjks=−∑k=1cwk⊤gjlogwk⊤gj,wherey^jks=wk⊤gj is the k-th dimension y^js and wk is the k-th column of W. We minimize the entropy of the source samples in Us as follows:(18)minW,GO2W,G=−∑j:xjs∈Us∑k=1cwk⊤gjlogwk⊤gj.Target Domain. In the target domain, we also have labeled set denoted as ℒt and unlabeled set denoted as Ut. To use the labeled samples of the target domain, instead of learning a classifier in the target domain space, we design a classifier in the target domain sample proxy space. For a proxy z, we apply a linear classifier function to approximate its label vector:(19)y^t=V⊤z,where V∈Rdt×c is the classifier parameter and y^t∈ℝc is the estimated class label response vector. For a labeled sample, xit∈ℒt, its class label vector yit∈1,0c is known, and we minimize the classification loss measured by the squared loss between yit and y^it:(20)minV,ZO3V,Z=∑qi:xit∈Ltyit−V⊤zi22.Meanwhile, for the samples in the target domainUt, we also hope their uncertainty can be minimized, and accordingly, we minimize the entropy of the classification responses:(21)minV,ZO4V,Z=−∑i:xit∈Ut∑k=1cvk⊤zilogvk⊤zi.


#### 2.1.3. Manifold Regularization

Moreover, we also hope the classification responses of samples in both source and target domains can be smooth over their manifold. To be specific, the classification responses of two neighboring samples should also be similar. To this end, we first construct the nearest neighbor graphs for both the source and target domains and then use the graphs to regularize their classification responses.Source Domain. We firstly build a convolutional graph from the convolutional representations of the source domain samples g1,…,gm. For each sample xjs, its nearest neighbor set is defined as Njs, and we calculate its affinity between itself and each neighbor xj′s∈Njs based on a Gaussian kernel:(22)Ajj′s=ψgj−gj′∑j″:xj′'s∈Njsψgj−gj″,ifxj′s∈Njs,0,otherwise,where ψx=exp−x2/σ2 is the Gaussian kernel function. Based on the neighborhood affinity, we hope the squared ℓ2 norm distance between gj and its neighbors can be minimized so that the neighborhood structure can be kept in the convolutional representation space:(23)minGO5G=∑j,j′=1mAjj′sgj−gj′22.Target Domain. In the target domain, we firstly construct the graph from the original convolutional representations of the target samples f1,…,fn, not the proxies. Given a target sample xit, we find its nearest neighbors and denote the set of nearest neighbors Nit. We calculate the similarity between xit and a sample xi′t∈Nit also by kernel function:(24)Aii′t=ψfi−fi′∑i″:xi″t∈Nitψfj−fi″,ifxi′t∈Nit,0,otherwise.With this affinity measure, different from the source domain, we use it to regularize the learning of the proxies of the target domain,z1,…,zn, i.e.,(25)minZO6Z=∑i,i′=1mAii′tzi−zi′22.


#### 2.1.4. Overall Objective Function

Our overall object is the combination of the above objectives:(26)OF,G,Z,A,W,V,θ,Φ=MF,G,Z,A;ϕ∗+C1O1W,G+C2O2W,G+C3O3V,Z+C4O4V,Z+C5O5G+C6O6Z+C7W22+V22+Θ22+Φ22=2n∑i=1nhϕ1+ϕ3Tfi+ϕ2−ϕ3TZi+1n2∑i,i′=1nhϕ1+ϕ3Tfi+ϕ2−ϕ3TZihϕ1+ϕ3Tfi′+ϕ2−ϕ3T+C1∑j:xJs∈Lsyjs−WT−gj22−C2∑j:xJs∈Us∑k=1cWkTgjlogWkTgj+C3∑i:xit∈Ltyit−VTZi22−C4∑i:xit∈Ut∑k=1cVkTZilogVkTZi+C5∑j,j′−1mAjj′sgj−gj′22+C6∑i,i′=1mAii′tZ−Zi′22+C7W22+V22+Θ22+Φ22,s.t.∀i=1,…,n:fi=fxit;Θ,zi=∑j=1mαijgj,∑j=1mαij=1,and1≥αij≥0,∀j=1,…,m:gj=gxjs;Φ,where A=α11,⋯,αn1⋮⋱⋮α1m,⋯,αnm,∈ℝ+m×n and Ci,i=1,…,7 are the tradeoff parameters, which weight the loss terms of the objective. The last term is the squared ℓ2 norms of the mode parameters to prevent the overfitting problem. In this objective, we impose the following condition:(1)
In the target domain, we hope the SMI between the samples and their proxies constructed from the source domain can be maximized so that the source and target domains can be matched well.(2)
In both target and source domains, we hope the classification function can approximate the ground truth of labeled well, while the uncertainty of classification results of unlabeled can be minimized.(3)
Again, in both source and target domains, we hope the classification results can be similar among the neighborhoods.(4)
Finally, we hope the overall complexity of the model can be minimized as much as possible; thus, we minimize the squaredℓ2 norm of the parameters of the model in the last term.With this objective, we model the learning problem as a minimization problem:(27)minF,G,Z,A,W,V,Θ,ΦOF,G,Z,A,W,V,Θ,Φ,s.t.∀i=1,…,n:fi=fxit;Θ,zi=∑j=1mαijgj,∑j=1mαij=1,and1≥αij≥0,∀j=1,…,m:gj=gxjs;Φ.By solving this problem, we can learn the source and target CNN modelsW and V and the classification layer parameters Θ and Φ. Moreover, the proxy parameters of the target samples, A, are also optimized together with the model parameters to match the source and target domains.


### 2.2. Problem Optimization

To solve the problem at (27), we adopt the ADMM algorithm, and according, we have the following argued Lagrangian function:(28)ℒF,G,Z,A,W,V,Θ,Φ,π,Ψ,Ξ,ς=2n∑i=1nhϕ1+ϕ3Tfi+ϕ2+ϕ3Tzi+1n2∑i,i′=1nhϕ1+ϕ3Tfi+ϕ2+ϕ3Tzihϕ1+ϕ3Tfi′+ϕ2+ϕ3Tzi′+C1∑j:xsj∈Lsyjs−WTgj22+C2∑j:xsj∈Us∑k=1cWkTgjlogWkTgj+C3∑i:xit∈LTyit−VTzi22−C4∑i:xit∈UT∑k=1cVkTzilogVkTzi+C5∑j,j′=1mAjj′sgj−gj′22+C6∑i,i′=1mAii′tgz−gz′22+C7W22+V22+Θ22+Φ22+∑i=1nπiTfi−fxit;Θ+ρ12∑i=1nfi−fxit;Θ22+∑j=1mϖjTgj−gxjs;Φ+ρ22∑j=1ngj−gxjs;Φ22+∑i=1nτiTzi−∑j=1mαijgj+ρ32∑i=1nzi−∑j=1mαijgj22+∑i=1nςiT∑j=1mαij−1+ρ42∑i=1n∑j=1mαij−1s.t.∀i=1,…,n,j=1,…,m:1≥αij≥0,where the following holds:(i)
πi is the dual variable of constraint fi=fxit;Θ, ϖj is the dual variable of constraint gj=gxjs;Φ, τi is the dual variable of constraint zi=∑j=1mαijgj, and ςi is the dual variable of constraint ∑j=1mαij=1.(ii)
Π=π1,…,πn,Ψ=ϖ1,…,ϖm,Ξ=τ1,…,τn, and =ς1,⋯,ςn⊤.(iii)
ρk,k=1,…,4 are the corresponding penalty parameters of the constraints.According to the ADMM algorithm, we optimize the parameters and the dual variables iteratively as follows.Optimizing F. To optimize the target convolutional vectors of F, we update fi one by one. The subgradient of ℒ with regard to fi is(29)∇fiℒfi=1n2∑i′=1nhϕ1+ϕ3⊤fi′+ϕ2−ϕ3⊤zi′−2n×∇hϕ1+ϕ3⊤fi+ϕ2−ϕ3⊤ziϕ1+ϕ3+πi+ρ1fi−fxit;Θ,where∇hx=hx×1−hx.The subgradient descent step to updatefi is(30)fi←fi−η∇fiℒfi.Optimizing G. Similar to F, we also optimize gj of G one by one, and the following subgradient algorithm is used to update gj:(31)gj←gj−η∇gjℒgj,where∇gjℒgj=2C1Ixjs∈ℒsWyjs−W⊤gj−C2Ixjs∈Us∑k=1clogwk⊤gj+wk⊤gjlogwk⊤gjwk+2C5∑j′=1m+ϖj+ρ2∑j=1mAjj′sgj−gj′gj−gxjs;Φ−∑i=1nαijτi+ρ32∑i=1n∑j′=1mαijαij′gj′−2αijzi,where Ix=1 if x is true and 0 otherwise.Optimizing Z. We use the subgradient descent algorithm to update zi one by one, and the updating rule is(32)zi←zi−η∇ziℒziwhere∇ziℒzi=1n2∑i′=1nhϕ1+ϕ3⊤fi′+ϕ2−ϕ3⊤zi′−2n×∇hϕ1+ϕ3⊤fi+ϕ2−ϕ3⊤ziϕ2−ϕ3−2C3Ixit∈LtVyit−V⊤zi−C4Ixit∈Ut∑k=1clogvk⊤zi+vk⊤zilogvk⊤zivk+2C6∑i′=1mAii′tzi−zi′+τi+ρ3zi−∑j=1mαijgj.OptimizingΘ. The gradient descent algorithm is applied to update Θ:(33)Θ←Θ−η∇ΘℒΘ,where∇ΘℒΘ=∑i=1n∇fxit;Θℋfxit;Θ×∇Θfxit;Θ+2C7Θ,ℋfxit;Θ=πi⊤fi−fxit;Θ+ρ12fi−fxit;Θ22,∇fxit;Θℋfxit;Θ=−πi−ρ1fi−fxit;Θ,and ∇Θfxit;Θ is the gradient function of the convolutional network function with regard to the network parameters, usually the convolutional filters.OptimizingΦ. We use the gradient descent algorithm to optimize Φ to minimize ℒΦ:(34)Φ←Φ−ηΦ∇ΦℒΦ,where∇Φℒ3Φ=∇gxjs;ΦGgxjs;Φ×∇Φgxjs;Φ+2C7ΦGgxjs;Φ=ϖj⊤gj−gxjs;Φ+ρ22gj−gxjs;Φ22,∇gxjs;ΦGgxjs;Φ=−∑j=1mϖj−ρ2∑j=1mgj−gxjs;Φ.OptimizingW. To optimize the classifier parameter in W, we also update the columns one by one. To optimize the k-th column, we use the following subgradient descent step:(35)wk←wk−η∇wkℒwk,where∇wkℒwk=−2C1∑j:xjs∈Lsyjsk−wk⊤gjgj−C2∑j:xjs∈Uswk⊤gj+1logwk⊤gjgj+2C7wk.OptimizingV. To optimize V, we also update the columns one by one. To update the k-th column vk, we have the following subgradient descent step:(36)vk←vk−η∇vkOvk,where∇vkOvk=−2C1∑j:xit∈Ltyitk−vk⊤zizi−C2∑j:xit∈Usvk⊤zi+1logvk⊤zizi+2C7vk.Optimizing A. The optimization of the proxy construction coefficient matrix is a constrained minimization problem as follows:(37)minALA=∑i=1nτi⊤zi−∑j=1mαijgj+ρ32∑i=1nzi−∑j=1mαijgj22+∑i=1nςi⊤∑j=1mαij−1+ρ42∑i=1n∑j=1mαij−122,s.t.∀i=1,…,n,j=1,…,m:1≥αij≥0.Again, we optimize the columns ofA one by one. To optimize the i-th column αi of A, we have the following subproblem:(38)minαiτi⊤zi−Gαi+ρ32zi−Gαi22+ςi⊤1⊤αi−1+ρ421⊤αi−122,s.t.1≥αi≥0,where 1 is an all-one vector and 1 is an all-zero vector. This is a typical quadratic programming (QP) problem, and we use the active set algorithm to solve it.OptimizingΠ,Ψ,Ξ,andς. To optimize the dual variables, we have the following maximization problem:(39)maxΠ,Ψ,Ξ,ςℒΠ,Ψ,Ξ,ς=∑i=1nπi⊤fi−fxit;Θ+∑j=1mϖj⊤gj−gxjs;Φ+∑i=1nτi⊤zi−∑j=1mαijgj+∑i=1nςi⊤∑j=1mαij−1=TrΠ⊤N+TrΨ⊤J+TrΞ⊤D+ς⊤e,whereN=f1−fx1t;Θ,…,fn−fxnt;Θ,J=g1−gx1s;Φ,…,gm−gxms;Φ,D=z1−∑j=1mα1jgj,…,zn−∑j=1mαnjgj,ande=∑j=1mα1j−1,…,∑j=1mαnj−1,are the residual matrices and vector and TrX is the trace of matrix X. To solve this problem, we use the gradient-ascent algorithm, and the updating rules are(40)Π←Π+ε∇ΠℒΠ,Ψ,Ξ,ς,∇ΠℒΠ,Ψ,Ξ,ς=N,Ψ←Ψ+ε∇ΨℒΠ,Ψ,Ξ,ς,∇ΨℒΠ,Ψ,Ξ,ς=J,Ξ←Ξ+ε∇ΞℒΠ,Ψ,Ξ,ς,∇ΞℒΠ,Ψ,Ξ,ς=D,ς←ς+ε∇ςℒΠ,Ψ,Ξ,ς,and∇ςℒΠ,Ψ,Ξ,ς=e,where ϵ is the ascent step size.Besides solving the minimization problem, we also need to solve the parameter of the SMI term of (13) ϕ∗. To learn the parameter of SMI^,ϕ∗, according to (10), we have the following minimization problem:(41)ϕ∗=minϕSϕ=−2n∑i=1nrfi,zi;ϕ+1n2∑i,i′=1nrfi,zi;ϕrfi′,zi′;ϕ=−2n∑i=1nhϕ⊤ri+1n2∑i,i′=1nhϕ⊤rihϕ⊤ri′,whereri=fi;zi;fi−zi.The gradient function ofSϕ regarding ϕ is(42)∇Sϕϕ=−2n∑i=1n∇ϕhϕ⊤riri+1n2∑i,i′=1n∇ϕhϕ⊤rihϕ⊤ri′ri+hϕ⊤ri∇ϕhϕ⊤ri′ri′.We use the gradient descent algorithm to updateϕ to minimize Sϕ:(43)ϕ←ϕ−η∇Sϕϕ.


## 2.1. Objective Function

Suppose we have a training set of two domains. In the target domain, we haven training samples x1t,…,xnt, where xit is the i-th training sample of the target domain. To represent a target domain sample xt, we use a deep CNN model, f, to convert it to a dt-dimensional vector:(1)f=fxt;Θ∈Rdt.Meanwhile, we have a set of source domain training setx1s,…,xms, which has m samples and xjs is the j-th source domain training sample. Similar to the target domain, we use another deep CNN model, g, to convert a source domain sample, xs to a ds-dimensional vector:(2)g=gxs;Φ∈Rds.To learn the network parameters Θ and Φ, we discuss the following problems.


### 2.1.1. Squared-Loss Mutual Information

Due to the mismatch of the source and target domains, we propose to leverage the two domains by constructing a proxy for each target domain sample in the source domain space. To this send, for a target domain sample,xti, we denote its proxy zi∈Rds, as a ds-dimensional vector. Moreover, we impose that it can be reconstructed by a learning combination of the convolutional representations of source domain samples:(3)zi=∑j=1mαijgj,s.t.∑j=1mαij=1,1≥αij≥0,where gj=gxjs;Φ is the convolutional vector of the j-th source domain sample and αij is the nonnegative weight of the j-th source domain sample for the construction of the i-th proxy.To encourage the matching of the source and target domains, we argue to measure their quality of matching by the density ration of the proxies and convolutional vectors of the target domain samplesz and f. For this purpose, we firstly measure the density ratio of z and f, denoted as rf,z. According to the definition of density ratio [18],(4)rf,z=pf,zpfpz,where pf,z is the joint probability of f and z, pf is the probability of f, and pz is the probability of z. Since it is difficult to directly estimate the density ratio, we propose to learn a parametric density ratio estimator function:(5)r^f,z;ϕ=hϕ⊤fzf−z,where pf,z is the joint probability of f and z, pf is the probability of f, and pz is the probability of z. Since it is difficult to directly estimate the density ratio, we propose to learn a parametric density ratio estimator function. fzf−z is the concatenation of the f, z, and their element-wise difference vector, ϕ∈ℝ3ds is the parameter vector, and hx=1/1+exp−x is the Sigmoid activation function. To learn ϕ, we minimize the SMI between rf,z and r^f,z;ϕ, defined as follows:(6)SMIϕ=∫f∫zrf,z−r^f,z;ϕ2pfpzdfdz=∫f∫zrf,z2−2rf,zr^f,z;ϕ+r^f,z;ϕ2pfpzdfdz=∫f∫zrf,z2pfpzdfdz−2∫f∫zrf,zr^f,z;ϕpfpzdfdz+∫f∫zr^f,z;ϕ2pfpzdfdz=∫f∫zrf,z2pfpzdfdz−2∫f∫zpf,zr^f,z;ϕdfdz+∫f∫zr^f,z;ϕ2pfpzdfdz.To estimate the SMI given the training samples of target domain and their proxies, we collect the set of convolutional vectors and corresponding proxiesf1,z1,…,fn,zn. We assume the distributions of f, z, and f,z are uniform distributions, which lead the probability functions as(7)pf=1n,pz=1n,pf,z=1n.With probabilities in (7) and the empirical approximation, we rewrite the second and third terms of (6) as follows:(8)∫f∫zpf,zrf,z;ϕdfdz=1n∫f∫zrf,z;ϕdfdz=1n∑i=1nrfi,zi;ϕ,∫f∫zr^f,z;ϕ2pfpzdfdz=1n2∑i,i′=1nr^fi,zi;ϕr^fi′,zi′;ϕ.By substituting (8) to (6), we have the approximated SMI as(9)SMIϕ=C−2n∑i=1nrfi,zi;ϕ+1n2∑i,i′=1nrfi,zi;ϕrfi′,zi′;ϕ,where C=∫f∫zrf,z2pfpzdfdz is a constant. To obtain a good quality estimator, we propose to minimize the squared-loss mutual information learned from the parameter of r^f,z;ϕ:(10)ϕ∗=argminϕSMIϕ.With this optimalϕ∗, we have to use the SMI as a measure of the matching quality between f and z. Using this measure as a term of loss for the purpose of learning the convolutional representations f and proxy z, we organize these parameters as matrices as F=f1,…,fn and Z=z1,…,zn. With these variables and the SMI as the matching measure, we seek to learn them to maximize the SMI. In other words, we minimize a loss function derived from SMI, defined as ℳF,Z;ϕ∗:(11)ℳF,Z;ϕ∗=2n∑i=1nr^fi,zi;ϕ∗−1n2∑i,i′=1nr^fi,zi;ϕ∗r^fi′,zi′;ϕ∗=−2n∑i=1nhϕ∗⊤fizifi−zi+1n2∑i,i′=1nhϕ∗⊤fizifi−zihϕ∗⊤fi′zi′fi′−zi′.We rewrite(12)ϕ∗⊤fizifi−zi=ϕ1⊤fi+ϕ2⊤zi+ϕ3⊤fi−zi=ϕ1+ϕ3⊤fi+ϕ2−ϕ3⊤zi,where ϕ∗=ϕ1ϕ2ϕ3 accordingly.(13)ℳF,Z;ϕ∗=−2n∑i=1nhϕ1+ϕ3⊤fi+ϕ2−ϕ3⊤zi+1n2∑i,i′=1nhϕ1+ϕ3⊤fi+ϕ2−ϕ3⊤zihϕ1+ϕ3⊤fi′+ϕ2−ϕ3⊤zi′.The squared-loss mutual information regularization problem is modelled as(14)minF,ZℳF,Z;ϕ∗.By minimizing this objective, we seek to learn the convolutional representation vectors of both source and target domain samples, target domain sample proxies, and other parameters to match the two domains as much as possible. Please note that, in this problem, we are not matching the source and target domain samples directly but seek to match target domain samples and the proxies of target domain samples in the source domain space.


### 2.1.2. Semisupervised Classification

In both the source and target domains, there are both labeled and unlabeled data samples. To learn from them, we use the semisupervised method. We discuss the semisupervised learning problems in both source and target domains as follows.Source Domain. To approximate the class label vector, ys∈1,0c, of a source domain sample, xs, from its convolutional feature vector, g, we design a linear classification function:(15)ys=W⊤g,where W∈ℝds×c is the parameter vector of the classifier and y^s∈ℝc is the classification response. To learn the classifier parameter W, we utilize both the labeled and unlabeled samples of the source domain. We denote the set of the labeled samples of source domain as ℒs and the set of unlabeled as Us. For the samples in ℒs, we minimize their classification errors measured by the squared loss:(16)minW,GO1W,G=∑j:xjs∈Lsyjs−W⊤gj22,where yjs is the class label vector of the xj and G=g1,…,gm. For the samples in ℒs, even we do not have the available labels to constrain the classification responses in y^s, and we still can regularize the learning of the parameter by its entropy. For a sample xjs, we hope the uncertainty of its classification responses in y^js is as small as possible. The uncertainty is measured by its entropy:(17)Hyjs=−∑k=1cyjkslogyjks=−∑k=1cwk⊤gjlogwk⊤gj,wherey^jks=wk⊤gj is the k-th dimension y^js and wk is the k-th column of W. We minimize the entropy of the source samples in Us as follows:(18)minW,GO2W,G=−∑j:xjs∈Us∑k=1cwk⊤gjlogwk⊤gj.Target Domain. In the target domain, we also have labeled set denoted as ℒt and unlabeled set denoted as Ut. To use the labeled samples of the target domain, instead of learning a classifier in the target domain space, we design a classifier in the target domain sample proxy space. For a proxy z, we apply a linear classifier function to approximate its label vector:(19)y^t=V⊤z,where V∈Rdt×c is the classifier parameter and y^t∈ℝc is the estimated class label response vector. For a labeled sample, xit∈ℒt, its class label vector yit∈1,0c is known, and we minimize the classification loss measured by the squared loss between yit and y^it:(20)minV,ZO3V,Z=∑qi:xit∈Ltyit−V⊤zi22.Meanwhile, for the samples in the target domainUt, we also hope their uncertainty can be minimized, and accordingly, we minimize the entropy of the classification responses:(21)minV,ZO4V,Z=−∑i:xit∈Ut∑k=1cvk⊤zilogvk⊤zi.


### 2.1.3. Manifold Regularization

Moreover, we also hope the classification responses of samples in both source and target domains can be smooth over their manifold. To be specific, the classification responses of two neighboring samples should also be similar. To this end, we first construct the nearest neighbor graphs for both the source and target domains and then use the graphs to regularize their classification responses.Source Domain. We firstly build a convolutional graph from the convolutional representations of the source domain samples g1,…,gm. For each sample xjs, its nearest neighbor set is defined as Njs, and we calculate its affinity between itself and each neighbor xj′s∈Njs based on a Gaussian kernel:(22)Ajj′s=ψgj−gj′∑j″:xj′'s∈Njsψgj−gj″,ifxj′s∈Njs,0,otherwise,where ψx=exp−x2/σ2 is the Gaussian kernel function. Based on the neighborhood affinity, we hope the squared ℓ2 norm distance between gj and its neighbors can be minimized so that the neighborhood structure can be kept in the convolutional representation space:(23)minGO5G=∑j,j′=1mAjj′sgj−gj′22.Target Domain. In the target domain, we firstly construct the graph from the original convolutional representations of the target samples f1,…,fn, not the proxies. Given a target sample xit, we find its nearest neighbors and denote the set of nearest neighbors Nit. We calculate the similarity between xit and a sample xi′t∈Nit also by kernel function:(24)Aii′t=ψfi−fi′∑i″:xi″t∈Nitψfj−fi″,ifxi′t∈Nit,0,otherwise.With this affinity measure, different from the source domain, we use it to regularize the learning of the proxies of the target domain,z1,…,zn, i.e.,(25)minZO6Z=∑i,i′=1mAii′tzi−zi′22.


### 2.1.4. Overall Objective Function

Our overall object is the combination of the above objectives:(26)OF,G,Z,A,W,V,θ,Φ=MF,G,Z,A;ϕ∗+C1O1W,G+C2O2W,G+C3O3V,Z+C4O4V,Z+C5O5G+C6O6Z+C7W22+V22+Θ22+Φ22=2n∑i=1nhϕ1+ϕ3Tfi+ϕ2−ϕ3TZi+1n2∑i,i′=1nhϕ1+ϕ3Tfi+ϕ2−ϕ3TZihϕ1+ϕ3Tfi′+ϕ2−ϕ3T+C1∑j:xJs∈Lsyjs−WT−gj22−C2∑j:xJs∈Us∑k=1cWkTgjlogWkTgj+C3∑i:xit∈Ltyit−VTZi22−C4∑i:xit∈Ut∑k=1cVkTZilogVkTZi+C5∑j,j′−1mAjj′sgj−gj′22+C6∑i,i′=1mAii′tZ−Zi′22+C7W22+V22+Θ22+Φ22,s.t.∀i=1,…,n:fi=fxit;Θ,zi=∑j=1mαijgj,∑j=1mαij=1,and1≥αij≥0,∀j=1,…,m:gj=gxjs;Φ,where A=α11,⋯,αn1⋮⋱⋮α1m,⋯,αnm,∈ℝ+m×n and Ci,i=1,…,7 are the tradeoff parameters, which weight the loss terms of the objective. The last term is the squared ℓ2 norms of the mode parameters to prevent the overfitting problem. In this objective, we impose the following condition:(1)
In the target domain, we hope the SMI between the samples and their proxies constructed from the source domain can be maximized so that the source and target domains can be matched well.(2)
In both target and source domains, we hope the classification function can approximate the ground truth of labeled well, while the uncertainty of classification results of unlabeled can be minimized.(3)
Again, in both source and target domains, we hope the classification results can be similar among the neighborhoods.(4)
Finally, we hope the overall complexity of the model can be minimized as much as possible; thus, we minimize the squaredℓ2 norm of the parameters of the model in the last term.With this objective, we model the learning problem as a minimization problem:(27)minF,G,Z,A,W,V,Θ,ΦOF,G,Z,A,W,V,Θ,Φ,s.t.∀i=1,…,n:fi=fxit;Θ,zi=∑j=1mαijgj,∑j=1mαij=1,and1≥αij≥0,∀j=1,…,m:gj=gxjs;Φ.By solving this problem, we can learn the source and target CNN modelsW and V and the classification layer parameters Θ and Φ. Moreover, the proxy parameters of the target samples, A, are also optimized together with the model parameters to match the source and target domains.


## 2.1.1. Squared-Loss Mutual Information

Due to the mismatch of the source and target domains, we propose to leverage the two domains by constructing a proxy for each target domain sample in the source domain space. To this send, for a target domain sample,xti, we denote its proxy zi∈Rds, as a ds-dimensional vector. Moreover, we impose that it can be reconstructed by a learning combination of the convolutional representations of source domain samples:(3)zi=∑j=1mαijgj,s.t.∑j=1mαij=1,1≥αij≥0,where gj=gxjs;Φ is the convolutional vector of the j-th source domain sample and αij is the nonnegative weight of the j-th source domain sample for the construction of the i-th proxy.To encourage the matching of the source and target domains, we argue to measure their quality of matching by the density ration of the proxies and convolutional vectors of the target domain samplesz and f. For this purpose, we firstly measure the density ratio of z and f, denoted as rf,z. According to the definition of density ratio [18],(4)rf,z=pf,zpfpz,where pf,z is the joint probability of f and z, pf is the probability of f, and pz is the probability of z. Since it is difficult to directly estimate the density ratio, we propose to learn a parametric density ratio estimator function:(5)r^f,z;ϕ=hϕ⊤fzf−z,where pf,z is the joint probability of f and z, pf is the probability of f, and pz is the probability of z. Since it is difficult to directly estimate the density ratio, we propose to learn a parametric density ratio estimator function. fzf−z is the concatenation of the f, z, and their element-wise difference vector, ϕ∈ℝ3ds is the parameter vector, and hx=1/1+exp−x is the Sigmoid activation function. To learn ϕ, we minimize the SMI between rf,z and r^f,z;ϕ, defined as follows:(6)SMIϕ=∫f∫zrf,z−r^f,z;ϕ2pfpzdfdz=∫f∫zrf,z2−2rf,zr^f,z;ϕ+r^f,z;ϕ2pfpzdfdz=∫f∫zrf,z2pfpzdfdz−2∫f∫zrf,zr^f,z;ϕpfpzdfdz+∫f∫zr^f,z;ϕ2pfpzdfdz=∫f∫zrf,z2pfpzdfdz−2∫f∫zpf,zr^f,z;ϕdfdz+∫f∫zr^f,z;ϕ2pfpzdfdz.To estimate the SMI given the training samples of target domain and their proxies, we collect the set of convolutional vectors and corresponding proxiesf1,z1,…,fn,zn. We assume the distributions of f, z, and f,z are uniform distributions, which lead the probability functions as(7)pf=1n,pz=1n,pf,z=1n.With probabilities in (7) and the empirical approximation, we rewrite the second and third terms of (6) as follows:(8)∫f∫zpf,zrf,z;ϕdfdz=1n∫f∫zrf,z;ϕdfdz=1n∑i=1nrfi,zi;ϕ,∫f∫zr^f,z;ϕ2pfpzdfdz=1n2∑i,i′=1nr^fi,zi;ϕr^fi′,zi′;ϕ.By substituting (8) to (6), we have the approximated SMI as(9)SMIϕ=C−2n∑i=1nrfi,zi;ϕ+1n2∑i,i′=1nrfi,zi;ϕrfi′,zi′;ϕ,where C=∫f∫zrf,z2pfpzdfdz is a constant. To obtain a good quality estimator, we propose to minimize the squared-loss mutual information learned from the parameter of r^f,z;ϕ:(10)ϕ∗=argminϕSMIϕ.With this optimalϕ∗, we have to use the SMI as a measure of the matching quality between f and z. Using this measure as a term of loss for the purpose of learning the convolutional representations f and proxy z, we organize these parameters as matrices as F=f1,…,fn and Z=z1,…,zn. With these variables and the SMI as the matching measure, we seek to learn them to maximize the SMI. In other words, we minimize a loss function derived from SMI, defined as ℳF,Z;ϕ∗:(11)ℳF,Z;ϕ∗=2n∑i=1nr^fi,zi;ϕ∗−1n2∑i,i′=1nr^fi,zi;ϕ∗r^fi′,zi′;ϕ∗=−2n∑i=1nhϕ∗⊤fizifi−zi+1n2∑i,i′=1nhϕ∗⊤fizifi−zihϕ∗⊤fi′zi′fi′−zi′.We rewrite(12)ϕ∗⊤fizifi−zi=ϕ1⊤fi+ϕ2⊤zi+ϕ3⊤fi−zi=ϕ1+ϕ3⊤fi+ϕ2−ϕ3⊤zi,where ϕ∗=ϕ1ϕ2ϕ3 accordingly.(13)ℳF,Z;ϕ∗=−2n∑i=1nhϕ1+ϕ3⊤fi+ϕ2−ϕ3⊤zi+1n2∑i,i′=1nhϕ1+ϕ3⊤fi+ϕ2−ϕ3⊤zihϕ1+ϕ3⊤fi′+ϕ2−ϕ3⊤zi′.The squared-loss mutual information regularization problem is modelled as(14)minF,ZℳF,Z;ϕ∗.By minimizing this objective, we seek to learn the convolutional representation vectors of both source and target domain samples, target domain sample proxies, and other parameters to match the two domains as much as possible. Please note that, in this problem, we are not matching the source and target domain samples directly but seek to match target domain samples and the proxies of target domain samples in the source domain space.


## 2.1.2. Semisupervised Classification

In both the source and target domains, there are both labeled and unlabeled data samples. To learn from them, we use the semisupervised method. We discuss the semisupervised learning problems in both source and target domains as follows.Source Domain. To approximate the class label vector, ys∈1,0c, of a source domain sample, xs, from its convolutional feature vector, g, we design a linear classification function:(15)ys=W⊤g,where W∈ℝds×c is the parameter vector of the classifier and y^s∈ℝc is the classification response. To learn the classifier parameter W, we utilize both the labeled and unlabeled samples of the source domain. We denote the set of the labeled samples of source domain as ℒs and the set of unlabeled as Us. For the samples in ℒs, we minimize their classification errors measured by the squared loss:(16)minW,GO1W,G=∑j:xjs∈Lsyjs−W⊤gj22,where yjs is the class label vector of the xj and G=g1,…,gm. For the samples in ℒs, even we do not have the available labels to constrain the classification responses in y^s, and we still can regularize the learning of the parameter by its entropy. For a sample xjs, we hope the uncertainty of its classification responses in y^js is as small as possible. The uncertainty is measured by its entropy:(17)Hyjs=−∑k=1cyjkslogyjks=−∑k=1cwk⊤gjlogwk⊤gj,wherey^jks=wk⊤gj is the k-th dimension y^js and wk is the k-th column of W. We minimize the entropy of the source samples in Us as follows:(18)minW,GO2W,G=−∑j:xjs∈Us∑k=1cwk⊤gjlogwk⊤gj.Target Domain. In the target domain, we also have labeled set denoted as ℒt and unlabeled set denoted as Ut. To use the labeled samples of the target domain, instead of learning a classifier in the target domain space, we design a classifier in the target domain sample proxy space. For a proxy z, we apply a linear classifier function to approximate its label vector:(19)y^t=V⊤z,where V∈Rdt×c is the classifier parameter and y^t∈ℝc is the estimated class label response vector. For a labeled sample, xit∈ℒt, its class label vector yit∈1,0c is known, and we minimize the classification loss measured by the squared loss between yit and y^it:(20)minV,ZO3V,Z=∑qi:xit∈Ltyit−V⊤zi22.Meanwhile, for the samples in the target domainUt, we also hope their uncertainty can be minimized, and accordingly, we minimize the entropy of the classification responses:(21)minV,ZO4V,Z=−∑i:xit∈Ut∑k=1cvk⊤zilogvk⊤zi.


## 2.1.3. Manifold Regularization

Moreover, we also hope the classification responses of samples in both source and target domains can be smooth over their manifold. To be specific, the classification responses of two neighboring samples should also be similar. To this end, we first construct the nearest neighbor graphs for both the source and target domains and then use the graphs to regularize their classification responses.Source Domain. We firstly build a convolutional graph from the convolutional representations of the source domain samples g1,…,gm. For each sample xjs, its nearest neighbor set is defined as Njs, and we calculate its affinity between itself and each neighbor xj′s∈Njs based on a Gaussian kernel:(22)Ajj′s=ψgj−gj′∑j″:xj′'s∈Njsψgj−gj″,ifxj′s∈Njs,0,otherwise,where ψx=exp−x2/σ2 is the Gaussian kernel function. Based on the neighborhood affinity, we hope the squared ℓ2 norm distance between gj and its neighbors can be minimized so that the neighborhood structure can be kept in the convolutional representation space:(23)minGO5G=∑j,j′=1mAjj′sgj−gj′22.Target Domain. In the target domain, we firstly construct the graph from the original convolutional representations of the target samples f1,…,fn, not the proxies. Given a target sample xit, we find its nearest neighbors and denote the set of nearest neighbors Nit. We calculate the similarity between xit and a sample xi′t∈Nit also by kernel function:(24)Aii′t=ψfi−fi′∑i″:xi″t∈Nitψfj−fi″,ifxi′t∈Nit,0,otherwise.With this affinity measure, different from the source domain, we use it to regularize the learning of the proxies of the target domain,z1,…,zn, i.e.,(25)minZO6Z=∑i,i′=1mAii′tzi−zi′22.


## 2.1.4. Overall Objective Function

Our overall object is the combination of the above objectives:(26)OF,G,Z,A,W,V,θ,Φ=MF,G,Z,A;ϕ∗+C1O1W,G+C2O2W,G+C3O3V,Z+C4O4V,Z+C5O5G+C6O6Z+C7W22+V22+Θ22+Φ22=2n∑i=1nhϕ1+ϕ3Tfi+ϕ2−ϕ3TZi+1n2∑i,i′=1nhϕ1+ϕ3Tfi+ϕ2−ϕ3TZihϕ1+ϕ3Tfi′+ϕ2−ϕ3T+C1∑j:xJs∈Lsyjs−WT−gj22−C2∑j:xJs∈Us∑k=1cWkTgjlogWkTgj+C3∑i:xit∈Ltyit−VTZi22−C4∑i:xit∈Ut∑k=1cVkTZilogVkTZi+C5∑j,j′−1mAjj′sgj−gj′22+C6∑i,i′=1mAii′tZ−Zi′22+C7W22+V22+Θ22+Φ22,s.t.∀i=1,…,n:fi=fxit;Θ,zi=∑j=1mαijgj,∑j=1mαij=1,and1≥αij≥0,∀j=1,…,m:gj=gxjs;Φ,where A=α11,⋯,αn1⋮⋱⋮α1m,⋯,αnm,∈ℝ+m×n and Ci,i=1,…,7 are the tradeoff parameters, which weight the loss terms of the objective. The last term is the squared ℓ2 norms of the mode parameters to prevent the overfitting problem. In this objective, we impose the following condition:(1)
In the target domain, we hope the SMI between the samples and their proxies constructed from the source domain can be maximized so that the source and target domains can be matched well.(2)
In both target and source domains, we hope the classification function can approximate the ground truth of labeled well, while the uncertainty of classification results of unlabeled can be minimized.(3)
Again, in both source and target domains, we hope the classification results can be similar among the neighborhoods.(4)
Finally, we hope the overall complexity of the model can be minimized as much as possible; thus, we minimize the squaredℓ2 norm of the parameters of the model in the last term.With this objective, we model the learning problem as a minimization problem:(27)minF,G,Z,A,W,V,Θ,ΦOF,G,Z,A,W,V,Θ,Φ,s.t.∀i=1,…,n:fi=fxit;Θ,zi=∑j=1mαijgj,∑j=1mαij=1,and1≥αij≥0,∀j=1,…,m:gj=gxjs;Φ.By solving this problem, we can learn the source and target CNN modelsW and V and the classification layer parameters Θ and Φ. Moreover, the proxy parameters of the target samples, A, are also optimized together with the model parameters to match the source and target domains.


## 2.2. Problem Optimization

To solve the problem at (27), we adopt the ADMM algorithm, and according, we have the following argued Lagrangian function:(28)ℒF,G,Z,A,W,V,Θ,Φ,π,Ψ,Ξ,ς=2n∑i=1nhϕ1+ϕ3Tfi+ϕ2+ϕ3Tzi+1n2∑i,i′=1nhϕ1+ϕ3Tfi+ϕ2+ϕ3Tzihϕ1+ϕ3Tfi′+ϕ2+ϕ3Tzi′+C1∑j:xsj∈Lsyjs−WTgj22+C2∑j:xsj∈Us∑k=1cWkTgjlogWkTgj+C3∑i:xit∈LTyit−VTzi22−C4∑i:xit∈UT∑k=1cVkTzilogVkTzi+C5∑j,j′=1mAjj′sgj−gj′22+C6∑i,i′=1mAii′tgz−gz′22+C7W22+V22+Θ22+Φ22+∑i=1nπiTfi−fxit;Θ+ρ12∑i=1nfi−fxit;Θ22+∑j=1mϖjTgj−gxjs;Φ+ρ22∑j=1ngj−gxjs;Φ22+∑i=1nτiTzi−∑j=1mαijgj+ρ32∑i=1nzi−∑j=1mαijgj22+∑i=1nςiT∑j=1mαij−1+ρ42∑i=1n∑j=1mαij−1s.t.∀i=1,…,n,j=1,…,m:1≥αij≥0,where the following holds:(i)
πi is the dual variable of constraint fi=fxit;Θ, ϖj is the dual variable of constraint gj=gxjs;Φ, τi is the dual variable of constraint zi=∑j=1mαijgj, and ςi is the dual variable of constraint ∑j=1mαij=1.(ii)
Π=π1,…,πn,Ψ=ϖ1,…,ϖm,Ξ=τ1,…,τn, and =ς1,⋯,ςn⊤.(iii)
ρk,k=1,…,4 are the corresponding penalty parameters of the constraints.According to the ADMM algorithm, we optimize the parameters and the dual variables iteratively as follows.Optimizing F. To optimize the target convolutional vectors of F, we update fi one by one. The subgradient of ℒ with regard to fi is(29)∇fiℒfi=1n2∑i′=1nhϕ1+ϕ3⊤fi′+ϕ2−ϕ3⊤zi′−2n×∇hϕ1+ϕ3⊤fi+ϕ2−ϕ3⊤ziϕ1+ϕ3+πi+ρ1fi−fxit;Θ,where∇hx=hx×1−hx.The subgradient descent step to updatefi is(30)fi←fi−η∇fiℒfi.Optimizing G. Similar to F, we also optimize gj of G one by one, and the following subgradient algorithm is used to update gj:(31)gj←gj−η∇gjℒgj,where∇gjℒgj=2C1Ixjs∈ℒsWyjs−W⊤gj−C2Ixjs∈Us∑k=1clogwk⊤gj+wk⊤gjlogwk⊤gjwk+2C5∑j′=1m+ϖj+ρ2∑j=1mAjj′sgj−gj′gj−gxjs;Φ−∑i=1nαijτi+ρ32∑i=1n∑j′=1mαijαij′gj′−2αijzi,where Ix=1 if x is true and 0 otherwise.Optimizing Z. We use the subgradient descent algorithm to update zi one by one, and the updating rule is(32)zi←zi−η∇ziℒziwhere∇ziℒzi=1n2∑i′=1nhϕ1+ϕ3⊤fi′+ϕ2−ϕ3⊤zi′−2n×∇hϕ1+ϕ3⊤fi+ϕ2−ϕ3⊤ziϕ2−ϕ3−2C3Ixit∈LtVyit−V⊤zi−C4Ixit∈Ut∑k=1clogvk⊤zi+vk⊤zilogvk⊤zivk+2C6∑i′=1mAii′tzi−zi′+τi+ρ3zi−∑j=1mαijgj.OptimizingΘ. The gradient descent algorithm is applied to update Θ:(33)Θ←Θ−η∇ΘℒΘ,where∇ΘℒΘ=∑i=1n∇fxit;Θℋfxit;Θ×∇Θfxit;Θ+2C7Θ,ℋfxit;Θ=πi⊤fi−fxit;Θ+ρ12fi−fxit;Θ22,∇fxit;Θℋfxit;Θ=−πi−ρ1fi−fxit;Θ,and ∇Θfxit;Θ is the gradient function of the convolutional network function with regard to the network parameters, usually the convolutional filters.OptimizingΦ. We use the gradient descent algorithm to optimize Φ to minimize ℒΦ:(34)Φ←Φ−ηΦ∇ΦℒΦ,where∇Φℒ3Φ=∇gxjs;ΦGgxjs;Φ×∇Φgxjs;Φ+2C7ΦGgxjs;Φ=ϖj⊤gj−gxjs;Φ+ρ22gj−gxjs;Φ22,∇gxjs;ΦGgxjs;Φ=−∑j=1mϖj−ρ2∑j=1mgj−gxjs;Φ.OptimizingW. To optimize the classifier parameter in W, we also update the columns one by one. To optimize the k-th column, we use the following subgradient descent step:(35)wk←wk−η∇wkℒwk,where∇wkℒwk=−2C1∑j:xjs∈Lsyjsk−wk⊤gjgj−C2∑j:xjs∈Uswk⊤gj+1logwk⊤gjgj+2C7wk.OptimizingV. To optimize V, we also update the columns one by one. To update the k-th column vk, we have the following subgradient descent step:(36)vk←vk−η∇vkOvk,where∇vkOvk=−2C1∑j:xit∈Ltyitk−vk⊤zizi−C2∑j:xit∈Usvk⊤zi+1logvk⊤zizi+2C7vk.Optimizing A. The optimization of the proxy construction coefficient matrix is a constrained minimization problem as follows:(37)minALA=∑i=1nτi⊤zi−∑j=1mαijgj+ρ32∑i=1nzi−∑j=1mαijgj22+∑i=1nςi⊤∑j=1mαij−1+ρ42∑i=1n∑j=1mαij−122,s.t.∀i=1,…,n,j=1,…,m:1≥αij≥0.Again, we optimize the columns ofA one by one. To optimize the i-th column αi of A, we have the following subproblem:(38)minαiτi⊤zi−Gαi+ρ32zi−Gαi22+ςi⊤1⊤αi−1+ρ421⊤αi−122,s.t.1≥αi≥0,where 1 is an all-one vector and 1 is an all-zero vector. This is a typical quadratic programming (QP) problem, and we use the active set algorithm to solve it.OptimizingΠ,Ψ,Ξ,andς. To optimize the dual variables, we have the following maximization problem:(39)maxΠ,Ψ,Ξ,ςℒΠ,Ψ,Ξ,ς=∑i=1nπi⊤fi−fxit;Θ+∑j=1mϖj⊤gj−gxjs;Φ+∑i=1nτi⊤zi−∑j=1mαijgj+∑i=1nςi⊤∑j=1mαij−1=TrΠ⊤N+TrΨ⊤J+TrΞ⊤D+ς⊤e,whereN=f1−fx1t;Θ,…,fn−fxnt;Θ,J=g1−gx1s;Φ,…,gm−gxms;Φ,D=z1−∑j=1mα1jgj,…,zn−∑j=1mαnjgj,ande=∑j=1mα1j−1,…,∑j=1mαnj−1,are the residual matrices and vector and TrX is the trace of matrix X. To solve this problem, we use the gradient-ascent algorithm, and the updating rules are(40)Π←Π+ε∇ΠℒΠ,Ψ,Ξ,ς,∇ΠℒΠ,Ψ,Ξ,ς=N,Ψ←Ψ+ε∇ΨℒΠ,Ψ,Ξ,ς,∇ΨℒΠ,Ψ,Ξ,ς=J,Ξ←Ξ+ε∇ΞℒΠ,Ψ,Ξ,ς,∇ΞℒΠ,Ψ,Ξ,ς=D,ς←ς+ε∇ςℒΠ,Ψ,Ξ,ς,and∇ςℒΠ,Ψ,Ξ,ς=e,where ϵ is the ascent step size.Besides solving the minimization problem, we also need to solve the parameter of the SMI term of (13) ϕ∗. To learn the parameter of SMI^,ϕ∗, according to (10), we have the following minimization problem:(41)ϕ∗=minϕSϕ=−2n∑i=1nrfi,zi;ϕ+1n2∑i,i′=1nrfi,zi;ϕrfi′,zi′;ϕ=−2n∑i=1nhϕ⊤ri+1n2∑i,i′=1nhϕ⊤rihϕ⊤ri′,whereri=fi;zi;fi−zi.The gradient function ofSϕ regarding ϕ is(42)∇Sϕϕ=−2n∑i=1n∇ϕhϕ⊤riri+1n2∑i,i′=1n∇ϕhϕ⊤rihϕ⊤ri′ri+hϕ⊤ri∇ϕhϕ⊤ri′ri′.We use the gradient descent algorithm to updateϕ to minimize Sϕ:(43)ϕ←ϕ−η∇Sϕϕ.


## 3. Experiments

In this section, we experimentally evaluate the proposed algorithm over benchmark data sets.


### 3.1. Benchmark Data Sets

In our experiment, we use the following benchmark data sets:(i)
Amazon review data set. This data set has four domains of four products, including books, DVD, and music. For each domain, there are 2,000 positive reviews and 2,000 negative reviews. To fit the content of the review to the CNN network, we firstly tokenize the review to a sequence of words and secondly represent each word to by word embedding and then use the 1D-CNN as the CNN model.(ii)
UCSD anomaly detection data set. This data set has 70 sequences of video, which contains 13,900 frames. This data set is used as a source domain. Meanwhile, we also collected 122 sequences of videos from subway stations, malls, and communities in China as a target domain. The target domain has 24,400 frames. Both target and source domain data sets have 6 classes of anomalies.(iii)
TRECVID video concept detection data set. This data set has two domains of data. One domain is the data of TRECVID 2005, which has 61,901 frames from video, while the other one domain is the data of TRECVID 2007, which has 21,532 frames. The classification problem is to categorize one frame to one of the 36 semantic concepts.(iv)
Protein subcellular localization data set. The last data set is a protein data set. It is composed of three domains. The first domain is the MultiLoc data set which has 5,859 proteins, the second one is the BaCelLoc date set with 4,286 proteins, and the last data set Euk-mPLoc has 5,618 proteins. The problem of this data is to predict the subcellular locations of each protein from the amino acid sequence. To this end, we first map each amino acid by embedding technology and then use the 1D-CNN model to extract the features.The statistics of the data sets are summarized in Table1.Table 1
Summary of data sets.
Data set# domain# sampleData typeAmazon312,000TextUCSD238,300VideoTRECVID283,433VideoProtein315,763Protein


### 3.2. Experimental Setting

To perform the experiment, we design a leave-one-out protocol to use each domain as a target domain, while the other domains are combined as one single source domain. With each target domain-source domain configuration, we use the 10-fold cross-validation protocol to perform the training test process. The target domain data set is split into ten folds, one fold is used as the test set, and the remaining nine folds are combined with the source domain as the training set. The model is learned over the training set and tested over the test set. To measure the performance of the method, we calculate the average classification accuracy over different target domains.


### 3.3. Experimental Results

In our experiments, we firstly study the properties of the proposed method experimentally and then compare it to the other state-of-the-art methods.


### 3.4. Convergence Analysis

Since the algorithm is an iterative algorithm, we are interested in the convergence of the algorithm. We plot the curves of the objective values against the iteration number in Figure1. From this figure, we can see that, for all four benchmark data sets, the algorithm converges well at 50 iterations. For the protein data set, the object value keeps decreasing but cannot reach a lower value after 50 iterations. The possible reason is that the algorithm finds a local minimum object instead of a global minimum. The overall conclusion for the convergence of the algorithm is for all the tested data sets, and the algorithm can converge at certain number of iterations. This is due to the effective alternate optimization method.Figure 1
Convergence curves.


### 3.5. Tradeoff Parameter Analysis

We have seven tradeoff parameters in the objective, which control the weights of different objective terms. We also study how the performance of the algorithm changes with the change in these tradeoff parameters. The classification accuracies with different values of the tradeoff parameter values are shown in Figure2.Figure 2
Tradeoff parameter analysis: (a) UCSD and (b) protein.
(a)(b)From this figure, we have the following observations:(i)
The increasingC1 and C3 improve the performance significantly since it is the supervision term of the labeled data. This means the supervision of the labeled plays an important role in the learning of predictive models in the transfer learning problem.(ii)
The increasingC2 and C4 also boost the performance; however, it is not as significant as the class error terms’ tradeoff parameters. The reduction of entropy of the unlabeled samples’ classification responses from both domains also improves the performance.(iii)
C5 and C6 are the tradeoff parameters of two domains’ manifold regularization terms. Their increasing values also improve the accuracy of the target domain prediction. This means the neighborhood smoothness is also important for the transfer learning problem.(iv)
The performance is stable to the change inC7, the tradeoff value of the squared ℓ2 norm term, which means this term did not play a critical role in the learning problem.


### 3.6. Comparison to Other Methods

We compare our proposed algorithm against several state-of-the-art CNN-based transfer learning methods, including the methods developed by Haque et al. [11, 12], Zhang et al. [14], Long et al. [21], and Pan and Yang [7]. The comparison results are shown in Figure 3.Figure 3
Comparison to state-of-the-arts.From this figure, we can see the proposed method always outperforms the other methods in all cases. The second best method is Long et al. method. The results show the advantage of our method, especially the SMI-guided domain-transfer CNN learning and the proxy mapping mechanism. Among the four data sets, the most difficult one is the TRECVID, where all the classification accuracies of different methods are below 0.75. However, in this data set, our methods give the most significant improvements over the other methods. It is the only method which achieves an accuracy higher than 0.72.


## 3.1. Benchmark Data Sets

In our experiment, we use the following benchmark data sets:(i)
Amazon review data set. This data set has four domains of four products, including books, DVD, and music. For each domain, there are 2,000 positive reviews and 2,000 negative reviews. To fit the content of the review to the CNN network, we firstly tokenize the review to a sequence of words and secondly represent each word to by word embedding and then use the 1D-CNN as the CNN model.(ii)
UCSD anomaly detection data set. This data set has 70 sequences of video, which contains 13,900 frames. This data set is used as a source domain. Meanwhile, we also collected 122 sequences of videos from subway stations, malls, and communities in China as a target domain. The target domain has 24,400 frames. Both target and source domain data sets have 6 classes of anomalies.(iii)
TRECVID video concept detection data set. This data set has two domains of data. One domain is the data of TRECVID 2005, which has 61,901 frames from video, while the other one domain is the data of TRECVID 2007, which has 21,532 frames. The classification problem is to categorize one frame to one of the 36 semantic concepts.(iv)
Protein subcellular localization data set. The last data set is a protein data set. It is composed of three domains. The first domain is the MultiLoc data set which has 5,859 proteins, the second one is the BaCelLoc date set with 4,286 proteins, and the last data set Euk-mPLoc has 5,618 proteins. The problem of this data is to predict the subcellular locations of each protein from the amino acid sequence. To this end, we first map each amino acid by embedding technology and then use the 1D-CNN model to extract the features.The statistics of the data sets are summarized in Table1.Table 1
Summary of data sets.
Data set# domain# sampleData typeAmazon312,000TextUCSD238,300VideoTRECVID283,433VideoProtein315,763Protein


## 3.2. Experimental Setting

To perform the experiment, we design a leave-one-out protocol to use each domain as a target domain, while the other domains are combined as one single source domain. With each target domain-source domain configuration, we use the 10-fold cross-validation protocol to perform the training test process. The target domain data set is split into ten folds, one fold is used as the test set, and the remaining nine folds are combined with the source domain as the training set. The model is learned over the training set and tested over the test set. To measure the performance of the method, we calculate the average classification accuracy over different target domains.


## 3.3. Experimental Results

In our experiments, we firstly study the properties of the proposed method experimentally and then compare it to the other state-of-the-art methods.


## 3.4. Convergence Analysis

Since the algorithm is an iterative algorithm, we are interested in the convergence of the algorithm. We plot the curves of the objective values against the iteration number in Figure1. From this figure, we can see that, for all four benchmark data sets, the algorithm converges well at 50 iterations. For the protein data set, the object value keeps decreasing but cannot reach a lower value after 50 iterations. The possible reason is that the algorithm finds a local minimum object instead of a global minimum. The overall conclusion for the convergence of the algorithm is for all the tested data sets, and the algorithm can converge at certain number of iterations. This is due to the effective alternate optimization method.Figure 1
Convergence curves.


## 3.5. Tradeoff Parameter Analysis

We have seven tradeoff parameters in the objective, which control the weights of different objective terms. We also study how the performance of the algorithm changes with the change in these tradeoff parameters. The classification accuracies with different values of the tradeoff parameter values are shown in Figure2.Figure 2
Tradeoff parameter analysis: (a) UCSD and (b) protein.
(a)(b)From this figure, we have the following observations:(i)
The increasingC1 and C3 improve the performance significantly since it is the supervision term of the labeled data. This means the supervision of the labeled plays an important role in the learning of predictive models in the transfer learning problem.(ii)
The increasingC2 and C4 also boost the performance; however, it is not as significant as the class error terms’ tradeoff parameters. The reduction of entropy of the unlabeled samples’ classification responses from both domains also improves the performance.(iii)
C5 and C6 are the tradeoff parameters of two domains’ manifold regularization terms. Their increasing values also improve the accuracy of the target domain prediction. This means the neighborhood smoothness is also important for the transfer learning problem.(iv)
The performance is stable to the change inC7, the tradeoff value of the squared ℓ2 norm term, which means this term did not play a critical role in the learning problem.


## 3.6. Comparison to Other Methods

We compare our proposed algorithm against several state-of-the-art CNN-based transfer learning methods, including the methods developed by Haque et al. [11, 12], Zhang et al. [14], Long et al. [21], and Pan and Yang [7]. The comparison results are shown in Figure 3.Figure 3
Comparison to state-of-the-arts.From this figure, we can see the proposed method always outperforms the other methods in all cases. The second best method is Long et al. method. The results show the advantage of our method, especially the SMI-guided domain-transfer CNN learning and the proxy mapping mechanism. Among the four data sets, the most difficult one is the TRECVID, where all the classification accuracies of different methods are below 0.75. However, in this data set, our methods give the most significant improvements over the other methods. It is the only method which achieves an accuracy higher than 0.72.


## 4. Conclusions

In this paper, we proposed a novel CNN-based transfer learning method. We construct a proxy for each target domain sample in the source domain space and use the SMI as a measure to match the constructed proxy and the convolutional representation of a target domain sample. The learning of the proxy construction parameters and the CNN parameters are learned simultaneously by a unified learning framework. The estimation and parameter learning of the SMI is also driven by the proxy and CNN outputs. Thus, this framework can optimize the CNN, proxy, and SMI parameters jointly and automatically. Experimental results show the advantages of the proposed framework and algorithm.



---

*Source: 8873137-2020-10-14.xml*