# Discriminative Extreme Learning Machine with Cross-Domain Mean Approximation for Unsupervised Domain Adaptation

Authors: Shaofei Zang; Xinghai Li; Jianwei Ma; Yongyi Yan; Jinfeng Lv; Yuan Wei
Journal: Complexity
 (2022)

---

## Abstract

Extreme Learning Machine (ELM) is widely used in various fields because of its fast training and high accuracy. However, it does not primarily work well for Domain Adaptation (DA) in which there are many annotated data from auxiliary domain and few even no annotated data in target domain. In this paper, we propose a new variant of ELM called Discriminative Extreme Learning Machine with Cross-Domain Mean Approximation (DELM-CDMA) for unsupervised domain adaptation. It introduces Cross-Domain Mean Approximation (CDMA) into the hidden layer of ELM to reduce distribution discrepancy between domains for domain bias elimination, which is conducive to train a high accuracy ELM on annotated data from auxiliary domains for target tasks. Linear Discriminative Analysis (LDA) is also adopted to improve the discrimination of learned model and obtain higher accuracy. Moreover, we further provide a Discriminative Kernel Extreme Learning Machine with Cross-Domain Mean Approximation (DKELM-CDMA) as the kernelization extension of DELM-CDMA. Some experiments are performed to investigate the proposed approach, and the result shows that DELM-CDMA and DKELM-CDMA could effectively extend ELM suitable for domain adaptation and outperform ELM and many other domain adaptation approaches.


---

## Body

## 1. Introduction

It is a challenge for mankind to extract or mine valuable information from massive data generated from mobile devices, Internet, and industrial sensors in the current society. Classifiers based on machine learning play an important role in data and information processing systems. Extreme Learning Machine (ELM) [1] attracts attention due to its faster learning and higher accuracy compared with k Nearest Neighbor (kNN) [2], Back-Propagating (BP) [3], Naive Bayes (NB) [4], Support Vector Machine (SVM) [5], and Decision Tree (DT) [6], and has been widely promoted in many fields including image classification [7], traffic system [8], COVID-19 detection [9], fault diagnosis [10, 11], hyperspectral remote sensing images [12, 13], industrial sensors [14], facial expression recognition [15], and brain-computer interface (BCI) [16, 17] etc.Due to its fast learning and strong generalization capability, Extreme Learning Machine attracted more attention. It randomly selects the input weights and the biases of hidden layer neurons without training data, and only obtains the optimal solution of output weight by minimizing the training error and the norm of output weights simultaneously [18]. Since the hidden layer parameters are randomly initialized and the output weights are solved by finding a least squares solution, the training time of the model is greatly reduced [1]. Recently, many variants of ELM have emerged for improving its performance and have been divided into three parts: supervised ELMs, semi-supervised, and unsupervised ones. Supervised ELMs need numerous labeled data to ensure its high performance, such as Kernel Extreme Learning Machine (KELM) [19], Weighted Extreme Learning Machine (WELM) [20], Twin Extreme Machines (TELM) [21], and Adaptive Regularized Extreme Learning Machine (A-RELM) [22]. Semi-supervised ELM usually requires unlabeled data together with labeled data to train models well, including Laplacian Twin Extreme Learning Machine (Lap-TELM) [23], Semi-Supervised Extreme Learning Machine (SS-ELM) [24], Robust Semi-Supervised Extreme Learning Machine (RSS-ELM) [25], and Adaptive Safe Semi-Supervised Extreme Learning Machine (AdSafe-SSELM) [26]. In some other cases where no labeled data are available, some Unsupervised ELM (USELM) algorithms are proposed for clustering, dimension reduction, or data representation, such as Unsupervised Extreme Learning Machine (USELM) [24], Extreme Learning Machine as an Auto-Encoder (ELM-AE) [27], Enhanced Unsupervised Extreme Learning Machine (EUELM) [28], and Unsupervised Feature Selection based Extreme Learning Machine (UFSELM) [29]. Moreover, as deep learning has been successful in many fields, deep ELMs are also developed to extract more abstract and expressive features, such as Kernel-based Multi-Layer Extreme Learning Machine ELM (ML-KELM) [30], Hierarchical-ELM (H-ELM) [31], DS-ELM (a deep and stable extreme learning machine) [32], and Deep Residual Compensation Extreme Learning Machine (DRC-ELM) [33]. Although the above algorithms expand the application of ELM in various scenarios, they perform not well when training data are not related to the testing data.In application, it is difficult to collect a large amount of data which keeps consistent with test data because of condition change, data noise, view variety, and so on. To address this issue, Domain Adaptation (DA) [34–36], as an important transfer learning technology, can remedy this shortcoming of ELM, in which abundant labeled data from other domains (source domain) is applied to help the current domain (target domain) and there is inconsistency in distribution between domains. Consequently, L. Zhang and D. Zhang [37] provided two ELMs with domain adaptation including Source Domain Adaptation Extreme Learning Machine (DAELM-S) and Target Domain Adaptation Extreme Learning Machine (DAELM-T). They can improve the generalization ability of ELM in multiple domains. Similar to DAELM, Zhao and Chen [38] also developed One Stage-Transfer-Learning ELM (OSTL-ELM) and Two-Stage-Transfer-Learning ELM (TSTL-ELM) to handle the problem of insufficient labeled data from target domains in aero engine fault diagnosis tasks. With the help of subspace alignment and the weight approximation, Zang et al. [39] put forward Transfer Extreme Learning Machine with Output Weight Alignment (TELM-OWA) to handle domain adaptation. TL-ELM (Transfer Learning-based ELM) was proposed by Li et al. [40], in which an output weight from two domains were forced to be close to each other for knowledge transfer. In the methods mentioned above, it is a requirement that there are few labeled data in target domains. However, annotating data are costly, laborious, and time-consuming. Consequently, some unsupervised models appear for unsupervised domain adaptation. Chen et al. [41] presented an transfer ELM, in which output weight alignment was applied to reduce domain bias and L2, 1-norm was imposed on output weight to enhance feature selection. For minimizing the distribution discrepancy between source and target domains, Li et al. [42], Chen et al. [43], and Zang et al. [44] utilized Maximum Mean Discrepancy (MMD) [45] to promote knowledge transfer in their respective models. Due to the insufficient target sample labels, the performance of unsupervised models is usually lower than that of supervised models, but it is hard to collect labeled target samples.In this paper, we propose a novel ELM called Discriminative Extreme Learning Machine with Cross-Domain Mean Approximation (DELM-CDMA) for unsupervised domain adaptation. It introduces Cross-Domain Mean Approximation (CDMA) [36] into the objective function of ELM to jointly adapt the marginal and conditional distributions discrepancy between the output of hidden layers from source and target samples. CDMA could enhance the ability of transferring knowledge across domains. Moreover, to improve the discrimination, Linear Discriminant Analysis (LDA) [46] is also added into the objective function. It separates the samples with different categories and clusters the samples with the same category, which can improve the accuracy of ELM. Finally, we solve the designed objective function and obtain an optimal ELM model for domain adaptation. Moreover, we further present a Discriminative Kernel Extreme Learning Machine with Cross-Domain Mean Approximation (DKELM-CDMA) which not only kernelize DELM-CDMA suitable for nonlinear data but also eliminate the sensitivity of its accuracy to parameter initialization. DELM-CDMA is illustrated in Figure 1. Extensive experiments are carried out to verify the effectiveness and superiority of DELM-CDMA and DKELM-CDMA for unsupervised domain adaptation.Figure 1
An illustration of DELM-CDMA. (1) Cross-domain mean approximation (CDMA) is adopted to reduce the domain shift. (2) Linear discriminant analysis (LDA) is added into object function to enhance the discrimination ability of DELM-CDMA. (3) The objective function is solved efficiently.Contributions of this paper are as follows:(1)
Joint distribution adaptation based on CDMA measure is introduced into ELM to narrow distribution differences between domains. We apply LDA to boost the class separability of DELM-CDMA and thus improves its accuracy.(2)
As a kernel version of DELM-CDMA, it is proposed to enhance the robustness for initialization of the parameters of hidden layers and the adaptability to nonlinear data.(3)
We efficiently solve a least-squares problem for the objective function, and obtain an optimal ELM for unsupervised domain adaptation. Moreover, classification experiments on object recognition and text data sets are performed to investigate our approaches, and the results show that DELM-CDMA and DKELM-CDMA can efficiently achieve the capability of transferring knowledge across domains.The rest of this paper is summarized as follows: We describe domain adaptation, CDMA and ELM very simply in Section2. Then, we represent DELM-CDMA and DKELM-CDMA in Section 3. Next, the experiment is performed in Section 4 to verify the validity of DELM-CDMA. Finally, Section 5 is the conclusion of this paper.


## 2. Related Work

Since our method mainly extends ELM to handle domain adaptation, we will briefly introduce domain adaptation, CDMA and ELM in this section.


### 2.1. Domain Adaptation

Transfer learning [34, 36, 47, 48], as one of the important research branches of machine learning, helps target tasks to learn a high-quality model with the knowledge from source domains which have rich labeled data/samples but different distributions with target domains. Domain adaptation [36] is a hot topic in transfer learning and aims to adjust the distribution between domains in order to eliminate domain shift. In recent years, a large number of approaches have emerged for domain adaptation and have been roughly divided into five categories according to what is adapted, that is, instance adaptation, feature adaptation, parameter adaptation, deep network adaptation, and adversarial domain adaptation.(1)
Instance Adaptation. Most instance-adaptation methods attempt to find a strategy in which each instance is assigned a weight to balance the distribution difference between domains. Li et al. [49] proposed Prediction Reweighting for Domain Adaptation (PRDA) which first reweighed classifier training on source domain and then adopted manifold regularization to diffuse the labels from high confidence samples to low confidence ones. TrAdaBoost [50], as a classic domain adaptation method, weighs each sample in source and target domains by a dynamic mechanism to minimize the distribution discrepancy between domains. Moreover, bad samples and good samples from source domains are distinguished by weight, which promotes the knowledge transferring across domains.(2)
Feature Adaptation. This method usually finds shared feature subspace to reduce the domain distribution bias and transfer knowledge across domains. Many technologies such as K-L discrepancy [51], Bregman divergence [52], MMD [53], and Subspace alignment [54–56] are used to minimize distribution discrepancy and seek optimal common features between domains. Transfer Component Analysis (TCA) [57] and Joint Distribution Adaptation (JDA) [53] solve a projection matrix minimizing MMD measurement from source and target domains and extract shared features to obtain effective target learner training on source domain.(3)
Parameter Adaptation. Those approaches do not transfer shared knowledge in instance or feature levels but find optimal shared parameters of learner across domains. The DAELM-S(T), TELM-OWA, TL-ELM, and so on mentioned above belong to this method.(4)
Deep Network Adaptation. This method combines the traditional domain adaptation technology with the deep learning model. It not only uses the former to reduce the distribution differences between domains but also applies the deep network structure of the latter to extract the high-level respective and semantic features across domains, which enables more efficient knowledge transfer. Domain Adaptation Network (DAN) [58], Joint Adaptation Network (JAN) [59], and Residual Transfer Network (RTN) [60] all utilize MMD to reduce the distribution differences between different domains and the data set bias.(5)
Adversarial Domain Adaptation. Based on the generative adversarial network, this method produces target sample/data using domain-invariant feature generators, and subsequently applies the domain discriminator training on source samples to find the weakness of generated samples for generator adaptation, which could reduce the domain gap. As an unsupervised domain adaptation method based on deep feed-forward architecture, Domain Adversarial Neural Network (DANN) [61] introduces adversarial learning for domain adaptation. It simultaneously learns classifiers, feature extractors, and domain discriminators and obtains domain-invariant feature representation by minimizing classifier errors and maximizing the discriminant errors. Collaborative and Adversarial Network (CAN) [62] first uses collaborative learning to distinguish the domain-informative features of the sample belonging to source domain or target domain, and then utilizes adversarial learning to learn difficult-distinguish domain-uninformative features. Finally, domain-invariant features could be extracted.In the methods mentioned in Section 2.1, the instance adaptation method has higher knowledge transferring efficiency, but it usually requires high-quality samples to ensure its excellent performance. Feature adaptation method has higher flexibility and wider application, but finding the shared feature with better generalization is a challenge. Due to the lack of direct utilization of sample and feature information, the parameter adaptation method usually has lower efficiency of knowledge transfer compared with the two former ones, but its domain knowledge utilization is more diversified. Deep network adaptation and adversarial domain adaptation have good performance owing to higher-level feature extraction. They strictly belong to feature adaptation method because obtaining good features is their primary task. However, Deep network adaptation usually requires plenty of labeled samples, more consuming-time and memory, simultaneously finding efficient feature extractors and discriminators is also a challenge for adversarial domain adaptation. In this paper, DELM-CDMA belongs to parameter adaptation method, and DKELM-CDMA is shallow feature adaptation method.


### 2.2. Cross-Domain Mean Approximation (CDMA)

To address the problem that MMD ignores sample individual difference, and thus insufficiently mining local information of data, Zang et al. [36] designed Cross-Domain Mean Approximation (CDMA), as shown in Figure 2. Given source domain xSi,ySii=1nS with nS sample xSi and according to its label ySi, and target domain xTj,yTjj=1nT including nT sample xTj and according to its label yTj.The distribution discrepancy between source and target domains based on CDMA can be defined as follows:(1)LCDMADS,DT=∑i=1nS+nTxi−μ22,where μ=μT,ifxi∈DSμS,ifxi∈DT, μTS is the mean vector of the target (source) domain samples. Moreover, to adapt the marginal and conditional distribution together, label information of sample is added into CDMA, and (1) is modified as given in the following equation:(2)LCDMADS,DT=∑i=1nS+nTxi−μ22+∑c=1C∑i=1nSic+nTicxic−μic22,where μc=μTc,ifxic∈DScμSc,ifxic∈DTc, μTSc is the mean vector of the target (source) domain DTSc with c category.Figure 2
An illustration of CDMA.ELM is a single-layer neural network with fast learning and high accuracy, without tuning input weight and bias. It efficiently obtains optimal output weight by solving a least squares problem. Given a data setxi,yii=1N including N samples xi with label yi. A classic ELM network is given:CDMA is an efficient evaluation metric of the distribution discrepancy between domains, which is adopted in our DELM-CDMA and DKELM-CDMA to reduce the marginal and conditional distribution discrepancy. Different from Joint Distribution Adaptation based on Cross Domain Mean Approximation (JDA-CDMA) [36] which uses CDMA to construct the objective function and incorporates subspace learning for extracting shared feature between the source and target domains, DELM-CDMA applies CDMA into the hidden layer of ELM for domain bias elimination and enhancing the generalization of the parameters of learned ELM and DKELM-CDMA applies CDMA together with reconstruction error for shared feature extraction in KELM framework.


### 2.3. Extreme Learning Machine (ELM)

ELM is a single-layer neural network with fast learning and high accuracy, without tuning input weight and bias. It efficiently obtains optimal output weight by solving a least squares problem. Given a data setxi,yii=1N including N samples xi with label yi. A classic ELM network is given by the following equation:(3)yiT=∑j=1LβjTgwjxi+bj,where xi is an input sample, wj and bj are the input layer weight and bias, respectively, gx represents the nonlinear activation function, L represents the number of nodes in the hidden layer, and βj represents the hidden layer output weight. We can obtain an optimal β∗ by solving the objective function as given by the following equation:(4)minβ:12β2+λ2∑i=1Nεi2s.t.gxiβ=yiT−εiT,i=1,…,N,where β2 is applied to avoid model overfitting, εi is the error vector corresponding to the empirical risk of sample xi with the tradeoff parameter λ. The constraint is removed and equation (4) changes into the following equation:(5)LELM=minβ:12β2+λ2Y−Hβ2,where H=gxiT,…,gxNTT, β=β1⋮βL, and Y=y1⋮yN.The objective function is considered as a ridge regression or a regular least square problem. We set∂LELM/∂β=0 and the optimal solution is got according to [18] as given by the following equation:(6)β∗=HTH+ILλHTY,N>L,HTHTH+INλY,N≤L.Then, the prediction result of test samplexTe can be determined by the following equation:(7)yTe=signhTeTβ∗,for binary classification,argmaxhTeTβ∗,for multi−classification,where hTe=gxTe.


## 2.1. Domain Adaptation

Transfer learning [34, 36, 47, 48], as one of the important research branches of machine learning, helps target tasks to learn a high-quality model with the knowledge from source domains which have rich labeled data/samples but different distributions with target domains. Domain adaptation [36] is a hot topic in transfer learning and aims to adjust the distribution between domains in order to eliminate domain shift. In recent years, a large number of approaches have emerged for domain adaptation and have been roughly divided into five categories according to what is adapted, that is, instance adaptation, feature adaptation, parameter adaptation, deep network adaptation, and adversarial domain adaptation.(1)
Instance Adaptation. Most instance-adaptation methods attempt to find a strategy in which each instance is assigned a weight to balance the distribution difference between domains. Li et al. [49] proposed Prediction Reweighting for Domain Adaptation (PRDA) which first reweighed classifier training on source domain and then adopted manifold regularization to diffuse the labels from high confidence samples to low confidence ones. TrAdaBoost [50], as a classic domain adaptation method, weighs each sample in source and target domains by a dynamic mechanism to minimize the distribution discrepancy between domains. Moreover, bad samples and good samples from source domains are distinguished by weight, which promotes the knowledge transferring across domains.(2)
Feature Adaptation. This method usually finds shared feature subspace to reduce the domain distribution bias and transfer knowledge across domains. Many technologies such as K-L discrepancy [51], Bregman divergence [52], MMD [53], and Subspace alignment [54–56] are used to minimize distribution discrepancy and seek optimal common features between domains. Transfer Component Analysis (TCA) [57] and Joint Distribution Adaptation (JDA) [53] solve a projection matrix minimizing MMD measurement from source and target domains and extract shared features to obtain effective target learner training on source domain.(3)
Parameter Adaptation. Those approaches do not transfer shared knowledge in instance or feature levels but find optimal shared parameters of learner across domains. The DAELM-S(T), TELM-OWA, TL-ELM, and so on mentioned above belong to this method.(4)
Deep Network Adaptation. This method combines the traditional domain adaptation technology with the deep learning model. It not only uses the former to reduce the distribution differences between domains but also applies the deep network structure of the latter to extract the high-level respective and semantic features across domains, which enables more efficient knowledge transfer. Domain Adaptation Network (DAN) [58], Joint Adaptation Network (JAN) [59], and Residual Transfer Network (RTN) [60] all utilize MMD to reduce the distribution differences between different domains and the data set bias.(5)
Adversarial Domain Adaptation. Based on the generative adversarial network, this method produces target sample/data using domain-invariant feature generators, and subsequently applies the domain discriminator training on source samples to find the weakness of generated samples for generator adaptation, which could reduce the domain gap. As an unsupervised domain adaptation method based on deep feed-forward architecture, Domain Adversarial Neural Network (DANN) [61] introduces adversarial learning for domain adaptation. It simultaneously learns classifiers, feature extractors, and domain discriminators and obtains domain-invariant feature representation by minimizing classifier errors and maximizing the discriminant errors. Collaborative and Adversarial Network (CAN) [62] first uses collaborative learning to distinguish the domain-informative features of the sample belonging to source domain or target domain, and then utilizes adversarial learning to learn difficult-distinguish domain-uninformative features. Finally, domain-invariant features could be extracted.In the methods mentioned in Section 2.1, the instance adaptation method has higher knowledge transferring efficiency, but it usually requires high-quality samples to ensure its excellent performance. Feature adaptation method has higher flexibility and wider application, but finding the shared feature with better generalization is a challenge. Due to the lack of direct utilization of sample and feature information, the parameter adaptation method usually has lower efficiency of knowledge transfer compared with the two former ones, but its domain knowledge utilization is more diversified. Deep network adaptation and adversarial domain adaptation have good performance owing to higher-level feature extraction. They strictly belong to feature adaptation method because obtaining good features is their primary task. However, Deep network adaptation usually requires plenty of labeled samples, more consuming-time and memory, simultaneously finding efficient feature extractors and discriminators is also a challenge for adversarial domain adaptation. In this paper, DELM-CDMA belongs to parameter adaptation method, and DKELM-CDMA is shallow feature adaptation method.


## 2.2. Cross-Domain Mean Approximation (CDMA)

To address the problem that MMD ignores sample individual difference, and thus insufficiently mining local information of data, Zang et al. [36] designed Cross-Domain Mean Approximation (CDMA), as shown in Figure 2. Given source domain xSi,ySii=1nS with nS sample xSi and according to its label ySi, and target domain xTj,yTjj=1nT including nT sample xTj and according to its label yTj.The distribution discrepancy between source and target domains based on CDMA can be defined as follows:(1)LCDMADS,DT=∑i=1nS+nTxi−μ22,where μ=μT,ifxi∈DSμS,ifxi∈DT, μTS is the mean vector of the target (source) domain samples. Moreover, to adapt the marginal and conditional distribution together, label information of sample is added into CDMA, and (1) is modified as given in the following equation:(2)LCDMADS,DT=∑i=1nS+nTxi−μ22+∑c=1C∑i=1nSic+nTicxic−μic22,where μc=μTc,ifxic∈DScμSc,ifxic∈DTc, μTSc is the mean vector of the target (source) domain DTSc with c category.Figure 2
An illustration of CDMA.ELM is a single-layer neural network with fast learning and high accuracy, without tuning input weight and bias. It efficiently obtains optimal output weight by solving a least squares problem. Given a data setxi,yii=1N including N samples xi with label yi. A classic ELM network is given:CDMA is an efficient evaluation metric of the distribution discrepancy between domains, which is adopted in our DELM-CDMA and DKELM-CDMA to reduce the marginal and conditional distribution discrepancy. Different from Joint Distribution Adaptation based on Cross Domain Mean Approximation (JDA-CDMA) [36] which uses CDMA to construct the objective function and incorporates subspace learning for extracting shared feature between the source and target domains, DELM-CDMA applies CDMA into the hidden layer of ELM for domain bias elimination and enhancing the generalization of the parameters of learned ELM and DKELM-CDMA applies CDMA together with reconstruction error for shared feature extraction in KELM framework.


## 2.3. Extreme Learning Machine (ELM)

ELM is a single-layer neural network with fast learning and high accuracy, without tuning input weight and bias. It efficiently obtains optimal output weight by solving a least squares problem. Given a data setxi,yii=1N including N samples xi with label yi. A classic ELM network is given by the following equation:(3)yiT=∑j=1LβjTgwjxi+bj,where xi is an input sample, wj and bj are the input layer weight and bias, respectively, gx represents the nonlinear activation function, L represents the number of nodes in the hidden layer, and βj represents the hidden layer output weight. We can obtain an optimal β∗ by solving the objective function as given by the following equation:(4)minβ:12β2+λ2∑i=1Nεi2s.t.gxiβ=yiT−εiT,i=1,…,N,where β2 is applied to avoid model overfitting, εi is the error vector corresponding to the empirical risk of sample xi with the tradeoff parameter λ. The constraint is removed and equation (4) changes into the following equation:(5)LELM=minβ:12β2+λ2Y−Hβ2,where H=gxiT,…,gxNTT, β=β1⋮βL, and Y=y1⋮yN.The objective function is considered as a ridge regression or a regular least square problem. We set∂LELM/∂β=0 and the optimal solution is got according to [18] as given by the following equation:(6)β∗=HTH+ILλHTY,N>L,HTHTH+INλY,N≤L.Then, the prediction result of test samplexTe can be determined by the following equation:(7)yTe=signhTeTβ∗,for binary classification,argmaxhTeTβ∗,for multi−classification,where hTe=gxTe.


## 3. Proposed Methodology

In recent decades, ELM has been better than traditional classifiers of machine learning in speed and accuracy because of its ability in the generalization and global approximation. Thus, many variants of ELM appear for classification and regression in machine learning and pattern recognition to improve it in both theory and application. Nevertheless, ELM will be unsuccessful in the case of insufficient labeled samples [37]. In response to this issue, we develop Discriminative Extreme Learning Machine with Cross-Domain Mean Approximation (DELM-CDMA) for unsupervised domain adaptation, which use CDMA to narrow the marginal and conditional distributions between domains and adopt LDA to enhance category separability. Next, we will introduce it in detail in this Section 3.1.


### 3.1. Objective Function of DELM-CDMA

In unsupervised domain adaptation, given two different but related data sets: source domainDS=xsi,ysii=1nS owning nS sample xSi with label ySi, and target domain DT=xTjj=1nT including nT unlabeled sample xTj. We hope DELM-CDMA training on DS perform well on DT.First, we transformDS and DT to HS and HT using activation function gx, and then construct the objective function of DELM-CDMA as given by the following equation:(8)LDELM−CDMA=LELM+α1LCDMA+α2LLDA.From equation (8), we can see that there are three parts in the objective function of DELM-CDMA. LELM represents the loss function of regularized ELM, LCDMA denotes marginal or conditional CDMA measure between source and target domains, and LLDA is LDA term. α1 and α2 are tradeoff parameters of LCDMA and LLDA, respectively.


#### 3.1.1. Loss Function of Regularized ELM

In this paper, we carry out unsupervised domain adaptation task in which no labeled samples appear on target domain, so we can obtain a Regularized ELM on source domain as given by the following equation:(9)LELM=minβ:12β2+λ12HSβ−YS2.In equation (9), the first term is a regularizer to avoid over-fitting, and the second term is classification error of samples from source domain. λ1 is a parameter that balances the two terms.In the iterative refinement process of labels, when the pseudo labels of the target samplesY∧T are obtained, we could add the classification error from target domain, and equation (9) becomes equation (10).(10)LELM=minβ:12β2+λ12HSβ−YS2+HTβ−Y∧T2=minβ:12β2+λ12Hβ−Y2,where H=HSHT and Y=YSY∧T.


#### 3.1.2. Cross-Domain Mean Approximation Measure

In unsupervised domain adaptation, since source domain and target domain have distribution discrepancy, the prediction error of model training onDS become increases on target domain. CDMA is an effective strategy for evaluation of inter-domain distribution differences. We apply it to measure the distribution discrepancy of output-layer data of ELM training on source and target domains, and compute it as given by the following equation:(11)LCDMA=minβ:12HTβ−HS_avβ2+HSβ−HT_avβ2+12∑c=1CHTcβ−HS_avcβ2+HScβ−HT_avcβ2.In equation (11), the first term measures the marginal distribution discrepancy, and the second term measures the conditional distribution discrepancy.


#### 3.1.3. Linear Discriminant Analysis on Output Layer

To further improve the discrimination of ELM, we add Linear Discriminant Analysis (LDA) into it. LDA forces the samples with the same class closer and the ones with different class farther so as to enhance ELM category separability in unsupervised domain adaptation. We apply LDA on data from the output layer, in whichSb and Sw, respectively, denote the between-class scatter and the within-class scatter and could been computed as given by the following equation:(12)Sb=∑c=1CncβHc−βHavTβHc−βHav,(13)Sw=∑c=1C∑i=1ncβHc−βHavcTβHc−βHavc.In equations (12) and (13), Hav=HT_avHS_av, Havc=HT_avcHS_avc, Hc represents samples with c category in HS and HT. HS_av and HT_av are mean vectors of HS and HT, respectively. HTS_avc is the mean vector of the target (source) domain HTSc with c category. Then, we construct the loss of LDA on the output layer as given by the following equation:(14)LLDA=minβ:12Sw−Sb.


#### 3.1.4. Total Objective Function

To improve the ability of knowledge transferring and discrimination of ELM, DELM-CDMA joins CDMA and LDA to ELM. Therefore, we bring equations (9) or (10), (11) and (14) into (8), and then the objective function of DELM-CDMA is established:(15)LDELM−CDMA=LELM+α1LCDMA+α2LLDA=minβ:12β2+λ12Hβ−Y2+α12Hβ−Havβ2+∑c=1CHcβ−Havcβ2+α22βT∑c=1C∑i=1ncHc−HavcTHc−Havc−∑c=1CHavc−HavTHavc−Havβ.


### 3.2. Model Learning

To solve optimal output weight, we let∂LDELM−CDMA/∂β=0 and yield(16)β−λ1HTHβ−Y+α12H−HavTH−Hav+∑c=1CHc−HavcTHc−Havc+α2∑c=1C∑i=1ncHc−HavcTHc−Havc−∑c=1CHavc−HavTHavc−Havβ=0According to [18], equation (16) has a closed form solution:(17)β∗=HTH+α1H−HavTH−Hav+∑c=1CHc−HavcTHc−Havc+α2∑c=1C∑i=1ncHc−HavcTHc−Havc−∑c=1CHavc−HavTHavc−Hav+IL×Lλ1HTY,nS+nT>L,HTHHT+α1H−HavH−HavT+∑c=1CHc−HavcHc−HavcT+α2∑c=1C∑i=1ncHc−HavcHc−HavcT−∑c=1CHavc−HavHavc−HavT+InS+nT×nS+nTλ1Y,nS+nT≤L.After optimalβ∗ is obtained, the test samples are classified by equation (7). A complete classification procedure of DELM-CDMA is summarized in Algorithm 1.Algorithm 1: DELM-CDMA
Input: Data setDS and DT, tradeoff parameters α1, α2, and λ1. Output: Output layer weight β∗Step 1: Randomly initialize input weightsw and biases b of the hidden layer with L nodes, and set tradeoff parameters α1, α2, and λ1.Step 2: TransformDS=xsi,ysii=1nS and DT=xTjj=1nT to HS and HT using gwx+b, and set Sb and Sw to zero.Step 3: Calculateβ∗ according to equation (17).Step 4: Useβ∗ to predict DT and get its label Y∧T.Step 5: SolveSb and Sw by using DS, DT, and Y∧T according to equations (12) and (13).Step 6: Repeat steps 3–5 untilY∧T unchanged.


### 3.3. Kernelization

Although DELM-CDMA can improve the performance of ELM with fewer labeled samples or lack of conditions, it has two shortcomings: (1) The random initialization of hidden layer parameters will cause performance instability. (2) Inability to enhance the high-dimensional separability of nonlinear data. Therefore, we present a kernel version of DELM-CDMA named Discriminative Kernel Extreme Learning Machine with Cross-Domain Mean Approximation (DKELM-CDMA) to address above problems. DKELM-CDMA has two stages: feature extraction and classification.


#### 3.3.1. Feature Extraction

At this stage, we letK=HHT, Kij=gxigxjT=Ωxi,xj, Ωxi,xj is a kernel function such as sigmoid kernel, linear kernel, and radial basis kernel. The objective function of DKELM-CDMA is given by the following equation:(18)LDKELM−CDMA=LKELM+αk1LCDMA+αk2LLDA=minβk:12βk2+λ22Kβk−X2+αk12Kβk−Kavβk2+∑c=1CKcβk−Kavcβk2+αk22βkT∑c=1C∑i=1ncKc−KavcTKc−Kavc−∑c=1CKavc−KavTKavc−Kavβk,where X=xii=1nS+nT and K are its kernel maps of Kav=KT_avKS_av and Kavc=KT_avcKS_avc; Kc represents samples with c category in KS and KT. KS and KT are the kernel maps of xSii=1nS and xTjj=1nT. KS_avand KT_av are mean vectors of KS and KT, respectively. KTS_avc is the mean vector of the target (source) domain KTSc with c category.The first term of equation (18) is LKELM=minβk:1/2βk2+λ2/2Kβk−X2, and the second and third terms are CDMA measure and LDA, respectively. αk1 and αk2 are tradeoff parameters of LCDMA and LLDA, respectively.The first part ofLKELM is the regularization parameter to prevent model overfitting, and the second part of LKELM denotes the reconstruction error. λ2 is a parameter that balances the two parts. We set ∂LDKELM−CDMA/∂βk=0, and get(19)βk−λ2KKβk−XT+αk12K−KavK−KavT+∑c=1CKc−KavcKc−KavcT+αk2∑c=1C∑i=1ncKc−KavcKc−KavcT−∑c=1CKavc−KavKavc−KavTβk=0,(20)βk∗=KTΩ+αk1K−KavK−KavT+∑c=1CKc−KavcKc−KavcT+αk2∑c=1C∑i=1ncKc−KavcKc−KavcT−∑c=1CKavc−KavKavc−KavT+InS+nT×nS+nTλ2X.After obtainingβk∗, we could learn a new feature representation of DS and DT by KS1=KSβk∗, KT1=KTβk∗.


#### 3.3.2. Classification

At this stage, we transformKS1 and KT1 to KS2 and KT2 by kernel mapping kij1⟶mappingΩkij1=kij2. We obtain a standard representation of kernel ELM according to [19].(21)fx=Ωkij1KS2TΩKS2+InSλ3YS,where fx is output of a test sample x, λ3 is the tradeoff parameter of empirical risk in kernel ELM. We summarize the DKELM-CDMA procedure in Algorithm 2.Algorithm 2: DKELM-CDMA.
Input: Data set DS and DT, tradeoff parameters αk1, αk2 , and λ2. Output: Target output Y∧T.Step 1: Map DS=xsi,ysii=1nS and DT=xTjj=1nT to KS and KT using Ωx, and set Sb and Sw to zero.Step 2: Calculate βk∗ according to equation (20).Step 3: Set KS1=KSβk∗, KT1=KTβk∗, and map KS1 and KT2 to KS2 and KT2 using Ωx.Step 4: Predict DT and get its label Y∧T according to equation (21).Step 5: Solve Sb and Sw by using KS,YS, KT,Y∧T and according to equations (12) and (13).Step 6: Repeat steps 2–5 until Y∧T unchanged.


## 3.1. Objective Function of DELM-CDMA

In unsupervised domain adaptation, given two different but related data sets: source domainDS=xsi,ysii=1nS owning nS sample xSi with label ySi, and target domain DT=xTjj=1nT including nT unlabeled sample xTj. We hope DELM-CDMA training on DS perform well on DT.First, we transformDS and DT to HS and HT using activation function gx, and then construct the objective function of DELM-CDMA as given by the following equation:(8)LDELM−CDMA=LELM+α1LCDMA+α2LLDA.From equation (8), we can see that there are three parts in the objective function of DELM-CDMA. LELM represents the loss function of regularized ELM, LCDMA denotes marginal or conditional CDMA measure between source and target domains, and LLDA is LDA term. α1 and α2 are tradeoff parameters of LCDMA and LLDA, respectively.


### 3.1.1. Loss Function of Regularized ELM

In this paper, we carry out unsupervised domain adaptation task in which no labeled samples appear on target domain, so we can obtain a Regularized ELM on source domain as given by the following equation:(9)LELM=minβ:12β2+λ12HSβ−YS2.In equation (9), the first term is a regularizer to avoid over-fitting, and the second term is classification error of samples from source domain. λ1 is a parameter that balances the two terms.In the iterative refinement process of labels, when the pseudo labels of the target samplesY∧T are obtained, we could add the classification error from target domain, and equation (9) becomes equation (10).(10)LELM=minβ:12β2+λ12HSβ−YS2+HTβ−Y∧T2=minβ:12β2+λ12Hβ−Y2,where H=HSHT and Y=YSY∧T.


### 3.1.2. Cross-Domain Mean Approximation Measure

In unsupervised domain adaptation, since source domain and target domain have distribution discrepancy, the prediction error of model training onDS become increases on target domain. CDMA is an effective strategy for evaluation of inter-domain distribution differences. We apply it to measure the distribution discrepancy of output-layer data of ELM training on source and target domains, and compute it as given by the following equation:(11)LCDMA=minβ:12HTβ−HS_avβ2+HSβ−HT_avβ2+12∑c=1CHTcβ−HS_avcβ2+HScβ−HT_avcβ2.In equation (11), the first term measures the marginal distribution discrepancy, and the second term measures the conditional distribution discrepancy.


### 3.1.3. Linear Discriminant Analysis on Output Layer

To further improve the discrimination of ELM, we add Linear Discriminant Analysis (LDA) into it. LDA forces the samples with the same class closer and the ones with different class farther so as to enhance ELM category separability in unsupervised domain adaptation. We apply LDA on data from the output layer, in whichSb and Sw, respectively, denote the between-class scatter and the within-class scatter and could been computed as given by the following equation:(12)Sb=∑c=1CncβHc−βHavTβHc−βHav,(13)Sw=∑c=1C∑i=1ncβHc−βHavcTβHc−βHavc.In equations (12) and (13), Hav=HT_avHS_av, Havc=HT_avcHS_avc, Hc represents samples with c category in HS and HT. HS_av and HT_av are mean vectors of HS and HT, respectively. HTS_avc is the mean vector of the target (source) domain HTSc with c category. Then, we construct the loss of LDA on the output layer as given by the following equation:(14)LLDA=minβ:12Sw−Sb.


### 3.1.4. Total Objective Function

To improve the ability of knowledge transferring and discrimination of ELM, DELM-CDMA joins CDMA and LDA to ELM. Therefore, we bring equations (9) or (10), (11) and (14) into (8), and then the objective function of DELM-CDMA is established:(15)LDELM−CDMA=LELM+α1LCDMA+α2LLDA=minβ:12β2+λ12Hβ−Y2+α12Hβ−Havβ2+∑c=1CHcβ−Havcβ2+α22βT∑c=1C∑i=1ncHc−HavcTHc−Havc−∑c=1CHavc−HavTHavc−Havβ.


## 3.1.1. Loss Function of Regularized ELM

In this paper, we carry out unsupervised domain adaptation task in which no labeled samples appear on target domain, so we can obtain a Regularized ELM on source domain as given by the following equation:(9)LELM=minβ:12β2+λ12HSβ−YS2.In equation (9), the first term is a regularizer to avoid over-fitting, and the second term is classification error of samples from source domain. λ1 is a parameter that balances the two terms.In the iterative refinement process of labels, when the pseudo labels of the target samplesY∧T are obtained, we could add the classification error from target domain, and equation (9) becomes equation (10).(10)LELM=minβ:12β2+λ12HSβ−YS2+HTβ−Y∧T2=minβ:12β2+λ12Hβ−Y2,where H=HSHT and Y=YSY∧T.


## 3.1.2. Cross-Domain Mean Approximation Measure

In unsupervised domain adaptation, since source domain and target domain have distribution discrepancy, the prediction error of model training onDS become increases on target domain. CDMA is an effective strategy for evaluation of inter-domain distribution differences. We apply it to measure the distribution discrepancy of output-layer data of ELM training on source and target domains, and compute it as given by the following equation:(11)LCDMA=minβ:12HTβ−HS_avβ2+HSβ−HT_avβ2+12∑c=1CHTcβ−HS_avcβ2+HScβ−HT_avcβ2.In equation (11), the first term measures the marginal distribution discrepancy, and the second term measures the conditional distribution discrepancy.


## 3.1.3. Linear Discriminant Analysis on Output Layer

To further improve the discrimination of ELM, we add Linear Discriminant Analysis (LDA) into it. LDA forces the samples with the same class closer and the ones with different class farther so as to enhance ELM category separability in unsupervised domain adaptation. We apply LDA on data from the output layer, in whichSb and Sw, respectively, denote the between-class scatter and the within-class scatter and could been computed as given by the following equation:(12)Sb=∑c=1CncβHc−βHavTβHc−βHav,(13)Sw=∑c=1C∑i=1ncβHc−βHavcTβHc−βHavc.In equations (12) and (13), Hav=HT_avHS_av, Havc=HT_avcHS_avc, Hc represents samples with c category in HS and HT. HS_av and HT_av are mean vectors of HS and HT, respectively. HTS_avc is the mean vector of the target (source) domain HTSc with c category. Then, we construct the loss of LDA on the output layer as given by the following equation:(14)LLDA=minβ:12Sw−Sb.


## 3.1.4. Total Objective Function

To improve the ability of knowledge transferring and discrimination of ELM, DELM-CDMA joins CDMA and LDA to ELM. Therefore, we bring equations (9) or (10), (11) and (14) into (8), and then the objective function of DELM-CDMA is established:(15)LDELM−CDMA=LELM+α1LCDMA+α2LLDA=minβ:12β2+λ12Hβ−Y2+α12Hβ−Havβ2+∑c=1CHcβ−Havcβ2+α22βT∑c=1C∑i=1ncHc−HavcTHc−Havc−∑c=1CHavc−HavTHavc−Havβ.


## 3.2. Model Learning

To solve optimal output weight, we let∂LDELM−CDMA/∂β=0 and yield(16)β−λ1HTHβ−Y+α12H−HavTH−Hav+∑c=1CHc−HavcTHc−Havc+α2∑c=1C∑i=1ncHc−HavcTHc−Havc−∑c=1CHavc−HavTHavc−Havβ=0According to [18], equation (16) has a closed form solution:(17)β∗=HTH+α1H−HavTH−Hav+∑c=1CHc−HavcTHc−Havc+α2∑c=1C∑i=1ncHc−HavcTHc−Havc−∑c=1CHavc−HavTHavc−Hav+IL×Lλ1HTY,nS+nT>L,HTHHT+α1H−HavH−HavT+∑c=1CHc−HavcHc−HavcT+α2∑c=1C∑i=1ncHc−HavcHc−HavcT−∑c=1CHavc−HavHavc−HavT+InS+nT×nS+nTλ1Y,nS+nT≤L.After optimalβ∗ is obtained, the test samples are classified by equation (7). A complete classification procedure of DELM-CDMA is summarized in Algorithm 1.Algorithm 1: DELM-CDMA
Input: Data setDS and DT, tradeoff parameters α1, α2, and λ1. Output: Output layer weight β∗Step 1: Randomly initialize input weightsw and biases b of the hidden layer with L nodes, and set tradeoff parameters α1, α2, and λ1.Step 2: TransformDS=xsi,ysii=1nS and DT=xTjj=1nT to HS and HT using gwx+b, and set Sb and Sw to zero.Step 3: Calculateβ∗ according to equation (17).Step 4: Useβ∗ to predict DT and get its label Y∧T.Step 5: SolveSb and Sw by using DS, DT, and Y∧T according to equations (12) and (13).Step 6: Repeat steps 3–5 untilY∧T unchanged.


## 3.3. Kernelization

Although DELM-CDMA can improve the performance of ELM with fewer labeled samples or lack of conditions, it has two shortcomings: (1) The random initialization of hidden layer parameters will cause performance instability. (2) Inability to enhance the high-dimensional separability of nonlinear data. Therefore, we present a kernel version of DELM-CDMA named Discriminative Kernel Extreme Learning Machine with Cross-Domain Mean Approximation (DKELM-CDMA) to address above problems. DKELM-CDMA has two stages: feature extraction and classification.


### 3.3.1. Feature Extraction

At this stage, we letK=HHT, Kij=gxigxjT=Ωxi,xj, Ωxi,xj is a kernel function such as sigmoid kernel, linear kernel, and radial basis kernel. The objective function of DKELM-CDMA is given by the following equation:(18)LDKELM−CDMA=LKELM+αk1LCDMA+αk2LLDA=minβk:12βk2+λ22Kβk−X2+αk12Kβk−Kavβk2+∑c=1CKcβk−Kavcβk2+αk22βkT∑c=1C∑i=1ncKc−KavcTKc−Kavc−∑c=1CKavc−KavTKavc−Kavβk,where X=xii=1nS+nT and K are its kernel maps of Kav=KT_avKS_av and Kavc=KT_avcKS_avc; Kc represents samples with c category in KS and KT. KS and KT are the kernel maps of xSii=1nS and xTjj=1nT. KS_avand KT_av are mean vectors of KS and KT, respectively. KTS_avc is the mean vector of the target (source) domain KTSc with c category.The first term of equation (18) is LKELM=minβk:1/2βk2+λ2/2Kβk−X2, and the second and third terms are CDMA measure and LDA, respectively. αk1 and αk2 are tradeoff parameters of LCDMA and LLDA, respectively.The first part ofLKELM is the regularization parameter to prevent model overfitting, and the second part of LKELM denotes the reconstruction error. λ2 is a parameter that balances the two parts. We set ∂LDKELM−CDMA/∂βk=0, and get(19)βk−λ2KKβk−XT+αk12K−KavK−KavT+∑c=1CKc−KavcKc−KavcT+αk2∑c=1C∑i=1ncKc−KavcKc−KavcT−∑c=1CKavc−KavKavc−KavTβk=0,(20)βk∗=KTΩ+αk1K−KavK−KavT+∑c=1CKc−KavcKc−KavcT+αk2∑c=1C∑i=1ncKc−KavcKc−KavcT−∑c=1CKavc−KavKavc−KavT+InS+nT×nS+nTλ2X.After obtainingβk∗, we could learn a new feature representation of DS and DT by KS1=KSβk∗, KT1=KTβk∗.


### 3.3.2. Classification

At this stage, we transformKS1 and KT1 to KS2 and KT2 by kernel mapping kij1⟶mappingΩkij1=kij2. We obtain a standard representation of kernel ELM according to [19].(21)fx=Ωkij1KS2TΩKS2+InSλ3YS,where fx is output of a test sample x, λ3 is the tradeoff parameter of empirical risk in kernel ELM. We summarize the DKELM-CDMA procedure in Algorithm 2.Algorithm 2: DKELM-CDMA.
Input: Data set DS and DT, tradeoff parameters αk1, αk2 , and λ2. Output: Target output Y∧T.Step 1: Map DS=xsi,ysii=1nS and DT=xTjj=1nT to KS and KT using Ωx, and set Sb and Sw to zero.Step 2: Calculate βk∗ according to equation (20).Step 3: Set KS1=KSβk∗, KT1=KTβk∗, and map KS1 and KT2 to KS2 and KT2 using Ωx.Step 4: Predict DT and get its label Y∧T according to equation (21).Step 5: Solve Sb and Sw by using KS,YS, KT,Y∧T and according to equations (12) and (13).Step 6: Repeat steps 2–5 until Y∧T unchanged.


## 3.3.1. Feature Extraction

At this stage, we letK=HHT, Kij=gxigxjT=Ωxi,xj, Ωxi,xj is a kernel function such as sigmoid kernel, linear kernel, and radial basis kernel. The objective function of DKELM-CDMA is given by the following equation:(18)LDKELM−CDMA=LKELM+αk1LCDMA+αk2LLDA=minβk:12βk2+λ22Kβk−X2+αk12Kβk−Kavβk2+∑c=1CKcβk−Kavcβk2+αk22βkT∑c=1C∑i=1ncKc−KavcTKc−Kavc−∑c=1CKavc−KavTKavc−Kavβk,where X=xii=1nS+nT and K are its kernel maps of Kav=KT_avKS_av and Kavc=KT_avcKS_avc; Kc represents samples with c category in KS and KT. KS and KT are the kernel maps of xSii=1nS and xTjj=1nT. KS_avand KT_av are mean vectors of KS and KT, respectively. KTS_avc is the mean vector of the target (source) domain KTSc with c category.The first term of equation (18) is LKELM=minβk:1/2βk2+λ2/2Kβk−X2, and the second and third terms are CDMA measure and LDA, respectively. αk1 and αk2 are tradeoff parameters of LCDMA and LLDA, respectively.The first part ofLKELM is the regularization parameter to prevent model overfitting, and the second part of LKELM denotes the reconstruction error. λ2 is a parameter that balances the two parts. We set ∂LDKELM−CDMA/∂βk=0, and get(19)βk−λ2KKβk−XT+αk12K−KavK−KavT+∑c=1CKc−KavcKc−KavcT+αk2∑c=1C∑i=1ncKc−KavcKc−KavcT−∑c=1CKavc−KavKavc−KavTβk=0,(20)βk∗=KTΩ+αk1K−KavK−KavT+∑c=1CKc−KavcKc−KavcT+αk2∑c=1C∑i=1ncKc−KavcKc−KavcT−∑c=1CKavc−KavKavc−KavT+InS+nT×nS+nTλ2X.After obtainingβk∗, we could learn a new feature representation of DS and DT by KS1=KSβk∗, KT1=KTβk∗.


## 3.3.2. Classification

At this stage, we transformKS1 and KT1 to KS2 and KT2 by kernel mapping kij1⟶mappingΩkij1=kij2. We obtain a standard representation of kernel ELM according to [19].(21)fx=Ωkij1KS2TΩKS2+InSλ3YS,where fx is output of a test sample x, λ3 is the tradeoff parameter of empirical risk in kernel ELM. We summarize the DKELM-CDMA procedure in Algorithm 2.Algorithm 2: DKELM-CDMA.
Input: Data set DS and DT, tradeoff parameters αk1, αk2 , and λ2. Output: Target output Y∧T.Step 1: Map DS=xsi,ysii=1nS and DT=xTjj=1nT to KS and KT using Ωx, and set Sb and Sw to zero.Step 2: Calculate βk∗ according to equation (20).Step 3: Set KS1=KSβk∗, KT1=KTβk∗, and map KS1 and KT2 to KS2 and KT2 using Ωx.Step 4: Predict DT and get its label Y∧T according to equation (21).Step 5: Solve Sb and Sw by using KS,YS, KT,Y∧T and according to equations (12) and (13).Step 6: Repeat steps 2–5 until Y∧T unchanged.


## 4. Experiment and Analysis

In this section, we will perform experiments on object recognition and text data sets (described in Tables1 and 2) for classification tasks under domain adaptation, and estimate DELM-CDMA and DKELM-CDMA. For fairness, all experiments were conducted on a PC with 8 GB memory and Windows 10 operating system. The algorithms are implemented in MATLAB 2017b. Each experiment runs 20 times, and the average is recorded. The accuracy rate [36, 39, 53] is adopted to evaluate each algorithm in experiments.Table 1
Description of image data sets.
Data setType of dataNumber of samplesDimensionClassContains subsetsUSPSDigit1,80025610USPSMNISTDigit2,00025610MNISTMSRCObject1,26924018MSRCVOC2007Object1,53024020VOCCaltech-256Object1,12380010CaltechOffice-10AMAZONObject95880010AMAZONWebcamObject29580010WebcamDSLRObject15780010DSLROffice-31AmazonObject2,8172,04831AmazonWebcamObject7952,04831WebcamDslrObject4982,04831DslrTable 2
Description of reuters-21578 data set.
Data setType of dataNumber of samplesDimensionClassContains subsetsReuters-21578:OrgsText1,2374,771BinaryOrgsPeopleText1,2084,771BinaryPeoplePlaceText1,0164,771BinaryPlace


### 4.1. Data Set Description

USPS + MNIST [53]: USPS and MNIST (shown in Figure 3 and Table 1) are two image data sets with different but related distributions for handwritten number recognition. They share 10 categories from 0 to 9. USPS data set contains 9,298 images with 16 × 16 pixels, and MNIST data set collects 70,000 images with 28 × 28 pixels. In this section, 1800 images from USPS data set and 2000 images from MNIST data set are selected for domain adaptation, and then we construct two domain adaptation tasks: USPS ⟶ MNIST, MNIST ⟶ USPS. Moreover, all the images are uniformly transformed into grayscale images with 16 × 16 pixels, and each image is represented by a 256-dimensional vector encoding the grayscale values of all pixels.Figure 3
Image samples from (a) MNIST and (b) USPS.
(a)(b)MSRC + VOC2007 [63]: MSRC and VOC2007 (shown in Figure 4 and Table 1) are two common object recognition image data sets. MSRC has 4,323 images from 18 categories, and VOC2007 contains 5,011 images from 20 categories. Both are different but related. In MSRC and VOC2007 data sets, we, respectively, choose 1269 pictures and 1530 pictures from 6 shared classes including airplanes, birds, cows, family cars, sheep, and bicycles, and set two domain adaptation task: MSRC ⟶ VOC and VOC ⟶ MSRC. In this experiment, all the pictures are transformed into 0∼256 gray pixels, and extract 240 dimensions. All images are rescaled to be 256 pixels in length, and extract 128-dimensional dense SIFT (DSIFT) features. A 240-dimensional codebook is created, where K-means clustering is used to obtain the codewords.Figure 4
Image samples from (a) MSRC and (b) VOC2007.
(a)(b)Office + Caltech [53]: Office + Caltech data set (shown in Figure 5 and Table 1) released by Gong [51] for visual cross-domain object recognition. Office data set includes three sub-data sets: Amazon (A), Webcam (W), and DSLR (D), in which 4,652 images with31 categories are collected. Caltech-256 (C) is also a benchmark data set for target recognition, in which 30,607 images from 256 categories are collected. In this experiment, we select 10 shared categories in four domains C (Caltech-256), A (Amazon), W (Webcam), and D (DSLR) with a total of 2,533 images with 800 SURF features, shown in Table 1. During the experiment, we can construct 12 groups of domain adaptation task, such as C ⟶ A, C ⟶ W, C ⟶ D, …, and D ⟶ W.Figure 5
Image samples from Amazon, Caltech, DSLR, and Webcam. (a) DSLR. (b) Amazon. (c) Webcam. (d) Caltech-256.
(a)(b)(c)(d)Office31 [64]: Office-31 (shown in Table 1) is a standard benchmark for domain adaptation, which contains 4,652 images with 31 categories and consists of three real-world object domains: amazon, webcam, and dslr. In this paper, we use 2,048-dim ResNet-50 [65] features to conduct 6 cross-domain tasks, that is, amazon vs dslr, amazon vs webcam, dslr vs amazon, dslr vs webcam, webcam vs amazon, webcam vs dslr.Reuters-21578 [66]: Reuters-21578 text data set is commonly used for text categorization in domain adaptation, in which 21,577 news articles were collected and annotated by Reuters and divided into 5 categories: exchanges, orgs, people, places, and topics. In every text, a corpus becomes a digital data processing that takes out every word that appears as a feature. In this section, we adopt articles from 3 largest classes (shown in Table 2): orgs, people, and place, and then construct 6 domain adaptation tasks: orgs vs people, people vs orgs, orgs vs place, place vs orgs, people vs place, and place vs people.


### 4.2. Compared Algorithm and Setting

To investigate our method, we choose several classification algorithms for comparison.


#### 4.2.1. Non-Adaptation Domain Classifiers

1NN:k Nearest Neighbor classifier; we choose one sample as target nearest neighbor.SVM: Support vector machine with linear kernel; we set SVM penalty parameter belong to0.1,0.5,1,5,10,50,100.ELM: Standard extreme learning machine.SSELM: Semi-Supervised ELM with graph regularization [24].


#### 4.2.2. Shallow Adaptation Domain Classifiers

TCA1, TCA2: Classifier combining TCA [57] with 1NN (TCA1) and classifier combining TCA with SVM (TCA2).JDA1, JDA2: Classifier combining JDA [53] with 1NN (JDA1) and classifier combining JDA with SVM (JDA2). We set parameters in TCA and JDA according to [53].DAELM_S, DAELM_T [37]: Supervised ELMs for domain adaptation.KMM [67]: Kernel Mean Matching with resampling weights. Its result in this section is cited from [42].LMPROJ [68]: A large margin hyperplane classifier based on SVM. We cite its result from [42].ARRLS [66]: A general transfer learning framework referred to adaptation regularization based on transfer learning using squared loss. We set its parameters according to [66].TELM-OWA [39]: Supervised ELM using output weight alignment for domain adaptation. We set its parameters referred [39].CDELM-M, CDELM-C: An unsupervised ELM using MMD and manifold regularization for domain adaptation. We cited its result in [42] for comparison.JUC-SDELM [69]: An unsupervised ELM using MMD and the scalable factor for domain adaptation. We cited its result in [69] for comparison.


#### 4.2.3. Deep Domain Adaptation Classifiers

DAN [58]: A deep domain adaptation network with multi-kernel MMD for domain adaptation. We cited its result in [59] for comparison.JAN [59]: A deep domain adaptation network for domain adaptation by aligning the joint distributions of multiple domain-specific layers across domains. We cited its result in [59] for comparison.DANN [61]: An adversarial domain adaptation network with domain-invariant feature extractor. We cited its result in [62] for comparison.CAN [62]: A adversarial domain adaptation network adopting collaborative learning and adversarial learning for domain adaptation. We cited its result in [62] for comparison.ELM-CDMA, DELM-CDMA, and DKELM-CDMA are our methods. ELM-CDMA is a special case of DELM-CDMA when the LDA term is 0.It is noteworthy that we set the penalty parameterλ1∈10−3,10−1 in ELM, SSELM, DAELM_S, DAELM_T, TELM-OWA, ELM-CDMA, and DELM-CDMA. In ELM-CDMA and DELM-CDMA, we set α1∈10−2,102, α2∈10−4,10−2, and L=2000 on Office + Caltech and offfice31 data sets, L=3000 on USPS + MNIST and MSRC + VOC2007 data sets, L=3000 on Reuters-21578 data set. We evaluate DAELM_S, DAELM_T, and TELM-OWA in unsupervised domain adaptation task by selecting 0.5% labeled target samples on USPS + MNIST, MSRC + VOC2007, and Reuters-21578 data sets and 1% labeled target samples on Office + Caltech data set to train model. For DKELM-CDMA, we set balance coefficient αk1∈10−1,101 on USPS + MNIST, Reuters-21578, Office + Caltech and offfice31 data sets, αk1=10−4 on MSRC + VOC2007 data set, and set αk2∈10−2,101, penalty coefficient λ2∈10−1,103 and kernel parameter σ1∈10−1,103 on 5 data sets at feature extraction stage. We let λ3∈101,103 and kernel parameter σ2∈101,104 on 5 data sets at classification stage.


### 4.3. Results and Analysis

We present the accuracy of all algorithms in Tables3–5 and Figures 6–10 and find that(1)
DKELM-CDMA achieves best average result in Tables3 and 4 with help of CDMA and LDA. Especially in Table 3, the average result is 56.7%, much higher than other methods, which indicate that CDMA could efficiently reduce the distribution discrepancy of output-layer data from source and target domains and significantly promote the knowledge transfer capability of ELM in unsupervised domain adaptation. DKELM-CDMA could further mine shared features across domains, so it achieves better accuracy than DELM-CDMA.(2)
DELM-CDMA and ELM-CDMA perform better than other methods on most tasks in total average, which shows again that CDMA is efficient for unsupervised domain adaptation. It extends ELM in wide application scope and scene. DELM-CDMA outperform ELM-CDMA on most tasks with the help of LDA, showing that LDA also improves discrimination of ELM for domain adaptation.(3)
Compared with DKELM-CDMA, DELM-CDMA, and ELM-CDMA, TELM-OWA and DAELM_T get bad results, because they are supervised classifiers requiring a few labeled samples on target domain. Moreover, the traditional classifiers, such as 1NN, SVM, and ELM, are unsuccessful because of inability in knowledge transfer. ELM gains slightly better performance than 1NN and SVM due to its good generation. SSELM has better performance than ELM by mining manifold structure information of data.(4)
Although ARRLS, CDELM-M(C), JUC-SDELM, KMM, and LMPROJ also can gain better results with the help of statistical adaptation mechanism like MMD, (D)ELM-CDMA are more successful duo to CDMA is more efficient than MMD. TCA1 (2) and JDA1 (2) have higher accuracy than 1NN and SVM, depending on cross-domain shared feature extraction.(5)
In Table5 and Figure 10, we run 1NN, SVM, ELM, SSELM, DAELM_S, DAELM_T, ARRLS, TELM-OWA, ELM-CDMA, DELM-CDMA, and DKELM-CDMA on the Office-31 data set with ResNet-50 features comparing with deep adaptation domain methods DAN, DANN, JAN, and CAN. As a machine learning classification model with shallow network structure, DELM-CDMA, although not extracting shared high-level features, performs well on more complex data sets combined with deep feature extraction networks. It clearly demonstrates that DELM-CDMA equipped with deep generic features could further reduce the cross-domain discrepancy and achieve the best adaptation performance, which demonstrates the potential of our method. However, DKELM-CDMA does not work well on data office31. A possible explanation is that its feature extraction process hurts the quality of features already extracted by ResNet-50.Table 3
Classification accuracies (%) of all algorithms on office + caltech and USPS + MNIST.
MethodsData setC ⟶ A (1)C ⟶ W (2)C ⟶ D (3)A ⟶ C (4)A ⟶ W (5)A ⟶ D (6)W ⟶ C (7)W ⟶ A (8)W ⟶ D (9)D ⟶ C (10)D ⟶ A (11)D ⟶ W (12)USPS ⟶ MNISTMNIST ⟶ USPSAverage1NN23.725.825.526.029.825.519.923.059.226.328.563.444.765.934.8SVM52.440.742.044.141.740.829.934.187.931.432.473.635.154.745.8ELM50.642.742.043.037.642.732.236.980.330.131.571.930.751.444.5SSELM52.945.438.941.835.938.934.337.179.030.432.180.029.143.344.2TCA155.948.851.644.240.336.932.338.481.536.239.482.435.653.348.3TCA255.048.549.744.441.043.332.638.979.634.638.383.934.752.848.4JDA153.346.852.942.736.340.833.747.686.936.543.382.734.854.949.5JDA254.851.951.241.239.744.635.145.580.333.440.983.133.953.149.2DAELM_S50.345.437.642.938.336.934.236.480.932.334.472.944.447.145.3DAELM_T27.931.922.925.319.336.341.946.553.548.151.254.231.160.339.3KMM48.445.853.642.342.542.731.632.072.031.732.242.933.042.042.3LMPROJ52.546.940.843.638.638.230.636.483.530.734.081.746.357.747.2ARRLS54.950.544.642.539.038.934.739.978.332.237.282.734.652.547.3TELM-OWA31.732.233.138.938.643.333.538.077.132.536.278.659.659.445.2CDELM-M52.151.145.942.342.945.930.339.881.230.335.781.846.562.549.2CDELM-C56.351.043.843.240.841.834.439.683.134.439.584.345.942.648.6JUC-SDELM49.639.844.542.539.044.129.735.681.527.736.474.639.862.746.2ELM-CDMA53.247.145.243.644.147.133.136.684.133.136.574.645.382.650.5DELM-CDMA55.348.545.942.244.845.233.937.384.134.038.181.047.584.451.6DKELM-CDMA57.956.659.947.445.846.538.741.493.638.042.889.555.679.856.7Table 4
Classification accuracies (%) of all algorithms on MSRC + VOC2007 and reuters-21578.
Data setMethods1NNSVMELMSSELMTCA1TCA2JDA1JDA2DAELM_SDAELM_TARRLSTELM-OWAELM-CDMADELM-CDMADKELM-CDMAMSRC ⟶ VOC36.035.138.637.527.934.927.134.838.333.634.641.438.038.239.0VOC ⟶ MSRC45.254.758.063.247.954.248.354.658.430.650.466.262.763.564.5Orgs vs people (1)72.975.377.280.376.577.580.181.876.648.780.663.782.583.482.2People vs orgs (2)72.077.179.384.777.379.485.185.378.447.784.664.386.186.385.4Orgs vs place (3)67.570.263.669.672.471.875.776.268.543.872.268.178.379.178.2Place vs orgs (4)61.163.864.671.268.265.876.476.168.842.574.174.779.580.683.3People vs place (5)52.760.657.066.061.260.666.965.859.142.565.259.068.969.367.2Place vs people (6)53.457.958.761.756.151.158.151.258.839.958.660.566.565.169.0Total average57.661.862.166.860.961.964.765.763.441.265.062.270.370.771.1Table 5
Accuracy (%) comparison of different algorithms on office-31 (resnet-50).
Data setMethods1NNSVMELMSSELMDANDANNJANCANDAELM_SDAELM_TARRLSTELM-OWAELM-CDMADELM-CDMADKELM-CDMAAmazon vs dslr (1)79.180.782.183.378.679.784.785.583.938.285.777.786.786.785.3Amazon vs webcam (2)75.975.477.577.680.582.085.481.577.732.183.476.080.684.478.5Dslr vs amazon (3)60.262.864.066.163.668.268.665.964.629.471.765.268.769.567.3Dslr vs webcam (4)96.096.294.295.597.196.997.498.294.012.897.194.295.497.796.1Webcam vs amazon (5)59.962.365.666.262.867.470.063.466.083.270.466.168.368.467.3Webcam vs dslr (6)99.499.698.698.099.699.199.899.799.081.399.098.499.099.499.4Total average78.479.580.381.180.482.284.382.480.946.284.679.683.184.482.3Figure 6
Classification accuracy of different algorithms on USPS + MNIST data set.Figure 7
Classification accuracy of different algorithms on office + caltech data set.Figure 8
Classification accuracy of different algorithms on MSRC + VOC data set.Figure 9
Classification accuracy of different algorithms on reuters-21578 data set.Figure 10
Classification accuracy of different algorithms on office-31 data set.We carry out experiment on MNIST⟶USPS data set to compare the running-time in 1NN,SVM, ELM,SSELM,TCA1(2), JDA1(2), DAELM_S, DAELM_T, ARRLS, DELM-CDMA, and DKELM-CDMA, and the results are shown in Table6. It can be seen: (1) ELM has least running-time because of fast learning speed, and the algorithm based on ELM also spend less time cost than ones based on 1NN and SVM, such as TCA1(2) and JDA1(2). (2) DELM-CDMA spends more running-time than DAELM_S and DAELM_T because of the additional calculation generated by CDMA and LDA, but its time cost is lower than ARRLS in which MMD matrix and graph regulation term are required to construct. (3) TELM-OWA and DKELM-CDMA have similar time consumption more time than ELM, SSELM, DAELM_S, and DAELM_T, because they all need to solve the output weights twice during training. (4) SSELM needs more time than ELM because of manifold regulation. (5) JDA2 has most running time because it need to construct the marginal and conditional MMD matrix, and SVM has a complex solving process.Table 6
Comparison of running time on MNIST ⟶ USPS.
Algorithm1NNSVMELMSSELMTCA1TCA2JDA1JDA2DAELM_SDAELM_TARRLSTELM-OWADELM-CDMADKELM-CDMATime(s)0.68.80.43.54.75.548.353.60.80.62.13.71.93.7Through the above analysis, it can be seen that compared with other methods, DKELM-CDMA and DELM-CDMA do not require more time consumption while achieving higher accuracy.


### 4.4. Parameter Sensitivity Analysis

To evaluate the sensitivity of DELM-CDMA to parametersα1, α2, λ1 and number of hidden layer nodes(L) and its convergence, we run experiments on org vs people, MSRC vs VOC, MNIST vs USPS, A vs D and amazon vs dslr data sets, and the results are shown in Figures 11(a)–11(e). We can see that: (1) the accuracy increases first and then little decreases with α1, α2, λ1 growing, and DELM-CDMA achieve optimal results when α1∈10−2,100, α2∈10−3,10−1, and λ1∈10−3,10−2, respectively. Because α1, α2, and λ1 adjust CDMA, LDA, and output weight regularization terms, results in Figures 11(a)–11(c) show that CDMA, LDA, and output weight regularization term, adjusted the appropriate range, can improve the knowledge transfer ability and discrimination of ELM and avoid model from overfitting. (2) From Figure 11(d), we can see that the accuracy of DELM-CDMA increases first and then slightly decreases with L growing on most data sets. Wider hidden layer as soon as possible can improve the approximation ability of the function, but it will weaken the ELM performance of domain adaptation by damaging cross-domain metric performance of CDMA. (3) As shown in Figure 11(e), by observing the variation of accuracy with the increase of iterations, we find that finally converge after several iterations, which shows that DELM-CDMA has good robustness.Figure 11
Effect of iteration, number of hidden layer nodes, and parametersα1, α2, and λ1 on the accuracy of DELM-CDMA.
(a)(b)(c)(d)(e)Similar to DELM-CDMA, we carry out some experiments on org vs people, MSRC vs VOC, MNIST vs USPS, A vs D data set, and amazon vs dslr data sets to observe influence of parameterαk1, αk2, and λ2 on the accuracy of DKELM-CDMA, its convergence, and CDMA distance. We record results in Figure 12 and can see that: (1) Generally, the sensitivity of parameter αk1, αk2, and λ2 are different for various data sets, but DKELM-CDMA mainly achieves good performance when αk1∈10−2,100, αk2∈10−3,10−1, and λ2∈10−3,100, respectively, shown in Figures 12(a)–12(c). (2) From Figure 12(d), we can see that the accuracy is increasing iteratively, and finally converges after several iterations, so the robustness of DKELM-CDMA is strong. (3) We investigate CDMA distance with iteration growing, and the result is shown in Figure 12(f). We can see that: with the increase of iteration number, the CDMA distance becomes small, and the accuracy becomes high, which indicate that DKELM-CDMA could extract effective shared feature representation across domains relying on CDMA to reduce differences in marginal and conditional distributions between domains, and facilitate the cross-domain transfer of knowledge.Figure 12
CDMA distance and Effect of iteration and parameterαk1, αk2, and λ2 on the accuracy of DKELM-CDMA.
(a)(b)(c)(d)(e)


## 4.1. Data Set Description

USPS + MNIST [53]: USPS and MNIST (shown in Figure 3 and Table 1) are two image data sets with different but related distributions for handwritten number recognition. They share 10 categories from 0 to 9. USPS data set contains 9,298 images with 16 × 16 pixels, and MNIST data set collects 70,000 images with 28 × 28 pixels. In this section, 1800 images from USPS data set and 2000 images from MNIST data set are selected for domain adaptation, and then we construct two domain adaptation tasks: USPS ⟶ MNIST, MNIST ⟶ USPS. Moreover, all the images are uniformly transformed into grayscale images with 16 × 16 pixels, and each image is represented by a 256-dimensional vector encoding the grayscale values of all pixels.Figure 3
Image samples from (a) MNIST and (b) USPS.
(a)(b)MSRC + VOC2007 [63]: MSRC and VOC2007 (shown in Figure 4 and Table 1) are two common object recognition image data sets. MSRC has 4,323 images from 18 categories, and VOC2007 contains 5,011 images from 20 categories. Both are different but related. In MSRC and VOC2007 data sets, we, respectively, choose 1269 pictures and 1530 pictures from 6 shared classes including airplanes, birds, cows, family cars, sheep, and bicycles, and set two domain adaptation task: MSRC ⟶ VOC and VOC ⟶ MSRC. In this experiment, all the pictures are transformed into 0∼256 gray pixels, and extract 240 dimensions. All images are rescaled to be 256 pixels in length, and extract 128-dimensional dense SIFT (DSIFT) features. A 240-dimensional codebook is created, where K-means clustering is used to obtain the codewords.Figure 4
Image samples from (a) MSRC and (b) VOC2007.
(a)(b)Office + Caltech [53]: Office + Caltech data set (shown in Figure 5 and Table 1) released by Gong [51] for visual cross-domain object recognition. Office data set includes three sub-data sets: Amazon (A), Webcam (W), and DSLR (D), in which 4,652 images with31 categories are collected. Caltech-256 (C) is also a benchmark data set for target recognition, in which 30,607 images from 256 categories are collected. In this experiment, we select 10 shared categories in four domains C (Caltech-256), A (Amazon), W (Webcam), and D (DSLR) with a total of 2,533 images with 800 SURF features, shown in Table 1. During the experiment, we can construct 12 groups of domain adaptation task, such as C ⟶ A, C ⟶ W, C ⟶ D, …, and D ⟶ W.Figure 5
Image samples from Amazon, Caltech, DSLR, and Webcam. (a) DSLR. (b) Amazon. (c) Webcam. (d) Caltech-256.
(a)(b)(c)(d)Office31 [64]: Office-31 (shown in Table 1) is a standard benchmark for domain adaptation, which contains 4,652 images with 31 categories and consists of three real-world object domains: amazon, webcam, and dslr. In this paper, we use 2,048-dim ResNet-50 [65] features to conduct 6 cross-domain tasks, that is, amazon vs dslr, amazon vs webcam, dslr vs amazon, dslr vs webcam, webcam vs amazon, webcam vs dslr.Reuters-21578 [66]: Reuters-21578 text data set is commonly used for text categorization in domain adaptation, in which 21,577 news articles were collected and annotated by Reuters and divided into 5 categories: exchanges, orgs, people, places, and topics. In every text, a corpus becomes a digital data processing that takes out every word that appears as a feature. In this section, we adopt articles from 3 largest classes (shown in Table 2): orgs, people, and place, and then construct 6 domain adaptation tasks: orgs vs people, people vs orgs, orgs vs place, place vs orgs, people vs place, and place vs people.


## 4.2. Compared Algorithm and Setting

To investigate our method, we choose several classification algorithms for comparison.


### 4.2.1. Non-Adaptation Domain Classifiers

1NN:k Nearest Neighbor classifier; we choose one sample as target nearest neighbor.SVM: Support vector machine with linear kernel; we set SVM penalty parameter belong to0.1,0.5,1,5,10,50,100.ELM: Standard extreme learning machine.SSELM: Semi-Supervised ELM with graph regularization [24].


### 4.2.2. Shallow Adaptation Domain Classifiers

TCA1, TCA2: Classifier combining TCA [57] with 1NN (TCA1) and classifier combining TCA with SVM (TCA2).JDA1, JDA2: Classifier combining JDA [53] with 1NN (JDA1) and classifier combining JDA with SVM (JDA2). We set parameters in TCA and JDA according to [53].DAELM_S, DAELM_T [37]: Supervised ELMs for domain adaptation.KMM [67]: Kernel Mean Matching with resampling weights. Its result in this section is cited from [42].LMPROJ [68]: A large margin hyperplane classifier based on SVM. We cite its result from [42].ARRLS [66]: A general transfer learning framework referred to adaptation regularization based on transfer learning using squared loss. We set its parameters according to [66].TELM-OWA [39]: Supervised ELM using output weight alignment for domain adaptation. We set its parameters referred [39].CDELM-M, CDELM-C: An unsupervised ELM using MMD and manifold regularization for domain adaptation. We cited its result in [42] for comparison.JUC-SDELM [69]: An unsupervised ELM using MMD and the scalable factor for domain adaptation. We cited its result in [69] for comparison.


### 4.2.3. Deep Domain Adaptation Classifiers

DAN [58]: A deep domain adaptation network with multi-kernel MMD for domain adaptation. We cited its result in [59] for comparison.JAN [59]: A deep domain adaptation network for domain adaptation by aligning the joint distributions of multiple domain-specific layers across domains. We cited its result in [59] for comparison.DANN [61]: An adversarial domain adaptation network with domain-invariant feature extractor. We cited its result in [62] for comparison.CAN [62]: A adversarial domain adaptation network adopting collaborative learning and adversarial learning for domain adaptation. We cited its result in [62] for comparison.ELM-CDMA, DELM-CDMA, and DKELM-CDMA are our methods. ELM-CDMA is a special case of DELM-CDMA when the LDA term is 0.It is noteworthy that we set the penalty parameterλ1∈10−3,10−1 in ELM, SSELM, DAELM_S, DAELM_T, TELM-OWA, ELM-CDMA, and DELM-CDMA. In ELM-CDMA and DELM-CDMA, we set α1∈10−2,102, α2∈10−4,10−2, and L=2000 on Office + Caltech and offfice31 data sets, L=3000 on USPS + MNIST and MSRC + VOC2007 data sets, L=3000 on Reuters-21578 data set. We evaluate DAELM_S, DAELM_T, and TELM-OWA in unsupervised domain adaptation task by selecting 0.5% labeled target samples on USPS + MNIST, MSRC + VOC2007, and Reuters-21578 data sets and 1% labeled target samples on Office + Caltech data set to train model. For DKELM-CDMA, we set balance coefficient αk1∈10−1,101 on USPS + MNIST, Reuters-21578, Office + Caltech and offfice31 data sets, αk1=10−4 on MSRC + VOC2007 data set, and set αk2∈10−2,101, penalty coefficient λ2∈10−1,103 and kernel parameter σ1∈10−1,103 on 5 data sets at feature extraction stage. We let λ3∈101,103 and kernel parameter σ2∈101,104 on 5 data sets at classification stage.


## 4.2.1. Non-Adaptation Domain Classifiers

1NN:k Nearest Neighbor classifier; we choose one sample as target nearest neighbor.SVM: Support vector machine with linear kernel; we set SVM penalty parameter belong to0.1,0.5,1,5,10,50,100.ELM: Standard extreme learning machine.SSELM: Semi-Supervised ELM with graph regularization [24].


## 4.2.2. Shallow Adaptation Domain Classifiers

TCA1, TCA2: Classifier combining TCA [57] with 1NN (TCA1) and classifier combining TCA with SVM (TCA2).JDA1, JDA2: Classifier combining JDA [53] with 1NN (JDA1) and classifier combining JDA with SVM (JDA2). We set parameters in TCA and JDA according to [53].DAELM_S, DAELM_T [37]: Supervised ELMs for domain adaptation.KMM [67]: Kernel Mean Matching with resampling weights. Its result in this section is cited from [42].LMPROJ [68]: A large margin hyperplane classifier based on SVM. We cite its result from [42].ARRLS [66]: A general transfer learning framework referred to adaptation regularization based on transfer learning using squared loss. We set its parameters according to [66].TELM-OWA [39]: Supervised ELM using output weight alignment for domain adaptation. We set its parameters referred [39].CDELM-M, CDELM-C: An unsupervised ELM using MMD and manifold regularization for domain adaptation. We cited its result in [42] for comparison.JUC-SDELM [69]: An unsupervised ELM using MMD and the scalable factor for domain adaptation. We cited its result in [69] for comparison.


## 4.2.3. Deep Domain Adaptation Classifiers

DAN [58]: A deep domain adaptation network with multi-kernel MMD for domain adaptation. We cited its result in [59] for comparison.JAN [59]: A deep domain adaptation network for domain adaptation by aligning the joint distributions of multiple domain-specific layers across domains. We cited its result in [59] for comparison.DANN [61]: An adversarial domain adaptation network with domain-invariant feature extractor. We cited its result in [62] for comparison.CAN [62]: A adversarial domain adaptation network adopting collaborative learning and adversarial learning for domain adaptation. We cited its result in [62] for comparison.ELM-CDMA, DELM-CDMA, and DKELM-CDMA are our methods. ELM-CDMA is a special case of DELM-CDMA when the LDA term is 0.It is noteworthy that we set the penalty parameterλ1∈10−3,10−1 in ELM, SSELM, DAELM_S, DAELM_T, TELM-OWA, ELM-CDMA, and DELM-CDMA. In ELM-CDMA and DELM-CDMA, we set α1∈10−2,102, α2∈10−4,10−2, and L=2000 on Office + Caltech and offfice31 data sets, L=3000 on USPS + MNIST and MSRC + VOC2007 data sets, L=3000 on Reuters-21578 data set. We evaluate DAELM_S, DAELM_T, and TELM-OWA in unsupervised domain adaptation task by selecting 0.5% labeled target samples on USPS + MNIST, MSRC + VOC2007, and Reuters-21578 data sets and 1% labeled target samples on Office + Caltech data set to train model. For DKELM-CDMA, we set balance coefficient αk1∈10−1,101 on USPS + MNIST, Reuters-21578, Office + Caltech and offfice31 data sets, αk1=10−4 on MSRC + VOC2007 data set, and set αk2∈10−2,101, penalty coefficient λ2∈10−1,103 and kernel parameter σ1∈10−1,103 on 5 data sets at feature extraction stage. We let λ3∈101,103 and kernel parameter σ2∈101,104 on 5 data sets at classification stage.


## 4.3. Results and Analysis

We present the accuracy of all algorithms in Tables3–5 and Figures 6–10 and find that(1)
DKELM-CDMA achieves best average result in Tables3 and 4 with help of CDMA and LDA. Especially in Table 3, the average result is 56.7%, much higher than other methods, which indicate that CDMA could efficiently reduce the distribution discrepancy of output-layer data from source and target domains and significantly promote the knowledge transfer capability of ELM in unsupervised domain adaptation. DKELM-CDMA could further mine shared features across domains, so it achieves better accuracy than DELM-CDMA.(2)
DELM-CDMA and ELM-CDMA perform better than other methods on most tasks in total average, which shows again that CDMA is efficient for unsupervised domain adaptation. It extends ELM in wide application scope and scene. DELM-CDMA outperform ELM-CDMA on most tasks with the help of LDA, showing that LDA also improves discrimination of ELM for domain adaptation.(3)
Compared with DKELM-CDMA, DELM-CDMA, and ELM-CDMA, TELM-OWA and DAELM_T get bad results, because they are supervised classifiers requiring a few labeled samples on target domain. Moreover, the traditional classifiers, such as 1NN, SVM, and ELM, are unsuccessful because of inability in knowledge transfer. ELM gains slightly better performance than 1NN and SVM due to its good generation. SSELM has better performance than ELM by mining manifold structure information of data.(4)
Although ARRLS, CDELM-M(C), JUC-SDELM, KMM, and LMPROJ also can gain better results with the help of statistical adaptation mechanism like MMD, (D)ELM-CDMA are more successful duo to CDMA is more efficient than MMD. TCA1 (2) and JDA1 (2) have higher accuracy than 1NN and SVM, depending on cross-domain shared feature extraction.(5)
In Table5 and Figure 10, we run 1NN, SVM, ELM, SSELM, DAELM_S, DAELM_T, ARRLS, TELM-OWA, ELM-CDMA, DELM-CDMA, and DKELM-CDMA on the Office-31 data set with ResNet-50 features comparing with deep adaptation domain methods DAN, DANN, JAN, and CAN. As a machine learning classification model with shallow network structure, DELM-CDMA, although not extracting shared high-level features, performs well on more complex data sets combined with deep feature extraction networks. It clearly demonstrates that DELM-CDMA equipped with deep generic features could further reduce the cross-domain discrepancy and achieve the best adaptation performance, which demonstrates the potential of our method. However, DKELM-CDMA does not work well on data office31. A possible explanation is that its feature extraction process hurts the quality of features already extracted by ResNet-50.Table 3
Classification accuracies (%) of all algorithms on office + caltech and USPS + MNIST.
MethodsData setC ⟶ A (1)C ⟶ W (2)C ⟶ D (3)A ⟶ C (4)A ⟶ W (5)A ⟶ D (6)W ⟶ C (7)W ⟶ A (8)W ⟶ D (9)D ⟶ C (10)D ⟶ A (11)D ⟶ W (12)USPS ⟶ MNISTMNIST ⟶ USPSAverage1NN23.725.825.526.029.825.519.923.059.226.328.563.444.765.934.8SVM52.440.742.044.141.740.829.934.187.931.432.473.635.154.745.8ELM50.642.742.043.037.642.732.236.980.330.131.571.930.751.444.5SSELM52.945.438.941.835.938.934.337.179.030.432.180.029.143.344.2TCA155.948.851.644.240.336.932.338.481.536.239.482.435.653.348.3TCA255.048.549.744.441.043.332.638.979.634.638.383.934.752.848.4JDA153.346.852.942.736.340.833.747.686.936.543.382.734.854.949.5JDA254.851.951.241.239.744.635.145.580.333.440.983.133.953.149.2DAELM_S50.345.437.642.938.336.934.236.480.932.334.472.944.447.145.3DAELM_T27.931.922.925.319.336.341.946.553.548.151.254.231.160.339.3KMM48.445.853.642.342.542.731.632.072.031.732.242.933.042.042.3LMPROJ52.546.940.843.638.638.230.636.483.530.734.081.746.357.747.2ARRLS54.950.544.642.539.038.934.739.978.332.237.282.734.652.547.3TELM-OWA31.732.233.138.938.643.333.538.077.132.536.278.659.659.445.2CDELM-M52.151.145.942.342.945.930.339.881.230.335.781.846.562.549.2CDELM-C56.351.043.843.240.841.834.439.683.134.439.584.345.942.648.6JUC-SDELM49.639.844.542.539.044.129.735.681.527.736.474.639.862.746.2ELM-CDMA53.247.145.243.644.147.133.136.684.133.136.574.645.382.650.5DELM-CDMA55.348.545.942.244.845.233.937.384.134.038.181.047.584.451.6DKELM-CDMA57.956.659.947.445.846.538.741.493.638.042.889.555.679.856.7Table 4
Classification accuracies (%) of all algorithms on MSRC + VOC2007 and reuters-21578.
Data setMethods1NNSVMELMSSELMTCA1TCA2JDA1JDA2DAELM_SDAELM_TARRLSTELM-OWAELM-CDMADELM-CDMADKELM-CDMAMSRC ⟶ VOC36.035.138.637.527.934.927.134.838.333.634.641.438.038.239.0VOC ⟶ MSRC45.254.758.063.247.954.248.354.658.430.650.466.262.763.564.5Orgs vs people (1)72.975.377.280.376.577.580.181.876.648.780.663.782.583.482.2People vs orgs (2)72.077.179.384.777.379.485.185.378.447.784.664.386.186.385.4Orgs vs place (3)67.570.263.669.672.471.875.776.268.543.872.268.178.379.178.2Place vs orgs (4)61.163.864.671.268.265.876.476.168.842.574.174.779.580.683.3People vs place (5)52.760.657.066.061.260.666.965.859.142.565.259.068.969.367.2Place vs people (6)53.457.958.761.756.151.158.151.258.839.958.660.566.565.169.0Total average57.661.862.166.860.961.964.765.763.441.265.062.270.370.771.1Table 5
Accuracy (%) comparison of different algorithms on office-31 (resnet-50).
Data setMethods1NNSVMELMSSELMDANDANNJANCANDAELM_SDAELM_TARRLSTELM-OWAELM-CDMADELM-CDMADKELM-CDMAAmazon vs dslr (1)79.180.782.183.378.679.784.785.583.938.285.777.786.786.785.3Amazon vs webcam (2)75.975.477.577.680.582.085.481.577.732.183.476.080.684.478.5Dslr vs amazon (3)60.262.864.066.163.668.268.665.964.629.471.765.268.769.567.3Dslr vs webcam (4)96.096.294.295.597.196.997.498.294.012.897.194.295.497.796.1Webcam vs amazon (5)59.962.365.666.262.867.470.063.466.083.270.466.168.368.467.3Webcam vs dslr (6)99.499.698.698.099.699.199.899.799.081.399.098.499.099.499.4Total average78.479.580.381.180.482.284.382.480.946.284.679.683.184.482.3Figure 6
Classification accuracy of different algorithms on USPS + MNIST data set.Figure 7
Classification accuracy of different algorithms on office + caltech data set.Figure 8
Classification accuracy of different algorithms on MSRC + VOC data set.Figure 9
Classification accuracy of different algorithms on reuters-21578 data set.Figure 10
Classification accuracy of different algorithms on office-31 data set.We carry out experiment on MNIST⟶USPS data set to compare the running-time in 1NN,SVM, ELM,SSELM,TCA1(2), JDA1(2), DAELM_S, DAELM_T, ARRLS, DELM-CDMA, and DKELM-CDMA, and the results are shown in Table6. It can be seen: (1) ELM has least running-time because of fast learning speed, and the algorithm based on ELM also spend less time cost than ones based on 1NN and SVM, such as TCA1(2) and JDA1(2). (2) DELM-CDMA spends more running-time than DAELM_S and DAELM_T because of the additional calculation generated by CDMA and LDA, but its time cost is lower than ARRLS in which MMD matrix and graph regulation term are required to construct. (3) TELM-OWA and DKELM-CDMA have similar time consumption more time than ELM, SSELM, DAELM_S, and DAELM_T, because they all need to solve the output weights twice during training. (4) SSELM needs more time than ELM because of manifold regulation. (5) JDA2 has most running time because it need to construct the marginal and conditional MMD matrix, and SVM has a complex solving process.Table 6
Comparison of running time on MNIST ⟶ USPS.
Algorithm1NNSVMELMSSELMTCA1TCA2JDA1JDA2DAELM_SDAELM_TARRLSTELM-OWADELM-CDMADKELM-CDMATime(s)0.68.80.43.54.75.548.353.60.80.62.13.71.93.7Through the above analysis, it can be seen that compared with other methods, DKELM-CDMA and DELM-CDMA do not require more time consumption while achieving higher accuracy.


## 4.4. Parameter Sensitivity Analysis

To evaluate the sensitivity of DELM-CDMA to parametersα1, α2, λ1 and number of hidden layer nodes(L) and its convergence, we run experiments on org vs people, MSRC vs VOC, MNIST vs USPS, A vs D and amazon vs dslr data sets, and the results are shown in Figures 11(a)–11(e). We can see that: (1) the accuracy increases first and then little decreases with α1, α2, λ1 growing, and DELM-CDMA achieve optimal results when α1∈10−2,100, α2∈10−3,10−1, and λ1∈10−3,10−2, respectively. Because α1, α2, and λ1 adjust CDMA, LDA, and output weight regularization terms, results in Figures 11(a)–11(c) show that CDMA, LDA, and output weight regularization term, adjusted the appropriate range, can improve the knowledge transfer ability and discrimination of ELM and avoid model from overfitting. (2) From Figure 11(d), we can see that the accuracy of DELM-CDMA increases first and then slightly decreases with L growing on most data sets. Wider hidden layer as soon as possible can improve the approximation ability of the function, but it will weaken the ELM performance of domain adaptation by damaging cross-domain metric performance of CDMA. (3) As shown in Figure 11(e), by observing the variation of accuracy with the increase of iterations, we find that finally converge after several iterations, which shows that DELM-CDMA has good robustness.Figure 11
Effect of iteration, number of hidden layer nodes, and parametersα1, α2, and λ1 on the accuracy of DELM-CDMA.
(a)(b)(c)(d)(e)Similar to DELM-CDMA, we carry out some experiments on org vs people, MSRC vs VOC, MNIST vs USPS, A vs D data set, and amazon vs dslr data sets to observe influence of parameterαk1, αk2, and λ2 on the accuracy of DKELM-CDMA, its convergence, and CDMA distance. We record results in Figure 12 and can see that: (1) Generally, the sensitivity of parameter αk1, αk2, and λ2 are different for various data sets, but DKELM-CDMA mainly achieves good performance when αk1∈10−2,100, αk2∈10−3,10−1, and λ2∈10−3,100, respectively, shown in Figures 12(a)–12(c). (2) From Figure 12(d), we can see that the accuracy is increasing iteratively, and finally converges after several iterations, so the robustness of DKELM-CDMA is strong. (3) We investigate CDMA distance with iteration growing, and the result is shown in Figure 12(f). We can see that: with the increase of iteration number, the CDMA distance becomes small, and the accuracy becomes high, which indicate that DKELM-CDMA could extract effective shared feature representation across domains relying on CDMA to reduce differences in marginal and conditional distributions between domains, and facilitate the cross-domain transfer of knowledge.Figure 12
CDMA distance and Effect of iteration and parameterαk1, αk2, and λ2 on the accuracy of DKELM-CDMA.
(a)(b)(c)(d)(e)


## 5. Conclusion

In this paper, we propose a novel ELM model called Discriminative Extreme Learning Machine with Cross-Domain Mean Approximation for Unsupervised Domain Adaptation. Based on traditional ELM, it introduces Cross-Domain Mean Approximation to jointly minimize the marginal and conditional distribution from the data of output layers between source and target domains which enhance the ability of ELM to transfer knowledge. We also add Linear Discriminant Analysis to further improve the discrimination which improves the accuracy of ELM. In addition, in order to overcome the sensitivity of DELM-CDMA to the initialization of hidden layer parameters, we further propose DKELM-CDMA as the kernel version of DELM-CDMA, which further facilitates the fitting for nonlinear data. Finally, we run DELM-CDMA and DKELM-CDMA on many experiments for unsupervised domain adaptation, and the results show that the proposed approach can effectively enhance the efficiency of cross-domain knowledge transfer in ELM. Although DELM-CDMA and DKELM-CDMA work well in domain adaptation, they cannot fully utilize high-level semantic features representation because of their own shallow network structure. Therefore, we design a DKELM-CDMA with deep learning for feature extraction in future.



---

*Source: 2463746-2022-10-03.xml*