# TSTELM: Two-Stage Transfer Extreme Learning Machine for Unsupervised Domain Adaptation

Authors: Shaofei Zang; Xinghai Li; Jianwei Ma; Yongyi Yan; Jiwei Gao; Yuan Wei
Journal: Computational Intelligence and Neuroscience
 (2022)

---

## Abstract

As a single-layer feedforward network (SLFN), extreme learning machine (ELM) has been successfully applied for classification and regression in machine learning due to its faster training speed and better generalization. However, it will perform poorly for domain adaptation in which the distributions between training data and testing data are inconsistent. In this article, we propose a novel ELM called two-stage transfer extreme learning machine (TSTELM) to solve this problem. At the statistical matching stage, we adopt maximum mean discrepancy (MMD) to narrow the distribution difference of the output layer between domains. In addition, at the subspace alignment stage, we align the source and target model parameters, design target cross-domain mean approximation, and add the output weight approximation to further promote the knowledge transferring across domains. Moreover, the prediction of test sample is jointly determined by the ELM parameters generated at the two stages. Finally, we investigate the proposed approach in classification task and conduct experiments on four public domain adaptation datasets. The result indicates that TSTELM could effectively enhance the knowledge transfer ability of ELM with higher accuracy than other existing transfer and non-transfer classifiers.


---

## Body

## 1. Introduction

In the current era of big data, the classification model constructed by machine learning can help human quickly identify and annotate a large number of images, texts, audios, and signal data rapidly generated by the Internet, sensors, and computers. Mining information from these data helps understand the relationship between things. Support vector machine (SVM) [1], k-nearest neighbor (kNN) [2], naive Bayes [3, 4], decision tree [5], logistic regression [6], and many other classifiers with high accuracy appear and have attracted much attention. Huang et al. [7] proposed extreme learning machine (ELM) which is a better classifier with powerful nonlinear fitting and approximation capabilities [8, 9] and has been widely studied and applied in brain-computer interfaces [10, 11], medical diagnosis [12, 13], fault diagnosis [14], hyperspectral [15], and other fields.ELM initializes randomly the input weight and bias and obtains optimal output weight by solving a least-squares problem [7–9, 16], which has the advantages of faster learning speed and better generalization, therefore becomes a hot research topic. There are many variants of ELM put forward both in theories and applications to enhance its performance for handling problems in different situations. In order to solve the problem that ELM is sensitive to the input weights and biases, Li et al. [17] proposed a WOA-ELM algorithm which applied the whale optimization algorithm (WOA) to optimize the input weights and biases of ELM for its performance improvement. In response to the class imbalance problem, weighted extreme learning machine (WELM) [18–20] was proposed, in which different weights are assigned for each training sample based on two different strategies. SMOTE based on class-specific extreme learning machine (SMOTE-CSELM) [21] was also presented by exploiting the benefit of both the minority oversampling and the class-specific regularization. To improve the generalization power and prevent the overtraining of ELM, some methods combined ensemble learning with it to improve its robustness, including voting-based extreme learning machine (V-ELM) [22, 23], AdaBoost extreme learning machine [24–26], and extreme ensemble of ELMs (EEoELMs) [27]. Moreover, affected by deep learning, ELMs with deep structure occur for high accuracy. ML-ELM [28, 29] was presented to resolve the time-consuming issue of deep learning and achieved faster speed and higher generalization than stacked autoencoders, deep belief network, and deep Boltzmann machine. Hierarchical ELM (H-ELM) [30, 31] was proposed to enhance the universal approximation capability of ELM. The kernel-based multilayer ELM (ML-KELM) [32] integrated the kernel learning technique into the ML-ELM and achieved a faster learning speed and a better recognition performance. Although the above ELM models have achieved great success in classification and regression tasks, they will degrade when training samples and test samples are taken from different domains with different distributions (i.e., cross-domain tasks).To handle this problem, domain adaptation (DA) [33–35], as an important branch of transfer learning, has attracted wide attention, in which efficient classifier is obtained with the help of the knowledge from source domain, which is different but related to target domain. L. Zhang and D. Zhang [36] put forward the domain adaptation extreme learning machine (DAELM) framework by extending ELM to handle domain adaptation problems for gas identification and drift compensation of E-nose system. Adaptive ELM (AELM) [37] was proposed by introducing the manifold regularization term into ELM for image classification. Zang et al. [38] proposed a supervised extreme learning machine called transfer extreme learning machine with output weight alignment (TELM-OWA), which aligned the output weight matrix of the ELM between domains and added the approximation between the inter-domain ELM parameters for knowledge transferring. However, these approaches are developed to solve semi-supervised domain adaptation problems because they require few labeled samples from the target domains. Due to its high cost in collecting labels and labeling samples, cross-domain ELM (CDELM) [39], domain space transfer ELM (DST-ELM) [40], cross-domain extreme learning machine (CdELM) [41], and extreme learning machine based on maximum weighted mean discrepancy (ELM-MWMD) [42] are proposed respectively for unsupervised domains by minimizing the classification loss and applying the maximum mean discrepancy (MMD) strategy on the prediction results. In the above methods, the supervised ELM model usually outperforms the unsupervised ones with the help of a few labeled samples from the target domain.In this article, inspired by pioneering works [38, 42], we propose a novel method denoted as two-stage transfer extreme learning machine (TSTELM), in which there are two stages of domain adaptation: statistical matching and subspace alignment. At the statistical matching stage, we first learn a domain adaptation ELM classifier via utilizing the MMD to simultaneously minimize the marginal and conditional distribution between domains. In addition, at the subspace alignment stage, we use a transformation matrix to align the output weights of inter-domain ELM models, simultaneously put forward target cross-domain mean approximation, and add an output weight approximation term into the objective function. Then we can obtain the other domain adaptation ELM. Finally, we fuse the DAELM parameters from two stages to realize the label prediction of test samples. TSTELM is illustrated in Figure 1. Extensive experiments have been conducted on real-world image and text datasets, and the results verify that our approach outperforms several existing domain adaptation and non-domain adaptation methods.Figure 1
An illustration of TSTELM. (1) At the statistical matching domain adaptation stage, we use the MMD to simultaneously minimize the marginal and conditional distribution between domains in output layers and get a DAELM. (2) At the subspace alignment domain adaptation stage, we align the output weights of source and target ELM model, simultaneously design target cross-domain mean approximation term, and add the output weight approximation term into the objective function. We can obtain the other DAELM. (3) In fusion decision, we fuse the DAELM parameters from two stages to realize the label prediction of test samples.In this article, TSTELM realizes knowledge transferring at two stages, and its contributions are summarized as follows:(1)
Similar to [39], our method is to use MMD proved as a general statistical distribution discrepancy measure, to minimize the marginal and conditional distribution discrepancy of the outputs of hidden lays of ELMs from two domains, which effectively extends ELM for unsupervised domain adaptation. Therefore, we can obtain one DAELM.(2)
Based on the first DAELM and inspired by [42], we introduce the output weight alignment, design target cross-domain mean approximation, and add the output weight approximation constraint into traditional ELM for enhancing knowledge transfer across domains. Hence, we can learn the other DAELM. It is worth emphasizing that we present target cross-domain mean approximation referred to [35] to adapt the distribution of the target domain for consistency with the source domains.(3)
At prediction stage, the above two DAELMs jointly determined the category of test samples. Our proposed method performs image classification experiments on object recognition and text datasets. The results verify its effectiveness and advantages.(4)
Compared with other the state-of-the-art DAELMs, our research has some distinct properties: (1) Many technologies including MMD, output weight alignment, output weight approximation, and target cross-domain mean approximation are utilized to jointly realize the efficient knowledge transfer across domains at two stages. (2) Output weight alignment organically bridges the DAELMs from the statistical matching stage and subspace alignment stage. (3) Joint decision from two DAELMs facilitates our approach to obtain robustness and high accuracy.The rest of this article is organized as follows: In Section2, we briefly review domain adaptation and ELM. We then present the proposed TSTELM in Section 3. In Section 4, the experiment and analysis are presented. Finally, Section 5 is the conclusion of this article.


## 2. A Brief Review of the Domain Adaptation and Extreme Learning Machine

### 2.1. Domain Adaptation

When the training and testing data are drawn from different distributions, a classifier directly learned on training data would have a poor performance for testing data. Domain adaptation is developed to deal with this scenario, in which an excellent classifier could be obtained from the source domain with rich labeled samples and perform well in the target domain task. The source domain and the target domain have different distributions but some correlations.In the past decades, many researches have conducted to address domain adaptation problems in classification task, which are mainly divided into three parts [35, 43]: (1) Sample-based adaptation. It directly assigns weights to each sample of two domains, which could adapt and minimize the distribution gap between domains. Many such approaches appeared such as domain adaptation (PRDA) [44], TrAdaBoost [45], and Kernel Mean Match (KMM) [46]. (2) Feature-based adaption. It seeks the shared subspace between domains, in which distribution discrepancy is alleviated and knowledge is easily transferred across domains. Transfer component analysis (TCA) [47] and joint distribution adaptation (JDA) [48] take MMD metric as an objective function to find an optimal projected matrix for shared low-dimensional subspace. Liang et al. [49] designed a relaxed domain-irrelevant class clustering (DICE) term and then combined it with MMD to obtain a domain-irrelevant projection for reducing distribution discrepancy between domains. Moreover, DICE was extended to ensemble learning with multiple projection obtained from sampling subsets of source and target domains, which help it achieve better performance. Progressive learning with Confidence-wEighted Targets (PACET) [50] improved DICE by adding a confidence-weight strategy with the posterior probability of target instance. (3) Classifier (or parameter-based) adaptation. Its purpose is to find optimal classifier or its parameter with a well-generalized ability between the source and target domains. Yang et al. [51] presented adaptive support vector machine (Adapt-SVM). It designed a regularizer to minimize the discrepancy between parameters of two classifiers trained on source and target labeled samples and then added into SVM’s objective function. Multi-model knowledge transfer (Multi-KT) [52], following the idea of Adapt-SVM, constructs a regularizer to force the parameter of target SVM close to multiple weighted source SVMs. In multiple kernel learning, Wang et al. [53] introduced multiple kernel MMD into the objective function to adapt distribution discrepancy between training samples and test samples, which prevents performance degradation because of inconsistent distribution of datasets and simultaneously obtain a multiple kernel classifier with strong generalization ability. Recently, deep network adaptation and adversarial learning adaptation have become successful in computer version and machine learning. Based on the assumption that samples with the same category are close each to other and the local geometry property of the data can be maintained in neural embedding subspace, Wang et al. [54] proposed the neural embedding match (NEM), which reduces cross-domain distribution divergence by projecting the source and target domains into a common subspace using deep neural network embedding model. In [55], a deep neural network with weighted MMD and the manifold embedding was proposed to handle domain adaptation for hyperspectral image classification. To address the problem in unsupervised partial domain adaptation (PDA), Liang et al. [56] put forward a domain adversarial neural networks called BA3US. It presented balanced adversarial alignment (BAA) and adaptive uncertainty suppression (AUS) to overcome negative transfer and propagation of uncertainty which usually appear in PDA.In the abovementioned approaches, sample-based adaptation methods are the most efficient ones for knowledge transfer because of direct utilization of source sample, while feature-based adaptation methods are widely applied. Classifier (or parameter)-based adaptation is the most potential one due to past related domain knowledge or experience is integrated into shared parameters of classifier. Deep network adaptation and adversarial learning adaptation strictly belong to feature-based adaptation method but they can extract (deep) domain-invariant feature with strong discrimination. However, these methods also have their own shortcomings. In sample-based adaptation methods, the effective evaluation mechanism about sample importance is a challenge. Generic shared features obtained from different domains is also a difficult task in feature-based adaptation methods. Since the useful information and knowledge from the auxiliary domain to the target domain is not directly applied, classifier (or parameter-based) adaptation is not efficient compared with two formers. Deep network adaptation usually needs massive labeled samples and sufficient computing resources for training deep model, which could hinder its application. Class misalignment and the simultaneous efficiency of feature extractor and discriminator are a challenge for adversarial learning adaptation. In this article, our approach belongs to classifier (or parameter)-based adaptation; it attempts to seek two output weights of shared ELM models across domains for knowledge transferring.In this article, we propose TSTELM to address problems in the unsupervised domain adaptation, in which the training data come from the source domain with labeled samples and the test data come from the target domain with unlabeled samples. Suppose the source domain dataset is denoted asxSi,yii=1nS∈DS and the target domain dataset is denoted as xTjj=1nT∈DT, where nS and nT represent the number of source and target samples, respectively. The source data and the target data belong to the same feature space XS=XT and label space YS=YT. The data distributions of the source and the target domains should be different but similar, that is the marginal distribution PXS≈PXT and conditional distribution PYS|XS≈PYT|XT. In TSTELM, we hope to construct an ELM model using xsi,ysii=1nS to obtain high accuracy on xTekk=1nTe.Table 1 summarizes other related notations in domain adaptation problems.Table 1
Notations.
TerminologySourceTargetDomainDS=XS,PXSDT=XT,PXTDataXS=xSi,yii=1nSXT=xTjj=1nTFeature spaceXSXTLabel spaceYSYTMarginal distributionPSXSPTXTConditional distributionPYS|XSPYT|XTTaskTS=YS,PYS|XSTT=YT,PYT|XT


### 2.2. Extreme Learning Machine

Unlike the conventional feedforward neural networks, ELM is an approach in which two characteristics are contained: (1) Hidden layer parameters (i.e., input weights and the biases) can be randomly initialized. (2) The output layer weight can be solved as the least-squares problem. As a result, it yields faster learning speed and better generalization performance compared with other classifiers.Given a training setxi,yii=1N with N samples, where yi is the label corresponding to xi, and C is the number of categories. The structure of the ELM with L hidden nodes and activation function hx is shown in Figure 2:Figure 2
The structure of ELM.In Figure2, xi is the input sample, w is the input layer weight, b is the hidden layer bias, and the hidden layer output hx is calculated as: hxi=gwxi+b. Here, gx is the nonlinear activation function, L is the number of nodes in the hidden layer, and βi is the hidden layer output weight. The outputs of the network are given by:(1)yj=∑i=1Lβihwi⋅xj+bi,j=1,2,…,N.The above formula can be written in matrix form:(2)Y=Hβ,where H=hw1⋅x1+b1⋯hwL⋅x1+bL⋮⋱⋮hw1⋅xN+b1⋯hwL⋅xN+bLN×L,β=β1⋮βL,andY=y1⋮yN.By adopting parameter regularization, the ELM can avoid the overfitting problem. Its corresponding objective function can be formulated as(3)minβLELM=12β2+θ2Y−Hβ2,where θ is a penalty constant on the training errors, and •2 denotes the L2-norm of a matrix or a vector.The minimization of equation (3) is a regularized least-squares problem. By setting the gradient of equation (3) with respect to β as zero, we have(4)∇LELM=β+θHTY−Hβ=0.The output weight vectorβ is obtained according to the Moore–Penrose principle. If N>L, the optimal solution of equation (3) is:(5)β∗=HTH+ILθHTY,where IL is a L-dimensional unit matrix.IfN≤L, the optimal solution of equation (3) is:(6)β∗=HTHTH+INθY,where IN is an N-dimensional unit matrix.


## 2.1. Domain Adaptation

When the training and testing data are drawn from different distributions, a classifier directly learned on training data would have a poor performance for testing data. Domain adaptation is developed to deal with this scenario, in which an excellent classifier could be obtained from the source domain with rich labeled samples and perform well in the target domain task. The source domain and the target domain have different distributions but some correlations.In the past decades, many researches have conducted to address domain adaptation problems in classification task, which are mainly divided into three parts [35, 43]: (1) Sample-based adaptation. It directly assigns weights to each sample of two domains, which could adapt and minimize the distribution gap between domains. Many such approaches appeared such as domain adaptation (PRDA) [44], TrAdaBoost [45], and Kernel Mean Match (KMM) [46]. (2) Feature-based adaption. It seeks the shared subspace between domains, in which distribution discrepancy is alleviated and knowledge is easily transferred across domains. Transfer component analysis (TCA) [47] and joint distribution adaptation (JDA) [48] take MMD metric as an objective function to find an optimal projected matrix for shared low-dimensional subspace. Liang et al. [49] designed a relaxed domain-irrelevant class clustering (DICE) term and then combined it with MMD to obtain a domain-irrelevant projection for reducing distribution discrepancy between domains. Moreover, DICE was extended to ensemble learning with multiple projection obtained from sampling subsets of source and target domains, which help it achieve better performance. Progressive learning with Confidence-wEighted Targets (PACET) [50] improved DICE by adding a confidence-weight strategy with the posterior probability of target instance. (3) Classifier (or parameter-based) adaptation. Its purpose is to find optimal classifier or its parameter with a well-generalized ability between the source and target domains. Yang et al. [51] presented adaptive support vector machine (Adapt-SVM). It designed a regularizer to minimize the discrepancy between parameters of two classifiers trained on source and target labeled samples and then added into SVM’s objective function. Multi-model knowledge transfer (Multi-KT) [52], following the idea of Adapt-SVM, constructs a regularizer to force the parameter of target SVM close to multiple weighted source SVMs. In multiple kernel learning, Wang et al. [53] introduced multiple kernel MMD into the objective function to adapt distribution discrepancy between training samples and test samples, which prevents performance degradation because of inconsistent distribution of datasets and simultaneously obtain a multiple kernel classifier with strong generalization ability. Recently, deep network adaptation and adversarial learning adaptation have become successful in computer version and machine learning. Based on the assumption that samples with the same category are close each to other and the local geometry property of the data can be maintained in neural embedding subspace, Wang et al. [54] proposed the neural embedding match (NEM), which reduces cross-domain distribution divergence by projecting the source and target domains into a common subspace using deep neural network embedding model. In [55], a deep neural network with weighted MMD and the manifold embedding was proposed to handle domain adaptation for hyperspectral image classification. To address the problem in unsupervised partial domain adaptation (PDA), Liang et al. [56] put forward a domain adversarial neural networks called BA3US. It presented balanced adversarial alignment (BAA) and adaptive uncertainty suppression (AUS) to overcome negative transfer and propagation of uncertainty which usually appear in PDA.In the abovementioned approaches, sample-based adaptation methods are the most efficient ones for knowledge transfer because of direct utilization of source sample, while feature-based adaptation methods are widely applied. Classifier (or parameter)-based adaptation is the most potential one due to past related domain knowledge or experience is integrated into shared parameters of classifier. Deep network adaptation and adversarial learning adaptation strictly belong to feature-based adaptation method but they can extract (deep) domain-invariant feature with strong discrimination. However, these methods also have their own shortcomings. In sample-based adaptation methods, the effective evaluation mechanism about sample importance is a challenge. Generic shared features obtained from different domains is also a difficult task in feature-based adaptation methods. Since the useful information and knowledge from the auxiliary domain to the target domain is not directly applied, classifier (or parameter-based) adaptation is not efficient compared with two formers. Deep network adaptation usually needs massive labeled samples and sufficient computing resources for training deep model, which could hinder its application. Class misalignment and the simultaneous efficiency of feature extractor and discriminator are a challenge for adversarial learning adaptation. In this article, our approach belongs to classifier (or parameter)-based adaptation; it attempts to seek two output weights of shared ELM models across domains for knowledge transferring.In this article, we propose TSTELM to address problems in the unsupervised domain adaptation, in which the training data come from the source domain with labeled samples and the test data come from the target domain with unlabeled samples. Suppose the source domain dataset is denoted asxSi,yii=1nS∈DS and the target domain dataset is denoted as xTjj=1nT∈DT, where nS and nT represent the number of source and target samples, respectively. The source data and the target data belong to the same feature space XS=XT and label space YS=YT. The data distributions of the source and the target domains should be different but similar, that is the marginal distribution PXS≈PXT and conditional distribution PYS|XS≈PYT|XT. In TSTELM, we hope to construct an ELM model using xsi,ysii=1nS to obtain high accuracy on xTekk=1nTe.Table 1 summarizes other related notations in domain adaptation problems.Table 1
Notations.
TerminologySourceTargetDomainDS=XS,PXSDT=XT,PXTDataXS=xSi,yii=1nSXT=xTjj=1nTFeature spaceXSXTLabel spaceYSYTMarginal distributionPSXSPTXTConditional distributionPYS|XSPYT|XTTaskTS=YS,PYS|XSTT=YT,PYT|XT


## 2.2. Extreme Learning Machine

Unlike the conventional feedforward neural networks, ELM is an approach in which two characteristics are contained: (1) Hidden layer parameters (i.e., input weights and the biases) can be randomly initialized. (2) The output layer weight can be solved as the least-squares problem. As a result, it yields faster learning speed and better generalization performance compared with other classifiers.Given a training setxi,yii=1N with N samples, where yi is the label corresponding to xi, and C is the number of categories. The structure of the ELM with L hidden nodes and activation function hx is shown in Figure 2:Figure 2
The structure of ELM.In Figure2, xi is the input sample, w is the input layer weight, b is the hidden layer bias, and the hidden layer output hx is calculated as: hxi=gwxi+b. Here, gx is the nonlinear activation function, L is the number of nodes in the hidden layer, and βi is the hidden layer output weight. The outputs of the network are given by:(1)yj=∑i=1Lβihwi⋅xj+bi,j=1,2,…,N.The above formula can be written in matrix form:(2)Y=Hβ,where H=hw1⋅x1+b1⋯hwL⋅x1+bL⋮⋱⋮hw1⋅xN+b1⋯hwL⋅xN+bLN×L,β=β1⋮βL,andY=y1⋮yN.By adopting parameter regularization, the ELM can avoid the overfitting problem. Its corresponding objective function can be formulated as(3)minβLELM=12β2+θ2Y−Hβ2,where θ is a penalty constant on the training errors, and •2 denotes the L2-norm of a matrix or a vector.The minimization of equation (3) is a regularized least-squares problem. By setting the gradient of equation (3) with respect to β as zero, we have(4)∇LELM=β+θHTY−Hβ=0.The output weight vectorβ is obtained according to the Moore–Penrose principle. If N>L, the optimal solution of equation (3) is:(5)β∗=HTH+ILθHTY,where IL is a L-dimensional unit matrix.IfN≤L, the optimal solution of equation (3) is:(6)β∗=HTHTH+INθY,where IN is an N-dimensional unit matrix.


## 3. Proposed Methods

In this section, the overall architecture of the proposed TSTELM is introduced in detail. As shown in Figure1, TSTELM consists of three parts: statistical matching stage, subspace alignment stage, and prediction based on weight fusion.


### 3.1. Statistical Matching Stage

In statistical matching stage, we hope to obtain an ELM for domain adaptation using labeled samples of the source domain and unlabeled samples of the target domain. First, the source data and target data are mapped into the hidden layer space of ELM, and then we could obtainHS=hSi,yii=1nS and HT=hTjj=1nT, respectively, where hSxi=gwxSi+b and hTxj=gwxTj+b, n=nS+nT, w∈Rd×L and b∈R1×L are randomly generated weights and bias, d is the original spatial dimension of the data.For labeled source data, we can learn an ELM, that is(7)minβS:12βS2+12HSβS−YS2,where βS is the output weight of the ELM learned on HS,YS. Since equation (7) just obtain an ELM classifier using the labeled source samples, it cannot perform well for the target domains due to distribution difference between the source and target domains. Therefore, we adopt MMD between HS and HT to reduce marginal and conditional distribution difference between domains [42]. MMD minimization is formulated as:(8)MMDβ=minβ1nS∑i=1nShxiβ−1nT∑j=1nThxjβℋ2+∑c=1C1nSc∑xi∈DSchxicβ−1nTc∑xj∈DTchxjcβℋ2=minβtrβTHTM0+∑c=1CMcHβ,where H=HST∪HTT; M0 and Mc are the MMD matrixes which are as follows:(9)M0ij=1nSnS,xi,xj∈DS,1nTnT,xi,xj∈DT,−1nSnT,otherwise,(10)Mcij=1nScnSc,xic,xjc∈DSc,1nTcnTc,xic,xjc∈DTc,−1nScnTc,xic∈DScandxjc∈DTc,xjc∈DScandxic∈DTc,0,otherwise,where DSc and DTc are the sample subsets with label category c in the source and target domains; xic and xjc are the samples in DSc and DTc, respectively; nSc and nTc are the number of samples in DSc and DTc, respectively.Here, replacingβS and β with β1 and incorporating equations (7) and (8), we can obtain the DAELM at statistical matching stage, and its objective function is(11)minβ12β12+θ2HSβ1−YS2+λ2Trβ1THTM0+∑c=1CMcHβ1.By setting the gradient of equation (11) with respect to β1 as zero, we have(12)β1=HTHEHT+λM0+∑c=1CMcHHT+ILθ−1EYST,n>L,HTEH+λHTM0+∑c=1CMcH+Inθ−1HTEYST,n≤L,where E is a diagonal label indicator matrix with each element Eii=1 if xi∈DS, and Eii=0 otherwise.


### 3.2. Subspace Alignment Stage

At subspace alignment stage, we train a DAELM on labeled samples of the source domains and unlabeled samples of the target domains.For the target sample, we can learn an ELM from the following formula:(13)minβT:12βT2+α2HTβT−HS_avβ12+α2∑c=1CHTcβT−HS_avcβ12,⇔minβT:12βT2+α2∑c=0CHTcβT−HS_avcβ12.Here, inspired by [35], we introduce cross-domain mean approximation to replace the prediction loss. When there are no labeled samples in the target domains (when c=0), we force target data HT close to source data mean point HS_av, which promotes domain adaptation seen from [35]. If the target sample obtains pseudo labels, it is drawn to source data mean point with the same category HS_avc.In order to further improve cross-domain knowledge transferring, similar to [38], we introduce a transformation matrix M to align the output weights of ELM between the source domain and the target domain. The function is established as follows:(14)fM=minβ1M−βTF2,where •F2 is Frobenius norm. It is invariant to the orthogonalization operation, so equation (14) can be rewritten as:(15)fM=minβ1Tβ1M−β1TβTF2=minM−β1TβTF2.Then, we can get the optimalM∗=β1TβT. Let βa=β1M=β1β1TβT, we can know that βa is closer to βT than β1 and facilitates cross-domain knowledge transfer.To align output layer of source ELM to target one, we combine the training errorHSβa−YS2, equation (13), and a regular term and replace β1 with βa to get:(16)Jβa,βT=minβa,βT12βT2+θ2HSβa−YS2+α2∑c=0CHTcβT−HS_avcβ12+12βT−βa2,where βT−βa2 is a parameter approximation term for facilitating knowledge transfer and preventing negative transfer, and λ and γ are the balance parametesr. We substitute βa=β1β1TβT to equation (16) and get:(17)JβT=minβT12βT2+θ2HSβ1β1TβT−YS2+α2∑c=0CHTcβT−HS_avcβ12+12βT−β1β1TβT2=minβT12βT2+θ2HSβ1β1TβT−YS2+α2∑c=0CHTcβT−HS_avcβ12+12I−β1β1TβT2.BecauseI−β1β1TβT2≤I−β1β1T2βT2, we change equation (17) into:(18)JβT=minβT12βT2+θ2HSβ1β1TβT−YS2+α2∑c=0CHTcβT−HS_avcβ12+12I−β1β1T2βT2=minβT12θHSβ1β1TαHTβT−YSα∑c=0CHS_avcβ12+I+I−β1β1TTI−β1β1T2βT2.LetQ=θHSβ1β1TαHT,T=YSα∑c=0CHS_avcβ1,andA=I+I−β1β1TTI−β1β1T, and equation (18) can be simplified as(19)JβT=minβT:12QβT−T2+A2βT2.Then:(20)β2=βT∗=QTQ+A−1QTT,n>L,IinAis anL−dimensional unit matrix,QTQQT+A−1T,n≤L,IinAis ann−dimensional unit matrix.


### 3.3. Prediction Based on Output Weight Fusion

In the classification task, a samplexTe is tested. After β1 and β2 are obtained, the output weight of final ELM model is dominated by β∗=β2+pβ1, and the classification result of xTe can be obtained:(21)yTe=signhTeTβ∗,for binary classification,argmaxhTeTβ∗,for multi−class classification,where hTe=gxTe and p is the scale factor to balance β1 and β2.TSTELM can be summarized in Algorithm 1.Algorithm 1: TSTELM.
Input: Source domain DS, source labels YS, target domain DT, maximum iterations T.(1)
Randomly initialize the input weightsw and biases b of the ELM network with L hidden nodes; set the trade-off parameters p, α, θ, and λ.(2)
Calculate the matrixHS and HT, obtain H0 using equation (9).(3)
Compute the optimal weightsβ1 by equation (12).(4)
Calculate the matrixHS_av and compute the optimal weights β2 using equation (20).(5)
Construct the MMD matrixesM0, Mc, and HS_avc.(6)
Compute the optimal weightsβ∗=β2+pβ1 and obtain the prediction Y^T of HT using equation (21).(7)
Repeat step 2–6 until the number of iterations reaches toT or Y^T no change.Output: The output weight β∗ and the predicted output Y^T.


### 3.4. Discussion

In order to solve the problem that the traditional ELM does not perform well in cross-domain tasks, we propose TSTELM and its objective function is equations (8) and (17). It can be seen:(1)
Compared with the classical ELM, TSTELM reduces the distribution difference between domains and transfers knowledge across domains via adopting MMD, output weight alignment, parameter approximation, and∑c=0CHTcβT−HS_avcβ12.(2)
Though TELM-OWA proposed by Zang et al. [38] also applies output weight alignment and parameter approximation for domain adaptation, it is a supervised domain adaptation algorithm requiring few target labeled samples unlike TSTELM. In addition, TSTELM replaces HTβT−YT2 with ∑c=0CHTcβT−HS_avcβ12, which is different from TELM-OWA.(3)
Different from DAELM-S, DAELM-T, TELM-OWA, and CDELM, in which only one output weightβ∗ is solved, our method needs the learned weights of two stages to fuse and then make decisions, which shows that TSTELM has strong robustness similar to ensemble learning.(4)
In domain adaptation, we consider samples from the source domains as training datasets and samples from the target domains as testing datasets, so the output matrixesHS and HT are computed which have time complexity of OLnSd and OLnTd. The main time cost of ELM is equation (5) or (6), and the time complexity is:(22)OnS3+2LnS2+mLnSwhen,nS<L,OL3+L2nS+mLnSwhen,nS>L.According to Algorithm 1, the main computation cost of our method is in steps 3 and 4.In step 3, we need to computeM0, ∑c=1CMc, E+M0+∑c=1CMcHHT, HTE+M0+∑c=1CMcH, the inverse of matrix with N×N or L×L size and HTEYS which has time complexity ON2, OCN2, ON3+LN2, OLN2+L2N, ON3 or OL3, and OCLN+LN2. Therefore, the time complexity of step 2 is(23)O2TN3+2TLN2+TC+1N2+TCLNwhen,N<L,OTL3+2TLN2+TL2N+TC+1N2+TCLNwhen,N>L,where T is the number of iterations.In step 4, the output weight is determined according to equation (20) and Q has the same size of H. Therefore, the time complexity of step 4 is:(24)OTN3+2TLN2+TCLNwhen,N<L,OTL3+TL2N+TCLNwhen,N>L.Given that TELM-OWA also has two stages to compute the output weight, it has time complexity in the first stage as follows:(25)OnS3+2LnS2+mLnSwhen,nS<L,OL3+L2nS+mLnSwhen,nS>L.The time complexity of TELM-OWA in the second stage is(26)ON3+2LN2+CLNwhen,N<L,OL3+L2N+CLNwhen,N>L.The above analysis indicates that the computational complexity of TSTELM is significantly higher than ELM and TELM-OWA.


## 3.1. Statistical Matching Stage

In statistical matching stage, we hope to obtain an ELM for domain adaptation using labeled samples of the source domain and unlabeled samples of the target domain. First, the source data and target data are mapped into the hidden layer space of ELM, and then we could obtainHS=hSi,yii=1nS and HT=hTjj=1nT, respectively, where hSxi=gwxSi+b and hTxj=gwxTj+b, n=nS+nT, w∈Rd×L and b∈R1×L are randomly generated weights and bias, d is the original spatial dimension of the data.For labeled source data, we can learn an ELM, that is(7)minβS:12βS2+12HSβS−YS2,where βS is the output weight of the ELM learned on HS,YS. Since equation (7) just obtain an ELM classifier using the labeled source samples, it cannot perform well for the target domains due to distribution difference between the source and target domains. Therefore, we adopt MMD between HS and HT to reduce marginal and conditional distribution difference between domains [42]. MMD minimization is formulated as:(8)MMDβ=minβ1nS∑i=1nShxiβ−1nT∑j=1nThxjβℋ2+∑c=1C1nSc∑xi∈DSchxicβ−1nTc∑xj∈DTchxjcβℋ2=minβtrβTHTM0+∑c=1CMcHβ,where H=HST∪HTT; M0 and Mc are the MMD matrixes which are as follows:(9)M0ij=1nSnS,xi,xj∈DS,1nTnT,xi,xj∈DT,−1nSnT,otherwise,(10)Mcij=1nScnSc,xic,xjc∈DSc,1nTcnTc,xic,xjc∈DTc,−1nScnTc,xic∈DScandxjc∈DTc,xjc∈DScandxic∈DTc,0,otherwise,where DSc and DTc are the sample subsets with label category c in the source and target domains; xic and xjc are the samples in DSc and DTc, respectively; nSc and nTc are the number of samples in DSc and DTc, respectively.Here, replacingβS and β with β1 and incorporating equations (7) and (8), we can obtain the DAELM at statistical matching stage, and its objective function is(11)minβ12β12+θ2HSβ1−YS2+λ2Trβ1THTM0+∑c=1CMcHβ1.By setting the gradient of equation (11) with respect to β1 as zero, we have(12)β1=HTHEHT+λM0+∑c=1CMcHHT+ILθ−1EYST,n>L,HTEH+λHTM0+∑c=1CMcH+Inθ−1HTEYST,n≤L,where E is a diagonal label indicator matrix with each element Eii=1 if xi∈DS, and Eii=0 otherwise.


## 3.2. Subspace Alignment Stage

At subspace alignment stage, we train a DAELM on labeled samples of the source domains and unlabeled samples of the target domains.For the target sample, we can learn an ELM from the following formula:(13)minβT:12βT2+α2HTβT−HS_avβ12+α2∑c=1CHTcβT−HS_avcβ12,⇔minβT:12βT2+α2∑c=0CHTcβT−HS_avcβ12.Here, inspired by [35], we introduce cross-domain mean approximation to replace the prediction loss. When there are no labeled samples in the target domains (when c=0), we force target data HT close to source data mean point HS_av, which promotes domain adaptation seen from [35]. If the target sample obtains pseudo labels, it is drawn to source data mean point with the same category HS_avc.In order to further improve cross-domain knowledge transferring, similar to [38], we introduce a transformation matrix M to align the output weights of ELM between the source domain and the target domain. The function is established as follows:(14)fM=minβ1M−βTF2,where •F2 is Frobenius norm. It is invariant to the orthogonalization operation, so equation (14) can be rewritten as:(15)fM=minβ1Tβ1M−β1TβTF2=minM−β1TβTF2.Then, we can get the optimalM∗=β1TβT. Let βa=β1M=β1β1TβT, we can know that βa is closer to βT than β1 and facilitates cross-domain knowledge transfer.To align output layer of source ELM to target one, we combine the training errorHSβa−YS2, equation (13), and a regular term and replace β1 with βa to get:(16)Jβa,βT=minβa,βT12βT2+θ2HSβa−YS2+α2∑c=0CHTcβT−HS_avcβ12+12βT−βa2,where βT−βa2 is a parameter approximation term for facilitating knowledge transfer and preventing negative transfer, and λ and γ are the balance parametesr. We substitute βa=β1β1TβT to equation (16) and get:(17)JβT=minβT12βT2+θ2HSβ1β1TβT−YS2+α2∑c=0CHTcβT−HS_avcβ12+12βT−β1β1TβT2=minβT12βT2+θ2HSβ1β1TβT−YS2+α2∑c=0CHTcβT−HS_avcβ12+12I−β1β1TβT2.BecauseI−β1β1TβT2≤I−β1β1T2βT2, we change equation (17) into:(18)JβT=minβT12βT2+θ2HSβ1β1TβT−YS2+α2∑c=0CHTcβT−HS_avcβ12+12I−β1β1T2βT2=minβT12θHSβ1β1TαHTβT−YSα∑c=0CHS_avcβ12+I+I−β1β1TTI−β1β1T2βT2.LetQ=θHSβ1β1TαHT,T=YSα∑c=0CHS_avcβ1,andA=I+I−β1β1TTI−β1β1T, and equation (18) can be simplified as(19)JβT=minβT:12QβT−T2+A2βT2.Then:(20)β2=βT∗=QTQ+A−1QTT,n>L,IinAis anL−dimensional unit matrix,QTQQT+A−1T,n≤L,IinAis ann−dimensional unit matrix.


## 3.3. Prediction Based on Output Weight Fusion

In the classification task, a samplexTe is tested. After β1 and β2 are obtained, the output weight of final ELM model is dominated by β∗=β2+pβ1, and the classification result of xTe can be obtained:(21)yTe=signhTeTβ∗,for binary classification,argmaxhTeTβ∗,for multi−class classification,where hTe=gxTe and p is the scale factor to balance β1 and β2.TSTELM can be summarized in Algorithm 1.Algorithm 1: TSTELM.
Input: Source domain DS, source labels YS, target domain DT, maximum iterations T.(1)
Randomly initialize the input weightsw and biases b of the ELM network with L hidden nodes; set the trade-off parameters p, α, θ, and λ.(2)
Calculate the matrixHS and HT, obtain H0 using equation (9).(3)
Compute the optimal weightsβ1 by equation (12).(4)
Calculate the matrixHS_av and compute the optimal weights β2 using equation (20).(5)
Construct the MMD matrixesM0, Mc, and HS_avc.(6)
Compute the optimal weightsβ∗=β2+pβ1 and obtain the prediction Y^T of HT using equation (21).(7)
Repeat step 2–6 until the number of iterations reaches toT or Y^T no change.Output: The output weight β∗ and the predicted output Y^T.


## 3.4. Discussion

In order to solve the problem that the traditional ELM does not perform well in cross-domain tasks, we propose TSTELM and its objective function is equations (8) and (17). It can be seen:(1)
Compared with the classical ELM, TSTELM reduces the distribution difference between domains and transfers knowledge across domains via adopting MMD, output weight alignment, parameter approximation, and∑c=0CHTcβT−HS_avcβ12.(2)
Though TELM-OWA proposed by Zang et al. [38] also applies output weight alignment and parameter approximation for domain adaptation, it is a supervised domain adaptation algorithm requiring few target labeled samples unlike TSTELM. In addition, TSTELM replaces HTβT−YT2 with ∑c=0CHTcβT−HS_avcβ12, which is different from TELM-OWA.(3)
Different from DAELM-S, DAELM-T, TELM-OWA, and CDELM, in which only one output weightβ∗ is solved, our method needs the learned weights of two stages to fuse and then make decisions, which shows that TSTELM has strong robustness similar to ensemble learning.(4)
In domain adaptation, we consider samples from the source domains as training datasets and samples from the target domains as testing datasets, so the output matrixesHS and HT are computed which have time complexity of OLnSd and OLnTd. The main time cost of ELM is equation (5) or (6), and the time complexity is:(22)OnS3+2LnS2+mLnSwhen,nS<L,OL3+L2nS+mLnSwhen,nS>L.According to Algorithm 1, the main computation cost of our method is in steps 3 and 4.In step 3, we need to computeM0, ∑c=1CMc, E+M0+∑c=1CMcHHT, HTE+M0+∑c=1CMcH, the inverse of matrix with N×N or L×L size and HTEYS which has time complexity ON2, OCN2, ON3+LN2, OLN2+L2N, ON3 or OL3, and OCLN+LN2. Therefore, the time complexity of step 2 is(23)O2TN3+2TLN2+TC+1N2+TCLNwhen,N<L,OTL3+2TLN2+TL2N+TC+1N2+TCLNwhen,N>L,where T is the number of iterations.In step 4, the output weight is determined according to equation (20) and Q has the same size of H. Therefore, the time complexity of step 4 is:(24)OTN3+2TLN2+TCLNwhen,N<L,OTL3+TL2N+TCLNwhen,N>L.Given that TELM-OWA also has two stages to compute the output weight, it has time complexity in the first stage as follows:(25)OnS3+2LnS2+mLnSwhen,nS<L,OL3+L2nS+mLnSwhen,nS>L.The time complexity of TELM-OWA in the second stage is(26)ON3+2LN2+CLNwhen,N<L,OL3+L2N+CLNwhen,N>L.The above analysis indicates that the computational complexity of TSTELM is significantly higher than ELM and TELM-OWA.


## 4. Experiment and Analysis

In this section, experiments are conducted on four cross-domain datasets including Office + Caltech object recognition, USPS and MNIST digital handwriting, MSRC and VOC2007 object recognition, and Reuters-21578 text dataset for classification, where image datasets are descripted in Table2. We compare our approach with several related unsupervised classification methods and semi-supervised and unsupervised domain adaptation methods. To be more objective, experiments are implemented on PC with 8 GB memory and Windows 10 operating system and MATLAB 2017b. Every experiment runs 20 times and the average value is recorded. We adopt the accuracy rate to evaluate the performance of every algorithm, and it is(27)Accuracy=correctly_classified_samplestotal_samples×100%.Table 2
Description of image dataset.
DatasetTypeSamplesDimensionClassContains subsetsUSPSDigit180025610USPSMNISTDigit200025610MNISTMSRCObject126924018MSRCVOC2007Object153024020VOCOfficeObject141080010A, W, DCaltechObject112380010C


### 4.1. Dataset Description

aOffice + Caltech256 (shown in Figure 3): Office is widely used dataset for visual cross-domain learning, which contains 4,652 images in 31 categories. These images come from 3 realistic aggregated item datasets: Amazon (images download from online chants https://www.amazon.com); DSLR (high-resolution images by a digital SLR camera in realistic environments); and Webcam (low-resolutions images by a simple webcam). Caltech256 is also a standard object recognition dataset which contains 30,607 images from 256 categories.Figure 3
Image samples from (b) Amazon, (d) Caltech256, (a) DSLR, and (c) Webcam.
(a)(b)(c)(d)In this article, we employ the Office + Caltech dataset released by Gong et al. [57]. SURF features are extracted and quantized into an 800-bin histogram with codebooks computed with K-means on a subset of images from Amazon. Then, the histograms are standardized by z-score. We select four domains C (Caltech256), A (Amazon), W (Webcam), and D (DSLR) for experiment, and two different domains are randomly selected as the source and the target domain datasets, and 12 cross-domain tasks for evaluation are constructed, namely C ⟶ A, C ⟶ W, C ⟶ D, …, and D ⟶ W.USPS + MNIST (as shown in Figure 4): USPS and MNIST are the two different but related handwritten datasets with 10 categories of 0–9. The USPS dataset contains 7,291 training samples and 2,007 test samples with 16 × 16 pixels. There are 60,000 training images and 10,000 test images with 28 × 28 pixels in the MNIST database.Figure 4
Image samples from (a) MNIST and (b) USPS.
(a)(b)During this experiment, we randomly select 1,800 pictures from USPS and 2,000 pictures from MNIST and convert them into 16 × 16 pixels. We construct two cross-domain tasks, that is USPS as the source domain and MNIST as the target domain (USPS vs MNIST) and vice versa (MNIST vs USPS).MSRC + VOC2007 (shown in Figure 5): MSRC dataset is provided by Microsoft Cambridge, which consists of 4323 images of 18 object classes. VOC2007 dataset contains 5,011 images annotated with 20 concepts. We can see from Figure 5 that MSRC and VOC2007 have distinct but different distributions. MSRC are standard images as benchmark data for evaluation. VOC2007 is randomly constructed by using the images in the network album.Figure 5
Image samples from (a) MSRC and (b) VOC2007.
(a)(b)In our experiments, we construct the domain adaptation dataset MSRC vs VOC in which 6 shared categories are selected including aircraft, birds, cows, family cars, sheep and bicycles. Among them, 1269 images are selected from the MSRC dataset as the source domain dataset and 1530 images are selected from the VOC2007 dataset as the target domain dataset. Then, the source domain and the target domain are exchanged to construct a new set of domain adaptation dataset VOC vs MSRC. We convert all images into 256 gray pixels; 240 dimensions are extracted as the spatial dimension of the sample.Reuters-21578: Reuters-21578 text dataset is a common dataset for text classification. It contains 21,577 news documents from Reuters in 1987. These documents have been manually labeled by Reuters as five classes, such as “exchanges,” “orgs,” “people,” “places,” and “topics,” including multiple categories and subclasses. Among them, the largest three categories are “orgs,” “people,” and “place,” which can construct six cross-domain text classification tasks orgs vs people, people vs orgs, orgs vs place, place vs orgs, people vs place, and place vs people. This article makes a more complete evaluation of the algorithm on 6 classification tasks.


### 4.2. Experimental Settings

To validate the efficiency of TSTELM, we compare it with some other classifiers.Classifiers for non-domain adaptation: 1NN, SVM, ELM, and SSELM [58] (ELM with graph regularization term for semi-supervised learning).Classifiers for domain adaptation: TCA1 [47](TCA with 1NN for classification), TCA2 [47] (TCA with SVM for classification), JDA1 [48] (JDA with 1NN for classification), JDA2 [48](JDA with SVM for classification), DAELM-S [36], DAELM-T [36], ARRLS [59], TELM-OWA [38], JPDA [60], AELM [61], DST-ELM [40], CDELM [39], and TSTELM.In order to achieve the optimal performance of each algorithm in the experiment, we set SVM penalty parameter belonging to0.1,0.5,1,5,10,50,100, and the penalty parameter θ∈0.001,0.1 in ELM, SSELM, DAELM_S, DAELM_T, TELM-OWA, and TSTELM. TCA(1,2) and JDA(1,2) are combined feature extraction with standard classifier for domain adaptation, in which the dimension of the feature subspace is 100 and the value range of the balanced-constraint parameter of the projection matrix in TCA and JDA is 0.1,1. The parameters of ARRLS, DAELM_S, DAELM_T, and TELM-OWA are set according to the corresponding literature. In TSTELM, we set α∈103,104, λ∈10−3,10−4,and L=1500 on Office + Caltech dataset, L=3000 on USPS + MNIST and MSRC + VOC2007 datasets, L=5000 on Reuters-21578 dataset. We cite results of JPDA, AELM, DST-ELM, and CDELM from those corresponding literature.In fact, DAELM_S, DAELM_T, and TELM-OWA, as supervised models, need a few labeled target samples to induce the target classifier. We test them with 0.5% labeled target samples on USPS + MNIST, MSRC + VOC2007, and Reuters-21578 datasets and 1% labeled target samples on Office + Caltech dataset, to approximate the performance of these methods in unsupervised domain adaptation.


### 4.3. Experimental Results and Analysis

We test TSTELM on Office + Caltech256, USPS + MNIST, MSRC + VOC2007, and Reuters-21578 datasets, and the comparison results are displayed in Tables3 and 4 and Figures 4–8, in which the best results of each task are bold. From the comparison of results, we have the following observations.(1)
The total average accuracy of TSTELM across 22 tasks is the highest among all methods, indicating that our approach can effectively deal with domain adaptation problem.(2)
In unsupervised domain adaptation, the proposed method outperforms AELM, DST-ELM, CDELM, TELM-OWA, DAELM_S, and DAELM_T, indicating the superiority of TSTELM in which MMD and output weight alignment are unified into the ELM learning framework to minimize distribution discrepancy between domains. AELM, DAELM_S, and DAELM_T obtain poor results, showing that they are highly dependent on labeled target samples. 1NN, SVM, and ELM perform unsuccessfully because of the problem of domain shift. SSELM performs better than ELM due to that the original geometry information of data is mined.(3)
TCA(1,2), JDA(1,2), and JPDA are better than 1NN and SVM, showing the importance of shared feature extraction across domains. JPDA and JDA(1,2) are generally higher than TCA(1,2), which indicates the superiority of simultaneously reducing the marginal and conditional distribution discrepancy. ARRLS performs well because of MMD and preserves the manifold consistency at the same time.Table 3
Accuracy of different algorithms on Office + Caltech256 and USPS + MNIST datasets.
MethodsDatasetC ⟶ A(1)C ⟶ W(2)C ⟶ D(3)A ⟶ C(4)A ⟶ W(5)A ⟶ D (6)W ⟶ C (7)W ⟶ A (8)W ⟶ D(9)D ⟶ C(10)D ⟶ A(11)D ⟶ W(12)AverageUSPS vs MNISTMNIST vs USPSAverageTotal average1NN23.7025.7625.4826.0029.8325.4819.8622.9659.2426.2728.5063.3931.3744.7065.9455.3234.79SVM52.4040.6842.0444.0841.6940.7629.9234.1387.9031.4332.3673.5645.9135.1054.6944.945.77ELM50.6342.7142.0443.0137.6342.6832.2436.8580.2530.1031.5271.8645.1330.7051.3941.0544.54SSELM52.9245.4238.8541.7635.9338.8534.2837.0678.9830.3732.0580.0045.5429.0543.2836.1644.2TCA155.8548.8151.5944.1740.3436.9432.3238.4181.5336.2439.3582.3748.9935.5553.3444.4548.34TCA255.0148.4849.6844.3541.0243.3132.5938.9479.6234.6438.3183.8949.1534.7052.7843.7448.38JDA153.3446.7852.8742.6536.2740.7633.6647.6086.8936.5143.3182.7347.3334.7754.9244.8549.5JDA254.8051.8651.2241.2339.6644.5935.0945.5180.2633.3940.9283.0550.1333.8653.1143.4949.18DAELM_S50.2645.4237.5842.8838.3136.9434.2036.4480.8932.3234.3572.8845.2144.3647.1145.7345.28DAELM_T27.8531.8622.9325.2519.3236.3141.9046.4953.5048.0851.2054.2438.2431.0860.3245.739.31ARRLS54.9150.5144.5942.4838.9838.8534.7339.8778.3432.2437.1682.7147.9534.6452.4843.5647.32JPDA47.6045.7646.540.7840.6836.9434.5533.8288.5434.7334.6691.1947.9859.268.9464.0750.28AELM52.4043.7347.7740.1634.9235.6730.5437.1686.6228.6729.5483.7345.9141.6562.8352.2446.81DST-ELM (mar)51.8846.144.5943.7244.4140.1333.9336.2274.5232.0634.4577.9746.6743.959.8351.8747.41DST-ELM (con)53.6551.5349.0437.4939.6650.9635.9842.0782.1729.4744.0584.0750.0150.8573.3962.1251.74CDELM-M52.0751.0545.8642.3342.8545.8630.3139.8381.1530.3135.7281.7648.2646.5362.4754.549.15CDELM-C56.2850.9843.8243.1740.7541.7834.4439.5883.0634.4439.5484.3449.3545.8942.5744.2348.62TELM-OWA31.7332.2033.1238.8538.6443.3133.4838.0177.0732.5036.2378.6442.8259.5559.3759.4645.19TSTELM56.2650.4749.142.2744.7547.2235.2639.2478.3435.4440.2983.3850.1758.5569.4462.9652.86The bold values in Table3 is best result in its column.Table 4
Accuracy of different algorithms on MSRC + VOC2007 and Reuters-21578 datasets.
Methods\datasetNon-transfer learning algorithmTransfer learning algorithm1NNSVMELMSSELMTCA1TCA2JDA1JDA2DAELM_SDAELM_TARRLSTELM-OWATSTELMMSRC vs VOC35.9535.1038.5637.5227.9134.9027.1234.7738.3433.6034.6441.3736.47VOC vs MSRC45.2354.6958.0063.2047.9154.2248.3154.6158.3730.6150.3566.2275.23Average40.5944.9048.2850.3637.9144.5637.7144.6948.3532.1042.5053.7955.85Orgs vs people (1)72.8575.2577.2480.3076.4977.5180.1381.7976.6448.7180.5563.7483.27People vs orgs (2)72.0377.1279.3084.7277.2879.3985.1385.2978.4147.7284.5664.2785.69Orgs vs place (3)67.5070.1863.5769.6172.3971.8175.7476.2268.5343.8172.2068.0973.92Place vs orgs (4)61.1263.7764.5771.1668.2165.7576.3876.0868.7742.5074.1174.6873.52People vs place (5)52.6560.6357.0166.0261.1960.6366.8565.8359.1442.4665.1859.0461.74Place vs people (6)53.3957.9458.6861.6556.0851.0758.1251.1658.7739.9358.5660.4560.81Average63.2667.4866.7372.2468.6167.6973.7372.7368.3844.1972.5365.0573.16Total average57.5961.8462.1266.7760.9361.9164.7265.7263.3741.1765.0262.2368.83The bold values in Table3 is best result in its row.Figure 6
Classification accuracy of different algorithms on (a) USPS + MNIST and (b) MSRC + VOC2007 dataset.
(a)(b)Figure 7
Classification accuracy of different algorithms on Office + Caltech256 dataset.Figure 8
Classification accuracy of different algorithms on Reuters-21578 dataset.We check the execution times of some methods on MNIST vs USPS, and the results are reported in Table5. It can be seen: (1) The speed of the methods based on ELM are significantly faster than other methods, and ELM is the fastest. (2) TSTELM consumes more time than TELM-OWA, ELM, SSELM, DAELM_S, and DAELM_T, because of label refinement iterative process. TELM-OWA is more time-consuming than ELM, SSELM, DAELM_S, and DAELM_T as a result of solving βS∗ and βT∗. (3) Since constructing the Laplacian matrix is the most time-consuming, SSELM is relatively inefficient. (4) TCA(1,2) and JDA(1,2) cost more time than 1NN and SVM because of additional feature extraction process. (5) JDA(1,2) has the highest time cost because it applies an iterative manner to refine the target pseudo label and extract cross-domain shared feature.Table 5
Average runtime of all the methods on MNIST vs USPS.
Algorithm1NNSVMELMSSELMTCA1TCA2JDA1JDA2DAELM_SDAELM_TARRLSTELM-OWATSTELMTime (s)0.558.790.373.494.725.4748.3253.60.810.642.083.736.5


### 4.4. Parameter Analysis

To evaluate the effects of scale factor (p); number of hidden layer nodes(L); parameter α, λ, and θ on TSTELM, we conduct some experiments on org vs people, MSRC vs VOC, MNIST vs USPS, and A vs D. The results are shown in Figures 9(a)–9(f). It can be seen that: (1) With the increase of p, the trend in TSTELM accuracy goes up first and then goes down on all test datasets and achieves optimal results when p∈0.1,1, as shown in Figure 9(a). It can be known that the results of joint decision of β1 and β2 are better than their separate decisions. (2) As shown in Figure 9(b), TSTELM accuracy increases first and then decreases with the number of L on all test datasets. Although a large network forces the ELM network to behave better on output function approximation, time cost of the algorithm and required memory become large, and too many hidden nodes will hurt the ELM performance of domain adaptation because of better output function approximation. (3) In Figures 9(c)–9(e), with the gradual increase of parameter α, λ, and θ, the accuracy increases first and then decreases and takes different optimal values on different test datasets to achieve the optimal accuracy, which indicates that the control term of these parameters are beneficial to TSTELM when the parameter values are reasonable. (4) We also provide the classification accuracy varying with the iteration number, and result is shown in Figure 9(f). It shows that the accuracy is increasing iterative with the number of iterations and finally converges after several iterations, which verifies that TSTELM has strong robustness.Figure 9
Classification accuracy of TSTELM with respect to scale factor (p); number of hidden layer nodes (L); parameter α, λ, and θ; and iteration.
(a)(b)(c)(d)(e)(f)


## 4.1. Dataset Description

aOffice + Caltech256 (shown in Figure 3): Office is widely used dataset for visual cross-domain learning, which contains 4,652 images in 31 categories. These images come from 3 realistic aggregated item datasets: Amazon (images download from online chants https://www.amazon.com); DSLR (high-resolution images by a digital SLR camera in realistic environments); and Webcam (low-resolutions images by a simple webcam). Caltech256 is also a standard object recognition dataset which contains 30,607 images from 256 categories.Figure 3
Image samples from (b) Amazon, (d) Caltech256, (a) DSLR, and (c) Webcam.
(a)(b)(c)(d)In this article, we employ the Office + Caltech dataset released by Gong et al. [57]. SURF features are extracted and quantized into an 800-bin histogram with codebooks computed with K-means on a subset of images from Amazon. Then, the histograms are standardized by z-score. We select four domains C (Caltech256), A (Amazon), W (Webcam), and D (DSLR) for experiment, and two different domains are randomly selected as the source and the target domain datasets, and 12 cross-domain tasks for evaluation are constructed, namely C ⟶ A, C ⟶ W, C ⟶ D, …, and D ⟶ W.USPS + MNIST (as shown in Figure 4): USPS and MNIST are the two different but related handwritten datasets with 10 categories of 0–9. The USPS dataset contains 7,291 training samples and 2,007 test samples with 16 × 16 pixels. There are 60,000 training images and 10,000 test images with 28 × 28 pixels in the MNIST database.Figure 4
Image samples from (a) MNIST and (b) USPS.
(a)(b)During this experiment, we randomly select 1,800 pictures from USPS and 2,000 pictures from MNIST and convert them into 16 × 16 pixels. We construct two cross-domain tasks, that is USPS as the source domain and MNIST as the target domain (USPS vs MNIST) and vice versa (MNIST vs USPS).MSRC + VOC2007 (shown in Figure 5): MSRC dataset is provided by Microsoft Cambridge, which consists of 4323 images of 18 object classes. VOC2007 dataset contains 5,011 images annotated with 20 concepts. We can see from Figure 5 that MSRC and VOC2007 have distinct but different distributions. MSRC are standard images as benchmark data for evaluation. VOC2007 is randomly constructed by using the images in the network album.Figure 5
Image samples from (a) MSRC and (b) VOC2007.
(a)(b)In our experiments, we construct the domain adaptation dataset MSRC vs VOC in which 6 shared categories are selected including aircraft, birds, cows, family cars, sheep and bicycles. Among them, 1269 images are selected from the MSRC dataset as the source domain dataset and 1530 images are selected from the VOC2007 dataset as the target domain dataset. Then, the source domain and the target domain are exchanged to construct a new set of domain adaptation dataset VOC vs MSRC. We convert all images into 256 gray pixels; 240 dimensions are extracted as the spatial dimension of the sample.Reuters-21578: Reuters-21578 text dataset is a common dataset for text classification. It contains 21,577 news documents from Reuters in 1987. These documents have been manually labeled by Reuters as five classes, such as “exchanges,” “orgs,” “people,” “places,” and “topics,” including multiple categories and subclasses. Among them, the largest three categories are “orgs,” “people,” and “place,” which can construct six cross-domain text classification tasks orgs vs people, people vs orgs, orgs vs place, place vs orgs, people vs place, and place vs people. This article makes a more complete evaluation of the algorithm on 6 classification tasks.


## 4.2. Experimental Settings

To validate the efficiency of TSTELM, we compare it with some other classifiers.Classifiers for non-domain adaptation: 1NN, SVM, ELM, and SSELM [58] (ELM with graph regularization term for semi-supervised learning).Classifiers for domain adaptation: TCA1 [47](TCA with 1NN for classification), TCA2 [47] (TCA with SVM for classification), JDA1 [48] (JDA with 1NN for classification), JDA2 [48](JDA with SVM for classification), DAELM-S [36], DAELM-T [36], ARRLS [59], TELM-OWA [38], JPDA [60], AELM [61], DST-ELM [40], CDELM [39], and TSTELM.In order to achieve the optimal performance of each algorithm in the experiment, we set SVM penalty parameter belonging to0.1,0.5,1,5,10,50,100, and the penalty parameter θ∈0.001,0.1 in ELM, SSELM, DAELM_S, DAELM_T, TELM-OWA, and TSTELM. TCA(1,2) and JDA(1,2) are combined feature extraction with standard classifier for domain adaptation, in which the dimension of the feature subspace is 100 and the value range of the balanced-constraint parameter of the projection matrix in TCA and JDA is 0.1,1. The parameters of ARRLS, DAELM_S, DAELM_T, and TELM-OWA are set according to the corresponding literature. In TSTELM, we set α∈103,104, λ∈10−3,10−4,and L=1500 on Office + Caltech dataset, L=3000 on USPS + MNIST and MSRC + VOC2007 datasets, L=5000 on Reuters-21578 dataset. We cite results of JPDA, AELM, DST-ELM, and CDELM from those corresponding literature.In fact, DAELM_S, DAELM_T, and TELM-OWA, as supervised models, need a few labeled target samples to induce the target classifier. We test them with 0.5% labeled target samples on USPS + MNIST, MSRC + VOC2007, and Reuters-21578 datasets and 1% labeled target samples on Office + Caltech dataset, to approximate the performance of these methods in unsupervised domain adaptation.


## 4.3. Experimental Results and Analysis

We test TSTELM on Office + Caltech256, USPS + MNIST, MSRC + VOC2007, and Reuters-21578 datasets, and the comparison results are displayed in Tables3 and 4 and Figures 4–8, in which the best results of each task are bold. From the comparison of results, we have the following observations.(1)
The total average accuracy of TSTELM across 22 tasks is the highest among all methods, indicating that our approach can effectively deal with domain adaptation problem.(2)
In unsupervised domain adaptation, the proposed method outperforms AELM, DST-ELM, CDELM, TELM-OWA, DAELM_S, and DAELM_T, indicating the superiority of TSTELM in which MMD and output weight alignment are unified into the ELM learning framework to minimize distribution discrepancy between domains. AELM, DAELM_S, and DAELM_T obtain poor results, showing that they are highly dependent on labeled target samples. 1NN, SVM, and ELM perform unsuccessfully because of the problem of domain shift. SSELM performs better than ELM due to that the original geometry information of data is mined.(3)
TCA(1,2), JDA(1,2), and JPDA are better than 1NN and SVM, showing the importance of shared feature extraction across domains. JPDA and JDA(1,2) are generally higher than TCA(1,2), which indicates the superiority of simultaneously reducing the marginal and conditional distribution discrepancy. ARRLS performs well because of MMD and preserves the manifold consistency at the same time.Table 3
Accuracy of different algorithms on Office + Caltech256 and USPS + MNIST datasets.
MethodsDatasetC ⟶ A(1)C ⟶ W(2)C ⟶ D(3)A ⟶ C(4)A ⟶ W(5)A ⟶ D (6)W ⟶ C (7)W ⟶ A (8)W ⟶ D(9)D ⟶ C(10)D ⟶ A(11)D ⟶ W(12)AverageUSPS vs MNISTMNIST vs USPSAverageTotal average1NN23.7025.7625.4826.0029.8325.4819.8622.9659.2426.2728.5063.3931.3744.7065.9455.3234.79SVM52.4040.6842.0444.0841.6940.7629.9234.1387.9031.4332.3673.5645.9135.1054.6944.945.77ELM50.6342.7142.0443.0137.6342.6832.2436.8580.2530.1031.5271.8645.1330.7051.3941.0544.54SSELM52.9245.4238.8541.7635.9338.8534.2837.0678.9830.3732.0580.0045.5429.0543.2836.1644.2TCA155.8548.8151.5944.1740.3436.9432.3238.4181.5336.2439.3582.3748.9935.5553.3444.4548.34TCA255.0148.4849.6844.3541.0243.3132.5938.9479.6234.6438.3183.8949.1534.7052.7843.7448.38JDA153.3446.7852.8742.6536.2740.7633.6647.6086.8936.5143.3182.7347.3334.7754.9244.8549.5JDA254.8051.8651.2241.2339.6644.5935.0945.5180.2633.3940.9283.0550.1333.8653.1143.4949.18DAELM_S50.2645.4237.5842.8838.3136.9434.2036.4480.8932.3234.3572.8845.2144.3647.1145.7345.28DAELM_T27.8531.8622.9325.2519.3236.3141.9046.4953.5048.0851.2054.2438.2431.0860.3245.739.31ARRLS54.9150.5144.5942.4838.9838.8534.7339.8778.3432.2437.1682.7147.9534.6452.4843.5647.32JPDA47.6045.7646.540.7840.6836.9434.5533.8288.5434.7334.6691.1947.9859.268.9464.0750.28AELM52.4043.7347.7740.1634.9235.6730.5437.1686.6228.6729.5483.7345.9141.6562.8352.2446.81DST-ELM (mar)51.8846.144.5943.7244.4140.1333.9336.2274.5232.0634.4577.9746.6743.959.8351.8747.41DST-ELM (con)53.6551.5349.0437.4939.6650.9635.9842.0782.1729.4744.0584.0750.0150.8573.3962.1251.74CDELM-M52.0751.0545.8642.3342.8545.8630.3139.8381.1530.3135.7281.7648.2646.5362.4754.549.15CDELM-C56.2850.9843.8243.1740.7541.7834.4439.5883.0634.4439.5484.3449.3545.8942.5744.2348.62TELM-OWA31.7332.2033.1238.8538.6443.3133.4838.0177.0732.5036.2378.6442.8259.5559.3759.4645.19TSTELM56.2650.4749.142.2744.7547.2235.2639.2478.3435.4440.2983.3850.1758.5569.4462.9652.86The bold values in Table3 is best result in its column.Table 4
Accuracy of different algorithms on MSRC + VOC2007 and Reuters-21578 datasets.
Methods\datasetNon-transfer learning algorithmTransfer learning algorithm1NNSVMELMSSELMTCA1TCA2JDA1JDA2DAELM_SDAELM_TARRLSTELM-OWATSTELMMSRC vs VOC35.9535.1038.5637.5227.9134.9027.1234.7738.3433.6034.6441.3736.47VOC vs MSRC45.2354.6958.0063.2047.9154.2248.3154.6158.3730.6150.3566.2275.23Average40.5944.9048.2850.3637.9144.5637.7144.6948.3532.1042.5053.7955.85Orgs vs people (1)72.8575.2577.2480.3076.4977.5180.1381.7976.6448.7180.5563.7483.27People vs orgs (2)72.0377.1279.3084.7277.2879.3985.1385.2978.4147.7284.5664.2785.69Orgs vs place (3)67.5070.1863.5769.6172.3971.8175.7476.2268.5343.8172.2068.0973.92Place vs orgs (4)61.1263.7764.5771.1668.2165.7576.3876.0868.7742.5074.1174.6873.52People vs place (5)52.6560.6357.0166.0261.1960.6366.8565.8359.1442.4665.1859.0461.74Place vs people (6)53.3957.9458.6861.6556.0851.0758.1251.1658.7739.9358.5660.4560.81Average63.2667.4866.7372.2468.6167.6973.7372.7368.3844.1972.5365.0573.16Total average57.5961.8462.1266.7760.9361.9164.7265.7263.3741.1765.0262.2368.83The bold values in Table3 is best result in its row.Figure 6
Classification accuracy of different algorithms on (a) USPS + MNIST and (b) MSRC + VOC2007 dataset.
(a)(b)Figure 7
Classification accuracy of different algorithms on Office + Caltech256 dataset.Figure 8
Classification accuracy of different algorithms on Reuters-21578 dataset.We check the execution times of some methods on MNIST vs USPS, and the results are reported in Table5. It can be seen: (1) The speed of the methods based on ELM are significantly faster than other methods, and ELM is the fastest. (2) TSTELM consumes more time than TELM-OWA, ELM, SSELM, DAELM_S, and DAELM_T, because of label refinement iterative process. TELM-OWA is more time-consuming than ELM, SSELM, DAELM_S, and DAELM_T as a result of solving βS∗ and βT∗. (3) Since constructing the Laplacian matrix is the most time-consuming, SSELM is relatively inefficient. (4) TCA(1,2) and JDA(1,2) cost more time than 1NN and SVM because of additional feature extraction process. (5) JDA(1,2) has the highest time cost because it applies an iterative manner to refine the target pseudo label and extract cross-domain shared feature.Table 5
Average runtime of all the methods on MNIST vs USPS.
Algorithm1NNSVMELMSSELMTCA1TCA2JDA1JDA2DAELM_SDAELM_TARRLSTELM-OWATSTELMTime (s)0.558.790.373.494.725.4748.3253.60.810.642.083.736.5


## 4.4. Parameter Analysis

To evaluate the effects of scale factor (p); number of hidden layer nodes(L); parameter α, λ, and θ on TSTELM, we conduct some experiments on org vs people, MSRC vs VOC, MNIST vs USPS, and A vs D. The results are shown in Figures 9(a)–9(f). It can be seen that: (1) With the increase of p, the trend in TSTELM accuracy goes up first and then goes down on all test datasets and achieves optimal results when p∈0.1,1, as shown in Figure 9(a). It can be known that the results of joint decision of β1 and β2 are better than their separate decisions. (2) As shown in Figure 9(b), TSTELM accuracy increases first and then decreases with the number of L on all test datasets. Although a large network forces the ELM network to behave better on output function approximation, time cost of the algorithm and required memory become large, and too many hidden nodes will hurt the ELM performance of domain adaptation because of better output function approximation. (3) In Figures 9(c)–9(e), with the gradual increase of parameter α, λ, and θ, the accuracy increases first and then decreases and takes different optimal values on different test datasets to achieve the optimal accuracy, which indicates that the control term of these parameters are beneficial to TSTELM when the parameter values are reasonable. (4) We also provide the classification accuracy varying with the iteration number, and result is shown in Figure 9(f). It shows that the accuracy is increasing iterative with the number of iterations and finally converges after several iterations, which verifies that TSTELM has strong robustness.Figure 9
Classification accuracy of TSTELM with respect to scale factor (p); number of hidden layer nodes (L); parameter α, λ, and θ; and iteration.
(a)(b)(c)(d)(e)(f)


## 5. Conclusion

To handle the problem that traditional ELM does not perform well in unsupervised domain adaptation, we in this article propose TSTELM including two domain adaptation stages. At the statistical matching stage, MMD is introduced into ELM learning frame to simultaneously minimize the marginal and conditional distribution between domains. At the subspace alignment stage, subspace alignment strategy, cross-domain mean approximation, and output weight approximation are adopted to further adjust the distribution consistency between domains. Finally, parameters of learned ELM models at two stages are fused and used to predict test samples. Extensive experiments have been conducted on real-world image and text datasets, and the results show that TSTELM has higher accuracy and better generalization performance. In the future, we will make further research that TSTELM is improved by stacking it into deep structure model for extracting deep feature.



---

*Source: 1582624-2022-07-18.xml*