# Joint Transfer Extreme Learning Machine with Cross-Domain Mean Approximation and Output Weight Alignment

Authors: Shaofei Zang; Dongqing Li; Chao Ma; Jianwei Ma
Journal: Complexity
 (2023)

---

## Abstract

With fast learning speed and high accuracy, extreme learning machine (ELM) has achieved great success in pattern recognition and machine learning. Unfortunately, it will fail in the circumstance where plenty of labeled samples for training model are insufficient. The labeled samples are difficult to obtain due to their high cost. In this paper, we solve this problem with transfer learning and propose joint transfer extreme learning machine (JTELM). First, it applies cross-domain mean approximation (CDMA) to minimize the discrepancy between domains, thus obtaining one ELM model. Second, subspace alignment (sa) and weight approximation are together introduced into the output layer to enhance the capability of knowledge transfer and learn another ELM model. Third, the prediction of test samples is dominated by the two learned ELM models. Finally, a series of experiments are carried out to investigate the performance of JTELM, and the results show that it achieves efficiently the task of transfer learning and performs better than the traditional ELM and other transfer or nontransfer learning methods.


---

## Body

## 1. Introduction

The fast development of mobile Internet, the Internet of things, and high-performance computers causes a large amount of data to emerge. How to mine the information of these data to help people make decisions has become a challenge. Machine learning uses numerous labeled data to train a statistical model for automatic prediction and this has become a hot topic in artificial intelligence (AI). As a high-performance model in machine learning, ELM has achieved success in pattern recognition, computation science, and machine vision. It includes the following two merits [1–4]: fast learning speed and outstanding generalization performance. There is no need for ELM to tune input weight and bias, and what it only needs is to optimize the output weight by solving a least square problem. Therefore, it has been widely recognized for classification and regression in various fields including industry fault diagnosis [5, 6], medical diagnosis [7], hyperspectral imagery classification [8, 9], facial expression recognition [10], and brain-computer interface [11, 12]. However, like the traditional machine learning model, ELM performs less satisfyingly when the training samples are insufficient.Transfer learning (TL) can handle this problem, in which the account of labeled samples (data) from other domains (source domain) related to the current domain (target domain) are adopted to train an efficient model for helping target tasks [13–15]. TL can not only reduce the cost of collecting training samples for data reuse but also enhance the generalization performance of the model. It is an expression of advanced intelligence. We commonly divide TL into three parts [14, 16], namely, instance-based transfer [17–19], feature-based transfer [20–23], and classifier (or parameter)-based transfer [24–26]. Moreover, with the success of deep learning and adversarial network in computer version and machine learning, some deep transfer learning [27] and transfer adversarial learning approaches [28] appear to further enrich the transfer learning in theory and application.TL could help ELM to solve the shortage of available training samples, and many variant ELMs with the ability of knowledge transfer have appeared. Depending on how to adapt between domains, we divide the transfer ELM (TELM) into the following three types. (1) The target supervised method: It usually requires a few of the labeled samples from the target domain to adjust the model training on the source domain. Domain adaptation extreme learning machine (DAELM) [29] was put forward to enhance ELM to handle the domain adaptation problems in the E-nose system. Online domain adaptation extreme learning machine (ODAELM) [30] and online weighted domain transfer extreme learning machine (OWDTELM) [31] extend DAELM to the online task. To further improve DAELM, Xia et al. [32] proposed the boosting for DAELM (BDAELM) which introduces boosting technology to ensemble DAELMs. (2) Parameter transformation or approximation: This method realizes the knowledge transferring across domains by a transform matrix or output weight approximation, such as transfer extreme learning machine with output weight alignment (TELM-OWA) [33], parameter transfer ELM (PTELM) [34], and extreme learning machine (ELM)-based domain adaptation (EDA) [35]. Li et al. [36] designed transfer learning based on the ELM algorithm (TL-ELM) by adding a constraint which forces the output weights of the two domains to be close to each other. (3) Statistical adaptation: It usually introduces a statistical distribution metric, such as MMD [37], into ELM to reduce the domain shift. Many methods including cross-domain extreme learning machines (CdELMs) [38], extreme learning machine based on maximum weighted mean discrepancy (ELM-MWMD) [39], and domain space transfer ELM (DST-ELM) [40] are applied MMD to reduce the distribution discrepancy of the output data in hidden layer from source and target domains.In this paper, we propose a novel ELM called joint transfer extreme learning machine (JTELM) for transfer learning. It first obtains one ELM model by introducing cross-domain mean approximation (CDMA) [41] into ELM, in which CDMA could effectively minimize the marginal and conditional distribution differences between the two domains. Second, we apply subspace alignment technology [42] to align output weights of two domains and to simultaneously add the approximation term to force the output weights to be close to each other, which could boost knowledge transfer. Then, we can obtain the other ELM model. Finally, the target samples are tested by two learned ELMs. JTELM is illustrated in Figure 1. We carry out some experiments on public datasets for transfer learning tasks to estimate the performance of JTELM, and the result demonstrates the superiority of our method.Figure 1
An illustration of JTELM. (1) CDMA is combined with ELM to minimize the distribution discrepancy of output data of hidden layer from two domains. (2) Output weight alignment and approximation could enhance the ability of knowledge transfer of JTELM. (3) Joint decisions of two ELMs improve the robustness of JTELM.We summarize our contributions as follows:(1)
CDMA measure is added to the objective function of ELM to reduce the distribution discrepancy of the output of hidden layers in the source and target domains, which could obtain one transfer ELM model.(2)
We apply output weight alignment and the approximation of the output weights from the two domains to improve the efficiency of knowledge transfer and simultaneously to get the other transfer ELM.(3)
We use the two obtained transfer ELMs to jointly predict test samples, which enhances the robustness of JTELM. To estimate the performance of our approach, we conduct classification experiments on object recognition and text datasets, and the result demonstrates that JTELM has a remarkable knowledge transfer ability.We organize the rest of the sections of this paper as follows. ELM, CDMA, and SA are briefly described in Section2. JTELM is described in detail in Section 3. Then,, the experiment is analyzed in Section 4 and the conclusion of this paper is presented in Section 5.


## 2. Related Work

In this section, we briefly introduce ELM, CDMA, and SA.


### 2.1. Extreme Learning Machine (ELM)

ELM, as a single-hidden-layer forward-feedback network, randomly initializes the input weight and bias and then solves the optimal output weight, which leads to its fast learning speed and high accuracy. If a labeled datasetxi,yii=1N with N samples of xi and a correspondent label yi is given, then we can construct a classic ELM model with L nodes in a hidden layer in the following manner:(1)oiT=∑j=1LβjTgwjxi+bj,where oi is the output of ELM according to the input samples xi, wj, and bj and these are the input weights and bias which are often randomly initialized. β is a vector representation of the output weight. If we want an optimal β, then the following loss function is solved:(2)minβ12β2+λ2∑i=1Noi−yi2,where β2 is a parameter sparse constraint avoiding model overfitting. ∑i=1Noi−yi2 is the classification error and λ is its tradeoff parameter. We then convert equation (2) into the following matrix form:(3)minβLELM=12β2+λ2Y−Hβ2,where H=gxiT,…,gxNTT, β=β1⋮βL, and Y=y1⋮yN.According to [2], we get the optimal β as(4)β∗=HTH+ILλHTY,N>L,HTHTH+INλY,N≤L.Finally, we predict the testing samplexTe as(5)yTe=signhTeTβ∗,forbinaryclassification,argmaxhTeTβ∗formulticlassification,where hTe=gxTe.


### 2.2. Cross-Domain Mean Approximation (CDMA)

The distribution discrepancy measure is very critical in transfer learning. Zang et al. [41] presented CDMA that is nonparametric, easy to understand, efficient, and beneficial for mining local information. In transfer learning, there are two datasets: DS=xSi,ySii=1nS from the source domain and DT=xTj,yTjj=1nT from the target domain, where nSnT is the number of xSixTj in DSDT and ySiyTj belonging to C classes which is the label of xSixTj. Then, we can get the CDMA measure as(6)LCDMADS,DT=∑i=1nS+nTxi−μ22,where μ=μT,ifxi∈DSμSifxi∈DT and μTS is the mean vector of the target (source) domain sample. If we further consider the label information of the samples, CDMA is also represented as(7)LCDMADS,DT=∑i=1nS+nTxi−μ22+∑c=1C∑i=1nSic+nTicxic−μic22,where μc=μTc,ifxic∈DScμScifxic∈DTc and μTSc is the mean vector of the target (source) domain DTSc with c category.


### 2.3. Subspace Alignment (SA)

In transfer learning, especially feature transfer, SA usually aligns two feature subspaces from the source and target domains obtained by other feature extraction methods, and realizes the distribution consistency of the two domains. If we have learned the two subspace transform matrixesAS and AT, then, a transformation matrix M is obtained to solve the following function:(8)fM=minASM−ATF2,where •F2 is the Frobenius norm. We add the orthogonalization operation into equation (8) and get(9)fM=minASTASM−ASTΑTF2=minM−ASTΑTF2.From equation (9), we can see that M∗=ASTΑT. We set Α=ASM=ASASTΑT and from this it is clear that the sample distribution in Α subspace is more similar to the one in ΑT than in AS, which facilitates knowledge transfer.


## 2.1. Extreme Learning Machine (ELM)

ELM, as a single-hidden-layer forward-feedback network, randomly initializes the input weight and bias and then solves the optimal output weight, which leads to its fast learning speed and high accuracy. If a labeled datasetxi,yii=1N with N samples of xi and a correspondent label yi is given, then we can construct a classic ELM model with L nodes in a hidden layer in the following manner:(1)oiT=∑j=1LβjTgwjxi+bj,where oi is the output of ELM according to the input samples xi, wj, and bj and these are the input weights and bias which are often randomly initialized. β is a vector representation of the output weight. If we want an optimal β, then the following loss function is solved:(2)minβ12β2+λ2∑i=1Noi−yi2,where β2 is a parameter sparse constraint avoiding model overfitting. ∑i=1Noi−yi2 is the classification error and λ is its tradeoff parameter. We then convert equation (2) into the following matrix form:(3)minβLELM=12β2+λ2Y−Hβ2,where H=gxiT,…,gxNTT, β=β1⋮βL, and Y=y1⋮yN.According to [2], we get the optimal β as(4)β∗=HTH+ILλHTY,N>L,HTHTH+INλY,N≤L.Finally, we predict the testing samplexTe as(5)yTe=signhTeTβ∗,forbinaryclassification,argmaxhTeTβ∗formulticlassification,where hTe=gxTe.


## 2.2. Cross-Domain Mean Approximation (CDMA)

The distribution discrepancy measure is very critical in transfer learning. Zang et al. [41] presented CDMA that is nonparametric, easy to understand, efficient, and beneficial for mining local information. In transfer learning, there are two datasets: DS=xSi,ySii=1nS from the source domain and DT=xTj,yTjj=1nT from the target domain, where nSnT is the number of xSixTj in DSDT and ySiyTj belonging to C classes which is the label of xSixTj. Then, we can get the CDMA measure as(6)LCDMADS,DT=∑i=1nS+nTxi−μ22,where μ=μT,ifxi∈DSμSifxi∈DT and μTS is the mean vector of the target (source) domain sample. If we further consider the label information of the samples, CDMA is also represented as(7)LCDMADS,DT=∑i=1nS+nTxi−μ22+∑c=1C∑i=1nSic+nTicxic−μic22,where μc=μTc,ifxic∈DScμScifxic∈DTc and μTSc is the mean vector of the target (source) domain DTSc with c category.


## 2.3. Subspace Alignment (SA)

In transfer learning, especially feature transfer, SA usually aligns two feature subspaces from the source and target domains obtained by other feature extraction methods, and realizes the distribution consistency of the two domains. If we have learned the two subspace transform matrixesAS and AT, then, a transformation matrix M is obtained to solve the following function:(8)fM=minASM−ATF2,where •F2 is the Frobenius norm. We add the orthogonalization operation into equation (8) and get(9)fM=minASTASM−ASTΑTF2=minM−ASTΑTF2.From equation (9), we can see that M∗=ASTΑT. We set Α=ASM=ASASTΑT and from this it is clear that the sample distribution in Α subspace is more similar to the one in ΑT than in AS, which facilitates knowledge transfer.


## 3. Joint Transfer Extreme Learning Machine (JTELM)

In response to the shortcoming of ELM with no ability of knowledge transfer, we propose a novel transfer ELM abbreviated as JTELM for handling unsupervised transfer learning tasks in which no labeled target samples appear. In unsupervised transfer learning, the source domainDS and DT are given but yTj disappears in DT, so we expect that JTELM learned from DS to precisely predict the samples in DT


### 3.1. Extreme Learning Machine with CDMA

To equip ELM with the ability of knowledge transfer, we first usegx to map DS and DT into HS and HT, and then construct ELM with CDMA by introducing CDMA into the loss function of ELM. CDMA minimizes the distribution difference of data in output layer from the source and target domains. Then, we get the loss function of ELM with CDMA as follows:(10)LELM−CDMA=minβS:12βS2+λ2HβS−Y2+α12HβS−HavβS2+∑c=1CHcβS−HavcβS.In equation (10), the first two items are the loss of ELM, and the third item is the loss of CDMA in the output layer. α1 is a tradeoff parameter between two losses. H=HSHT,Y=YSY¯T, Hav=HT_avHS_av, Havc=HT_avcHS_avc, Y¯T is the pseudo label of DT in the label refinement process. Hc represents samples with c category in HS and HT. HS_av, HT_av is the mean vector of HS and HT, respectively. HTS_avc is the mean vector of the target (source) domain HTSc with c category.We can obtain one ELM with knowledge transfer ability by using∂LELM−CDMA/∂βS=0 according to [2] as follows:(11)βS=β1∗=HTH+α1H−HavTH−Hav+∑c=1CHc−HavcTHc−Havc+ILλHTY,nS+nT>L,HTHTH+α1H−HavH−HavT+∑c=1CHc−HavcHc−HavcT+INλY,nS+nT≤L.


### 3.2. Extreme Learning Machine with Output Weight Alignment and Approximation

Suppose that there is weightβT in the target domain, then we can construct a loss function as follows:(12)JβT=minβT:12βT2+α22HSβS−YS2+γ2βS−βT2,where HSβS−YS2 denotes the classification error in the source domain, βS−βT2 denotes output weight approximation to force βS close to βT for the facilitation of knowledge transfer, and α2 and γ are the balance parameters.As a next step, we apply SA to align the output layer of the source ELM to target ones. First, we obtain a transform matrixM∗=βSTβT, and set βtemp=βSM=βSβSTβT and then replace βS with βtemp and substitute it into equation (12) to get(13)JβT=minβT12βT2+α22HSβtemp−YS2+γ2βtemp−βT2.At this moment,HSβtemp−YS change into the source classification error under the output weight alignment. We substitute βtemp=βSβSTβT to equation (13) and get(14)JβT=minβT12βT2+α22HSβSβSTβT−YS2+γ2βT−βSβSTβT2=minβT12βT2+α22HSβSβSTβT−YS2+γ2I−βSβSTβT2.BecauseI−βSβSTβT2≤I−βSβST2βT2, equation (14) can change to(15)JβT=minβT12βT2+α22HSβSβSTβT−YS2+γ2I−βSβST2βT2=minβTα22HSβSβSTβT−YS2+I+γI−βSβSTTI−βSβST2βT2.We setA=I+γI−βSβSTTI−βSβST and Htemp=HSβ1β1T, then equation (15) can be simplified as(16)JβT=minβT:α22HtempβT−YS2+A2βT2.Let∂JβT/∂βT=0, then we obtain(17)βΤ=β2∗=HtempTHtemp+Aα2−1HtempTYS,nS+nT>L,IinAisanL−dimensionalunitmatrix,HtempTHtempHtempT+Aα2−1YS,nS+nT≤L,IinAisann−dimensionalunitmatrix.


### 3.3. Joint Decision

Up to now, we have learned two transfer ELMs withβ1∗ and β2∗, but the target samples are predicted by β=β1∗+β2∗/2 according to equation (5). We summarize JTELM in Algorithm 1.Algorithm 1: JTELM.
Input: Dataset DS and DT, number of hidden-layer nodes L, tradeoff parameter α1, α2, λ, and γ.Output: Predicted result Y¯T.Step 1: Use DS and DT to calculate β1∗ according to equation (11).Step 2: Solve the output weight β2∗ according to equation (17);Step 3: Use β1∗ and β2∗ to predict DT and get its label Y¯T.Step 4: Repeat Steps 1–3 until no change on Y¯T.


### 3.4. Discussion

Inspired by TELM-OWA [33], we put forward JTELM to address the problem of unsupervised transfer learning. It has the following characteristics:(1)
Similar to TELM-OWA, output weight alignment (equation (8)) and weight approximation βS−βT2 are used to learn a transfer ELM parameter β2∗, but JTELM is an unsupervised TL method in which no labeled samples in the target domain exist. Therefore, JTELM has a higher difficulty and challenge.(2)
The authors in [41] have proved that CDMA is a more efficient distribution discrepancy metric than MMD. We apply it to ELM to add the transferring ability of knowledge from the source domain to the target domain. Thus, β1∗ in equation (11) is a parameter of the shared model between domains.(3)
JTELM utilizesβ1∗ and β2∗ to jointly make decisions for test samples, which not only unify statistical adaptation and parameter transformation into a learning framework to improve knowledge transfer, but also enhances the robustness of our approach similar to ensemble learning.


## 3.1. Extreme Learning Machine with CDMA

To equip ELM with the ability of knowledge transfer, we first usegx to map DS and DT into HS and HT, and then construct ELM with CDMA by introducing CDMA into the loss function of ELM. CDMA minimizes the distribution difference of data in output layer from the source and target domains. Then, we get the loss function of ELM with CDMA as follows:(10)LELM−CDMA=minβS:12βS2+λ2HβS−Y2+α12HβS−HavβS2+∑c=1CHcβS−HavcβS.In equation (10), the first two items are the loss of ELM, and the third item is the loss of CDMA in the output layer. α1 is a tradeoff parameter between two losses. H=HSHT,Y=YSY¯T, Hav=HT_avHS_av, Havc=HT_avcHS_avc, Y¯T is the pseudo label of DT in the label refinement process. Hc represents samples with c category in HS and HT. HS_av, HT_av is the mean vector of HS and HT, respectively. HTS_avc is the mean vector of the target (source) domain HTSc with c category.We can obtain one ELM with knowledge transfer ability by using∂LELM−CDMA/∂βS=0 according to [2] as follows:(11)βS=β1∗=HTH+α1H−HavTH−Hav+∑c=1CHc−HavcTHc−Havc+ILλHTY,nS+nT>L,HTHTH+α1H−HavH−HavT+∑c=1CHc−HavcHc−HavcT+INλY,nS+nT≤L.


## 3.2. Extreme Learning Machine with Output Weight Alignment and Approximation

Suppose that there is weightβT in the target domain, then we can construct a loss function as follows:(12)JβT=minβT:12βT2+α22HSβS−YS2+γ2βS−βT2,where HSβS−YS2 denotes the classification error in the source domain, βS−βT2 denotes output weight approximation to force βS close to βT for the facilitation of knowledge transfer, and α2 and γ are the balance parameters.As a next step, we apply SA to align the output layer of the source ELM to target ones. First, we obtain a transform matrixM∗=βSTβT, and set βtemp=βSM=βSβSTβT and then replace βS with βtemp and substitute it into equation (12) to get(13)JβT=minβT12βT2+α22HSβtemp−YS2+γ2βtemp−βT2.At this moment,HSβtemp−YS change into the source classification error under the output weight alignment. We substitute βtemp=βSβSTβT to equation (13) and get(14)JβT=minβT12βT2+α22HSβSβSTβT−YS2+γ2βT−βSβSTβT2=minβT12βT2+α22HSβSβSTβT−YS2+γ2I−βSβSTβT2.BecauseI−βSβSTβT2≤I−βSβST2βT2, equation (14) can change to(15)JβT=minβT12βT2+α22HSβSβSTβT−YS2+γ2I−βSβST2βT2=minβTα22HSβSβSTβT−YS2+I+γI−βSβSTTI−βSβST2βT2.We setA=I+γI−βSβSTTI−βSβST and Htemp=HSβ1β1T, then equation (15) can be simplified as(16)JβT=minβT:α22HtempβT−YS2+A2βT2.Let∂JβT/∂βT=0, then we obtain(17)βΤ=β2∗=HtempTHtemp+Aα2−1HtempTYS,nS+nT>L,IinAisanL−dimensionalunitmatrix,HtempTHtempHtempT+Aα2−1YS,nS+nT≤L,IinAisann−dimensionalunitmatrix.


## 3.3. Joint Decision

Up to now, we have learned two transfer ELMs withβ1∗ and β2∗, but the target samples are predicted by β=β1∗+β2∗/2 according to equation (5). We summarize JTELM in Algorithm 1.Algorithm 1: JTELM.
Input: Dataset DS and DT, number of hidden-layer nodes L, tradeoff parameter α1, α2, λ, and γ.Output: Predicted result Y¯T.Step 1: Use DS and DT to calculate β1∗ according to equation (11).Step 2: Solve the output weight β2∗ according to equation (17);Step 3: Use β1∗ and β2∗ to predict DT and get its label Y¯T.Step 4: Repeat Steps 1–3 until no change on Y¯T.


## 3.4. Discussion

Inspired by TELM-OWA [33], we put forward JTELM to address the problem of unsupervised transfer learning. It has the following characteristics:(1)
Similar to TELM-OWA, output weight alignment (equation (8)) and weight approximation βS−βT2 are used to learn a transfer ELM parameter β2∗, but JTELM is an unsupervised TL method in which no labeled samples in the target domain exist. Therefore, JTELM has a higher difficulty and challenge.(2)
The authors in [41] have proved that CDMA is a more efficient distribution discrepancy metric than MMD. We apply it to ELM to add the transferring ability of knowledge from the source domain to the target domain. Thus, β1∗ in equation (11) is a parameter of the shared model between domains.(3)
JTELM utilizesβ1∗ and β2∗ to jointly make decisions for test samples, which not only unify statistical adaptation and parameter transformation into a learning framework to improve knowledge transfer, but also enhances the robustness of our approach similar to ensemble learning.


## 4. Experiment and Analysis

In this section, we present the validity of our JTELM and perform some experiments on image and text datasets commonly used in transfer learning for the classification task. All experiments are run on a PC with 8 GB memory and Windows 10 operating system and MATLAB 2017b. Every experiment runs 20 times and the average value is recorded. We evaluate all algorithms in the experiment with an accuracy similar to [21].


### 4.1. Datasets Description

Office31 + Caltech256 (shown in Figure2): These datasets were first published in the year referred in [43]. It contains two domains, namely, Office31 and Caltech256. Office31 consists of 4,652 images in 31 categories. They have been collected from 3 subdomains, that is, Amazon (A), DSLR (D), and Webcam (W). Caltech (C) is also an object image dataset consisting of 30,607 images from 256 categories.Figure 2
Images from Office31+Caltech256.During the experiment, we select 1,410 images with 10 categories from office31and 1123 images with 10 categories from Caltech. Every picture is extracted using SURF features with 800 dimensions. Two subdomains in A, W,D, and C are randomly chosen as the source and target domain datasets, and 12 cross-domain tasks are built as C⟶A, C⟶W, C⟶D, …, and D⟶W (shown in Table 1).Table 1
Description of image and text datasets.
DatasetTypeSamplesDimensionClassContains subsetsUSPSDigit180025610USPSMNISTDigit200025610MNISTMSRCObject126924018MSRCVOC2007Object153024020VOCOfficeObject141080010A, W, DCaltechObject112380010CReuters-21578Text1,2374,771BinaryOrgsText1,2084,771BinaryPeopleText1,0164,771BinaryPlaceUSPS + MNIST (shown in Figure3): USPS and MNIST are the two image datasets describing numbers from 0 to 9, so they share 10 categories but have different distributions. USPS consists of 9,298 images with 16 × 16 pixels and MNIST has 70,000 images with 28 × 28 pixels. During the experiment, 1,800 pictures from USPS and 2,000 pictures from MNIST are selected as the source domain and the target domain (shown in Table 1). Every image is converted into 16 × 16 pixels and two cross-domain tasks, i.e., USPS vs. MNIST and MNIST vs. USPS are constructed for transfer learning tasks.Figure 3
Image from MNIST and USPS datasets.MSRC + VOC2007 (shown in Figure4): MSRC is an object image dataset consisting of 4,323 images from 18 categories. VOC2007 is an image dataset with photos in Flickr, consisting of 5,011 images from 18 categories. They have similar but different distributions as can be seen in Figure 4. In this experiment, we collect samples from shared 6 categories of the two datasets, including aircraft, birds, cows, family cars, sheep, and bicycles. Then, we construct two transfer learning tasks: MSRC vs. VOC and VOC vs. MSRC, in which 1,269 images are selected in MSRC and 1530 images are selected in VOC2007. In addition, we rescale all images to 256 gray pixels in length and extract 240 dimensions as a new feature representation (shown in Table 1).Figure 4
Images from MSRC and VOC2007 datasets.Reuters-21578: Reuters-21578 is a text dataset commonly used for text data mining and analysis. It has 21,577 news documents from 5 classes, such as “exchanges,” “orgs,” “people,” “places,” and “topics.” In this experiment, we select the largest three categories “orgs,” “people,” and “place,” and construct 6 transfer learning tasks, i.e., orgs vs. people, people vs. orgs, orgs vs. place, place vs. orgs, people vs. place, and place vs. people, as shown in Table 1.


### 4.2. Experimental Settings

We choose some classifiers to compare them with JTELM as follows:1NN: one nearest neighbor classifierSVM: support vector machine with linear kernel and penalty parameter belonging to0.1,0.5,1,5,10,50,100;ELM: standard extreme learning machineSSELM: semisupervised ELM with graph regularization [24]TCA1(2): TCA [20] + NN (SVM)JDA1(2): JDA [21] + NN (SVM); we set the dimension of the feature subspace in TCA and JDA and the value range of the sparsity constraint parameter of the projection matrix is 0.1,10;DAELM_S, DAELM_T [29]: domain adaptation ELMsAELM [44]: ELM with feature augmentation (AELM). Its result in this section is cited from [44]ARRLS [45]: a general transfer learning framework. We set its parameters according to [45]TELM-OWA [33]: supervised transfer ELM. We set its parameters as referred in [33]CdELM-C [38]: unsupervised transfer ELM using MMD. We cited its results in [38] for comparisonIn addition, we set the penalty parameterλ∈0.001,0.1 in ELM, SSELM, DAELM_S, DAELM_T, and TELM-OWA. In JTELM, we set α1∈10−2,10−1, α2∈10−4,10−1, γ∈10−2,10−1, L=2000, and λ=10−4 on Office + Caltech dataset, L=3000 and λ=10−4 on USPS + MNIST, L=3000 and λ=103 on Reuters-21578 dataset, and L=1000 and λ=10−1 on MSRC + VOC2007 datasets.To evaluate DAELM_S, DAELM_T, and TELM-OWA in the experiment for unsupervised transfer learning, we select few target samples such as 0.5% labeled target samples on USPS + MNIST, MSRC + VOC2007, and Reuters-21578 datasets and 1% labeled target samples on Office + Caltech dataset to train models.


### 4.3. Results and Analysis

To investigate the performance of JTELM, we carry out classification tasks on image and text datasets including Office + Caltech, USPS + MNIST, MSRC + VOC2007, and Reuters-21578 datasets, and the results are reported in Table2 and 3. From the results, the following are the observations:(1)
JTELM has the highest accuracy in the total average of all algorithms in Tables2 and 3. It, respectively, gains improvement of 10.54% and 8.73% compared to the baseline ELM in Tables 2 and 3, indicating that our method has a better ability of knowledge transfer with help of CDMA, output weight alignment, and weight approximation. It enriches ELM in theory and application.(2)
TELM-OWA and DAELM, as supervised transfer learning mechanisms which requires parts of the labeled target samples, are not ideal under unsupervised learning. TCA, JDA, ARRLS, and CdELM-C apply MMD to reduce the distribution discrepancy of the two domains and to gain good results. SSELM utilizes graph regularization to explore the information of unlabeled target samples and it performs well.(3)
TCA1 (2) and JDA1 (2) implement the classification task by combining the transfer feature extraction methods(TCA and JDA) with baseline classifier, therefore they outperform 1NN and SVM. ELM performs slightly better than 1NN and SVM due to its good generation ability.Table 2
Accuracy of different algorithms on USPS + MNIST and Office + Caltech datasets.
Methods\datasetNontransfer learning algorithmTransfer learning algorithm1NNSVMELMSSELMTCA1TCA2JDA1JDA2DAELM_SDAELM_TTELM-OWAAELMCdELM-CJTELMUSPS vs. MNIST44.7035.1030.7029.0550.3560.2046.8053.2544.3631.0859.5541.6545.8962.00MNIST vs. USPS65.9454.6951.3943.2862.2871.4457.8375.9447.1160.3259.3762.8342.5776.39Average55.3244.9041.0536.1656.3165.8252.3264.6045.7345.7059.4652.2444.2369.20C⟶A (1)23.7052.4050.6352.9246.1444.7857.9356.7850.2627.8531.7352.4056.2857.30C⟶W (2)25.7640.6842.7145.4237.9741.6948.4750.8545.4231.8632.2043.7350.9855.59C⟶D (3)25.4842.0442.0438.8545.8645.2249.6850.3237.5822.9333.1247.7743.8252.86A⟶C (4)26.0044.0843.0141.7640.6139.3645.6843.1042.8825.2538.8540.1643.1741.85A⟶W (5)29.8341.6937.6335.9340.0037.9741.0241.3638.3119.3238.6434.9240.7549.49A⟶D (6)25.4840.7642.6838.8531.8539.4942.0441.4036.9436.3143.3135.6741.7849.68W⟶C (7)19.8629.9232.2434.2831.2631.1731.4333.3034.2041.9033.4830.5434.4436.60W⟶A (8)22.9634.1336.8537.0631.9432.7837.3746.0336.4446.4938.0137.1639.5842.28W⟶D (9)59.2487.9080.2578.9889.1789.1785.9985.9980.8953.5077.0786.6283.0682.17D⟶C (10)26.2731.4330.1030.3733.3931.5234.9135.2632.3248.0832.5028.6734.4437.49D⟶A (11)28.5032.3631.5232.0531.2133.0938.9440.4034.3551.2036.2329.5439.5443.63D⟶W (12)63.3973.5671.8680.0087.4689.4981.6983.0572.8854.2478.6483.7384.3483.73Average31.3745.9145.1345.5445.5746.3149.6050.6545.2138.2442.8245.9149.3552.72Total average34.7945.7744.5444.2047.1149.1049.9852.6545.2839.3145.1946.8148.6255.08Bold values indicate that the value is the best result of the row in which it is located.Table 3
Accuracy of different algorithms on MSRC + VOC2007 and Reuters-21578 datasets.
Methods\datasetNontransfer learning algorithmTransfer learning algorithm1NNSVMELMSSELMTCA1TCA2JDA1JDA2DAELM_SDAELM_TARRLSTELM-OWAJTELMMSRC vs. VOC35.9535.1038.5637.5227.5227.1235.8237.5238.3433.6034.6441.3740.32VOC vs. MSRC45.2354.6958.0063.2047.4448.3154.2254.6158.3730.6150.3566.2270.76Average40.5944.9048.2850.3637.4837.7145.0246.0648.3532.1042.5053.7955.54Orgs vs. people (1)72.8575.2577.2480.3074.0973.7675.9980.6376.6448.7180.5563.7483.28People vs. orgs (2)72.0377.1279.3084.7273.6574.5477.1285.8578.4147.7284.5664.2785.53Orgs vs. place (3)67.5070.1863.5769.6170.6671.7270.9574.9868.5343.8172.2068.0976.32Place vs. orgs (4)61.1263.7764.5771.1668.8070.1867.5275.9868.7742.5074.1174.6878.74People vs. place (5)52.6560.6357.0166.0261.8462.5859.8964.0759.1442.4665.1859.0467.5Place vs. people (6)53.3957.9458.6861.6560.8260.8254.6057.3858.7739.9358.5660.4564.34Average63.2667.4866.7372.2468.3168.9367.6873.1568.3844.1972.5365.0575.95Total average57.5961.8462.1266.7760.6061.1362.0166.3863.3741.1765.0262.2370.85Bold values indicate that the value is the best result of the row in which it is located.In Table4, we test the running time of some compared algorithms and JTELM and record it. It can be seen that (1) ELM has the least running time among all methods without tuning the input weight and bias. (2) DAELM_S and DAELM_T are slightly higher than ELM because of the participation of part of the target samples in the training model. TELM-OWA consumes more time than ELM, DAELM_S, and DAELM_T because of solving the two weights which is similar to JTELM. JTELM has more time to solve weights and refine the target pseudo labels. (3) JDA has the most running time because it requires the construction of MMD scatter, feature extraction, and label refinement. TCA (1, 2) and JDA (1, 2) cost more time than 1NN and SVM to extract cross-domain features. JDA spends more time than TCA for label refinement. (4) Due to construction of the Laplacian matrix, SSELM costs more time than ELM.Table 4
Time consumption of all the methods on MNIST vs USPS.
Algorithm1NNSVMELMSSELMTCA1TCA2JDA1JDA2DAELM_SDAELM_TARRLSTELM-OWAJTELMTime (s)0.558.790.373.494.725.4748.3253.60.810.642.083.726.62


### 4.4. Ablation Study

In JTELM, three mechanisms including CDMA, output weight alignment (OWA), and weight approximation (WA) are applied to realize knowledge transferring. Thus, jointly making a decision needs to be executed under output weight alignment, so that we can regard both output weight alignment and jointly making a decision as OWA. It mainly depends on the parameterα2 to adjust its influence. The impact of different combinations of the three mechanisms on JTELM’s accuracy is shown in Table 5, and it shows that (1) both CDMA and OWA can independently enhance ELM’s knowledge transfer capability. (2) Combining CDMA and OWA for knowledge transfer is better than their own. (3) WA is also beneficial to knowledge transfer. Moreover, we compare ELM, ELM-CDMA, and JTELM in time consumption. ELM-CDMA is a special situation of JTELM, without output weight alignment and weight alignment, and the result in Table 6 shows that CDMA, OWA, and WA could help ELM perform better in transfer learning, but they need more time-cost.Table 5
Ablation study on different mechanisms of JTELM on USPS + MNIST and Office + Caltech datasets.
CDMA×√×√√OWA××√√√WA××××√USPS vs. MNIST30.7045.3052.5560.2062.00MNIST vs. USPS51.3982.5671.8975.6776.39C⟶A (1)50.6353.8654.8055.4357.30C⟶W (2)42.7152.2054.2454.5855.59C⟶D (3)42.0447.1347.1348.4152.86A⟶C (4)43.0142.8339.5740.1641.85A⟶W (5)37.6344.4148.1448.8149.49A⟶D (6)42.6847.1346.5047.7749.68W⟶C (7)32.2434.1136.2436.1536.60W⟶A (8)36.8538.1039.5641.3442.28W⟶D (9)80.2579.6277.7180.2582.17D⟶C (10)30.1033.1435.0037.1337.49D⟶A (11)31.5236.5441.7541.9643.63D⟶W (12)71.8674.5883.3985.0883.73Total average44.5450.8252.0353.7855.08Table 6
Time consumption of ELM, ELM-CDMA, and JTELM on MNIST vs. USPS.
AlgorithmELMELM-CDMAJTELMTime (s)0.371.5326.62


### 4.5. Parameter Analysis

We investigate the sensitivity of JTELM to parametersα1, α2, γ, and λ and to the number of hidden-layer nodes L and its convergence and perform experiments on org vs. people, MSRC vs. VOC, MNIST vs. USPS, and A vs. D datasets. The results are shown in Figures 5(a) to 5(f) and the following observations were the observations made: (1) With α1, α2, γ, and λ increasing, the accuracy of JTELM first rises up and then falls on 4 datasets, as shown in Figures 5(a) to 5(d). We can see that CDMA, the source classification error with OWA and output weight approximation, when adjusted to the appropriate range, can improve accuracy and the knowledge transfer ability of ELM in transfer learning mechanism. (2) As shown in Figure 5(b), the accuracy first increases and then slightly decreases on 4 datasets, with the number of L growing. When L increases, the nonlinear approximation of our network will perform better. However, for some datasets, a larger L may maximize the distribution discrepancy of the output data for the hidden layer from two domains, leading to model’s poor performance. (3) We also observed the accuracy varying with the iteration number in Figure 5(f). It shows that the accuracy of JTELM gradually becomes stable and finally converges after 10 iterations.Figure 5
Classification accuracy vs. number of hidden-layer nodesL, iterations, and parameters α1, α2, γ, and λ.
(a)(b)(c)(d)(e)(f)


## 4.1. Datasets Description

Office31 + Caltech256 (shown in Figure2): These datasets were first published in the year referred in [43]. It contains two domains, namely, Office31 and Caltech256. Office31 consists of 4,652 images in 31 categories. They have been collected from 3 subdomains, that is, Amazon (A), DSLR (D), and Webcam (W). Caltech (C) is also an object image dataset consisting of 30,607 images from 256 categories.Figure 2
Images from Office31+Caltech256.During the experiment, we select 1,410 images with 10 categories from office31and 1123 images with 10 categories from Caltech. Every picture is extracted using SURF features with 800 dimensions. Two subdomains in A, W,D, and C are randomly chosen as the source and target domain datasets, and 12 cross-domain tasks are built as C⟶A, C⟶W, C⟶D, …, and D⟶W (shown in Table 1).Table 1
Description of image and text datasets.
DatasetTypeSamplesDimensionClassContains subsetsUSPSDigit180025610USPSMNISTDigit200025610MNISTMSRCObject126924018MSRCVOC2007Object153024020VOCOfficeObject141080010A, W, DCaltechObject112380010CReuters-21578Text1,2374,771BinaryOrgsText1,2084,771BinaryPeopleText1,0164,771BinaryPlaceUSPS + MNIST (shown in Figure3): USPS and MNIST are the two image datasets describing numbers from 0 to 9, so they share 10 categories but have different distributions. USPS consists of 9,298 images with 16 × 16 pixels and MNIST has 70,000 images with 28 × 28 pixels. During the experiment, 1,800 pictures from USPS and 2,000 pictures from MNIST are selected as the source domain and the target domain (shown in Table 1). Every image is converted into 16 × 16 pixels and two cross-domain tasks, i.e., USPS vs. MNIST and MNIST vs. USPS are constructed for transfer learning tasks.Figure 3
Image from MNIST and USPS datasets.MSRC + VOC2007 (shown in Figure4): MSRC is an object image dataset consisting of 4,323 images from 18 categories. VOC2007 is an image dataset with photos in Flickr, consisting of 5,011 images from 18 categories. They have similar but different distributions as can be seen in Figure 4. In this experiment, we collect samples from shared 6 categories of the two datasets, including aircraft, birds, cows, family cars, sheep, and bicycles. Then, we construct two transfer learning tasks: MSRC vs. VOC and VOC vs. MSRC, in which 1,269 images are selected in MSRC and 1530 images are selected in VOC2007. In addition, we rescale all images to 256 gray pixels in length and extract 240 dimensions as a new feature representation (shown in Table 1).Figure 4
Images from MSRC and VOC2007 datasets.Reuters-21578: Reuters-21578 is a text dataset commonly used for text data mining and analysis. It has 21,577 news documents from 5 classes, such as “exchanges,” “orgs,” “people,” “places,” and “topics.” In this experiment, we select the largest three categories “orgs,” “people,” and “place,” and construct 6 transfer learning tasks, i.e., orgs vs. people, people vs. orgs, orgs vs. place, place vs. orgs, people vs. place, and place vs. people, as shown in Table 1.


## 4.2. Experimental Settings

We choose some classifiers to compare them with JTELM as follows:1NN: one nearest neighbor classifierSVM: support vector machine with linear kernel and penalty parameter belonging to0.1,0.5,1,5,10,50,100;ELM: standard extreme learning machineSSELM: semisupervised ELM with graph regularization [24]TCA1(2): TCA [20] + NN (SVM)JDA1(2): JDA [21] + NN (SVM); we set the dimension of the feature subspace in TCA and JDA and the value range of the sparsity constraint parameter of the projection matrix is 0.1,10;DAELM_S, DAELM_T [29]: domain adaptation ELMsAELM [44]: ELM with feature augmentation (AELM). Its result in this section is cited from [44]ARRLS [45]: a general transfer learning framework. We set its parameters according to [45]TELM-OWA [33]: supervised transfer ELM. We set its parameters as referred in [33]CdELM-C [38]: unsupervised transfer ELM using MMD. We cited its results in [38] for comparisonIn addition, we set the penalty parameterλ∈0.001,0.1 in ELM, SSELM, DAELM_S, DAELM_T, and TELM-OWA. In JTELM, we set α1∈10−2,10−1, α2∈10−4,10−1, γ∈10−2,10−1, L=2000, and λ=10−4 on Office + Caltech dataset, L=3000 and λ=10−4 on USPS + MNIST, L=3000 and λ=103 on Reuters-21578 dataset, and L=1000 and λ=10−1 on MSRC + VOC2007 datasets.To evaluate DAELM_S, DAELM_T, and TELM-OWA in the experiment for unsupervised transfer learning, we select few target samples such as 0.5% labeled target samples on USPS + MNIST, MSRC + VOC2007, and Reuters-21578 datasets and 1% labeled target samples on Office + Caltech dataset to train models.


## 4.3. Results and Analysis

To investigate the performance of JTELM, we carry out classification tasks on image and text datasets including Office + Caltech, USPS + MNIST, MSRC + VOC2007, and Reuters-21578 datasets, and the results are reported in Table2 and 3. From the results, the following are the observations:(1)
JTELM has the highest accuracy in the total average of all algorithms in Tables2 and 3. It, respectively, gains improvement of 10.54% and 8.73% compared to the baseline ELM in Tables 2 and 3, indicating that our method has a better ability of knowledge transfer with help of CDMA, output weight alignment, and weight approximation. It enriches ELM in theory and application.(2)
TELM-OWA and DAELM, as supervised transfer learning mechanisms which requires parts of the labeled target samples, are not ideal under unsupervised learning. TCA, JDA, ARRLS, and CdELM-C apply MMD to reduce the distribution discrepancy of the two domains and to gain good results. SSELM utilizes graph regularization to explore the information of unlabeled target samples and it performs well.(3)
TCA1 (2) and JDA1 (2) implement the classification task by combining the transfer feature extraction methods(TCA and JDA) with baseline classifier, therefore they outperform 1NN and SVM. ELM performs slightly better than 1NN and SVM due to its good generation ability.Table 2
Accuracy of different algorithms on USPS + MNIST and Office + Caltech datasets.
Methods\datasetNontransfer learning algorithmTransfer learning algorithm1NNSVMELMSSELMTCA1TCA2JDA1JDA2DAELM_SDAELM_TTELM-OWAAELMCdELM-CJTELMUSPS vs. MNIST44.7035.1030.7029.0550.3560.2046.8053.2544.3631.0859.5541.6545.8962.00MNIST vs. USPS65.9454.6951.3943.2862.2871.4457.8375.9447.1160.3259.3762.8342.5776.39Average55.3244.9041.0536.1656.3165.8252.3264.6045.7345.7059.4652.2444.2369.20C⟶A (1)23.7052.4050.6352.9246.1444.7857.9356.7850.2627.8531.7352.4056.2857.30C⟶W (2)25.7640.6842.7145.4237.9741.6948.4750.8545.4231.8632.2043.7350.9855.59C⟶D (3)25.4842.0442.0438.8545.8645.2249.6850.3237.5822.9333.1247.7743.8252.86A⟶C (4)26.0044.0843.0141.7640.6139.3645.6843.1042.8825.2538.8540.1643.1741.85A⟶W (5)29.8341.6937.6335.9340.0037.9741.0241.3638.3119.3238.6434.9240.7549.49A⟶D (6)25.4840.7642.6838.8531.8539.4942.0441.4036.9436.3143.3135.6741.7849.68W⟶C (7)19.8629.9232.2434.2831.2631.1731.4333.3034.2041.9033.4830.5434.4436.60W⟶A (8)22.9634.1336.8537.0631.9432.7837.3746.0336.4446.4938.0137.1639.5842.28W⟶D (9)59.2487.9080.2578.9889.1789.1785.9985.9980.8953.5077.0786.6283.0682.17D⟶C (10)26.2731.4330.1030.3733.3931.5234.9135.2632.3248.0832.5028.6734.4437.49D⟶A (11)28.5032.3631.5232.0531.2133.0938.9440.4034.3551.2036.2329.5439.5443.63D⟶W (12)63.3973.5671.8680.0087.4689.4981.6983.0572.8854.2478.6483.7384.3483.73Average31.3745.9145.1345.5445.5746.3149.6050.6545.2138.2442.8245.9149.3552.72Total average34.7945.7744.5444.2047.1149.1049.9852.6545.2839.3145.1946.8148.6255.08Bold values indicate that the value is the best result of the row in which it is located.Table 3
Accuracy of different algorithms on MSRC + VOC2007 and Reuters-21578 datasets.
Methods\datasetNontransfer learning algorithmTransfer learning algorithm1NNSVMELMSSELMTCA1TCA2JDA1JDA2DAELM_SDAELM_TARRLSTELM-OWAJTELMMSRC vs. VOC35.9535.1038.5637.5227.5227.1235.8237.5238.3433.6034.6441.3740.32VOC vs. MSRC45.2354.6958.0063.2047.4448.3154.2254.6158.3730.6150.3566.2270.76Average40.5944.9048.2850.3637.4837.7145.0246.0648.3532.1042.5053.7955.54Orgs vs. people (1)72.8575.2577.2480.3074.0973.7675.9980.6376.6448.7180.5563.7483.28People vs. orgs (2)72.0377.1279.3084.7273.6574.5477.1285.8578.4147.7284.5664.2785.53Orgs vs. place (3)67.5070.1863.5769.6170.6671.7270.9574.9868.5343.8172.2068.0976.32Place vs. orgs (4)61.1263.7764.5771.1668.8070.1867.5275.9868.7742.5074.1174.6878.74People vs. place (5)52.6560.6357.0166.0261.8462.5859.8964.0759.1442.4665.1859.0467.5Place vs. people (6)53.3957.9458.6861.6560.8260.8254.6057.3858.7739.9358.5660.4564.34Average63.2667.4866.7372.2468.3168.9367.6873.1568.3844.1972.5365.0575.95Total average57.5961.8462.1266.7760.6061.1362.0166.3863.3741.1765.0262.2370.85Bold values indicate that the value is the best result of the row in which it is located.In Table4, we test the running time of some compared algorithms and JTELM and record it. It can be seen that (1) ELM has the least running time among all methods without tuning the input weight and bias. (2) DAELM_S and DAELM_T are slightly higher than ELM because of the participation of part of the target samples in the training model. TELM-OWA consumes more time than ELM, DAELM_S, and DAELM_T because of solving the two weights which is similar to JTELM. JTELM has more time to solve weights and refine the target pseudo labels. (3) JDA has the most running time because it requires the construction of MMD scatter, feature extraction, and label refinement. TCA (1, 2) and JDA (1, 2) cost more time than 1NN and SVM to extract cross-domain features. JDA spends more time than TCA for label refinement. (4) Due to construction of the Laplacian matrix, SSELM costs more time than ELM.Table 4
Time consumption of all the methods on MNIST vs USPS.
Algorithm1NNSVMELMSSELMTCA1TCA2JDA1JDA2DAELM_SDAELM_TARRLSTELM-OWAJTELMTime (s)0.558.790.373.494.725.4748.3253.60.810.642.083.726.62


## 4.4. Ablation Study

In JTELM, three mechanisms including CDMA, output weight alignment (OWA), and weight approximation (WA) are applied to realize knowledge transferring. Thus, jointly making a decision needs to be executed under output weight alignment, so that we can regard both output weight alignment and jointly making a decision as OWA. It mainly depends on the parameterα2 to adjust its influence. The impact of different combinations of the three mechanisms on JTELM’s accuracy is shown in Table 5, and it shows that (1) both CDMA and OWA can independently enhance ELM’s knowledge transfer capability. (2) Combining CDMA and OWA for knowledge transfer is better than their own. (3) WA is also beneficial to knowledge transfer. Moreover, we compare ELM, ELM-CDMA, and JTELM in time consumption. ELM-CDMA is a special situation of JTELM, without output weight alignment and weight alignment, and the result in Table 6 shows that CDMA, OWA, and WA could help ELM perform better in transfer learning, but they need more time-cost.Table 5
Ablation study on different mechanisms of JTELM on USPS + MNIST and Office + Caltech datasets.
CDMA×√×√√OWA××√√√WA××××√USPS vs. MNIST30.7045.3052.5560.2062.00MNIST vs. USPS51.3982.5671.8975.6776.39C⟶A (1)50.6353.8654.8055.4357.30C⟶W (2)42.7152.2054.2454.5855.59C⟶D (3)42.0447.1347.1348.4152.86A⟶C (4)43.0142.8339.5740.1641.85A⟶W (5)37.6344.4148.1448.8149.49A⟶D (6)42.6847.1346.5047.7749.68W⟶C (7)32.2434.1136.2436.1536.60W⟶A (8)36.8538.1039.5641.3442.28W⟶D (9)80.2579.6277.7180.2582.17D⟶C (10)30.1033.1435.0037.1337.49D⟶A (11)31.5236.5441.7541.9643.63D⟶W (12)71.8674.5883.3985.0883.73Total average44.5450.8252.0353.7855.08Table 6
Time consumption of ELM, ELM-CDMA, and JTELM on MNIST vs. USPS.
AlgorithmELMELM-CDMAJTELMTime (s)0.371.5326.62


## 4.5. Parameter Analysis

We investigate the sensitivity of JTELM to parametersα1, α2, γ, and λ and to the number of hidden-layer nodes L and its convergence and perform experiments on org vs. people, MSRC vs. VOC, MNIST vs. USPS, and A vs. D datasets. The results are shown in Figures 5(a) to 5(f) and the following observations were the observations made: (1) With α1, α2, γ, and λ increasing, the accuracy of JTELM first rises up and then falls on 4 datasets, as shown in Figures 5(a) to 5(d). We can see that CDMA, the source classification error with OWA and output weight approximation, when adjusted to the appropriate range, can improve accuracy and the knowledge transfer ability of ELM in transfer learning mechanism. (2) As shown in Figure 5(b), the accuracy first increases and then slightly decreases on 4 datasets, with the number of L growing. When L increases, the nonlinear approximation of our network will perform better. However, for some datasets, a larger L may maximize the distribution discrepancy of the output data for the hidden layer from two domains, leading to model’s poor performance. (3) We also observed the accuracy varying with the iteration number in Figure 5(f). It shows that the accuracy of JTELM gradually becomes stable and finally converges after 10 iterations.Figure 5
Classification accuracy vs. number of hidden-layer nodesL, iterations, and parameters α1, α2, γ, and λ.
(a)(b)(c)(d)(e)(f)


## 5. Conclusion

In this paper, we propose JTELM to address the problem that ELM degrades in transfer learning. It first applies CDMA to ELM and one transfer ELM model is learned. Then, similar to TELM-OWA, it uses output weight alignment and out weight approximation to learn the other transfer ELM on the source domain. Finally, it adopts two learned transfer ELMs to predict the samples from the target ones. Extensive experiments have been performed on the open image and text datasets, and the results show that JTELM has a higher accuracy and strong knowledge transfer ability than several state-of-the-art classifiers.



---

*Source: 5072247-2023-03-01.xml*