# Active Learning Query Strategies for Linear Regression Based on Efficient Global Optimization

Authors: Tianxin Zong; Na Li; Zhigang Zhang
Journal: Journal of Electrical and Computer Engineering
 (2022)

---

## Abstract

Active learning, a subfield of machine learning, can train a good model by selecting a minimum number of labeled samples. In many machine learning scenarios, needed information (such as the best value in unlabeled datasets) is acquired by prediction. When there is too little data in the training model, the prediction accuracy would obviously affect the accuracy of the results. To establish a high-performance regression model for a small dataset while accelerating the search for the best sample, a new active learning query strategy, EGO-ALR, that combines efficient global optimization (EGO) and active learning for regression (ALR) was proposed. It was found that the performance of EGO-ALR was significantly better than the original ALR query strategies in terms of the root mean square error (RMSE), correlation coefficient (CC), and opportunity cost (Oppo Cost). Specifically, EGO-ALR increased the Oppo Cost by an average of 25.27% when the RMSE and CC values were not more than 1.07% different from the original ALR. This study validated the efficiency and robustness of EGO-ALR approaches using 19 datasets from various domains and three distinct linear regression models (Ridge regression, Lasso, and Elastic network).


---

## Body

## 1. Introduction

Regression refers to estimating the value of a dependent variable (output) from one or more independent variables (characteristics). In a practical regression problem, some labeled samples (the independent variables and dependent variables are known) need to be trained by an appropriate approach to establish an accurate regression model. In general, the quality of performance of the trained model is proportional to the number of labeled samples. Data annotation is usually the biggest bottleneck in machine learning. Searching, managing, and labeling large amounts of data are often time-consuming and expensive to train a good model [1]. For example, in emotion estimation from speech signals, it is easy to record several speech utterances, but multiple assessors are needed to evaluate the emotion primitives [2]. In the problem of new alloy design, one can freely adjust the material composition within the range, but the synthesis and characterization of new materials should simultaneously consider the synthesis difficulty and cost of materials [3]. Similarly, in the research of video recommendation systems, users can simply upload videos, but few people manually annotate the metadata in detail; thus, costly annotation by experts is required, which will lead to a severe lack of text views and to a lack of training data for recommender systems [4].To enable use of applications with missing labeled samples, investigators propose ALR [5]. ALR can sequentially select some of the most beneficial samples for labeling, so that the trained model gives the most accurate predictions for the remaining unlabeled samples. ALR is iterative: first, one builds an initial model from a small number of labeled training samples, and then by some selection strategies, the most valuable samples among the unlabeled samples are labeled and added to the training set for the next round of modeling. This process will iterate until stopping conditions are met, such as the maximum number of iterations, the maximum number of labeled samples, and the cross-validation accuracy of the model.According to different query scenarios, investigators divide ALR into population-based ALR, flow-based ALR, and pool-based ALR [6]. The pool-based situation is considered in this study, in which a pool of unlabeled samples is given, and the goal is to select some training samples from the pool to improve a linear regression model.When a machine learning regression model is established, people usually use this model to predict samples with unknown labels and then obtain the needed information from the prediction results. In material design problems, machine learning is frequently used to extrapolate to a vast unexplored search space to search for the best performing material [7], but the accuracy of predictions is closely related to the performance of the regression models. To guide the experiment to the ideal material quickly, Balachandran et al. combined active learning with experimental design and proposed an adaptive iterative design strategy [3] to accelerate the material discovery process. This strategy is used in the scenario of multiple material designs [8–11].The adaptive strategy first defines a utility equation as the key to selecting the subsequent experimental sample. Then, the strategy predicts the most beneficial sample for experimental verification, and finally the strategy feeds the verified data to the machine learning model to improve the accuracy of predicting the best value. In this way, the samples with the best target performance are screened out with the least number of experiments. The utility functions commonly use the EGO algorithm [12] and the knowledge gradient algorithm [13].ALR can only solve the problem using as few samples as possible to improve the ability of the machine learning model. However, training a high-precision regression model is not its ultimate goal in practical application. People prefer to predict the samples in an unknown space to get the best beyond the existing one. Although EGO can meet the needs of objective optimization, this strategy often sacrifices the precision of the model’s prediction while performing rapid optimization [3]. So, there is currently no query strategy that perfectly balances the prediction performance and optimization performance under small sample conditions.It is easy to find that the principle of adaptive strategy is similar to ALR. Both strategies select the most beneficial samples from some approaches. If the utility function of global optimization is integrated with ALR, can we accelerate the progress to finding the best samples while building a model with high predictive performance? To address this question, this paper studied a class of ALR approaches based on optimization algorithms. The principal contributions are the following:(i)
A brand-new AL query strategy, combining the EGO algorithm with the ALR approach, was proposed to optimally balance the needs of “exploitation” (aims at improving the predictive model) and “exploration” (aims at finding the best sample).(ii)
The EGO-ALR inherits the usage of EGO: it can freely change the optimization direction, and its performance is stable regardless of finding the minimum or maximum value.(iii)
Extensive experiments were carried out on three common linear regression models and 19 datasets from different application domains, demonstrating the effectiveness and robustness of EGO-ALR. It also shows that this query strategy can even outperform the original ALR in both prediction and optimization performance in some cases.The remainder of this paper is organized as follows. Section2 introduces the EGO algorithm and some existing pool-based ALR approaches. Section 3 elaborates on the combined framework of the proposed approach. Section 4 conducts extensive experiments on 19 datasets, elucidating the experimental results and superiority performance of our approach. Finally, the conclusion and future work are given in Section 5.


## 2. Related Work

### 2.1. Existing Pool-Based ALR Query Strategies

The existing pool-based ALR approaches can be classified into two scenarios: supervised and unsupervised. Most existing ALR approaches are supervised; these approaches need some ground-truth labels to guide the sample selection. Unsupervised ALR does not require any label information when selecting samples. Next, several commonly used supervised and unsupervised ALR approaches are introduced below.


#### 2.1.1. Supervised ALR Query Strategies

Query by committee (QBC) [14] is widely used in different fields [15–19]. Assuming that N is the number of samples in the dataset, QBC first randomly selects and labels K0 samples, then establishes a committee of l learners from the existing training set (usually by bootstrapping), and predicts the samples in the unlabeled pool. QBC will select the samples from the pool on which the committee disagrees the most to label, that is:(1)σn2=1l∑i=1lyni−μn2,n=K0+1,…,N,where μn=1/l∑i=1lyni and ynl is the prediction result of the ith model built by bootstrap for the nth unlabeled sample xn. QBC selects the sample with the largest σn2 to label.Expected model change (EMCM) [20] is an ALR approach for regression and classification [21–23]; EMCM has a variety of algorithms [24–26]. Expected model change for linear regression is considered in this report. Expected model change first randomly selects and labels K0 samples to train a linear regression model. The prediction result of the model for the nth unlabeled sample is set as y^n. Then, EMCM uses bootstrap to build l linear regression models. In each sequential iteration, the labeled samples are those that change the linear regression parameters the most, that is:(2)gxn=1l∑i=1lyni−y^nxn,n=K0+1,…,N.EMCM labels the sample with the largestg (xn).


#### 2.1.2. Unsupervised ALR Query Strategies

Yu and Kim proposed an unsupervised ALR approach based on greedy sampling [27], which is also applied to image signal processing [28]. Greedy sampling initially labels at least one sample, but Yu and Kim do not define the first sample. Therefore, the study used an improved method, GSx, proposed by Wu et al. [29]. The idea of GSx is to take the sample closest to the centroid in the pool as the first labeled sample and then select a sample in a greedy way such that it is farthest from all the selected samples at each sequential iteration:(3)dnm=xn−xm,m=1,…,K0,n=K0+1,…,N,where xm is the labeled sample. GSx first calculates the distance between xn and xm of all unlabeled samples, then it computes the minimum distance from xn to xm:(4)dn=minmdmn,n=K0+1,…,N,and selects the sample with the largest dn.Representation-diversity (RD) proposed by Wu is also unsupervised and can be used for linear regression [30], and RD derived some excellent unsupervised ALRs such as IRD [31] and iRDM [1]. It performs k-means clustering (k = K0) and selects the first K0 samples closest to the centroid from each cluster. When selecting the K0 + 1th sample, RD performs k-means clustering (k = K0 + 1) on all samples in the pool, identifies the largest cluster that does not contain the labeled samples, and selects the one closest to the centroid as the K0 + 1th sample. The basic RD algorithm can also be combined with other supervised ALR approaches for better performance [30]. For example, RD-EMCM combines RD and EMCM.The ALR approach improves only the prediction performance as the criterion for selecting samples. Due to multiple limitations such as too few labeled samples, the complexity of the dataset distribution, and the defects of the algorithm itself, it is difficult for even the ALR to achieve the required prediction accuracy in only a few iterations. The results obtained by such a model have many deviations and cannot be used as a reference.


### 2.2. Efficient Global Optimization

EGO [12] is an algorithm with many related extensions to different types of research [32, 33]. We first introduce the expected improvement [34] before introducing EGO. Let f ∗ = min (y1, …, ym) be the current best value in the training set. Before labeling xn, its value yn is uncertain. The uncertainty at yn is modeled as the realization of a normally distributed random variable Y with mean and standard deviation determined by bootstrap. If the normal density function with the mean and standard deviation is plotted at xn, yn has a certain probability to be better than f ∗. Expected improvement (EI) weighs all possible improvements by the associated density value at the point. Formally, the improvement at xn is I = max (f∗-Y, 0). Because Y is a random variable, this expression is also a random variable. Simply take the expected value to obtain the EI:(5)EIxn=Emaxμ∗−Y,0.To compute this expectation, the notationsμn and σn are introduced to denote the expectation and standard deviation at xn. Y is normal (μn, σn2). By performing some integrals by parts on the right side of equation (5), one can expand as(6)EIxn=μ∗−μnΦμ∗−μnσn+σnϕμ∗−μnσn.In the above equation,ϕ (·) and Φ (·) are the standard normal density and distribution function. EGO will select the sample with the largest E[I(xn)].The EGO considers both the predicted (uncertain) and optimized (best) value, but the prediction error of the EGO-constructed model is still high [3]. The advantage of EGO is that in the case of large regression error, the approach can also be more effective than random selection or direct selection of the best value of the prediction. Because of this characteristic, EGO is used mostly in data-driven material design applications, which require selecting samples with better performance by a small number of iterations.


## 2.1. Existing Pool-Based ALR Query Strategies

The existing pool-based ALR approaches can be classified into two scenarios: supervised and unsupervised. Most existing ALR approaches are supervised; these approaches need some ground-truth labels to guide the sample selection. Unsupervised ALR does not require any label information when selecting samples. Next, several commonly used supervised and unsupervised ALR approaches are introduced below.


### 2.1.1. Supervised ALR Query Strategies

Query by committee (QBC) [14] is widely used in different fields [15–19]. Assuming that N is the number of samples in the dataset, QBC first randomly selects and labels K0 samples, then establishes a committee of l learners from the existing training set (usually by bootstrapping), and predicts the samples in the unlabeled pool. QBC will select the samples from the pool on which the committee disagrees the most to label, that is:(1)σn2=1l∑i=1lyni−μn2,n=K0+1,…,N,where μn=1/l∑i=1lyni and ynl is the prediction result of the ith model built by bootstrap for the nth unlabeled sample xn. QBC selects the sample with the largest σn2 to label.Expected model change (EMCM) [20] is an ALR approach for regression and classification [21–23]; EMCM has a variety of algorithms [24–26]. Expected model change for linear regression is considered in this report. Expected model change first randomly selects and labels K0 samples to train a linear regression model. The prediction result of the model for the nth unlabeled sample is set as y^n. Then, EMCM uses bootstrap to build l linear regression models. In each sequential iteration, the labeled samples are those that change the linear regression parameters the most, that is:(2)gxn=1l∑i=1lyni−y^nxn,n=K0+1,…,N.EMCM labels the sample with the largestg (xn).


### 2.1.2. Unsupervised ALR Query Strategies

Yu and Kim proposed an unsupervised ALR approach based on greedy sampling [27], which is also applied to image signal processing [28]. Greedy sampling initially labels at least one sample, but Yu and Kim do not define the first sample. Therefore, the study used an improved method, GSx, proposed by Wu et al. [29]. The idea of GSx is to take the sample closest to the centroid in the pool as the first labeled sample and then select a sample in a greedy way such that it is farthest from all the selected samples at each sequential iteration:(3)dnm=xn−xm,m=1,…,K0,n=K0+1,…,N,where xm is the labeled sample. GSx first calculates the distance between xn and xm of all unlabeled samples, then it computes the minimum distance from xn to xm:(4)dn=minmdmn,n=K0+1,…,N,and selects the sample with the largest dn.Representation-diversity (RD) proposed by Wu is also unsupervised and can be used for linear regression [30], and RD derived some excellent unsupervised ALRs such as IRD [31] and iRDM [1]. It performs k-means clustering (k = K0) and selects the first K0 samples closest to the centroid from each cluster. When selecting the K0 + 1th sample, RD performs k-means clustering (k = K0 + 1) on all samples in the pool, identifies the largest cluster that does not contain the labeled samples, and selects the one closest to the centroid as the K0 + 1th sample. The basic RD algorithm can also be combined with other supervised ALR approaches for better performance [30]. For example, RD-EMCM combines RD and EMCM.The ALR approach improves only the prediction performance as the criterion for selecting samples. Due to multiple limitations such as too few labeled samples, the complexity of the dataset distribution, and the defects of the algorithm itself, it is difficult for even the ALR to achieve the required prediction accuracy in only a few iterations. The results obtained by such a model have many deviations and cannot be used as a reference.


## 2.1.1. Supervised ALR Query Strategies

Query by committee (QBC) [14] is widely used in different fields [15–19]. Assuming that N is the number of samples in the dataset, QBC first randomly selects and labels K0 samples, then establishes a committee of l learners from the existing training set (usually by bootstrapping), and predicts the samples in the unlabeled pool. QBC will select the samples from the pool on which the committee disagrees the most to label, that is:(1)σn2=1l∑i=1lyni−μn2,n=K0+1,…,N,where μn=1/l∑i=1lyni and ynl is the prediction result of the ith model built by bootstrap for the nth unlabeled sample xn. QBC selects the sample with the largest σn2 to label.Expected model change (EMCM) [20] is an ALR approach for regression and classification [21–23]; EMCM has a variety of algorithms [24–26]. Expected model change for linear regression is considered in this report. Expected model change first randomly selects and labels K0 samples to train a linear regression model. The prediction result of the model for the nth unlabeled sample is set as y^n. Then, EMCM uses bootstrap to build l linear regression models. In each sequential iteration, the labeled samples are those that change the linear regression parameters the most, that is:(2)gxn=1l∑i=1lyni−y^nxn,n=K0+1,…,N.EMCM labels the sample with the largestg (xn).


## 2.1.2. Unsupervised ALR Query Strategies

Yu and Kim proposed an unsupervised ALR approach based on greedy sampling [27], which is also applied to image signal processing [28]. Greedy sampling initially labels at least one sample, but Yu and Kim do not define the first sample. Therefore, the study used an improved method, GSx, proposed by Wu et al. [29]. The idea of GSx is to take the sample closest to the centroid in the pool as the first labeled sample and then select a sample in a greedy way such that it is farthest from all the selected samples at each sequential iteration:(3)dnm=xn−xm,m=1,…,K0,n=K0+1,…,N,where xm is the labeled sample. GSx first calculates the distance between xn and xm of all unlabeled samples, then it computes the minimum distance from xn to xm:(4)dn=minmdmn,n=K0+1,…,N,and selects the sample with the largest dn.Representation-diversity (RD) proposed by Wu is also unsupervised and can be used for linear regression [30], and RD derived some excellent unsupervised ALRs such as IRD [31] and iRDM [1]. It performs k-means clustering (k = K0) and selects the first K0 samples closest to the centroid from each cluster. When selecting the K0 + 1th sample, RD performs k-means clustering (k = K0 + 1) on all samples in the pool, identifies the largest cluster that does not contain the labeled samples, and selects the one closest to the centroid as the K0 + 1th sample. The basic RD algorithm can also be combined with other supervised ALR approaches for better performance [30]. For example, RD-EMCM combines RD and EMCM.The ALR approach improves only the prediction performance as the criterion for selecting samples. Due to multiple limitations such as too few labeled samples, the complexity of the dataset distribution, and the defects of the algorithm itself, it is difficult for even the ALR to achieve the required prediction accuracy in only a few iterations. The results obtained by such a model have many deviations and cannot be used as a reference.


## 2.2. Efficient Global Optimization

EGO [12] is an algorithm with many related extensions to different types of research [32, 33]. We first introduce the expected improvement [34] before introducing EGO. Let f ∗ = min (y1, …, ym) be the current best value in the training set. Before labeling xn, its value yn is uncertain. The uncertainty at yn is modeled as the realization of a normally distributed random variable Y with mean and standard deviation determined by bootstrap. If the normal density function with the mean and standard deviation is plotted at xn, yn has a certain probability to be better than f ∗. Expected improvement (EI) weighs all possible improvements by the associated density value at the point. Formally, the improvement at xn is I = max (f∗-Y, 0). Because Y is a random variable, this expression is also a random variable. Simply take the expected value to obtain the EI:(5)EIxn=Emaxμ∗−Y,0.To compute this expectation, the notationsμn and σn are introduced to denote the expectation and standard deviation at xn. Y is normal (μn, σn2). By performing some integrals by parts on the right side of equation (5), one can expand as(6)EIxn=μ∗−μnΦμ∗−μnσn+σnϕμ∗−μnσn.In the above equation,ϕ (·) and Φ (·) are the standard normal density and distribution function. EGO will select the sample with the largest E[I(xn)].The EGO considers both the predicted (uncertain) and optimized (best) value, but the prediction error of the EGO-constructed model is still high [3]. The advantage of EGO is that in the case of large regression error, the approach can also be more effective than random selection or direct selection of the best value of the prediction. Because of this characteristic, EGO is used mostly in data-driven material design applications, which require selecting samples with better performance by a small number of iterations.


## 3. ALR Query Strategies Integrated with EGO

ALR is a subfield of machine learning that can train a good model by selecting a minimum number of labeled samples, and EGO is a global optimization algorithm that can quickly find the best sample. This study proposes a query strategy that integrates EGO with ALR, called EGO-ALR. By combining the advantages of the EGO and the ALR during sample selection, the method can accelerate the optimization process while maintaining prediction quality, which effectively reduces the influence of model performance on the outcome.The complete framework of the active learning method using EGO-ALR query strategy is shown in Figure1. First, resample the training set by bootstrap for l times. Second, train the resampled set into a regression model through machine learning. Then, predict all samples in the pool and use the EGO-ALR query strategy to calculate the information of each unlabeled sample. Finally, select and mark the most informative sample and add it to the training set.Figure 1
Framework of the active learning approach using EGO-ALR query strategy.Due to the different principles of supervised and unsupervised ALR query strategies, this paper needs to discuss the different approaches of EGO combined with the two kinds of ALR query strategies and point out some special changes.


### 3.1. Combination of Supervised ALR and EGO

The pseudocode of the supervised ALR combined with EGO is given in Algorithm 1. Pool U firstly consists ofN unlabeled samples and 0 labeled samples. Set K0 as the number of samples in the initial training set. Because it is combined with the supervised ALR approach, all samples in the initial training set will be randomly selected and labeled. Assuming that the first K samples (K ≥ K0) have been labeled, for the remaining N-K unlabeled samples, EGO-ALR first computes separately the “information” in both EGO and supervised ALR.The “information” is defined as a measure of how valuable a sample is to be labeled, for example,σn2 used in QBC, g (xn) used in EMCM, and E[I(xn)] used in EGO. Note that the “information” between each approach may have significantly different dimensions, and a larger scale may dominate the other “information.” Thus, EGO-ALR normalizes the “information” by min-max normalization and then adds it after weighting by the parameter c to reduce the sensitivity of the formula to scale. Here is an example of the “information” after the combination of EGO and QBC:(7)Tn=c⋅σn2∗+EIx∗,where c is the adjustable weight and ∗ represents the normalized value. For labeling, EGO-ALR selects the sample with the largest Tn.Because the value of parameterc is the most effective way to balance the prediction performance of ALR and the optimization performance of EGO, the value has a crucial influence on the sampling results. Generally, the larger the c value, the closer the sample selection to ALR, and model prediction performance increases while optimization performance (find the best value) decreases; the smaller the c value, the closer the sample selection to EGO, and model prediction performance decreases while optimization performance increases. The effect of different parameters c on the results is explained in Section 4.7.Algorithm 1: The proposed supervised EGO-ALR approach.
(1)
Input:xn, a pool of N unlabeled samples; K, maximum number of samples to label;(2)
c, weighting parameters.(3)
Output: regression model f(x)(4)
Randomly select and labelK0 samples;(5)
Construct the initial regression modelf(x) with K0 samples;(6)
form = K0 + 1, …, Kdo(7)
BuildL regression models using bootstrap from the training set(8)
forn = m, …, Kdo(9)
EGO-QBC: compute σn2 in (1) and E[I(xn)] in (6);(10)
min-max normalization ofσn2 and E[I(xn)], marked as σn2∗ and EIx∗(11)
ComputeTn=c⋅σn2∗+EIx∗(12)
EGO-EMCM: compute g (xn) in (2) and E[I(xn)] in (6);(13)
min-max normalization ofg (xn) and E[I(xn)], marked as g (xn)∗ and EIx∗(14)
ComputeTn=c⋅gxn∗+EIx∗(15)
end(16)
Label the sample with the largestTn and add it to the training set.(17)
end(18)
Update the regression modelf(x) with the labeled K samples.


### 3.2. Combination of Unsupervised ALR and EGO

The combined query strategy of unsupervised ALR is similar to supervised ALR, with the difference in the initial training samples. “Unsupervised” means the selection of samples is independent of the label information. When the pools are all unlabeled samples, unsupervised ALR can still select samples for labeling by corresponding strategies. Algorithm 2 gives the pseudocode. The combination of GSx and EGO uses the method described in Section3.1. The combination of RD and EGO adopts the method described by Wu [30], that is, it selects RD to initialize and uses EGO to select samples in the largest cluster lacking labeled samples.Algorithm 2: The proposed unsupervised EGO-ALR approach.
(1)
Input:xn, a pool of N unlabeled samples; K, maximum number of samples to label;(2)
c, weighting parameters.(3)
Output: regression model f(x)(4)
Select and label the initialK0 samples with the GSx (or RD) algorithm;(5)
Construct the initial regression modelf(x) with K0 samples;(6)
form = K0  + 1, …, Kdo(7)
BuildL regression models using bootstrap from the training set(8)
forn = m, …, Kdo(9)
EGO-GSx: compute dn in (4) and E[I(xn)] in (6);(10)
min-max normalization ofdn and E[I(xn)], marked as dn∗ and EIx∗(11)
ComputeTn=c⋅dn∗+EIx∗(12)
RD-EGO: perform k-means (k = n) clustering on all samples in the pool;(13)
Identify the largest cluster that does not contain labeled samples(14)
ComputeE[I(xn)] in (6) for the samples in the cluster(15)
end(16)
Label the sample with the largestTn (or E[I(xn)]) and add it to the training set.(17)
end(18)
Update the regression modelf(x) with the labeled K samples.In summary, EGO-ALR randomly (or using unsupervised ALR) selects the firstK0 samples to build an initial regression model, and, in each subsequent iteration, EGO-ALR chooses the sample with the largest Tn (or E[I(xn)]) to achieve the combination and balance of “exploitation” and “exploration.”


## 3.1. Combination of Supervised ALR and EGO

The pseudocode of the supervised ALR combined with EGO is given in Algorithm 1. Pool U firstly consists ofN unlabeled samples and 0 labeled samples. Set K0 as the number of samples in the initial training set. Because it is combined with the supervised ALR approach, all samples in the initial training set will be randomly selected and labeled. Assuming that the first K samples (K ≥ K0) have been labeled, for the remaining N-K unlabeled samples, EGO-ALR first computes separately the “information” in both EGO and supervised ALR.The “information” is defined as a measure of how valuable a sample is to be labeled, for example,σn2 used in QBC, g (xn) used in EMCM, and E[I(xn)] used in EGO. Note that the “information” between each approach may have significantly different dimensions, and a larger scale may dominate the other “information.” Thus, EGO-ALR normalizes the “information” by min-max normalization and then adds it after weighting by the parameter c to reduce the sensitivity of the formula to scale. Here is an example of the “information” after the combination of EGO and QBC:(7)Tn=c⋅σn2∗+EIx∗,where c is the adjustable weight and ∗ represents the normalized value. For labeling, EGO-ALR selects the sample with the largest Tn.Because the value of parameterc is the most effective way to balance the prediction performance of ALR and the optimization performance of EGO, the value has a crucial influence on the sampling results. Generally, the larger the c value, the closer the sample selection to ALR, and model prediction performance increases while optimization performance (find the best value) decreases; the smaller the c value, the closer the sample selection to EGO, and model prediction performance decreases while optimization performance increases. The effect of different parameters c on the results is explained in Section 4.7.Algorithm 1: The proposed supervised EGO-ALR approach.
(1)
Input:xn, a pool of N unlabeled samples; K, maximum number of samples to label;(2)
c, weighting parameters.(3)
Output: regression model f(x)(4)
Randomly select and labelK0 samples;(5)
Construct the initial regression modelf(x) with K0 samples;(6)
form = K0 + 1, …, Kdo(7)
BuildL regression models using bootstrap from the training set(8)
forn = m, …, Kdo(9)
EGO-QBC: compute σn2 in (1) and E[I(xn)] in (6);(10)
min-max normalization ofσn2 and E[I(xn)], marked as σn2∗ and EIx∗(11)
ComputeTn=c⋅σn2∗+EIx∗(12)
EGO-EMCM: compute g (xn) in (2) and E[I(xn)] in (6);(13)
min-max normalization ofg (xn) and E[I(xn)], marked as g (xn)∗ and EIx∗(14)
ComputeTn=c⋅gxn∗+EIx∗(15)
end(16)
Label the sample with the largestTn and add it to the training set.(17)
end(18)
Update the regression modelf(x) with the labeled K samples.


## 3.2. Combination of Unsupervised ALR and EGO

The combined query strategy of unsupervised ALR is similar to supervised ALR, with the difference in the initial training samples. “Unsupervised” means the selection of samples is independent of the label information. When the pools are all unlabeled samples, unsupervised ALR can still select samples for labeling by corresponding strategies. Algorithm 2 gives the pseudocode. The combination of GSx and EGO uses the method described in Section3.1. The combination of RD and EGO adopts the method described by Wu [30], that is, it selects RD to initialize and uses EGO to select samples in the largest cluster lacking labeled samples.Algorithm 2: The proposed unsupervised EGO-ALR approach.
(1)
Input:xn, a pool of N unlabeled samples; K, maximum number of samples to label;(2)
c, weighting parameters.(3)
Output: regression model f(x)(4)
Select and label the initialK0 samples with the GSx (or RD) algorithm;(5)
Construct the initial regression modelf(x) with K0 samples;(6)
form = K0  + 1, …, Kdo(7)
BuildL regression models using bootstrap from the training set(8)
forn = m, …, Kdo(9)
EGO-GSx: compute dn in (4) and E[I(xn)] in (6);(10)
min-max normalization ofdn and E[I(xn)], marked as dn∗ and EIx∗(11)
ComputeTn=c⋅dn∗+EIx∗(12)
RD-EGO: perform k-means (k = n) clustering on all samples in the pool;(13)
Identify the largest cluster that does not contain labeled samples(14)
ComputeE[I(xn)] in (6) for the samples in the cluster(15)
end(16)
Label the sample with the largestTn (or E[I(xn)]) and add it to the training set.(17)
end(18)
Update the regression modelf(x) with the labeled K samples.In summary, EGO-ALR randomly (or using unsupervised ALR) selects the firstK0 samples to build an initial regression model, and, in each subsequent iteration, EGO-ALR chooses the sample with the largest Tn (or E[I(xn)]) to achieve the combination and balance of “exploitation” and “exploration.”


## 4. Results

This section conducted experiments on 19 datasets and three linear regression models to establish the performance of the proposed EGO-ALR. The experimental device was a personal computer, and the programming language was MATLAB R2018b.


### 4.1. Data Sources

A total of 19 datasets were used in the experiment. Sixteen datasets were from the UCI Machine Learning Library and three were from the CMU StatLib Datasets Archive. These sources have been used in many ALR studies [1, 20, 23, 27, 29–31]. Table 1 summarizes the datasets. Before the experiment, all datasets were removed of samples with missing features, special characters, and garbled characters. Two datasets, AutoMPG and CPS, contained some categorical features, which needed to be converted into numerical features by one-hot coding before the experiment (this conversion increased the number of features).Table 1
Summary of the 19 regression datasets.
DatasetSourceNo. of samplesNo. of raw featuresNo. of final featuresConcrete-CSUCI110377Concrete-FlowUCI110377Concrete-SlumpUCI110377YatchUCI230866AutoMPGUCI339279RealEstateUCI441466NO2StatLib550077PM10StatLib650077HousingUCI75061313CPSStatLib85341019Energy-CoolingUCI976877Energy-HeatingUCI976877ConcreteUCI10103088AirfoilUCI11150355Wine-redUCI1215991111Wine-whiteUCI1248981111HEAJournal1316599DirectJournal145341212IndirectJournal14183615151https://archive.ics.uci.edu/ml/datasets/Concrete+Slump+Test. 2https://archive.ics.uci.edu/ml/datasets/Auto+MPG. 3https://archive.ics.uci.edu/ml/datasets/Real+estate+valuation+data+set. 4https://archive.ics.uci.edu/ml/datasets/Real+estate+valuation+data+set. 5http://lib.stat.cmu.edu/datasets/NO2.dat. 6http://lib.stat.cmu.edu/datasets/PM10.dat. 7https://archive.ics.uci.edu/ml/machine-learning-databases/housing/. 8http://lib.stat.cmu.edu/datasets/CPS_85_Wages. 9https://archive.ics.uci.edu/ml/datasets/energy+efficiency. 10https://archive.ics.uci.edu/ml/datasets/Concrete+Compressive+Strength. 11https://archive.ics.uci.edu/ml/datasets/airfoil+self-noise. 12https://archive.ics.uci.edu/ml/datasets/wine+quality. 13https://www.sciencedirect.com/science/article/abs/pii/S1359645419301430. 14https://pubs.acs.org/doi/abs/10.1021/acsami.1c15021.In addition, there are three datasets from the field of materials collected from the literature: HEA [10], Direct [11], and Indirect [11]. Note that the features of the HEA dataset were calculated from the data and feature formula provided by the literature. Before the experiment, each dimension of the feature space was normalized by Z-score, so that the mean of the feature dimension was zero and the standard deviation was one.


### 4.2. Comparison Algorithm

The study compared the performance of 10 different approaches as follows:(1)
Base line, BL, which randomly selects all samples for labeling.(2)
EGO, which is introduced in Section2.(3)
EMCM: supervised ALR, which is introduced in Section2.1.(4)
EGO-EMCM (c = 2): the combination of EGO and EMCM, which is introduced in Section 3.1.(5)
QBC: supervised ALR, which is introduced in Section2.1.(6)
EGO-QBC (c = 2): the combination of EGO and QBC, which is introduced in Section 3.1.(7)
GSx: unsupervised ALR, which is introduced in Section2.2.(8)
EGO-GSx (c = 2): the combination of EGO and GSx, which is introduced in Section 3.2.(9)
RD: unsupervised ALR, which is introduced in Section2.2.(10)
RD-EGO: the combination of EGO and RD, which is introduced in Section3.2.


### 4.3. Evaluation Process

For each dataset, first randomly select 50% of the total samples as the training pool U and the remaining 50% as the test set T: U (50%) +T (50%). Because the benefits of the ALR method are reflected in modeling with a small number of samples, each approach selectedK∈[5, 50] sample from the training set U. The entire process was repeated 100 times to eliminate the effect of randomness on the results.After each iteration of each approach, RMSE and CC were computed as measures of prediction performance. To measure the ability to find the best value of different approaches, Oppo Cost was also used in the evaluation. Oppo Cost was defined as the modulus difference between the current best and the overall best [3]. Powell and Ryzhov [35] also used Oppo Cost to compare the performance of knowledge gradient and EGO algorithms. To horizontally compare different approaches on different datasets, the Oppo Cost in this paper specifically refers to the normalized Oppo Cost.The formulas of the three evaluation indicators are as follows:(8)RMSE=1n∑i=1nyi−y^i2,(9)CC=n∑i=1nyiy^i−∑i=1nyi∑i=1ny^in∑i=1nyi2−∑i=1nyi2·n∑i=1ny^i2−∑i=1ny^i2,(10)Oppo Cost=μ∗∗−μ∗ymax−ymin,where yi is the actual value of the test set sample, y^i is the predicted value of the test set sample, and n is the number of samples in the test set; i = 1, 2, …, n. μ∗ is the best-so-far and μ∗∗ is the overall best value in the training pool. ymax and ymin are the maximum and minimum values in the training pool. Note that RMSE and CC evaluate the ability of the regression model to predict the samples in the test set, whereas Oppo Cost evaluates the ability to find the samples with the best performance during the iteration. Thus, (10) is calculated for the training pool, not the test set.Note that CC was not directly optimized in the objective function of these approaches [1, 29, 30]. Generally, a regression model with a CC close to 1 should have a smaller RMSE, but there is no guarantee (see the experimental results below for details). Thus, the CC can be viewed only as a secondary measure of prediction performance for reference.For each approach, three regularized linear regression models were used for training:(i)
Ridge regression, Ridge [36]: regularization coefficient λ = 0.1.(ii)
Lasso regression, Lasso [37]: regularization coefficient λ = 0.1.(iii)
Elastic network, Enet [38]: regularization coefficient λ = 0.1, penalty item mixing parameter α = 0.5.The regularized regression was chosen over ordinary linear regression because the number of labeled samples was too small. Thus, the model, which regularized the coefficients, usually achieved better performance compared with the ordinary linear regression model.


### 4.4. Experimental Result on Ridge

Figure2 shows the average RMSE, CC, and Oppo Cost of different optimization directions for 10 methods with 19 datasets when using Ridge as the regression model.(1)
The performance of all ten approaches improved as the value ofK increased (smaller RMSE, Oppo Cost, and larger CC), which was intuitive. However, because of the small number of samples in the early stage, there were some fluctuations in the result. For example, when K ∈ [5, 10], the RMSE and CC results on 15 of the 19 datasets showed unexpected changes (larger RMSE and smaller CC when K increases). Of course, this problem was lessened after K continued to increase.(2)
Intuitively, the prediction performance of the ten approaches was better than BL in most datasets, which suggested that the samples selected by the strategy could indeed improve the performance of the regression model.(3)
Most algorithms with better optimization performance are related to EGO. EGO-ALR approaches in different optimization directions all had smaller Oppo Costs compared with ALR approaches. EGO achieved the smallest Oppo Costs on 15 of the 19 datasets (the remaining four datasets had the second smallest Oppo Cost). The above shows that the EGO and the approaches combined with EGO were the best sample selection approaches for optimization, no matter finding the maximum or the minimum.Figure 2
Mean RMSEs, CCs, and Oppo Costs of 19 datasets, averaged over 100 runs and different optimization directions. The horizontal axis representsK. Ridge (λ = 0.1) was used as the regression model.This study additionally computed the area under the curve (AUC) of the mean RMSEs, CCs, and Oppo Costs for the Ridge regression model, denoted as AUC-RMSE, AUC-CC, and AUC-OPPO, respectively, to compare the prediction and optimization performance more concretely between the approaches (Figure3). Because the AUCs from different datasets varied greatly, the AUC results were normalized to the AUC of BL; thus, the AUC of BL was always 1.Figure 3
Normalized AUCs of the mean RMSEs, CCs, and Oppo Costs for the 19 datasets. Ridge (λ = 0.1) was used.We made the following observations:(1)
On average, GSx had the largest AUC-CC (1.0857) and the smallest AUC-RMSE (0.8384) for most datasets. The prediction performance of QBC (AUC-CC = 1.0648, AUC-RMSE = 0.8831) was slightly better than EMCM (AUC-CC = 1.0588, AUC-RMSE = 0.8897), and both were better than RD (AUC-CC = 1.0224, AUC-RMSE = 0.9314). The performance of EGO (AUC-CC = 1.0297, AUC-RMSE = 0.9691) was only better than BL.(2)
For most datasets, EGO-EMCM, EGO-QBE, and EGO-GSx had similar prediction performance relative to their original ALR. Specifically, the maximum absolute value of AUC-CC between the three combined approaches and their original ALR was 0.176, and the average was 0.007. The maximum absolute value of the AUC-RMSE difference was 0.111, and the average was 0.009. The results of RD-EGO were peculiar; its average AUC-CC was 1.0271 and greater than RD in 13 of the 19 datasets. Meanwhile, the average AUC-RMSE of RD-EGO was 0.9143 and smaller than RD in 13 datasets of the 19 datasets. The prediction performance of RD-EGO was better than the RD on average, which was consistent with the description of the performance of RD combined with other approaches reported by Wu [30].(3)
The optimization results of RD (average 1.1638) were the worst for 17 of the 19 datasets and the second lowest on the other two datasets. The optimization performance of QBC (average 0.6542) was also slightly better than EMCM (average 0.6625). GSx (average 0.5803) was the best among the four ALR approaches, whereas EGO (average 0.43957), as a global optimization algorithm, had better performance compared with all approaches.(4)
EGO-EMCM, EGO-QBC, and RD-EGO had significantly smaller AUC-OPPO than their original ALR in all datasets. EGO-GSx had smaller AUC-OPPO (average 0.4829) than GSx for 17 of 19 datasets. Of the remaining two datasets, the AUC-OPPO difference between EGO-GSx and GSx was at most 0.067. Generally, the optimization performance of the four EGO-ALRs was always better than all ALRs. EGO-GSx, as the combination approach of EGO and GSx, had better optimization performance than the other three EGO-ALRs. The optimization performance of the remaining three approaches, ranked from the best to the worst, was RD-EGO, EGO-QBC, and EGO-EMCM.In summary, the rank of the prediction performance of the 10 approaches could be sorted as follows: GSx ≈ EGO-GSx > QBC ≈ EGO-QBC > EMCM ≈ EGO-EMCM > RD-EGO > RD > EGO > BL. The optimization performance ranking was EGO > EGO-GSx > RD-EGO > GSx > EGO-QBC > EGO-EMCM > QBC > EMCM > BL > RD. The rank confirms that our proposed method, whether combined with supervised ALR or unsupervised ALR, exhibited strong advantages in improving the optimization performance without significant loss of prediction performance.The measurement standard of the algorithm is not only accuracy but also stability. In the case of similar algorithm performance, the more stable algorithm is usually chosen. Table2 shows the percent improvement of the AUCs of the mean RMSEs and CCs over BL. Ridge was the regression model.Table 2
Percent improvements of the AUCs of the mean RMSEs, CCs, and Oppo Costs over BL (Ridge was the regression model; the best performances are in bold).
EGOEMCMEGO-EMCMQBCEGOGSxEGORDRD-QBC-GSx-EGORMSEMean3.0911.0310.5711.6911.1616.1715.196.868.57std0.6511.129.929.779.9715.2714.454.635.81CCMean1.814.084.405.005.017.656.414.201.96std2.353.443.423.553.402.141.58−1.81−1.88Oppo CostMean56.0433.7545.4834.5844.6841.9751.72−16.3850.65std34.7316.9924.3723.0726.8311.6820.07−5.1931.17As seen from Table2, the improvement of RMSE and CC of all ALR approaches was better than EGO (RMSE = 3.09%, CC = 1.81%). According to the results of RMSE, GSx (mean = 16.17%, std = 15.27%) had the largest and most stable improvement compared with BL, followed by EGO-GSx. CC showed that the improvement of GSx (mean = 7.65%, std = 2.14%) was the largest, the improvement of QBC (mean = 4.08%, std = 3.44%) was the most stable, and the improvement of the standard deviation of RD (−1.81%) and RD-EGO (−1.88%) was negative, which indicated that these two approaches were very unstable. EGO-ALR and its original ALRs had a similar improvement in CC and RMSE, and the difference in improvement was less than 1%.EGO had the largest and most stable improvement in Oppo Cost compared to BL, while EGO-ALRs had a larger and more stable improvement than the four ALR approaches. These results correspond to the ranking of the performance.


### 4.5. Experimental Results on Lasso and Enet

All the foregoing experiments were repeated with Lasso and Enet models. The conclusions were similar to Section4.4. For additional results, see Figures S1 and S2 and Tables S1 and S2 in the Supplementary Material. Compared with Ridge, the optimization performance of the 10 approaches was improved significantly with Lasso and Enet, and the standard deviation of RMSE was improved most obviously with Lasso.To quantify the performance improvement of the four EGO-ALRs compared with their original ALRs, this study computed their percent improvements with the three regression models (Table3). The lowest promotion percent on RMSE was −1.07%, and the lowest on CC was −1.99%. The lowest on Oppo Cost was 10.92%, and the highest was 55.45%. Regardless of the regression model, the percent improvement of EGO-ALRs in RMSE and CC was not less than −2%, but the percent improvement of Oppo Cost was more than 10%.Table 3
Percent improvements of the AUCs of the mean RMSEs, CCs, and Oppo Costs compared with each original ALR approach.
EGO-EMCMEGO-QBCEGO-GSxRD-EGORidgeRMSE−0.96−0.29−1.071.31CC−0.16−0.27−0.45−0.13Oppo Cost15.3713.4214.5754.37LassoRMSE−0.99−0.81−0.823.48CC1.20−0.36−0.48−0.90Oppo Cost22.1710.9216.3555.45EnetRMSE−0.70−0.68−0.75−0.88CC0.37−0.55−0.62−1.99Oppo Cost18.6011.8215.8254.40RD-EGO had the most significant improvement over RD; the Oppo Cost increased by 55.45% and the RMSE increased by more than 1% on both Ridge and Lasso. The improvement in the Oppo Cost of EGO-EMCM was second only to RD-EGO, and the CC of EGO-EMCM was also positive on the Lasso and Enet models.


### 4.6. Statistical Analysis

This section established the test groups that compared EGO-ALR with its original ALR, EGO, and BL to see if the differences in Oppo Cost between EGO-ALR and other approaches were statistically significant (EGO-ALR was used as the control approach). There were four EGO-ALRs in our work, so there were four test groups.First, the Friedman test was performed on 19 datasets, and the calculated statistic Ff (Table 4) was proposed by Iman and Davenport [39]. All calculated Ff values were always greater than the critical value F (3,54) = 2.78, which suggested that, regardless of which regression model was used, there were statistical differences among these approaches in each group.Table 4
Ff values of Friedman test on the AUCs of the Oppo Costs in four test groups (the critical value was 2.78 (α = 0.05)).
EGO-EMCMEGO-QBCEGO-GSxRD-EGORidge136.71217.4355.5194.81Lasso110.93—155.5194.81Enet37.4457.9112.7733.901The denominator is 0.After that, the post hoc test was performed to compare methods. The power of the post hoc test is greater when all methods are compared only to the control method and not between each other [40]. Thus, we used the Bonferroni–Dunn test as post hoc test. At q = 0.1, the critical difference (CD) for comparing four approaches on 19 datasets was 0.8880, and the visualization of the post hoc test is shown in Figure 4.Figure 4
Comparison of EGO-ALR against the others with the Bonferroni–Dunn test: (a) Ridge, (b) Lasso, and (c) Enet. All approaches with ranks outside the marked interval are significantly different (p < 0.1) from the control.
(a)(b)(c)Irrespective of the regression model, the opportunity cost of EGO-ALR was significantly better than that of the original ALR and BL. In addition, there was no significant difference in the average ranking between EGO-ALRs and EGO except for EGO-QBC on Lasso and Enet models. Thus, it is concluded that the performance improvement of the original ALR and BL by the EGO-ALR was statistically significant. However, the improvement was not significantly different from that of EGO.


### 4.7. Parameterc Sensitivity

EGO-ALR in Section3.1 has a parameter c, the weighted value of “information.” This section investigated the effect of c on the performance of the EGO-ALR. Figure 5 shows the normalized AUCs of EGO-ALR on Ridge, Lasso, and Enet when the parameter c ∈ [0, 5]. The corresponding results of EGO are also marked on the figure for comparison. Note that EGO-ALR is equivalent to the EGO when c = 0.Figure 5
Ratios of AUCs of the mean RMSEs, CCs, and Oppo Costs with differentc, average of 19 datasets, 100 runs, and different optimization directions.The performance of EGO-ALR improved asc increased, and performance converged after c = 3. In general, the result of c = 1 was closer to EGO; too much emphasis on optimization performance leads to a large loss of prediction performance. So, c = 1 is not a recommended value. c > 2 can maximize the optimization performance without excessive loss of prediction performance. When c > 2, the AUC-CC results can even outperform ALR (c ≥ 2 on the Lasso model, c ≥ 3 on the Enet model). The larger the c, the stronger the model prediction performance; the smaller the c, the stronger the model optimization performance. This result is in line with the respective sample selection characteristics of EGO and ALR. To find the balance between predicting and optimizing that makes the choice from the EGO-ALR algorithm more meaningful, c = 2 was chosen as the parameter of EGO-ALR.


### 4.8. Visualization of Sample Selection

This section explains the selection behavior of different approaches by the visualization of sample selection, to better visualize the advantages of EGO-ALR.Taking the results of modeling a typical dataset (HEA) using Ridge regression as an example, the study compared EGO, EMCM, EGO-EMCM, GSx, and EGO-GSx, including supervised ALR and unsupervised ALR. Figure6 shows the visualization of the sample selection results after iterating to the 20th round (the number of labeled samples is 25). The first 24 samples selected by an approach are marked in blue. The samples selected in the 20th round are marked in red. The black dotted line represents that the predicted value was equal to the actual value, and the red dotted line is the optimization goal of this experiment, that is, the minimum of Y.Figure 6
Visualization of sampling results from different approaches during active learning iteration to the 20th round.Driven by the global optimization strategy, EGO collected samples with lowY and hardly selected samples with high Y, which caused the overall sampling of EGO to be biased. Most of the unlabeled samples are above the black dotted line, which indicates that the predicted value of unlabeled samples was significantly lower than the actual value.EMCM-selected samples were distributed uniformly in the entire space. However, because the early sampling of EMCM was random and the subsequent sampling was small, the prediction of unlabeled samples in this experiment was also lower than the actual value. This situation would be improved after increasing the number of samples.GSx-selected samples were more uniform than EMCM and EGO samples, so the prediction results were significantly better. This selection strategy caused it to seldom focus on a cluster for sampling, and it was difficult to find the best value.The sampling of EGO-EMCM and EGO-GSx followed the original ALR while favoring lowerY clusters. EGO-EMCM and EGO-GSx not only ensured the prediction performance but also had more opportunities to select the best sample. It is further confirmed that the EGO-based ALR approach selects more reasonable samples than the original ALR, which results in better regression performances.


## 4.1. Data Sources

A total of 19 datasets were used in the experiment. Sixteen datasets were from the UCI Machine Learning Library and three were from the CMU StatLib Datasets Archive. These sources have been used in many ALR studies [1, 20, 23, 27, 29–31]. Table 1 summarizes the datasets. Before the experiment, all datasets were removed of samples with missing features, special characters, and garbled characters. Two datasets, AutoMPG and CPS, contained some categorical features, which needed to be converted into numerical features by one-hot coding before the experiment (this conversion increased the number of features).Table 1
Summary of the 19 regression datasets.
DatasetSourceNo. of samplesNo. of raw featuresNo. of final featuresConcrete-CSUCI110377Concrete-FlowUCI110377Concrete-SlumpUCI110377YatchUCI230866AutoMPGUCI339279RealEstateUCI441466NO2StatLib550077PM10StatLib650077HousingUCI75061313CPSStatLib85341019Energy-CoolingUCI976877Energy-HeatingUCI976877ConcreteUCI10103088AirfoilUCI11150355Wine-redUCI1215991111Wine-whiteUCI1248981111HEAJournal1316599DirectJournal145341212IndirectJournal14183615151https://archive.ics.uci.edu/ml/datasets/Concrete+Slump+Test. 2https://archive.ics.uci.edu/ml/datasets/Auto+MPG. 3https://archive.ics.uci.edu/ml/datasets/Real+estate+valuation+data+set. 4https://archive.ics.uci.edu/ml/datasets/Real+estate+valuation+data+set. 5http://lib.stat.cmu.edu/datasets/NO2.dat. 6http://lib.stat.cmu.edu/datasets/PM10.dat. 7https://archive.ics.uci.edu/ml/machine-learning-databases/housing/. 8http://lib.stat.cmu.edu/datasets/CPS_85_Wages. 9https://archive.ics.uci.edu/ml/datasets/energy+efficiency. 10https://archive.ics.uci.edu/ml/datasets/Concrete+Compressive+Strength. 11https://archive.ics.uci.edu/ml/datasets/airfoil+self-noise. 12https://archive.ics.uci.edu/ml/datasets/wine+quality. 13https://www.sciencedirect.com/science/article/abs/pii/S1359645419301430. 14https://pubs.acs.org/doi/abs/10.1021/acsami.1c15021.In addition, there are three datasets from the field of materials collected from the literature: HEA [10], Direct [11], and Indirect [11]. Note that the features of the HEA dataset were calculated from the data and feature formula provided by the literature. Before the experiment, each dimension of the feature space was normalized by Z-score, so that the mean of the feature dimension was zero and the standard deviation was one.


## 4.2. Comparison Algorithm

The study compared the performance of 10 different approaches as follows:(1)
Base line, BL, which randomly selects all samples for labeling.(2)
EGO, which is introduced in Section2.(3)
EMCM: supervised ALR, which is introduced in Section2.1.(4)
EGO-EMCM (c = 2): the combination of EGO and EMCM, which is introduced in Section 3.1.(5)
QBC: supervised ALR, which is introduced in Section2.1.(6)
EGO-QBC (c = 2): the combination of EGO and QBC, which is introduced in Section 3.1.(7)
GSx: unsupervised ALR, which is introduced in Section2.2.(8)
EGO-GSx (c = 2): the combination of EGO and GSx, which is introduced in Section 3.2.(9)
RD: unsupervised ALR, which is introduced in Section2.2.(10)
RD-EGO: the combination of EGO and RD, which is introduced in Section3.2.


## 4.3. Evaluation Process

For each dataset, first randomly select 50% of the total samples as the training pool U and the remaining 50% as the test set T: U (50%) +T (50%). Because the benefits of the ALR method are reflected in modeling with a small number of samples, each approach selectedK∈[5, 50] sample from the training set U. The entire process was repeated 100 times to eliminate the effect of randomness on the results.After each iteration of each approach, RMSE and CC were computed as measures of prediction performance. To measure the ability to find the best value of different approaches, Oppo Cost was also used in the evaluation. Oppo Cost was defined as the modulus difference between the current best and the overall best [3]. Powell and Ryzhov [35] also used Oppo Cost to compare the performance of knowledge gradient and EGO algorithms. To horizontally compare different approaches on different datasets, the Oppo Cost in this paper specifically refers to the normalized Oppo Cost.The formulas of the three evaluation indicators are as follows:(8)RMSE=1n∑i=1nyi−y^i2,(9)CC=n∑i=1nyiy^i−∑i=1nyi∑i=1ny^in∑i=1nyi2−∑i=1nyi2·n∑i=1ny^i2−∑i=1ny^i2,(10)Oppo Cost=μ∗∗−μ∗ymax−ymin,where yi is the actual value of the test set sample, y^i is the predicted value of the test set sample, and n is the number of samples in the test set; i = 1, 2, …, n. μ∗ is the best-so-far and μ∗∗ is the overall best value in the training pool. ymax and ymin are the maximum and minimum values in the training pool. Note that RMSE and CC evaluate the ability of the regression model to predict the samples in the test set, whereas Oppo Cost evaluates the ability to find the samples with the best performance during the iteration. Thus, (10) is calculated for the training pool, not the test set.Note that CC was not directly optimized in the objective function of these approaches [1, 29, 30]. Generally, a regression model with a CC close to 1 should have a smaller RMSE, but there is no guarantee (see the experimental results below for details). Thus, the CC can be viewed only as a secondary measure of prediction performance for reference.For each approach, three regularized linear regression models were used for training:(i)
Ridge regression, Ridge [36]: regularization coefficient λ = 0.1.(ii)
Lasso regression, Lasso [37]: regularization coefficient λ = 0.1.(iii)
Elastic network, Enet [38]: regularization coefficient λ = 0.1, penalty item mixing parameter α = 0.5.The regularized regression was chosen over ordinary linear regression because the number of labeled samples was too small. Thus, the model, which regularized the coefficients, usually achieved better performance compared with the ordinary linear regression model.


## 4.4. Experimental Result on Ridge

Figure2 shows the average RMSE, CC, and Oppo Cost of different optimization directions for 10 methods with 19 datasets when using Ridge as the regression model.(1)
The performance of all ten approaches improved as the value ofK increased (smaller RMSE, Oppo Cost, and larger CC), which was intuitive. However, because of the small number of samples in the early stage, there were some fluctuations in the result. For example, when K ∈ [5, 10], the RMSE and CC results on 15 of the 19 datasets showed unexpected changes (larger RMSE and smaller CC when K increases). Of course, this problem was lessened after K continued to increase.(2)
Intuitively, the prediction performance of the ten approaches was better than BL in most datasets, which suggested that the samples selected by the strategy could indeed improve the performance of the regression model.(3)
Most algorithms with better optimization performance are related to EGO. EGO-ALR approaches in different optimization directions all had smaller Oppo Costs compared with ALR approaches. EGO achieved the smallest Oppo Costs on 15 of the 19 datasets (the remaining four datasets had the second smallest Oppo Cost). The above shows that the EGO and the approaches combined with EGO were the best sample selection approaches for optimization, no matter finding the maximum or the minimum.Figure 2
Mean RMSEs, CCs, and Oppo Costs of 19 datasets, averaged over 100 runs and different optimization directions. The horizontal axis representsK. Ridge (λ = 0.1) was used as the regression model.This study additionally computed the area under the curve (AUC) of the mean RMSEs, CCs, and Oppo Costs for the Ridge regression model, denoted as AUC-RMSE, AUC-CC, and AUC-OPPO, respectively, to compare the prediction and optimization performance more concretely between the approaches (Figure3). Because the AUCs from different datasets varied greatly, the AUC results were normalized to the AUC of BL; thus, the AUC of BL was always 1.Figure 3
Normalized AUCs of the mean RMSEs, CCs, and Oppo Costs for the 19 datasets. Ridge (λ = 0.1) was used.We made the following observations:(1)
On average, GSx had the largest AUC-CC (1.0857) and the smallest AUC-RMSE (0.8384) for most datasets. The prediction performance of QBC (AUC-CC = 1.0648, AUC-RMSE = 0.8831) was slightly better than EMCM (AUC-CC = 1.0588, AUC-RMSE = 0.8897), and both were better than RD (AUC-CC = 1.0224, AUC-RMSE = 0.9314). The performance of EGO (AUC-CC = 1.0297, AUC-RMSE = 0.9691) was only better than BL.(2)
For most datasets, EGO-EMCM, EGO-QBE, and EGO-GSx had similar prediction performance relative to their original ALR. Specifically, the maximum absolute value of AUC-CC between the three combined approaches and their original ALR was 0.176, and the average was 0.007. The maximum absolute value of the AUC-RMSE difference was 0.111, and the average was 0.009. The results of RD-EGO were peculiar; its average AUC-CC was 1.0271 and greater than RD in 13 of the 19 datasets. Meanwhile, the average AUC-RMSE of RD-EGO was 0.9143 and smaller than RD in 13 datasets of the 19 datasets. The prediction performance of RD-EGO was better than the RD on average, which was consistent with the description of the performance of RD combined with other approaches reported by Wu [30].(3)
The optimization results of RD (average 1.1638) were the worst for 17 of the 19 datasets and the second lowest on the other two datasets. The optimization performance of QBC (average 0.6542) was also slightly better than EMCM (average 0.6625). GSx (average 0.5803) was the best among the four ALR approaches, whereas EGO (average 0.43957), as a global optimization algorithm, had better performance compared with all approaches.(4)
EGO-EMCM, EGO-QBC, and RD-EGO had significantly smaller AUC-OPPO than their original ALR in all datasets. EGO-GSx had smaller AUC-OPPO (average 0.4829) than GSx for 17 of 19 datasets. Of the remaining two datasets, the AUC-OPPO difference between EGO-GSx and GSx was at most 0.067. Generally, the optimization performance of the four EGO-ALRs was always better than all ALRs. EGO-GSx, as the combination approach of EGO and GSx, had better optimization performance than the other three EGO-ALRs. The optimization performance of the remaining three approaches, ranked from the best to the worst, was RD-EGO, EGO-QBC, and EGO-EMCM.In summary, the rank of the prediction performance of the 10 approaches could be sorted as follows: GSx ≈ EGO-GSx > QBC ≈ EGO-QBC > EMCM ≈ EGO-EMCM > RD-EGO > RD > EGO > BL. The optimization performance ranking was EGO > EGO-GSx > RD-EGO > GSx > EGO-QBC > EGO-EMCM > QBC > EMCM > BL > RD. The rank confirms that our proposed method, whether combined with supervised ALR or unsupervised ALR, exhibited strong advantages in improving the optimization performance without significant loss of prediction performance.The measurement standard of the algorithm is not only accuracy but also stability. In the case of similar algorithm performance, the more stable algorithm is usually chosen. Table2 shows the percent improvement of the AUCs of the mean RMSEs and CCs over BL. Ridge was the regression model.Table 2
Percent improvements of the AUCs of the mean RMSEs, CCs, and Oppo Costs over BL (Ridge was the regression model; the best performances are in bold).
EGOEMCMEGO-EMCMQBCEGOGSxEGORDRD-QBC-GSx-EGORMSEMean3.0911.0310.5711.6911.1616.1715.196.868.57std0.6511.129.929.779.9715.2714.454.635.81CCMean1.814.084.405.005.017.656.414.201.96std2.353.443.423.553.402.141.58−1.81−1.88Oppo CostMean56.0433.7545.4834.5844.6841.9751.72−16.3850.65std34.7316.9924.3723.0726.8311.6820.07−5.1931.17As seen from Table2, the improvement of RMSE and CC of all ALR approaches was better than EGO (RMSE = 3.09%, CC = 1.81%). According to the results of RMSE, GSx (mean = 16.17%, std = 15.27%) had the largest and most stable improvement compared with BL, followed by EGO-GSx. CC showed that the improvement of GSx (mean = 7.65%, std = 2.14%) was the largest, the improvement of QBC (mean = 4.08%, std = 3.44%) was the most stable, and the improvement of the standard deviation of RD (−1.81%) and RD-EGO (−1.88%) was negative, which indicated that these two approaches were very unstable. EGO-ALR and its original ALRs had a similar improvement in CC and RMSE, and the difference in improvement was less than 1%.EGO had the largest and most stable improvement in Oppo Cost compared to BL, while EGO-ALRs had a larger and more stable improvement than the four ALR approaches. These results correspond to the ranking of the performance.


## 4.5. Experimental Results on Lasso and Enet

All the foregoing experiments were repeated with Lasso and Enet models. The conclusions were similar to Section4.4. For additional results, see Figures S1 and S2 and Tables S1 and S2 in the Supplementary Material. Compared with Ridge, the optimization performance of the 10 approaches was improved significantly with Lasso and Enet, and the standard deviation of RMSE was improved most obviously with Lasso.To quantify the performance improvement of the four EGO-ALRs compared with their original ALRs, this study computed their percent improvements with the three regression models (Table3). The lowest promotion percent on RMSE was −1.07%, and the lowest on CC was −1.99%. The lowest on Oppo Cost was 10.92%, and the highest was 55.45%. Regardless of the regression model, the percent improvement of EGO-ALRs in RMSE and CC was not less than −2%, but the percent improvement of Oppo Cost was more than 10%.Table 3
Percent improvements of the AUCs of the mean RMSEs, CCs, and Oppo Costs compared with each original ALR approach.
EGO-EMCMEGO-QBCEGO-GSxRD-EGORidgeRMSE−0.96−0.29−1.071.31CC−0.16−0.27−0.45−0.13Oppo Cost15.3713.4214.5754.37LassoRMSE−0.99−0.81−0.823.48CC1.20−0.36−0.48−0.90Oppo Cost22.1710.9216.3555.45EnetRMSE−0.70−0.68−0.75−0.88CC0.37−0.55−0.62−1.99Oppo Cost18.6011.8215.8254.40RD-EGO had the most significant improvement over RD; the Oppo Cost increased by 55.45% and the RMSE increased by more than 1% on both Ridge and Lasso. The improvement in the Oppo Cost of EGO-EMCM was second only to RD-EGO, and the CC of EGO-EMCM was also positive on the Lasso and Enet models.


## 4.6. Statistical Analysis

This section established the test groups that compared EGO-ALR with its original ALR, EGO, and BL to see if the differences in Oppo Cost between EGO-ALR and other approaches were statistically significant (EGO-ALR was used as the control approach). There were four EGO-ALRs in our work, so there were four test groups.First, the Friedman test was performed on 19 datasets, and the calculated statistic Ff (Table 4) was proposed by Iman and Davenport [39]. All calculated Ff values were always greater than the critical value F (3,54) = 2.78, which suggested that, regardless of which regression model was used, there were statistical differences among these approaches in each group.Table 4
Ff values of Friedman test on the AUCs of the Oppo Costs in four test groups (the critical value was 2.78 (α = 0.05)).
EGO-EMCMEGO-QBCEGO-GSxRD-EGORidge136.71217.4355.5194.81Lasso110.93—155.5194.81Enet37.4457.9112.7733.901The denominator is 0.After that, the post hoc test was performed to compare methods. The power of the post hoc test is greater when all methods are compared only to the control method and not between each other [40]. Thus, we used the Bonferroni–Dunn test as post hoc test. At q = 0.1, the critical difference (CD) for comparing four approaches on 19 datasets was 0.8880, and the visualization of the post hoc test is shown in Figure 4.Figure 4
Comparison of EGO-ALR against the others with the Bonferroni–Dunn test: (a) Ridge, (b) Lasso, and (c) Enet. All approaches with ranks outside the marked interval are significantly different (p < 0.1) from the control.
(a)(b)(c)Irrespective of the regression model, the opportunity cost of EGO-ALR was significantly better than that of the original ALR and BL. In addition, there was no significant difference in the average ranking between EGO-ALRs and EGO except for EGO-QBC on Lasso and Enet models. Thus, it is concluded that the performance improvement of the original ALR and BL by the EGO-ALR was statistically significant. However, the improvement was not significantly different from that of EGO.


## 4.7. Parameterc Sensitivity

EGO-ALR in Section3.1 has a parameter c, the weighted value of “information.” This section investigated the effect of c on the performance of the EGO-ALR. Figure 5 shows the normalized AUCs of EGO-ALR on Ridge, Lasso, and Enet when the parameter c ∈ [0, 5]. The corresponding results of EGO are also marked on the figure for comparison. Note that EGO-ALR is equivalent to the EGO when c = 0.Figure 5
Ratios of AUCs of the mean RMSEs, CCs, and Oppo Costs with differentc, average of 19 datasets, 100 runs, and different optimization directions.The performance of EGO-ALR improved asc increased, and performance converged after c = 3. In general, the result of c = 1 was closer to EGO; too much emphasis on optimization performance leads to a large loss of prediction performance. So, c = 1 is not a recommended value. c > 2 can maximize the optimization performance without excessive loss of prediction performance. When c > 2, the AUC-CC results can even outperform ALR (c ≥ 2 on the Lasso model, c ≥ 3 on the Enet model). The larger the c, the stronger the model prediction performance; the smaller the c, the stronger the model optimization performance. This result is in line with the respective sample selection characteristics of EGO and ALR. To find the balance between predicting and optimizing that makes the choice from the EGO-ALR algorithm more meaningful, c = 2 was chosen as the parameter of EGO-ALR.


## 4.8. Visualization of Sample Selection

This section explains the selection behavior of different approaches by the visualization of sample selection, to better visualize the advantages of EGO-ALR.Taking the results of modeling a typical dataset (HEA) using Ridge regression as an example, the study compared EGO, EMCM, EGO-EMCM, GSx, and EGO-GSx, including supervised ALR and unsupervised ALR. Figure6 shows the visualization of the sample selection results after iterating to the 20th round (the number of labeled samples is 25). The first 24 samples selected by an approach are marked in blue. The samples selected in the 20th round are marked in red. The black dotted line represents that the predicted value was equal to the actual value, and the red dotted line is the optimization goal of this experiment, that is, the minimum of Y.Figure 6
Visualization of sampling results from different approaches during active learning iteration to the 20th round.Driven by the global optimization strategy, EGO collected samples with lowY and hardly selected samples with high Y, which caused the overall sampling of EGO to be biased. Most of the unlabeled samples are above the black dotted line, which indicates that the predicted value of unlabeled samples was significantly lower than the actual value.EMCM-selected samples were distributed uniformly in the entire space. However, because the early sampling of EMCM was random and the subsequent sampling was small, the prediction of unlabeled samples in this experiment was also lower than the actual value. This situation would be improved after increasing the number of samples.GSx-selected samples were more uniform than EMCM and EGO samples, so the prediction results were significantly better. This selection strategy caused it to seldom focus on a cluster for sampling, and it was difficult to find the best value.The sampling of EGO-EMCM and EGO-GSx followed the original ALR while favoring lowerY clusters. EGO-EMCM and EGO-GSx not only ensured the prediction performance but also had more opportunities to select the best sample. It is further confirmed that the EGO-based ALR approach selects more reasonable samples than the original ALR, which results in better regression performances.


## 5. Conclusions

This study presents the EGO-ALR query strategy, which combines the ALR and EGO via weighted addition of normalization “information.” EGO-ALR combines the benefits of the two original approaches, speeding up the process of optimizing samples while also establishing a high-precision regression model. EGO-ALR circumvents the complexities of sample labeling and the impact of model performance on the accuracy of subsequent results. Furthermore, depending on the demand, EGO-ALR can vary the search direction of the ideal value. The study used multiple ALR approaches and conducted extensive experiments with 19 datasets in different domains. The performance of the EGO-ALR was significantly better than the original ALR as evaluated by RMSE, CC, and opportunity cost. Specifically, EGO-ALR increased the opportunity cost by an average of 25.27% when the RMSE and CC values were not more than 1.07% different from the original ALR. Whether combined with supervised or unsupervised ALR, EGO-ALR had strong adaptability. In addition, the EGO-ALR evaluation results on Ridge are similar to those on Lasso and Enet regression models, demonstrating the stability of this approach in the linear regression model. To make the results of EGO-ALR meaningful, the value range of the parameterc≥ 2 is recommended.As one of the future steps, EGO or more optimization algorithms can be combined with ALR approaches not mentioned in this report. Alternatively, the method of combined query strategy can be extended to a nonlinear regression model or classification problem. The single objective optimization problem in this paper can also be extended to multiobjective optimization problems.



---

*Source: 2891463-2022-06-09.xml*