Package: baggingrnet
baggingrnet: Library for Bagging of Deep Residual Neural NetworksIntroductionThis package provides The python Library for Bagging of Deep Residual Neural Networks (baggingrnet). Current version just supports the KERAS package of deep learning and will extend to the others in the future. The following functionaity is provoded in this package: * model multBagging: Major class to parallel bagging of autoencoder-based deep residual networks. You can setup its aruments for optimal effects. See the class and its member functions' help for details.
resAutoencoder: Major class of the base model of autoencoder-based deep residual network. See the specifics for its details. ensPrediction: Major class to ensemble predictions and optional evaluation for independent test.
* util pmetrics: main metrics including rsquare and rmse etc.data data: function to access two sample datas to test and demonstrate parallel training and predictions of multiple models by bagging. simData: function to simulate the dataset for a test.Installation of the packageYou can directly install this package using the following command for the latest version:pip install baggingrnetYou can also clone the repository and then install:git clone --recursive https://github.com/lspatial/baggingrnet.git
 pip install ./setup.py installModeling FrameworkThe modeling is based on bagging of the encoding-decoding antoencoder based deep residual multilayer percepton (MLP). Residual connections were used from the encoding to decoding layers to improve the learning efficiency and use of bagging is to achieve the stable and improved ensemble predictions, with uncertainty metric (standard deviation).The relevant paper will be published and will update here once published.Example 1: Regression of Simulated DataThe dataset is simulated using the following formula:each covariate defined as:x1∼U(1, 100),x2∼U(0, 100),x3∼U(1, 10),x4∼U(1, 100),x5∼U(9, 100),x6∼U(1, 1009),x7∼U(5, 300),x8U(6 ∼ 200)
This example is to illustrate how to use bagging class to train a model and compare the results by the models with and without use of residual connections in the models.1) Load the dataset:frombaggingrnet.dataimportdatasim_train=data('sim_train')sim_train['gindex']=np.array([iforiinrange(sim_train.shape[0])])knitr::kable(py$sim_train[c(1:5),],format="html")x1x2x3x4x5x6x7x8ygindex984269.598936.3686965.95072097.9769881.7767038.1257838.71023124.90578168.76974480251388.8358047.6193858.10734823.9538941.00300256.75319203.75759146.79040184.84722121911665.3266449.4736795.98241875.9940180.56275849.48435204.52137161.61705-444.53906462267321.7282764.9466802.59234870.3206742.27824387.4206013.1585288.47877-166.35536313560769.4531718.8116485.62437339.8183584.80446333.4381189.2259177.25155-0.54054264###### 2) Set bagging path, list of predictor names, get the bagging class instance and input data:# Load the major class for parallel bagging trainingfrombaggingrnet.model.baggingimportmultBaggingfeasList=['x'+str(i)foriinrange(1,9)]#List of the covariates used in trainingtarget='y'# Name of the target variablebagpath='/tmp/sim_bagging/res'# Path used tochkpath(bagpath)mbag=multBagging(bagpath)mbag.getInputSample(sim_train,feasList,None,'gindex',target)3) Define the arguments of a model and append it to the list of modeling duties:name=str(0)# model name as unique identifiernodes=[32,16,8,4]# List of number of nodes for the encoding and coding layers, adjustable optionally;minibatch=512# Size for mini batchisresidual=True# Whether to use residual connections in the modelnepoch=200#Number of epochessampling_fea=False# Whether to bootstrap the predictors/featuresnoutput=1# Number of the output nodeislog=False# Whether to make the log transformation# The following is to add the model's arguments to the list of duties.mbag.addTask(name,noutput,sampling_fea,nepoch,nodes,minibatch,isresidual,islog)4) Initiate the training:mbag.startMProcess(1)Here, just one core is used for one model.5) Prediction using the trained models and optional evaluation of the trained model:frombaggingrnet.model.baggingpreimportensPrediction# Load the test datasetsim_test=data('sim_test')sim_test['gindex']=np.array([iforiinrange(sim_test.shape[0])])# Generate the unique id for merging the predicitons of multiple models# Setup the path and target variableprepath="/tmp/sim_bagging/res_pre"chkpath(prepath)#Load the prdiction classmbagpre=ensPrediction(bagpath,prepath)#Load the test datambagpre.getInputSample(sim_test,feasList,'gindex')#Start to make predictions for multiple trained models.mbagpre.startMProcess(1)#Obtain the ensemble predictions from those of multiple models and optional evaluation of the models.mbagpre.aggPredict(isval=True,tfld='y')The above five steps illustrate the process of loading data, training, testing, and predicting. To compare with the results of residual models, the following code is to get the results for the non-residual models.mbag.removeTask(name)bagpath='/tmp/sim_bagging/nores'chkpath(bagpath)mbag_nores=multBagging(bagpath)mbag_nores.getInputSample(sim_train,feasList,None,'gindex','y')isresidual=False# This is to set no use of residual connections in the models.mbag_nores.addTask(name,noutput,sampling_fea,nepoch,nodes,minibatch,isresidual,islog)mbag_nores.startMProcess(1)prepath="/tmp/sim_bagging/nores_pre"chkpath(prepath)mbagpre=ensPrediction(bagpath,prepath)mbagpre.getInputSample(sim_test,feasList,'gindex')mbagpre.startMProcess(1)mbagpre.aggPredict(isval=True,tfld='y')The comparison of the training/learning curves for residual and non-residual models:The comparison of the independent test for residual and non-residual models: performance (R2 and RMSE)## [1] "non residual model   r2: 0.78, rmse: 150.17"

## [1] "residual model   r2: 0.91, rmse: 98.37"

## [1] "Residual model improved R2 by 12.48%, compared with non-residual model"

## [1] "Residual model decreased rmse by -51.8, compared with non-residual model"The scatter comparison of residual vs. non-residual models for the independent test:Example 2: Spatiotemporal Estimation of PM2.5This dataset is the real dataset of the 2015 PM2.5and the relevant covariates for the Beijing-Tianjin-Tangshan area. Due to data security reason, it has been added with small Gaussian noise.1) Load input data:Here the PM2.5dataset is used to test the proposed methods.frombaggingrnet.dataimportdatapm25_train=data('pm2.5_train')pm25_train['gindex']=np.array([iforiinrange(pm25_train.shape[0])])sitessite\_namecitylonlatpm25\_davgeleprstemrhuwinaod231231010A昌平镇北京116.230040.19526.8000057.01007.70920.08598520.760995217.394270.287737213391014A南口路天津117.193039.173084.590918.51021.859-0.28946220.656514140.612960.2245625118431062A铁路承德117.966440.916121.27273362.0969.87615.30923650.528807116.616830.42728319373榆垡京南榆垡，京南区域点北京116.300039.520012.0869618.01013.11614.00859740.810076839.460790.5075859195961069A环境监测监理中心廊坊116.715039.557164.2083335.01005.24924.49604990.860404714.010481.5149391###### 2) Set bagging path, list of predictor names, get the bagging class instance and input data:frombaggingrnet.model.baggingimportmultBaggingimportrandomasrfeasList=['lat','lon','ele','prs','tem','rhu','win','pblh_re','pre_re','o3_re','aod','merra2_re','haod','shaod','jd','lat2','lon2','latlon']target='pm25_avg_log'bagpath='/tmp/baggingpm25_2/res'chkpath(bagpath)mbag=multBagging(bagpath)## initializing ...mbag.getInputSample(pm25_train,feasList,None,'gindex',target)## (29475, 31)3) Define the arguments of multiple models (here 100 models) and append them to the list of modeling duties:importrandomasrforiinrange(1,81):name=str(i)nodes=[128+r.randint(-5,5),128+r.randint(-5,5),96,64,32,12]minibatch=2560+r.randint(-5,5)isresidual=Falsenepoch=120sampling_fea=Falsenoutput=1islog=Truembag.addTask(name,noutput,sampling_fea,nepoch,nodes,minibatch,isresidual,islog)4) Initiate the training:Initiate the parallel programs using 10 coresmbag.startMProcess(10)5) Prediction using the trained models and optional evaluation of the trained model:frombaggingrnet.model.baggingpreimportensPredictionprepath="/tmp/baggingpm25_2p/res"chkpath(prepath)mbagpre=ensPrediction(bagpath,prepath)mbagpre.getInputSample(pm25_test,feasList,'gindex')mbagpre.startMProcess(10)mbagpre.aggPredict(isval=True,tfld='pm25_davg')Finally, the following results were obtaned.The results are shown as the following:1) Typical learning curves of non-residual vs. residual models are shown as the following:2) Mean performance (R2 and RMSE) of the predictions of multiple non-residual vs residual models for the independent dataset :3) Performance (R2 and RMSE) of the ensembled predictions based on multiple models for the independent dataset:## [1] "non residual model   r2: 0.88, rmse: 23.55"

## [1] "residual model   r2: 0.91, rmse: 20.35"

## [1] "Residual model improved R2 by 2.97%, compared with non-residual model"

## [1] "Residual model decreased rmse by -3.2, compared with non-residual model"4) Scatter plots for the ensemble predictions of non-residual vs residual models:5) Comparison of ensemble predictions vs. predictions of single models:Statistics of the performance for the predictions of multiple models and ensemble predictions are made. The following shows R2and RMSE, barplots and scatter plots.Performance digits:## [1] "Ensemble predictions: R2=0.91, RMSE=20.35"

## [1] "Mean performance of predictions of multiple single models: R2=0.86, RMSE=26.07"

## [1] "Ensemble predictions averagely improved the single predictions by 6% for R2, and reduced -5.72ug/m3 for RMSE"The boxplot shows considerable improvement by bagging (6% in R2and -5.72μg/m3), in comparison with single models.The following shows the scatter plots of observed PM2.5vs. ensemble predictions/residuals:ContactFor this library and its relevant complete applications, welcome to contact Dr. Lianfa Li. Email:lspatial@gmail.comorlilf@lreis.ac.cn