TITLE: Protocol for Downscaling Satellite Soil Moisture Estimates using Geomorphometry and Machine Learning
AUTHORS: Mario Guevara, Rodrigo Vargas

[DESCRIPTION]
<div class = "text-blocks"><div class = "text-block">This protocol is composed of five steps for downscaling satellite soil moisture estimates using digital terrain parameters derived from a digital elevation models. We provide an alternative approach to predict soil moisture spatial patterns at higher spatial resolution (compared with current satellite soil moisture estimates) across areas where no information is otherwise available. This approach relies on geomorphometry derived terrain parameters and machine learning models to improve the statistical accuracy and the spatial resolution (from 27km to 1km grids) of satellite soil moisture information. This approach has been tested for this study across the conterminous United States on an annual basis (1991-2016). </div><div class = "text-block"><ul style = "list-style-type:disc;"><li style = "counter-reset:ol0;list-style-type:disc;">We first derived 15 primary and secondary terrain parameters from a digital elevation model. </li><li style = "counter-reset:ol0;list-style-type:disc;">We trained a machine learning algorithm (i.e., kernel weighted nearest neighbors) for each year. </li><li style = "counter-reset:ol0;list-style-type:disc;">Terrain parameters were used as predictors and annual satellite soil moisture estimates were used to train the models. </li><li style = "counter-reset:ol0;list-style-type:disc;">We validate the models using cross-validation strategies and independent validation. </li></ul></div><div class = "text-block">The explained variance for all models-years was >70% (10-fold cross-validation). The 1km soil moisture grids (compared to the original satellite soil moisture estimates) had higher correlations with field soil moisture observations from the North American Soil Moisture Database (n=668 locations with available data between 1991-2013; 0-5cm depth) than the original product. We conclude that the fusion of geomorphometry methods and satellite soil moisture estimates is useful to increase the spatial resolution and accuracy of satellite-derived soil moisture. This approach can be applied to other satellite-derived soil moisture estimates and regions across the world.</div></div>

[STEPS]
?. PREPARE PREDICTION FACTORS: This first stetp is to prepare the prediction factors. Once R is installed and running, please load the library raster and import the DEM terrain parameters. This example is using a geographical coordinate system (lat, long) in degrees ('+proj=longlat +datum=WGS84 +no_defs'). We will consider (as in this case) that the topographic prediction factors are in a specified path in the local computer (e.g., 'path/to/topographic/files') and that all of them (n=15) have a SAGA-GIS native *.sdat format (these can be *.tif, *.asc or any other generic raster files).  If you have your own set of prediction factors in the same projection system and pixel size, reproject (if needed) to a geographical coordinate system (because that is the projection of the ESA-CCI satellite soil moisture product that we are using for testing this protocol), convert to a 'SpatialPixelDataFrame' object and go to step 2.  ##library to manage raster fileslibrary(raster)##path to prediction factors (e.g., a folder with *.tif predictors)lis1 ##object with the names of the predictor raster files #(can be *.tif or *.asc, or *.img files)lis2 ##make a raster stack of the filesx #function to remove NAsNA2mean #convert to spatial pixels dataframe (this will take overnight on conventional systems, but will work)x=as(x,'SpatialPixelsDataFrame')#for global downscaling use this line#x=as(s,'SpatialPixelsDataFrame')#transform NAs to median values of layerx@data[] The resulting object x (from this step) contains the prediction factors used as the basis of the downscaling process. This object containing all terrain parameters is an internal R format tyme known as 'SpatialPixelsDataFrame'.
?. PREPARE RESPONSE VARIABLE: The next step is about getting and importing the soil moisture dataset that is going to be downscaled to a finer pixel size. For our paper we used the European Space Agency-Climate Change Initiative soil moisture product (ESA-CCI). These data sets are provided here: https://www.esa-soilmoisture-cci.org/ . These data is provided yearly folders and in *.nc files (one file per day) and we can import these files in R using the following code for listing the folders with the satellite soil moisture data. Download the satellite soil moisture maps of your convenience and place them in your working directory. #list.dirs is a function used for generating a list of the annual soil moisture #folders as provided by the ESA-CCI initiativedirs #we can start a loop but first test with a yearly mean or median first#test with i=1 before the loop#i=1#for (i in 1:26){#if you want it to make it daily use 1:366, #1:26 means 26 years between 1991 and 2016#stack soil moisture records for a given folder/year#in this case i = 1 = year 1991#change the nc extention if your satellite data is in another formatr #crop, if needed, to area of interes using an extent #this is the extent of the prediction factors#consider that satellite soil moisture has 27km pixels, select at least 30 - 50 of #this pixels to build the models. Higher pixels for training models higher accuracy.  r #calculate median year value from daily estimates, #avoid this line if you want to go in day or month basisr The resulting raster object r will be harmonized with the result of step 1 in order to build a regression matrix for soil moisture. In this case r is the mean value of soil moisture estimates avaialable in the ESA-CCI product for the year 1991.
?. REGRESSION MATRIX: Next step is building a regresion matrix. This is a data frame with columns for coordinates, colum for soil moisture data and columns of terrain parameters. This regression matrix will be used for building a model in the next step. #define lat long projectionproj4string(r) #convert to data framedf=as.data.frame(r, xy=T)#remove no data valuesdf=na.omit(df)#define column coordinates coordinates(df)=~x+y#and lat long projection systemproj4string(df) #overlay soil moisture centroids and prediction factors (x)ov=over(df, x)#generate a data framed=as.data.frame(df)#combine extracted valuesy=cbind(d,  ov)#remove no data valuesy=na.omit(y)#training set year iz=data.frame(z=y[,3],y[,1:2],  y[,4:18])#print dimensions of the regression matrixprint(dim(z))ion (correlation obs vs mod [cd] and RMSE as well as N) in a data frame named results#empty dataframe to store resultsresultsThe resulting object is a data frame z, with the coordinate colums of centroids of ESA-CCI pixels and the corresponding values for those coordinates of the terrain parameters. This regression is used for building models and generating digital soil moisture maps. We additionally generate an empty data frame results for saving the modeling results (cross validation accuracy).
?. MODELING SOIL MOISTURE: Next we build a prediction model. We used as in our paper a kernel based machine learning model, but it can be any model able to account for non linear relationships see methos section of our paper. We start loading the required library kknn. The 'optimal' model parameters k and the kernel function will be stored in the previously generated results object. In this results object we will also save the accuracy indicators (cor and rmse) indicating the correlation between observed and predicted and the corresponding root mean squared error. This accuracy metrics are derived from a 10-fold cross validation strategy. library(kknn)#you must select the best parameters by tunning them with CV,  the parameter K and the parameter kernel###k between 1 and 50 kmax=50#find optimal k and kernel type via 10 fold cross validationknnTuning kernel = c("rectangular", "triangular", "epanechnikov","gaussian", "rank", "optimal"),           ykernel = NULL, scale = TRUE,kcv=10)#identify best kerneln#store best parametersmejoresresultados #lowest rmsermse #highest correlationcd #modeling year (1991-2016)year=1990+i#prepare best kernelas.character(unlist(knnTuning$best.parameters[1]))#prepare best kk=as.numeric(knnTuning$best.parameters[2])#store results for year i in the data frame results[i,1]results[i,2]results[i,3]results[i,4]results[i,5]results[i,6]#print accuracy results on screen print(results)The resulting object of this section is a model that we can use to generate predictions. This model is cross validated and the accuracy report is in the object results. In this example we predict across all CONUS using 1km grids.
?. PREDICTION ACROSS ALL AREA OF INTEREST: This is the final step. We apply the coefficients of the model (for each day or year) to all the prediction domain (defined by the terrain parameters). Thus we obtain a soil prediction for each pixel across the study area. These prediction values are acoupled with the coordinates of the center of each pixel and a digital soil moisture map is generated. ##build a model with best parameters, predict to all CONUS and make a map of itmejorKKNN                         scale=TRUE,k=as.numeric(knnTuning$best.parameters[2]))#add fitted values to the prediction domain x$kknn=mejorKKNN$fitted.values#make a raster of the predictionr #visualize the map #plot(r)#proj4string(r) #save resultswriteRaster(r, file=paste0('predicted-', 1990+i, '-esa-sm-topo-CONUS.tif'), overwrite=TRUE) ### change the name of the output files if necessary#print accuracy results on screenprint(results)#save accuracy resultswrite.csv(results, file=paste0('accuracy-', 1990+i, '-esa-sm-topo-CONUS.csv')) The resulting objects from this step are a data frame results with the accuracy report of the model for each year, and the object r, a raster file containing the predictions of soil moisture across the study area. These objects are going to be saved in the working directory as a *.tif file for the raster object and in a *.csv for the results file.