Xgboost with AWS Sagemaker 2.20 ‘Debugger’ | by Philippe Bouaziz,Data Scientist at Africa4Data,PhD | Towards Data Science

Article Co-author with : @bonnefoypy , CEO at Olexya.
Due to the high quantity of data, finding tricks for faster analysis using automatizations library is a key advantage for becoming a unicorn data scientist. AWS sagemaker offers various tools for developing machine and deep learning models in few lines of code. The process of machine learning is quite challenging for any data scientist from handling missing data, coding data types, calculate associations-correlations between features, shaping data structures, choosing the right models, and make a cost-efficient model deployment for inference. In this article, we will review the last version (Sagemaker 2.20) of AWS ‘debugger’ with the well-known XGboost algorithm to generate a first-class data science analysis report using few python code lines.
For this article, we will be analyzing the adult-diabetes dataset you can read with this code :
import pandas as pd   data = pd.read_csv('diabetes.csv') data   X_display = data.drop('Diabetes', axis = 1) y_display= data['Diabetic']display(X_display.describe())hist = X_display.hist(bins=30, sharey=True, figsize=(20, 10))
We after split the data into train, validation and test set and save it to our AWS S3 data-storage with the code below (for a complete analysis check the complete code at ...) :
import sagemaker, boto3, osbucket = sagemaker.Session().default_bucket()prefix = "demo-sagemaker-xgboost-diabetes -prediction"boto3.Session().resource('s3').Bucket(bucket).Object(os.path.join(prefix, 'data/train.csv')).upload_file('train.csv')boto3.Session().resource('s3').Bucket(bucket).Object(os.path.join(prefix, 'data/validation.csv')).upload_file('validation.csv')
Output :
AWS Region: eu-west-3RoleArn: arn:aws:iam::61xxxxxx :role/service-role/AmazonSageMaker-ExecutionRole-eu-west-3-philippe-bouaziz
AWS Region: eu-west-3
RoleArn: arn:aws:iam::61xxxxxx :role/service-role/AmazonSageMaker-ExecutionRole-eu-west-3-philippe-bouaziz
from sagemaker.debugger import Rule, rule_configsfrom sagemaker.session import TrainingInputs3_output_location='s3://{}/{}/{}'.format(bucket, prefix, 'xgboost_model')container=sagemaker.image_uris.retrieve("xgboost", region, "1.2-1")print(container)
Output :
61xxxxxx.dkr.ecr.eu-west-3.amazonaws.com/sagemaker-xgboost:1.2-1
We can easily personalize our model with various parameters and rules as shown in the code below:
xgb_model=sagemaker.estimator.Estimator(image_uri=container,role=role,instance_count=1,instance_type='ml.m5.large',volume_size=5,output_path=s3_output_location,sagemaker_session=sagemaker.Session(),rules=[Rule.sagemaker(rule_configs.create_xgboost_report())])
The set of hyperparameters can be done in one step:
xgb_model.set_hyperparameters(max_depth = 5,eta = 0.2,gamma = 4,min_child_weight = 6,subsample = 0.7,objective = "binary:logistic",num_round = 1000)
And finally, we can fit our model easily:
xgb_model.fit({"train": train_input, "validation": validation_input}, wait=True)
Output :
The majority of data scientists claim the real work is starting in this step by iterations, finding the best hyperparameters is a key to deploy the best model. A clear data science analysis report can be an important transition to successful results. AWS sagemaker debugger respond to this need with 2 reports in few lines of code as shown below:
from IPython.display import FileLink, FileLinksdisplay("Click link below to view the XGBoost Training report",FileLink("CreateXgboostReport/xgboost_report.html"))profiler_report_name = [rule["RuleConfigurationName"]for rule in xgb_model.latest_training_job.rule_job_summary()if "Profiler" in rule["RuleConfigurationName"]][0]profiler_report_namedisplay("Click link below to view the profiler report",FileLink(profiler_report_name+"/profiler-output/profiler-report.html"))xgb_model.model_data
XGboost profiler report
The profiler report shows statistics of resource utilization per worker (node), such as the total CPU and GPU utilization, and the memory utilization on CPU and GPU. The table shows also descriptive statistics of the data including min and max values as well as p99, p90, and p50 percentiles.
The second part of the report demonstrates a full profiling summary of the Debugger built-in rules used during your training job. The best parameters including to quote a few: the batch size, outliers, data loader configurations in a clear display table suggesting future cost-efficient improvements in the dataflow.
The report also analyzed the number of data-loading processes that have been running in parallel on the training instance and compares it against the total number of cores. If the number is less than 70% it can slow down the data processing leading to higher expenses, if the number is larger than 200% the total number of cores may reduce the performance of the model on CPU.
Refer to this github for more details.
XGBoost Training Report
This report provides a summary of the XGBoost model training evaluation results, insights of the model performance, and interactive graphs. During the creation of the report, outputs including plots are automatically saved to S3 as write in the code below:
# Parameterspath = "/opt/ml/processing/input/tensors"plot_step = 995s3_path = "s3://sagemaker-eu-west-3-xxxx/demo-sagemaker-xgboost-adult-diabetes-prediction/xgboost_model/sagemaker-xgboost-2021-06-04-08-56-51-612/debug-output"
The report includes the histogram of the true (target) labels of our dataset. You can see if the distribution of the target label is imbalanced (for classification) suggesting correcting our dataset by generating synthetic samples, resampling, or collecting more data. The distribution shows also the left or right skewness (for regression) of our dataset suggesting using numpy.log or Numpy. exp to reduce the skewness of our data.
Another important plot of this report is Feature importance. This method assigns scores to input features on how useful they are in predicting the label variable. For our Xgboost algorithm 3 parameters are taking under consideration:
· Weight (or Frequency) is the percentage of the relative feature that occurs in the model.
· Gain is the most relevant attribute to interpret the relative importance of each feature.
· Coverage measures the number of observations by features.
SageMaker Debugger also describes the loss values curves of the XGBoost Learning Task Parameters, for more details please referred to this link.
These loss curves give us valuable insights. If the model overfits our model fits too much of the training data which results in negative model estimation on the validation set. If the model under fits, the model parameters are not set up properly.
For Xgboost, if the model under fits you might want to tune the parameters by increasing the depth of your trees (max_depth), decreasing the regularizations parameters (lambda, alpha), or even reduce the gamma and eta parameters. If the model overfits you may want to reduce the depth of each tree, max_depth, increase the min_child_weight or even increase other valuable parameters such as gamma and eta, lambda and alpha, subsample, and colsample_bytree.
Another key element of the report is evaluating performance metrics, such as Confusion Matrix, Precision, Recall, F-score, and Operating Characteristic curves. As shown below:
The following Confusion Matrix shows the classification results. With the True Positive counts on the diagonal cells and the number of misclassified predictions at the off-diagonal cells.
A complete summary of key components of the performance model score metrics including accuracy, precision, recall, and F1-score also provides using the Scikit-learn Metrics and Scoring APIs as shown below :
The Receiver Operating Characteristic curve shows the ratio of TPR over FPR and the Area Under Curve (AUC) for the evaluation of binary classification. If the AUC value is close to 1 the model performance is optimal otherwise if the AUC is less than 0.6 it means your model underperforms and needs further parameters tunning or even changing model algorithms.
Finally, the report shows the distribution of residuals histogram . For regression, the histogram should show a normal distribution for a good model training at a value of zero. For classification, the residuals show the ratios of correct prediction counts by the number of wrong prediction counts.
In conclusion, AWS sagemaker debugger generates profile reports from a panda dataframe easily giving essential insights for finding model best parameters with optimal technical conditions for model training which is an essential step for efficient model training and deployment.
This article is a review of AWS Xgboost Debugger with another dataset for education purposes only, the codes used had been modified (charts sizes, colors, quality, formats ) for design purposes only . For more about SageMaker Debugger, see SageMaker Debugger XGBoost, Training Report and SageMaker Debugger Profiling Report.