Machine Learning on AWS SageMaker | by Mahsa Mir | Towards Data Science

In this tutorial, we’re going to create an ML model on AWS SageMaker using Amazon SageMaker Studio to predict employee salary based on years of experience.
Imagine a potential employee who has passed the interview process and now HR want to make them an offer. What would the salary be?
Assumptions: The only factor that has an effect on salary is years of experience.
In this tutorial, I’ll come up with a simple linear regression model. You feed years of experience in this model and it will make an according salary prediction.
Note: You can find the full explanation of linear regression in my previous post TK here.
Data Used: Kaggle-Salary-Data
To make this more exciting, we will build, train and deploy the ML model on Amazon SageMaker — yay!
Sagemaker Studio a fully integrated development environment (IDE) for Machine Learning, that allows us to write code, track experiments, visualize data, and perform debugging.
Follow along:
From your AWS management Console search bar find SageMaker Service.
Click on Amazon SageMaker Studio.
Hit the + icon on the top left and launch your Jupyter notebook.
Predict the employee salary based on the number of years of experience.
We’re using the simple linear regression model: Predict the Salary(variable Y, dependent) based on years of experience(variable X, independent).
#import required librariesimport pandas as pdimport numpy as npimport seaborn as snsimport matplotlib.pyplot as plt#if you need to install some packages, use this command:!pip install tensorflow#read the Datasetsalary_df = pd.read_csv('salary.csv')salary_df.head(10)salary_df.tail(10)
#get some information about our DataSetsalary_df.info()salary_df.describe()#check if there are any Null valuessalary_df.isnull().sum()
#visualizing datasns.pairplot(salary_df)sns.heatmap(salary_df.corr(), annot=True)sns.regplot(salary_df['YearsExperience'], salary_df['Salary'], data = salary_df )
We use the Train set to make the algorithm learn the data’s behavior and then we check the accuracy of our model on the Test set.
Features(X): the columns that are inserted into our model will be used to make predictions.
Prediction(y): target variable that will be predicted by the features
#define X variables and our target(y)X = salary_df[['YearsExperience']]y = salary_df[['Salary']]#splitting Train and Test from sklearn.model_selection import train_test_splitX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=0)#need to have y_train & y_test as an vector(for SageMaker linear learner) y_train = y_train[:,0]y_test = y_test[:,0]
Before we jump into this, let’s explain what we need to have in place — I’ll be quick, promise!
Amazon S3
Amazon S3 is a storage service allowing us to store and protect our data in directories (Buckets). We will need this service to go forward
Buckets: is a container for objects stored in Amazon S3 and each bucket has global unique name.
Create bucket
First search for S3 service.
Select create a bucket and give it a name and region.
Hit next till you get to configure options, you will block all public access and create your bucket.
Create prefix (subfolder within the bucket)
Within the created bucket, hit “create folder” and give it a name.
So, we have:
bucket: sagemaker-practical-m
prefix: linear-learner
IAM (Identity and Access Management)
We will need to give an IAM role to our Sagemaker so it can perform tasks on our behalf (i.e. reading training results, calling model artifacts from the S3 bucket, and writing training results to the S3)
Let’s get back to our code:
#import required kitsimport sagemakerimport boto3     #create a Sagemaker sessionsagemaker_session = sagemaker.Session()#define the S3 bucket and prefix(subfolder within the bucket)bucket = 'sagemaker-practical-m'prefix = 'linear-learner' #define IAM rolerole = sagemaker.get_execution_role()print(role)
Boto3 an AWS Python SDK that allows developers to write apps that make use of services like Amazon S3 or EC2.
Let’s convert our training data into a Sagemaker compatible input format (RecordIO). More info on this here.
#import module in terms of dealing with various types of I/Oimport io#import sagemaker common libraryimport sagemaker.amazon.common as smac #converts the data in numpy array format to RecordIO formatbuf = io.BytesIO()smac.write_numpy_to_dense_tensor(buf, X_train, y_train)#reset in-memory byte arrays to zerobuf.seek(0)
The final command is used to clear the in-memory buffer.
After our data has been converted into a RecordIO format and contained within the buffer. We can now upload our data to S3:
#import moduleimport os #Key refers to the name of the file    key = 'linear-train-data'#uploads the data in record-io format to S3 bucketboto3.resource('s3').Bucket(bucket).Object(os.path.join(prefix, 'train', key)).upload_fileobj(buf)#training data location in s3s3_train_data = 's3://{}/{}/train/{}'.format(bucket, prefix, key)print('uploaded training data location: {}'.format(s3_train_data))
We just created the folder “train” which contains our uploaded data (linear-train-data).
Let’s get back to S3 storage and check if the file is there:
We need to specify the S3 location to store our linear learner output:
#create an output location in S3 bucket to store the linear learner outputoutput_location = 's3://{}/{}/output'.format(bucket, prefix)print('Training artifacts will be uploaded to: {}'.format(output_location))
Now, to obtain the reference to the linear Learner container, just specify the algorithm name:
#specify the name of the algorithm, that we want to usefrom sagemaker.amazon.amazon_estimator import get_image_uricontainer = get_image_uri(boto3.Session().region_name, 'linear-learner')
We then pass the container and type of instance we want to use for training. The output path and SageMaker session variables have already been defined.
linear = sagemaker.estimator.Estimator(container,          role,           train_instance_count = 1,           train_instance_type = 'ml.c4.xlarge',          output_path = output_location,          sagemaker_session = sagemaker_session)
These parameters can be tuned :
Number of features passing in.
Type of predictor (regressor or classifier).
Mini batch size (the number of observations per mini-batch).
#Train 32 different versions of the model and will get the best out of themlinear.set_hyperparameters(feature_dim = 1,                           predictor_type = 'regressor',                           mini_batch_size = 5,                           epochs = 5,                           num_models = 32,                           loss = 'absolute_loss')#pass in the training data from S3 to train the linear learner modellinear.fit({'train': s3_train_data})
linear_regressor = linear.deploy(initial_instance_count = 1,                                 instance_type = 'ml.m4.xlarge')
To make inferences in our model, we need to convert the data into text/CSV format. Read more here.
from sagemaker.predictor import csv_serializer, json_deserializerlinear_regressor.content_type = 'text/csv'linear_regressor.serializer = csv_serializerlinear_regressor.deserializer = json_deserializer
Prediction on testing set:
# making prediction on the test dataresult = linear_regressor.predict(X_test)result
Let’s take out the result number and save it in the Prediction plate variable:
#access the scores by iterating through the scores in predictionspredictions = np.array([r['score'] for r in result['predictions']])predictions
Let’s check the result as a DataFrame:
#compare actual output values with predicted valuesdf = pd.DataFrame({'Actual': y_test.flatten(), 'Predicted': predictions.flatten()})df
# visualize comparison result as a bar graphdf1 = df.head(20)df1.plot(kind='bar',figsize=(12,8))plt.grid(which='major', linestyle='-', linewidth='0.5', color='green')plt.grid(which='minor', linestyle=':', linewidth='0.5', color='black')plt.show()
#prediction vs test setplt.scatter(X_test, y_test,  color='blue')plt.plot(X_test, predictions, color='red', linewidth=2)plt.xlabel('Years of Experience (Testing Dataset)')plt.ylabel('salary')plt.title('Salary vs. Years of Experience')plt.show()
That’s pretty much it!
IMPORTANT: Always remember to delete the endpoint if you don’t want to be charged by Amazon!
# Delete the end-pointlinear_regressor.delete_endpoint()
I hope this article has helped you understand the topic a little better. Any feedback is welcome as it allows me to get new insights!