How to build and use custom python packages for data science project | by Priya Brata Sen | Towards Data Science

In data science projects, we use similar steps depending on the nature of the problem. Usually, a data science project consists of problem identification, data collection and preparation, feature engineering, training and testing model, and deployment. In these steps, several functions are used. A data scientist needs to use those functions over and over again in different projects. For example, a data scientist imports different models to train and then fits the model with training data and then predicts based on the fitted model. To evaluate the model, he/she needs to use some form of evaluation metrics also such as Mean Absolute Percentage Error(MAPE). Therefore, if the codes used repeatedly can be saved somewhere and reused all the time, it will save a lot of time and make the code much cleaner than last time. This is where building custom packages can help a data scientist create reusable codes.
Package and Module:
Package in python is a collection of modules. On the other hand, The python module is a file that consists of python statements and definitions. Python has its own packages which we import and use all the time in our code. Similarly, we can create our own modules and save them as a package so that we can use them over and over again by importing them into our code. This can help to manage codes better and to increase the reusability of the codes. This post will show how we can create a custom package that consists of one module and use it in our regression problem.
Building our custom package:
In this problem, we will use xgboost and random forest regression models to train the model for demonstration purposes. We will then evaluate the model accuracy using mean absolute percentage error (MAPE). Finally, we will try to optimize the accuracy using grid search for both the model. Therefore, we will create all those functions to train and evaluate models as well as to optimize the hyper-parameters of the models. We will put all those functions in a python file which will be our module and then we will create a package by inserting that module. We can keep adding modules in that package going forward. Below are the functions that we will add to our module/ python file.
import xgboost as xgbfrom sklearn.ensemble import RandomForestRegressorfrom sklearn.metrics import mean_absolute_percentage_error#functions to train random forest regression modeldef model_rf(x_train,y_train): #Function fits the random forest regression    model = RandomForestRegressor(random_state=0)    return model.fit(x_train,y_train)#functions to train xgboost regression modeldef model_xgb(x_train,y_train):  #Function fits the xgboost regressor      model=xgb.XGBRegressor(objective ='reg:squarederror')    return model.fit(x_train,y_train)#functions to calculate MAPE valuedef result(model,x_test,y_true): #Funtion takes a fitted model and predicts based on x_test values and calculates the MAPE    y_pred=model.predict(x_test)    mape=mean_absolute_percentage_error(y_true, y_pred)*100    return mape#functions to do grid search to optimize model hyper-parameters def optimization(model,params,cv,x_train,y_train): # setup the grid search    grid_search = GridSearchCV(model,                           param_grid=params,                           cv=cv,                           verbose=1,                           n_jobs=1,                           return_train_score=True)    return grid_search.fit(x_train, y_train)
From the above code, we can see that we imported all the necessary libraries and then created 4 functions. The model_rf function creates a random forest regression object and then fits the training data. model_xgb function creates a xgboost regression object and then fits training data. The result function takes a fitted model, x_test, and y_true value. It predicts the y values using x_test and then calculates the MAPE comparing y_true and prediction value.
Let’s save the file with the name module.py and put it in a folder named my_module. We can use any name we want.
Let’s now create a blank python file and name it as __init__.py. So, this is very important. Python will read a folder as a package when it has an ‘__init__.py’ file in it. So, we create the file and save it in the same folder my_module.
This is all we need. Now, we want to put the folder in our python directory so that every time we need it, we can import it as a built-in python package. So, we need to go to our directory and put our my_module folder in the site-packages folder. I am using anaconda3. So, I will go to that folder and then find my site packages for anaconda distribution inside the lib folder as shown below.
After that, I just pasted my my_module folder inside the site-packages folder. As soon as, I dropped that I can now import the my_module as a package inside my IDE. After that, I can use all those functions in the module created inside the my_module package.
In the above picture, we can see that I have imported a module from the my_module package inside my Jupiter notebook. After that, I called the help function to see what we have inside the module. We can see that it is showing all the four function that is saved inside module file. It also shows the path where we have that module on our computer. Now, we can use these functions in our problem again and again instead of writing exclusively or copy-pasting. We can create more functions for further usage inside our module file or we can create different module files inside our my_module package. Let’s use our custom package to solve a machine learning problem
Problem:
Zoo management is trying to understand the number of llamas that will be available in different habitats. The projection will help them to plan for the foods so that they can feed the llamas properly. We have 20 months of historical data on the number of llamas in different habitats and also some weather features. The data files can be found in the following link
github.com
Data Preprocessing:
In this step, we just import the data and do some preprocessing to prepare the data for our machine learning model.
#Importing some packagesimport numpy as npimport pandas as pdimport matplotlib.pyplot as plt import seaborn as sns import datetime as dtfrom my_module import module# Importing the two files and merging them togetherdf1=pd.read_csv('llama_forecast_train.csv')df2=pd.read_csv('historical_weather.csv')df2.rename(columns={'HOUR':'DAY'},inplace='True')df1=pd.merge(df1,df2,on=['DAY'], how= "left")#Breaking the date time variablesdf1['Date']=pd.to_datetime(df1['DAY'])df1.sort_values(by='Date',ascending=True,inplace=True)df1['Year']=df1['Date'].dt.yeardf1['Month']=df1['Date'].dt.monthdf1['Week']=df1['Date'].dt.weekdf1['Day']=df1['Date'].dt.daydf1['Week Day']=df1['Date'].dt.dayofweekdf1['Year Day']=df1['Date'].dt.dayofyeardf1.drop(['Date','DAY'],axis=1,inplace=True)#Creating dummies for the habitat names becasue it is a qualitative variabledf3=pd.concat([df1, pd.get_dummies(df1[['HABITAT NAME']])], axis=1).drop(['HABITAT NAME'],axis=1)df3=df3.reset_index(drop=True)df3.head()
Model Building:
Now, we split the data into train and test. In the second section, we fitted both the random forest and xgboost regression models with our training data. We can see here that we called model_rf and model_xgb functions from our module which we imported from our my_module custom package. There is no need to create the regression object and then fit them with training data. The imported function will do this job
# Splitting data in tarin test splity=df3['AVAILABLE LLAMAS']x=df3.drop(['AVAILABLE LLAMAS'],axis=1)b=int(len(df1)*.75)x_train, x_test=x.iloc[0:b, :], x.iloc[b:-1, :]y_train, y_test=y.iloc[0:b], y.iloc[b:-1]# train both random forest and xgboost modelrf=module.model_rf(x_train,y_train)xgboost=module.model_xgb(x_train,y_train)
Model Evaluation:
Let’s calculate the MAPE for both the random forest and the xgboost model using the result function from our custom package. We just pass the model and the test values for both x and y
print('MAPE for random forest:',module.result(rf, x_test, y_test))print('MAPE for xgboost:',module.result(xgboost, x_test, y_test))
We can see the MAPE for the random forest is 20.01% and xgboost is 20.07%. Let’s use grid search to improve the results.
Hyper-parameter Optimization:
In this section, we will use the optimization function from our module to tune hyperparameters for both models.
#Splitting data for hyperparameter tuningtss=TimeSeriesSplit(n_splits=3)# define the grid for the xgboostparams_xgb = {    "learning_rate": [0.01, 0.1],    "max_depth": [2, 4, 6],    "n_estimators": [10, 100,150],    "subsample": [0.8, 1],    "min_child_weight": [1, 3],    "reg_lambda": [1, 3],    "reg_alpha": [1, 3]}# define the grid random forestparams_rf = {    "n_estimators": [50, 100, 150],    "max_features": [0.33, 0.66, 1.0],    "min_samples_split": [2, 8, 14],    "min_samples_leaf": [1, 5, 10],    "bootstrap": [True, False]}# fit the new models with the optimized hyper-parametersgrid_xgb=module.optimization(xgboost, params_xgb, tss, x_train, y_train)grid_rf=module.optimization(rf, params_rf, tss, x_train, y_train)
In the above block, we first created two-parameter grids. These will be passed in the param_grid parameter of our grid search function. Finally, we call our optimization function which mainly calculates the best values for the hyperparameters of the models and fits the refined models. Again, we are able to avoid writing long code for the hyperparameter tuning functions. We just used the function from our module. Below are the results of the tuned model
print('MAPE for the tuned xgboost model:',module.result(grid_xgb, x_test, y_test))print('MAPE for the tuned random forest model:',module.result(grid_rf, x_test, y_test))
Conclusion:
From the above example, we can see that how easily we can create our own python data science packages and use them in our projects repeatedly. This process will also help to keep your code clean and save a lot of time. It is very important to place the custom package inside the site package of the python directory so that we can use our custom package just like the built-in packages in python.