Intro to Machine Learning🖥️ & Azure Notebooks📑
By: Vijayakumar Pattanashetti
Tags: Data Science, Supervised Learning, Machine Learning, Microsoft Azure, Sklearn

Let’s us now start with coding 😍.

Step 1: If you remember, our first step is to collect the data & visualize. So, let’s execute the below command to collect data, flightdata.csv.

Now with the help of pandas library, we will try to read the data in the dataset & explore by storing as a dataframe, 2D data array.

The raw data that is not preprocessed/cleaned is as useless as waste in a bin.

Step 2: We will now try to clean the data & prepare it for training.

#1. It’s good to know and remove missing values.

Here, the 4th code line checks for null/missing values, while 5th code line tells the total number of missing/null values per column.

Interestingly, you may observe that “Unnamed: 25 column” has all it’s data missing, that is because it previously had commas which were omitted. This resulted in missing values.

So, we will delete/drop that particular column.

Step 3: We now select only desired columns which we feel have some impact on prediction.

We can see that only one column has missing values, so we reduced our burden 😀.

What actually is there in the column, ARR_DEL15? Let us check.

#2. Oh, NaN! It means “Not a Number”. But, how to resolve that🤔. Will replace with “1” if there is delay, else “0”.

Still there are two more problems. Which are they? Airport IDs(letters), and Departure time(huge numbers).

#3. So, we will bin or quantize the Departure time column by dividing entire column by 100.

What to do with Airport IDs? #4. We will replace with “dummies/dummy” variables as “1” if the flight originated/departed from that particular airport, else “0”.

Huh, done with data part🥳.

What’s next?

Step 4: Till now we have prepared the data, but not selected the features used for training(x) and the feature to be predicted(y). We will do this now. Also, to train any model and avoid overfitting(100% accuracy only over the training dataset), we will be splitting the dataset into two parts viz., training data & testing data.

You may observe that we dropped the data column to be predicted from training data and included as label/to be predicted.

Step 5: Now it’s time to select/build the model. We will be using sklearn library, which has inbuilt callable functions for classic ML algorithms such as Linear Regression, Logistics Regression, Random Forests, KNN, Decision Trees, etc... In our case here, we will be using Random Forests Classifier.

Step 6: We then train the model by calling fit function.

Need not worry, if arguments wonder you. Because they are by default, you should bother if you would like to change any of them.

Step 7: Let’s score & evaluate our model over test data.

You may observe that we used different metrics to evaluate model performance viz., accuracy, precision, recall, ROC AUC, confusion matrix. Refer here for more details about the metrics.

We will also plot ROC curve for better understanding. For this we make use of matplotlib & seaborn libraries, and use roc_curve function from metrics in sklearn.

Step 8: Tune Hyperparameters i.e., arguments of the function, RandomForestClassifier which we saw earlier for better model performance.

Step 9: To deploy a working model, either we need to pack it as an API or define as a callable function. We will define the whole model as callable function, predict_delay which takes departure date & time, origin airport id and destination airport id as inputs and gives the probability of one-time arrival of the flight.

Let’s test the function.

Hurrah! It works cool🥳.