Data Science Project | Will It Rain Tomorrow?
By: Mazen Ahmed
Tags: Logistic Regression, AI, Statistics, Data Science, Machine Learning

2. Processing our data

Dropping Data

The dataset description states that RISK_MM should be dropped from the dataset since it can result in something called data leakage. This basically means that RISK_MM may contain information about our target variable, rain tomorrow, which we are not suppose to know — which may result in models that are “too good to be true”.

Often it is a good idea to remove columns that contain large amounts of missing data since these tend to be less useful to constructing our model.

The following code gives the number of values per column.

df.count().sort_values()

We see here that Sunshine, Evaporation, Cloud3pm, Cloud9am have less than 60% of data, so we will remove these from our dataset. We will also remove the Date column.

df = df.drop(columns=['Sunshine','Evaporation','Cloud3pm','Cloud9am', 'RISK_MM','Date'],axis=1)

df.shape

Here we have removed 7 columns of data.

Next we will remove all rows containing na values using the following code:

Here we have remove around 30,000 rows of data.

Since we initially have quite a large dataset, removing chunks of data should not cause too large of an issue for our model.

Removing Outliers

Large outliers can heavily damage the accuracy of logistic and linear regression models.

To remove outliers we can look at something called a Z score.

Every column of data can be standardized into a normal distribution:

Around 99.8% of all data is within 3 standard deviations of the mean of the column . Any data which is beyond 3 standard deviations can be considered outliers and removed from the dataset.

In this case, if we find any row of data (observation) with an outlier we will remove the entire row from the dataset.

We can do this with the following code:

from scipy import stats z = np.abs(stats.zscore(df._get_numeric_data()))

print(z)

df= df[(z < 3).all(axis=1)]

print(df.shape)

Standardizing our data

Next, we standardize our data — since logistic and linear regression works best with data that is put all on the same scale. This is because when we apply gradient descent with regularization it assumed that all our features are cantered around 0 (mean = 0) and have the same order of variance.

We only standardize our numerical data , for this we create a list called numerical that stores all columns that contain numerical data .

, for this we create a list called numerical that stores all columns that contain . We then go through each numerical column using a for loop and standardizing each.

from sklearn import preprocessing numerical = [var for var in df.columns if df[var].dtype=='float64'] for col in numerical:

df[col] = preprocessing.scale(df[col])



df.head()

Dealing with Categorical Data

As in the previous episode, we change all yes and no data to 1’s and 0’s so it is readable by our model.

df['RainToday'].replace({'No': 0, 'Yes': 1},inplace = True)

df['RainTomorrow'].replace({'No': 0, 'Yes': 1},inplace = True)

One Hot encoding

One hot encoding is the method we use to convert categorical data into numerical data:

We see from the above image that each categorical variable is split into their own column. We then use 1’s and 0’s to represent where each colour is present in the Colour column.

We will now apply this method to all of our categorical data in our dataset.

To view the categorical data in our dataset we can use the following code:

Categorical variables have the data type (dtype) as object

Numerical variables have the data type (dtype) as float64

categorical = [var for var in df.columns if df[var].dtype=='object'] print("Number of categorical variables: ", len(categorical)) print(categorical)

We have dealt with RainToday and RainTomorrow columns. We now apply one hot encoding to the other columns.

To implement One Hot Encoding we can use the function pd.getdummies.

We implement this function on all remaining categorical variables with the following code:

Here we have also displayed all unique values in each categorical column.

We can check if our One Hot Encoding has worked by printing out a few of the newly created columns.

From df.shape we see we now have a total of 106 columns in our dataset.

Model Features

In this case we will be using all 106 features to predict if it will rain tomorrow:

# set X to all features

X = df.loc[:,df.columns!='RainTomorrow'] # set y to our target RainTomorrow

y = df.RainTomorrow

Even if any of our features to do not have any influence on if it will rain tomorrow we can still include it in our model.

have any influence on if it will rain tomorrow we can still include it in our model. This is because when calculating our model’s parameters, the model is able to identify features that have less influence on the final output (rainTomorrow) and will assign a very small parameter values to these features — reducing their influence on the final output.

Splitting our data

After manipulating all of our data it is now time to split our data into a training and test set.

We split our data into training and test data to evaluate our model’s performance. This is known as cross validation.

In this case we will be using 25% of our data for testing

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)

3. Building our Logistic Regression Model

To build our model we import Logistic Regression from the sci-kit learn library and fit it to our training data.

from sklearn.model_selection import train_test_split

from sklearn.linear_model import LogisticRegression # set X to all features

X = df.loc[:,df.columns!='RainTomorrow'] # set y to our target RainTomorrow

y = df.RainTomorrow X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42) logreg = LogisticRegression()

logreg.fit(X_train,y_train)

4. Evaluating our Model

In project 1 we used the train_test split method to evaluate our model. In this case we are going to be using K_fold cross validation, which is a more accurate method of evaluating our model since it uses each all of our data to build and test our model.

We will be splitting our data into 5 folds.

For each fold we produce an evaluation metric, in this case we will be using accuracy to evaluate our model.

Accuracy is essentially the percentage of correct predictions and is given by the following formula:

Because we are using 5 folds we will be producing 5 accuracy scores. We take the mean of these 5 scores to evaluate our model.

# import relevant functions

from numpy import mean

from sklearn.model_selection import KFold

from sklearn.model_selection import cross_val_score # split data into 5 folds and shuffle to avoid bias cv = KFold(n_splits=5, random_state=1, shuffle=True) scores = cross_val_score(logReg, X, y, scoring='accuracy', cv=cv)

average_score = mean(scores) print('Overall Accuracy:', average_score)

Now that we have established that our model has a good accuracy of 0.85657.. , meaning our model gets roughly 86% of predictions correct, we can now use it to make predictions.