A Beginner’s Look at Kaggle
By: Wesley Neill
Tags: Machine Learning, Data Science, Python, Begginer, Kaggle

The above heatmaps show our strength of association between each variable. While there is no rigid standard for “Highly Associated” or “Weakly Associated”, we will use a cut-off value of |0.1| between our independent variables and survival. We will likely drop features whose association is lower than |0.1|. This is an entirely arbitrary guess, and I may return to raise or lower the bar later (In fact, I decided to keep Age after noticing improved performance when I did).

For now, the feature that meets the criteria for dropping are SibSp. Additionally, I am choosing to drop Name, Ticket and Cabin, mostly on a hunch that they don’t add much.

It should be noted that correlation between independent (predictor) variables can mean redundant information. This can cause a drop in performance in some algorithms. Some might choose to drop highly correlated predictors. I did not take the time to do that, but you might try it at home!

todrop = ['SibSp', 'Ticket', 'Cabin', 'Name']

train_df = train_df.drop(todrop, axis=1)

Let’s take a look at our transformed data frame, replete with new features, categories converted to numerical data, and old features dropped:

Setup for Machine Learning:

During this phase, we will begin to format our data for feeding into a machine learning algorithm. We will then use this formatted data to get a picture of what a few different models can do for us, and pick the best one. This phase is broken into the following parts:

Train/Test Split Normalize Data of each split Impute missing values

Let’s go.

Train/Test Split

We will split our data once into training and testing sets. Within the training set, we will use stratified k-fold cross validation to find average performance of our models.

The test set will not be touched until after we have fully tuned each of our candidate models using the training data and k-fold cross validation. Once training and tuning is complete, we will compare the results of each model on the held-out test set. The one that performs the best will be used for the competition.

# Split dependant and independant variables

X = train_df.drop(['Survived'], axis = 1)

Y = train_df.loc[:, 'Survived'] # Split data into training and validation sets

x_train, x_test, y_train, y_test = model_selection.train_test_split(X, Y, test_size=0.2, random_state=333)

Normalizing the Data

Some Machine Learning models require all of our predictors to be on the same scale, while others do not. Most notably, models like Logistic Regression and SVM will probably benefit from scaling, while decision trees will simply ignore scaling. Because we are going to be looking at a mixed bag of algorithms, I’m going to go ahead and scale our data.

# We normalize the training and testing data separately so as to avoid data leaks. Ask at the end! x_train = pd.DataFrame(pre.scale(x_train),

columns=x_train.columns,

index=x_train.index)

x_test = pd.DataFrame(pre.scale(x_test),

columns=x_test.columns,

index=x_test.index)

Imputing Missing Data

You might recall that there were a significant amount of missing Age values in our data. Let’s fill this in with the median age:

# Again, applying changes to the now separate datasets helps us avoid data leaks. x_train.loc[x_train.Age.isnull(), 'Age'] = x_train.loc[:, 'Age']

.median() x_test.loc[x_test.Age.isnull(), 'Age'] = x_test.loc[:, 'Age']

.median()

Let’s make sure our missing data is filled in:

x_train.info() # Output:

<class 'pandas.core.frame.DataFrame'>

Int64Index: 712 entries, 466 to 781

Data columns (total 11 columns):

# Column Non-Null Count Dtype

--- ------ -------------- -----

0 Pclass 712 non-null float64

1 Sex 712 non-null float64

2 Age 712 non-null float64

3 Parch 712 non-null float64

4 Fare 712 non-null float64

5 Embarked 712 non-null float64

6 Title 712 non-null float64

7 FamilySize 712 non-null float64

8 Alone 712 non-null float64

9 LName 712 non-null float64

10 NameLength 712 non-null float64

dtypes: float64(11)

Now we see that each and every variable that we chose to keep has 712 valid data entries.

Model Selection

Now that we have prepared our data, we want to look at different options available to us for solving classification problems. Some common ones are:

K-Nearest Neighbors

Support Vector Machines

Decision Trees

Logistic Regression

We will train and tune each of these models on our training data by way of k-fold cross-validation. When complete, we will compare the tuned models’ performance on a held out test set.

Training and Comparing Base Models:

First, we want to get a feel model’s performance before tuning. We will write two functions to help us describe our results. The first will evaluate the model several times over random splits in the data, and return the average performance as a dictionary. The second will simply nicely print our dictionary.

# A function that evaluates each model and gives us the results:

def kfold_evaluate(model, folds=5):

eval_dict = {}

accuracy = 0

f1 = 0

AUC = 0



skf = model_selection.StratifiedKFold(n_splits=folds)



# perform k splits on the training data.

for train_idx, test_idx in skf.split(x_train, y_train):

xk_train, xk_test =

x_train.iloc[train_idx], x_train.iloc[test_idx] yk_train, yk_test =

y_train.iloc[train_idx], y_train.iloc[test_idx]



# Test performance on this fold:

model.fit(xk_train, yk_train)

y_pred = model.predict(xk_test)

report = metrics.classification_report(yk_test,

y_pred,

output_dict=True)

# Gather performance metrics for output

prob_array = model.predict_proba(xk_test)



fpr, tpr, huh = metrics.roc_curve(yk_test,

model.predict_proba(xk_test)[:,1])

auc = metrics.auc(fpr, tpr)

accuracy += report['accuracy']

f1 += report['macro avg']['f1-score']

AUC += auc



# Average performance metrics over the k folds

measures = np.array([accuracy, f1, AUC])

measures = measures/folds # Add metric averages to dictionary and return.

eval_dict['Accuracy'] = measures[0]

eval_dict['F1 Score'] = measures[1]

eval_dict['AUC'] = measures[2]

eval_dict['Model'] = model



return eval_dict # a function to pretty print our dictionary of dictionaries:

def pprint(web, level):

for k,v in web.items():

if isinstance(v, dict):

print('\t'*level, f'{k}: ')

level += 1

pprint(v, level)

level -= 1

else:

print('\t'*level, k, ": ", v)

Putting our kfold evaluation function to use:

# Perform evaluation on each model:

evals = {}

evals['KNN'] = kfold_evaluate(KNeighborsClassifier())

evals['Logistic Regression'] = kfold_evaluate(LogisticRegression(max_iter=1000))

evals['Random Forest'] = kfold_evaluate(RandomForestClassifier())

evals['SVC'] = kfold_evaluate(SVC(probability=True)) # Plot results for visual comparison:

result_df = pd.DataFrame(evals)

result_df

.drop('Model', axis=0)

.plot(kind='bar', ylim=(0.7, 0.9))

.set_title("Base Model Performance")

plt.xticks(rotation=0)

plt.show()

Base Model Summary

It appears that we have a clear winner in our Random Forest classifier.

Hyper-parameter Tuning:

Let’s tune up our current champion’s hyper-parameters in hopes of eking out a little bit more performance. We will use scikit-learn’s RandomizedSearchCV which has some speed advantages over using an exhaustive GridSearchCV . Our first step is to create our grid of parameters over which we will randomly search for the best settings:

# Number of trees in random forest

n_estimators = [int(x) for x in np.linspace(start = 200, stop = 2000, num = 10)]

# Number of features to consider at every split

max_features = ['auto', 'sqrt']

# Maximum number of levels in tree

max_depth = [int(x) for x in np.linspace(10, 110, num = 11)]

max_depth.append(None)

# Minimum number of samples required to split a node

min_samples_split = [2, 5, 10]

# Minimum number of samples required at each leaf node

min_samples_leaf = [1, 2, 4]

# Method of selecting samples for training each tree

bootstrap = [True, False] # Create the random grid from above parameters

random_grid = {'n_estimators': n_estimators,

'max_features': max_features,

'max_depth': max_depth,

'min_samples_split': min_samples_split,

'min_samples_leaf': min_samples_leaf,

'bootstrap': bootstrap} pprint(random_grid, 0) #Output:

n_estimators : [200, 400, 600, 800, 1000,

1200, 1400, 1600, 1800, 2000]

max_features : ['auto', 'sqrt']

max_depth : [10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110, None]

min_samples_split : [2, 5, 10]

min_samples_leaf : [1, 2, 4]

bootstrap : [True, False]

Next, we want to create our RandomizedSearchCV object which will use the grid we just created above. It will randomly sample 10 combinations of parameters, test them over 3 folds and return the set of parameters that performed the best on our training data.

# create RandomizedSearchCV object

searcher = model_selection.RandomizedSearchCV(

estimator = RandomForestClassifier(),

param_distributions = random_grid,

n_iter = 10, # Number of parameter settings to sample

cv = 3, # Number of folds for k-fold validation

n_jobs = -1, # Use all processors to compute in parallel

random_state=0 # Fore reproducible results

) # Look for the best parameters

search = searcher.fit(x_train, y_train)

params = search.best_params_

params #Output:

{'n_estimators': 1600,

'min_samples_split': 10,

'min_samples_leaf': 4,

'max_features': 'auto',

'max_depth': 30,

'bootstrap': False}

After performing our parameter tuning, we can verify whether or not the parameters provided by the search actually improve the base model or not. Let’s compare the performance of the two models before and after tuning.

tuning_eval = {}

tuned_rf = RandomForestClassifier(**params)

basic_rf = RandomForestClassifier() tuning_eval['Tuned'] = kfold_evaluate(tuned_rf)

tuning_eval['Basic'] = kfold_evaluate(basic_rf) result_df = pd.DataFrame(tuning_eval)

result_df.drop('Model', axis=0).plot(kind='bar', ylim=(0.7, 0.9)).set_title("Tuning Performance")

plt.xticks(rotation=0)

plt.show()

result_df

Final Steps:

Now that we have chosen and tuned a Random Forest classifier, we want to test it on data it has never before seen. This will tell us how we might expect the model to perform in the future, on new data. It’s time to use that held out test set.

Then, we will combine the test and training data, and re-fit our model to the combined data set, hopefully giving it the greatest chance of success on the unlabeled data from the competition.

Finally, we will make our predictions on the unlabeled data for submission to the competition.

Final Test on Held Out Data

# Get tuned model predictions on held out data

y_pred = tuned_rf.predict(x_test) # Compare predictions to actual answers and show performance

results = metrics.classification_report(y_test, y_pred,

labels = [0, 1],

target_names = ['Died', 'Survived'],

output_dict = True) pprint(results, 0)

And here is how our model performed:

Died:

precision : 0.7815126050420168

recall : 0.8532110091743119

f1-score : 0.8157894736842106

support : 109

Survived:

precision : 0.7333333333333333

recall : 0.6285714285714286

f1-score : 0.6769230769230768

support : 70

accuracy : 0.7653631284916201

macro avg:

precision : 0.757422969187675

recall : 0.7408912188728702

f1-score : 0.7463562753036437

support : 179

weighted avg:

precision : 0.7626715490665541

recall : 0.7653631284916201

f1-score : 0.761484178861421

support : 179

It looks like we may have experienced some over-fitting. Our model’s performance on the test data is roughly 7–9% lower across the board, but we should expect that our model performs about this well on real world data that it has never before seen.

Combine Training and Testing Datasets for Final Model Fit

Now that we have ascertained that our tuned model performs with about 76% accuracy and has an f1-score of 0.74 on new data, we can proceed to train our model on the entire labeled training set. More (good) data is almost always better for an algorithm.

X = pd.concat([x_train, x_test], axis=0).sort_index()

Y = pd.concat([y_train, y_test], axis=0).sort_index()

tuned_rf.fit(X, Y)

Format and Standardize Unlabeled Data

Now that our model has been completely fitted on the training data, it’s time to get ready to make the predictions that we will submit to the competition.

We need to transform our unlabeled competition data in the same manner as when we were formatting our training data. This includes encoding categorical variables, dropping the same features and normalization. The idea here is consistency. What we did to the data that we trained the model on, we need to do to the data we will use to make our final predictions.

# Feature Engineering:

test_df['Title'] = test_df.Name.str.extract(r'([A-Za-z]+)\.')

test_df['LName'] = test_df.Name.str.extract(r'([A-Za-z]+),')

test_df['NameLength'] = test_df.Name.apply(len)

test_df['FamilySize'] = 1 + test_df.SibSp + test_df.Parch

test_df['Alone'] = test_df.FamilySize.apply(lambda x: 1 if x==1 else 0)

test_df.Title = test_df.Title.map(title_dict) # Feature Selection

test_df = test_df.drop(todrop, axis=1) # Imputation of missing age and fare data

test_df.loc[test_df.Age.isna(), 'Age'] = test_df.Age.median()

test_df.loc[test_df.Fare.isna(), 'Fare'] = test_df.Fare.median() # encode categorical data

for i in test_df.columns:

if test_df[i].dtype == 'object':

test_df[i], _ = pd.factorize(test_df[i])



# center and scale data

test_df = pd.DataFrame(pre.scale(test_df), columns=test_df.columns, index=test_df.index) # ensure columns of unlabeled data are in same order as training data.

test_df = test_df[x_test.columns]

test_df

Make Final Predictions and Common Sense Check:

Roughly 32 percent of the passengers aboard the Titanic lived. We will do a last, common sense check to see if our algorithm predicts roughly the same distribution of survivals. Since Survived variable with value 1 implies survival, we can simply add all instances of survival and divide by the total number of passengers to get a rough idea of our predicted distribution.

Keep in mind, the competition organizers could have been tricky and given us uneven distributions for training and testing. In that case, this might not work, but I’m assuming they did not.

# Make final predictions

final = tuned_rf.predict(test_df) # Check the probability of survival according to our predictions. It should be roughly 32% (we get 36.6% which is a bit optimistic)

final.sum()/len(final) # Get our predictions in the competition rules format:

submission = pd.DataFrame({'PassengerId':test_df.index,

'Survived':final}) # Output our submission data to a .csv file:

submission.to_csv('submission2.csv', index=False)

Summary

My focus was not to win a competition, but to learn a way of thinking. After all, if you are like me, you aspire to become a data-scientist. Therefore, machine learning experiments should be rigorous and repeatable, but just as important, the process should be uniquely defined by the questions being asked and the data on hand to answer those questions.

However, if you are wondering how well this set-up performed, it achieved an accuracy of 77%. That’s far from perfect!

Again, if you have any feedback, I would love to hear your questions, comments and critiques of my process. Writing this article is part of my own learning process, and I hope you join in.

Thanks!