Machine Learning #1 — Supervised Learning, EDA, Cross-Validation
Tags: Artificial Intelligence, Python, Software Development, Machine Learning, Classification

EDA — Exploratory Data Analysis

It is an approach to interpret the data through a summary using visual or numerical methods. It enables us to have an idea on many issues, from the steps to be taken at the point of pre-processing the data, to the method and model to be used to make inferences.

Numerical EDA

It is an approach to summarize data numerically. We use several pandas method for this.

df.info()

Output of df.info()

df.info() is one of them, it allows us to get an overview of the data. Here we can see the names of the features, how many rows and columns we have in a data set. We have a dataset of 435 rows and 17 columns. We will use 16 variables in this set as predictors and 1 variable (‘party’) as target variable. We see that all 435x17 data is non-null, which is important, because libraries such as numpy, pandas, scikit-learn need to be non-null and of the same type for libraries like numpy, pandas, scikit-learn to work correctly.

df.describe()

Output of df.describe()

df.describe() provides another summary of the general information we get with info. Such as how many of each variable or how many values of these variables are reserved. A point that draws our attention here is that while there are 2 unique values (Democrat, Republican) for ‘party’, there are 3 values in other titles.

This means that even if the type is not “null” or “NaN” among the property values, there are data that need pre-processing. Let’s take a closer look at the data.

You can see all the titles voted on by scrolling to the right in the original work. Again, as an intermediate information note, I find gist, a service of GitHub, very useful. For information on how to add a Jupyter Notebook to a Medium post, you can read Ersin Yıldız’s article “How to Embed Jupyter Notebook in Medium Articles?”(In Turkish only)

With the df.head() command, we can see the data in the first 5 lines by default. If we cannot find the answer we are looking for in this data and need more data, we can see more data by typing the number we want in the command. For example;

This is how we saw the first 10 lines. The point that should draw our attention is that some values are given as ‘?’. In the next step, after doing this analysis visually, we will focus on this issue.

Visual EDA

Let’s visually look at our dataset, which we saw numerically in the previous step. For example, let’s look at the distribution of the votes on “immigration”.

A note for those who are not familiar with the Jupyter notebook environment: You do not have to repeat the import and read from csv process at every step. As you can see in the original work notebook, it is sufficient to have done these operations in a cell at the beginning of the file. I ran it all in a single cell just to be able to draw this table.

Confirming the information we obtained in the previous step, we see a third voting pattern other than ‘yes’ or ‘no’. This is the data in the form of “?” That we need to solve. In order to solve this problem, we use the SimpleImputer method of the scikit-learn library.

from sklearn.impute import SimpleImputer

imputer = SimpleImputer(missing_values='?', strategy='most_frequent')

We created a variable called imputer from the SimpleImputer object. We gave the missing_values parameter the string value “?” that we accept as the missing value and want to change. The default of this parameter is “NaN” (Not a Number). We set the strategy parameter as ‘most_frequent’. Thus, the values indicated by ‘?’ Will be equal to whatever value is the most in that column (ie according to frequency if ‘no’ votes are the most, ’n’ or ‘yes’ votes are ‘y’).

Note: In your own research, you can see that in some sources on the internet, an invocation from sklearn.preprocessing import Imputer is made for this process. However, this method has been removed in the new version and has been replaced by the sklearn.impute.SimpleImputer method we use.

A screenshot taken from the Scikit-learn documentation. Here you can see what values missing_values and strategy parameters can take and what these values mean. For other parameters of the method; You can read the scikit-learn documentation and examine the examples.

Next is determining and training our model. Since this is a classification problem, we can use the k-NN (k-Nearest Neighbors / k-Nearest Neighborhood) algorithm to solve this problem. If we talk about the working principles of this algorithm;

It looks at the classes of the nearest k points,

Whichever class most of these k points belong to, the algorithm determines the class of the point it examines as that class.

First step

Suppose we have a dataset of red and green points. And we have a point where we do not yet know which class it belongs to. If we set the value of our k parameter to 3,

Second step

Our algorithm looks at these 3 closest points, sees that the majority is red and assigns the red color to the unassigned point. If we choose our k value as 5,

Third step

This time, since the majority is green, our algorithm will decide that the class we are looking for is green.

Fourth step

The kNN algorithm examines the training set, looking for the nearest neighbors in the entire data set to make predictions. Now let’s include our algorithm in our working environment.

from sklearn.neighbors import KNeighborsClassifier

knn_model = KNeighborsClassifier(n_neighbors=2)

Again, using the scikit-learn library, we called our kNN algorithm and defined the variable that we call knn_model by setting the n_neighbors parameter, which indicates the neighborhood value to be used while making the decision, as 2. Next is to create our steps.

steps = [(‘imputation’, imputer),(‘knn’, knn_model)]

In this way, we have clustered our steps into a list of Python tuples. We will use the steps variable we have defined to create a pipeline instead of coding the guessing process step by step.

from sklearn.pipeline import Pipeline

pipeline = Pipeline(steps)

By using the sklearn.pipeline method, you can group your algorithm, parameters, data preprocessing steps into a single variable (we call steps) and write more compact code.

Let’s talk about creating training and test sets from the data set we have. For this, we will use the train_test_split method. In addition, we will determine the ‘party’ feature as target and other features as predictors.

from sklearn.model_selection import train_test_split

y=df['party'].values

X=df.drop('party', axis=1).values

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

We assign the ‘party’ values of each data in our data set to the variable y. Then, we deduct these values from our data set and assign the remaining values to variable X. Thus, we have created our predictor and target variables.

Then we divided our data into train and test using the train_test_split method. While separating, we set the size of our test set to be 30% of all data with the test_size parameter. We have separated the rest as train data.

random_state is simply an integer parameter that we use to shuffle the data before splitting it. In materials such as training and documents you are examining, you will usually see that this parameter is equal to 42.

Finally, another critical point we need to know is that our model / algorithm cannot process string-type data. As we can see in the visual representation of the kNN algorithm, the whole operation is based on a distance calculation (Euclidean distance) in the vector space. In order to mathematically model each point in our data set, we change the values given as ’n’ and ‘y’ to 0 and 1. Since I want to touch on some of the more popular Python libraries and applications, I will not include much mathematical explanations in this series of articles. Next, the algorithm learns the data and makes inferences.

from sklearn.metrics import accuracy_score

knn_pipeline = pipeline.fit(X_train, y_train)

y_pred = pipeline.predict(X_test)

accuracy_score(y_test, y_pred)

We created a pipeline with data pre-processing and learning stages. At this point, if you want to work without using pipeline, you can also make the model infer from the data directly with the knn_model.fit() method. Note that you will have to code the previous steps step by step in this case.

Then we give the set of inputs we reserved for testing to the predict method to make predictions. Finally, we can see that our success rate is approximately %92.37 by using the accuracy_score method that we have included from the scikit-learn library.

Well, what can we do to increase this rate even more? What could have caused our success to be lower than it could have been?

The first point we need to pay attention to is that the n_neighbors parameter is selected as 2. A higher parameter for this parameter would perhaps have enabled us to achieve a higher score.

The second point is the data separated as train / test. We allocated 30% of the data as test data, but keep in mind that the allocated data is randomly selected. Perhaps the result would have been different if we had set different 30% as test data, that is, if we had trained our algorithm with a different 70%. How can we solve these two problems?

Here is a sample scheme for the Cross-Validation method we use to increase our performance.

Our first solution is cross-validation. This technique is based on making a different train / test distinction at each step and choosing the best learning. As an example, I have based a demonstration where train / test data is split 80/20. We take a metric by selecting train / test sets differently in each split. Whichever of these distinctions has the highest score, we choose our distinction as that.

The main technique we will combine this technique with is grid search cross-validation. By splitting our dataset into grids, we not only select the train / test set differently in each step, but also try it with a different n_neighbor parameter. We use sklearn.model_selection.GridSearchCV for the Python implementation of this technique.

from sklearn.model_selection import GridSearchCV

parameters = {'knn__n_neighbors': np.arange(1, 50)}

cv = GridSearchCV(pipeline, param_grid=parameters, cv=5)

We will value the n_neighbors parameter of the knn algorithm from 1 to 50 with the help of the numpy library. If you want to use a different parameter of a different algorithm, you should write ‘modelname__parameter’. We assigned our pipeline first to the GridSearchCV method, and then our parameters variable, which we created in the Python dictionary type, as the value for the param_grid parameter. By entering how many different ways we want to make our train / test separation into the cv parameter, we go back to the fit stage.

cv.fit(X_train, y_train)

print("Tuned KNN Parameters: {}".format(cv.best_params_))

print("Best score is {}".format(cv.best_score_))

Here, our algorithm worked by processing the data in 5 different ways for each n_neighbor value, determining the best parameter and score. With the cv.best_params_ method, we saw that the best n_neighbor value was 6, and with cv.best_score_ our best score was about 93.76%.

In the next topics of the Machine Learning article series we will examine different ML approaches to different problems, different methods that can be used to measure performance. I wish it to be useful.

Resources

[1] https://archive.ics.uci.edu/ml/datasets/