From Scikit-learn to TensorFlow: Part 2 | by Karthik M Swamy | Towards Data Science

Continuing from where we left, we delve deeper into how to develop machine learning (ML) algorithms using TensorFlow from a scikit-learn developer’s perspective. If you’d like to know the reasons to move to TensorFlow, motivations, do read my earlier post for Reasons to move to TensorFlow and a simple classification program that highlights similarities of developing for scikit-learn and TensorFlow.
In the earlier post, we compared the fit and predict paradigm similarities in scikit-learn and TensorFlow. In this post, I want to show we can develop a TensorFlow classification framework with Scikit-learn’s data processing and reporting tools. This will give a good method to interweave both the frameworks to come up with a neat and concise framework.
Machine learning (ML) problems are pervasive. However, the catch here is that once you start solving problems with ML, every new problem might seem like an ML problem, which is something like the old adage goes:
If all you have is a hammer, everything looks like a nail
There is an important distinction between a person who understands ML and a person who uses ML, which is, when not to use ML to solve a problem. You might not need to use ML if you can easily arrive at the solution with simple rules. For example, you don’t need an ML algorithm to predict all bikes and cars are modes of transport or to replace commonly misspelt words with the right spelling.
We will require ML if the rules become unwieldy and if there are too many parameters to determine such rules. For example, we will require ML to predict tomorrow’s weather as it depends on a number of factors such as season, location, other factors such as El Niño which makes writing rules difficult and often inaccurate. Another instance when we might need ML is when the data becomes humanly impossible to peruse and find patterns, which is an area ML seems to do a good job. Take the email spam or ham example, where we classify email as spam or not, based on the text present in the email (body + subject). This task is easy if the patterns for spammy emails were finite and possibly limited. However, the “princes” who send such emails tend to find new methods to make us all “rich”, making it necessary for us to develop ML frameworks to help us fight spam.
Having said all this, let’s take a look at a problem which is inherently difficult to find rules and a tough problem to get high precision and recall.
We will be using the Wisconsin Diagnostic Data for Breast Cancer detecting presence of cancer. This dataset is a binary classification problem (malignant or benign) and has 569 instances (data points) that allow us to perform the classification task.
All the code described in this post is available in my GitHub repo here.
We leverage Scikit-learn’s efficient data loading tools to obtain our data. The breast cancer dataset is available here. A list of other datasets that Scikit-learn allows us to load can be found here. Let us load the dataset and view some attributes of the data.
from sklearn.datasets import load_breast_cancercancer_data = load_breast_cancer()print("Number of instances in the dataset: %d" % len(cancer_data.target))>> Number of instances in the dataset: 569
Let’s discuss the impact of using the data as-is versus scaling the features prior to training the data. Scaling the data allows the features to be normalised. What this means is that data is centred around zero and scaled to have a standard deviation of one. In other words, we restrict the data to fall between [0, 1] without changing the original distribution of the data. This ensures that the classifier is not going to wildly search a large dimensional space for the optimal weights and instead restricts the search space between [0, 1]. This scaling has a big impact on how well a classifier performs and this can be seen in the experiments section below. We try two experiments, one with feature scaling and another without feature scaling.
We use the data from SKLearn as-is, simply loading variables using the in-built train, test splitter
from sklearn.model_selection import train_test_splitin_train, in_test, out_train, out_test = train_test_split(cancer_data[‘data’],  cancer_data[‘target’])
For the second experiment, we scale the data using pre-processing functions available in SKLearn.
from sklearn.preprocessing import StandardScalerdata_scaler = StandardScaler()# Fit train data data_scaler.fit(in_train)in_train = data_scaler.transform(in_train)in_test = data_scaler.transform(in_test)
We design a neural network similar to my earlier post to keep a simple approach to the network and instead understand the effects of data pre-processing and overfitting. We use TensorFlow’s DNNClassifier which is available in the contrib module of TensorFlow. We define a network with three layers where each layer has <units> number of hidden units. In our experiments, we first evaluate performance of the network below by changing how the inputs are pre-processed, with a units=50. The best performing input is then chosen to experiment further to understand overfitting where we keep the data constant (best among D1 vs. D2) and vary units. The network structure is described below:
feature_columns = tf.contrib.learn.infer_real_valued_columns_from_input(in_train)classifier_tf = tf.contrib.learn.DNNClassifier(feature_columns=feature_columns,  hidden_units=[units, units, units],  n_classes=2)classifier_tf.fit(in_train, out_train, steps=1000)
What the network above does is define a fully-connect network also known as a multi-layer perceptron with the input from the data we just loaded. A multi-layer perceptron can be described as below:
The input in our case is the data we loaded from the cancer dataset while the output is a binary value indicating if the input is malignant or benign.
The aim of these two experiments is to understand the impact of the number of hidden units on each layer which will also provide insight into overfitting.
For this experiment, we first run to choose either D1 or D2 (described above) and then vary the number of hidden units on each layer of the network. The number of hidden units for this experiment is kept low at H1=10.
We perform our fourth experiment with a larger number of hidden units than H1. This would be able to accommodate more variations in the data but might end up memorising the data and hence unable to perform well on test data. We run the experiment similar to H1 above and then run the network with H2=50.
We use Scikit-learn’s reporting functionality to understand how our classifier performed. We use the classification report and the confusion matrix to understand how our classifier performed.
The classification report produces a matrix with key metrics calculated using the predicted output and the actual output values. The metrics reported are precision, recall and f1-scores for each class as well as the average across all classes.
report = metrics.classification_report(out_test, predictions,  target_names=cancer_data.target_names)print(report)
While the classification report provides key metrics, the confusion matrix provides the class in which each test data point was classified under. The confusion matrix shows how strong the classifier is in making a prediction for a class while at the same time, shows the vulnerabilities of the classifier in classifying the data under other classes.
# Plotting the confusion matrix using matplotlib%matplotlib inlineconfusion = metrics.confusion_matrix(out_test, predictions)# Plot non-normalized confusion matrixplt.figure()plot_confusion_matrix(confusion, classes=cancer_data.target_names, title=’Confusion matrix, without normalization’)
The classification report describes a holistic performance while the confusion matrix provides an exact number of data points classified under each class.
We first perform the experiment to understand if data normalisation helps with better classification. Hence, we choose H1 and experiment with D1 and D2. The results are provided below:
Performance with H1+D1
              precision    recall  f1-score   support  malignant       0.94      0.88      0.91        51     benign       0.94      0.97      0.95        92avg / total       0.94      0.94      0.94       143Prediction Accuracy: 0.937063
Performance with H1+D2
              precision    recall  f1-score   support  malignant       0.96      0.93      0.95        46     benign       0.97      0.98      0.97        97avg / total       0.96      0.97      0.96       143Prediction Accuracy: 0.965035
From our experiments, it is evident that data scaling provides a ~3% improvement in classification accuracy. This lets us choose D2 (data with scaling) for our experiments with H2.
Performance with H2+D2
              precision    recall  f1-score   support  malignant       0.96      0.92      0.94        53     benign       0.96      0.98      0.97        90avg / total       0.96      0.96      0.96       143Prediction Accuracy: 0.958042
The performance metrics of H1+D2 and H2+D2 does not provide much insight into how the classifier performed across other classes in terms of number of data points. This is where the confusion matrix provides a better picture for further analysis.
The above matrices provide a clearer picture of the differences in the performance of the classifiers H1 and H2. Although H2 has five times more hidden units than H1, it performed poorer than H1 on the test data. The confusion matrix provides an exact distribution of the number of points that were incorrectly classified under each category.
In this post, we saw how to use Scikit-learn with TensorFlow to perform different classification experiments and view the performances of each of the classifier. We also performed simple experiments with data pre-processing and the network parameters to understand different concepts.
All the code described in this post is available in my GitHub repo here.
Let me know what topics you’d like me to discuss in my next post, as comments below. I’m on Twitter, in case you’d like to connect with me there!