Logistic Regression for malignancy prediction in cancer | by Luca Zammataro | Towards Data Science

In Linear Regression with one or more variables, we have introduced the concept of Linear Regression, a statistical model used in Machine Learning that belongs to the Supervised Learning class of algorithms. Also, we have discussed the possible applications of the Linear Regression to biomedical data, in particular, in predicting continuous-valued outputs.
In this article, which ideally represents a continuation of that dedicated to the Linear Regression, we will explain the Logistic Regression, another Supervised Learning model, used for predicting discrete-valued outputs.
We will introduce the mathematical concepts underlying the Logistic Regression, and through Python, step by step, we will make a predictor for malignancy in breast cancer. We will use the “Breast Cancer Wisconsin (Diagnostic)” (WBCD) dataset, provided by the University of Wisconsin, and hosted by the UCI, Machine Learning Repository.
Drs Mangasarian, Street, and Wolberg, the creators of the database, intended to utilize 30 characteristics of individual cells of breast cancer obtained from a minimally invasive fine needle aspirate (FNA), to discriminate benign from malignant lumps of a breast mass, using Machine Learning [1].
Briefly, an FNA is a kind of biopsy, in which physicians use a needle attached to a syringe to withdraw a small number of cells from a suspicious area. The biopsy is then checked for cancer identification [2].
Using an image analysis software called Xcyt based on a curve-fitting algorithm, the authors were able to determine the boundaries of the cell nuclei from a digitized 640×400, 8-bit-per-pixel grayscale image of the FNA.
The 30 features describe characteristics of the cell nuclei present in the scanned images.
As reported in their paper, the authors have used machine learning and image processing techniques to accomplish malignancy outcome prediction. Still, an in-depth on their computational approaches go beyond the aim of this article.
In this post, we will not remake the work of Mangasarian, Street, and Wolberg. We will use their dataset to implement a Logistic Regression predictor based on some of the 30 features of the WBCD, in Python. We will use the outcome Bening/Malignant to predict if a new patient has a probability of developing malignancy or not, basing on the FNA data. Furthermore, our predictor will be an exciting occasion of exposing some basic concepts of Logistic Regression and implementing a code around the biomedical problem: what features are essential in predicting malignant outcomes.
The first column of the dataset corresponds to the patient ID, while the last column represents the diagnosis (the outcome can be “Benign” or “Malignant” based on the type of diagnosis reported). The resulting dataset consists of 569 patients: 212 (37.2%) have an outcome of Malignancy, and 357 (62.7) are Benign. Figure 2 describes the dataset structure:
In detail, the dataset consists of ten real-valued features computed for each cell nucleus. They are 1) Radius (mean of distances from center to points on the perimeter), 2) Texture (standard deviation of gray-scale values), 3) Perimeter, 4) Area, 5) Smoothness (local variation in radius lengths), 6) Compactness (perimeter2 / area — 1.0), 7) Concavity (severity of concave portions of the contour), 8) Concave points (number of concave portions of the contour), 9) Symmetry and 10) Fractal Dimension (“coastline approximation” — 1). The ten real-valued features correspond to the Mean, (values from columns 2 to column 11), to the Standard Errors (values from columns 12 to 21), and the Worst or largest (mean of the three largest values), (columns from 22 to 31). Column 32 contains the Bening/Malignant outcome.
For more simplicity, I have formatted the WBDC in a comma-separated-values file. You can download the formatted version following this link (filename: wdbc.data.csv).
Before starting, I suggest readers following the interesting course in Machine Learning at Coursera by Andrew NG [3]. The course provides an excellent explanation of all the arguments treated in this post.
All the code presented in this article is written in Python 2.7. For the implementation environment, I recommend the use of the Jupyter Notebook.
Import all the packages required for the Python code of this article: Pandas, NumPy, matplotlib, and SciPy. These packages belong to SciPy.org, which is a Python-based ecosystem of open-source software for mathematics, science, and engineering. Also, we will import seaborn, which is a Python data visualization library based on matplotlib. Moreover, an object op from the scipy.optimize package will be created, to make optimizations of the Gradient.
Once we have downloaded the wdbc.data.csv file, use it to create a DataFrame using pandas:
With pandas we can visualize the first lines of a df content using the .head() method:
Now that we have a DataFrame containing our data, we want to identify which of the 30 features are important for our prediction model. Selecting the best features of a dataset is a critical step in Machine Learning, to get a useful classification and to avoiding a predictive bias.
Visualization Per Pair
One method is the Visualization Per Pairs. As suggested by Ahmed Qassim in his article, an elegant display of data is gettable using the pairplot set of functions provided by seaborn. The Python code that produces the complete features combination plot is the following:
Here we have created the seaborn object sns; then, we have used the pandas DataFrame (df) produced in Step 2 as an argument for the sns pairplot method. Note that, specifying the argument “hue = Diagnosis,” the pairplot method has access to the df column containing the Diagnosis values (0, 1). Moreover, pairplot will use the argument “palette” to color points with b blue or r red basing on the Diagnosis values.
For simplifying the visualization, the map resulting from Code 3 shows the combinations of the first five pairs of features. They represent mean values of parameters Radius, Texture, Perimeter, Area, and Smoothness. Depending on your computer capabilities, the production of the complete features map, running Code 3, could take time.
Each tile represents a scatter plot of a couple of parameters. This visualization makes more accessible the identification of essential elements for the classification. Some of these pairs, like Radius vs. Texture or Perimeter vs. Smoothness, have a right separation level concerning the Diagnosis (blue points = Benign; red spots = Malignant). Visualizing features in plots of pairs represent an excellent tool, primarily because of its immediacy.
Bivariate Analysis
Visualization apart, a more efficient method for selecting non-redundant features is the Bivariate Analysis, which is a method based on the correlation, that analyzes the relationship between pairs of elements. Here I propose a code that I have readapted from AN6U5, which implements a function for the Bivariate Analysis. This function, here called features_correlation_matrix, takes the WBCD df as an argument and returns a correlation matrix for the 30 features:
Take a glance at the resulting plot in Figure 4: we aim to understand how the 30 characteristics of the dataset are related to each other. In calculating correlations between variables, we can observe that some of them result in being notably correlated (values greater than 0.9). The general assumption of the Bivariate Analysis is that features that are highly associated provide redundant information: for this reason, we want to eliminate them, avoiding a predictive bias.
As suggested in this post by Kalshtein Yael, dedicated to the WBCD analysis with R, we should remove all features with a correlation higher than 0.9, keeping those with the lower mean. Code 5 is a readaptation of a Python code example, from the Chris Albon book [4]:
The “to_drop” list contains ten redundant features with a correlation > 0.9. They are listed as follows:
['Perimeter', 'Area', 'Concave_points', 'Perimeter_SE', 'Area_SE', 'W_Radius', 'W_Texture', 'W_Perimeter', 'W_Area', 'W_Concave_points']
Now we want to eliminate these features from our DataFrame and re-plot the df with the remaining 20:
Figure 5 shows the plot resulting from Code 6:
We have now removed highly correlated feature pairs, so only features with low correlation, like Radius and Texture, will be kept. We can directly access to the correlation matrix values, typing in a new notebook cell:
The resulting output is the following:
Thanks to the Visualization Per Pairs and the Bivariate Analysis, now we know what features deserve to be considered in our analysis, then let’s go ahead with the construction of our Logistic Regression outcome predictor. In the next Steps, we will write a code to predict the diagnosis outcome for a pair of features chosen on the base of their non-redundancy.
Theoretically, for the Logistic Regression, we could use the same hypothesis model that we have used for the Linear Regression (see Linear Regression with one or more variables). Still, there are several reasons why we cannot use the linear hypothesis for the logistic, and now we are going to explain why. Take a glance at Figure 6:
The plot in Figure 6A explains why we cannot apply the linear Hypothesis to binary classification. Imagine that we want to plot our samples with an outcome that can be Benign or Malignant (red circles). We could apply the Linear Hypothesis model to separate the samples into two distinct groups. In attempting to classifying binary values as 0 and 1, the Linear Regression tries to predict values greater than 0.5 as “1” and all less than 0.5 as “0” because the threshold classifier output for hθ(x) is at 0.5.
Theoretically, the Linear Regression model that we have chosen could work well also for the binary classification, as the blue line demonstrates. But let see what happens if we insert another sample with Malignant diagnosis (the green circle in Figure 6A). The Linear Regression Hypothesis adapts the line to include the new sample (the magenta line).
Still, this Hypothesis model couldn’t work correctly for all the new samples that we are going to upload, because the Linear Hypothesis seems not to add further information to our predictions. That happens because classification is not a linear function. What we need is a new Hypothesis hθ(x) that can calculate the probability that the Diagnosis output can be 0 or 1: this new hypothesis model is the Logistic regression.
The Logistic Regression Hypothesis model in Equation 1, looks similar to that of the Linear Regression. But the real difference is in the g function that uses the product of the translated θ vector with the x vector (we will call this product z) as an argument. The g function is defined as in Equation 2:
The g(z) function, which is a sigmoid function (Logistic Function) is non-linear. It calculates the probability that the Diagnosis output can be 0 or 1(Figure 6B).
The Python Code for the implementation of the Logistic Function is the following:
This Step will answer to the question: How to choose parameters θ with Logistic Regression? Just like for the Linear Regression, the only way to select θ is by calculating its Cost Function, and then try to optimize the search using sophisticated algorithms for the Gradient descent optimization. (argument of Step 6).
Again, we could use the same Cost Function that we used for Linear Regression, and it should look like the Cost Function of the Equation 4 in the article dedicated to the Linear Regression, except for a difference in the sigmoid function that characterizes the Hypothesis model hθ(x).
The y vector, as usual, represents the output with the difference that, here, y is a vector of binary outcomes (0/1, or Benign/Malignant), and not a vector of continuous-valued outputs:
Concretely, we cannot use the Linear Regression Cost Function for the Logistic. The non-linearity of the sigmoid function, which handles hθ(x), leads to a J(θ) having a non-convex pattern, and it will look like the curve in the graph of Figure 7A:
This non-convex J(θ) is a function with many local optima. It’s not guaranteed that the Gradient descent will converge to the global minimum. What we want is a convex J(θ) like that in Figure 7B, which is a function that converges to the global minimum. So, we have to write the Cost Function in a way that guarantees a convex J(θ):
Implementing the Equation 4, the Cost Function that is calculated as log(z), looks like the red curve of Figure 8:
Because we have two binary conditions for the Benign or Malignant outcome (y), the Cost Function in Equation 4 states the cost of our Hypothesis prediction concerning y is.
If y = 1, but we predict hθ(x) = 0, we will penalize the learning algorithm by a considerable cost (see the red curve in Figure 8) because, in this case, the cost will tend to infinite. Instead, if our prediction is hθ(x) = 1, (thus equal to y), then the cost is going to be 0.
In the case of y = 0, we have the opposite: if y = 0 and we predict hθ(x) = 0, the cost is going to be 0, because our Hypothesis matches with y, while if our prediction is hθ(x) = 1, we end up paying a very large cost.
The simplified version of the Cost Function is the following:
If y = 0, the Cost Function will be equal to -(log(1-hθ(x)). If y = 1, the second part of Equation 5 will be 0, then the cost will be equal to -y * log(hθ(x)). A Python implementation of the simplified and vectorized Cost Function is the following:
In the post dedicated to Linear Regression, we have introduced the Gradient Descent, which is an algorithm that calculates the derivative of the Cost Function, updating the vector θ by mean of the parameter α, that is the learning rate.
But in the Machine Learning scenario, the Gradient Descent is not the only algorithm for minimizing the Cost Function: Conjugate Gradient, BFGS, L-BGFS, TNC, for example, represent some of the more sophisticated algorithms for minimizing the Cost Function, in automatizing the search of θ.
Based on different statistics, these algorithms try to optimize the Gradient Function, which is the difference between the actual vector y of the dataset, and the h vector (the prediction), to learn how to find the minimum J. Generalizing, an optimization algorithm will repeat until it will converge. Importantly: the updating of θ always has to be simultaneous.
Optimization algorithms are arguments of the Numerical Optimization field, and an in-depth study on the use of algorithms goes beyond this article. The Gradient function that we are going to implement looks identical to that used for the Linear Regression. But also here, as for the Cost Function, the difference is in the definition of hθ(x) that needs the sigmoid function:
The equation for the vectorized implementation of the Gradient Function is:
and the Python code for calculating the Gradient is the following:
In Step 9, we will see how to optimize the Gradient Function, using one of the algorithms provided by scipy.optimize.
We could need to rescale and normalized our data. The Python code for calculating Feature Scaling and Normalization for two variables is the following:
This function calculates the accuracy of our algorithm:
In the previous steps, we have created all the fundamental functions for the implementation of the Logistic Regression. Let’s briefly summarize all of them:
sigmoid(z)calcCostFunction(theta, X, y)calcGradient(theta, X, y)FeatureScalingNormalization(X)CalcAccuracy(theta, X)
Now we will write the code that wraps all of these functions, to predict outcomes of Malignancy based on two of the 20 not-redundant features of our dataset. For the choice of the features, we could refer to one of the pairs that we have found in Step 3, Radius, and Texture, which have a correlation score = 0.32. The following code produces the features numpy array X and the output numpy vector y, from the DataFrame df:
Let’s plot the two features:
In Figure 7, the resulting plot is shown:
The yellow dots represent the Benigns, the blacks, the Malignants.
Now let’s Normalize and Scale, our data. Also, we need collecting the mu, which is the average values of X in our training set, and sigma that is the Standard Deviation. In a new notebook cell, let’s type:
Now, we have to update the array X adding a column of “ones,” with the method .vstack:
Testing
Let’s do some tests: in order to test our code, let’s try to calculate the Cost Function and the Gradient, starting with a θ = [0, 0, 0]:
The new θ vector is now = [ 0.12741652, -0.35265304, -0.20056252], with an associated J(θ) = 0.69.
Also, we can try non-zero values of θ, and see what happens:
The updated θ vector now is = [-0.37258348, -0.35265304, -0.20056252], with an associated J(θ) = 8.48.
Gradient Descent Advanced Optimization
The optimization algorithm that we will use for finding θ is the BFGS, which is based on the quasi-Newton method of Broyden, Fletcher, Goldfarb, and Shanno [5]. Code 19 will implement the function Scypy minimize, which internally will call the BFGS method:
If we don’t specify the type of Method we want to use in the argument “method” of the .minimize function, the BFGS algorithm is used as default. Another method, i.e. TNC, uses a truncated Newton algorithm for minimizing a function with variables subject to bounds. Users can experiment with trying the various types of optimizing algorithms available with the .minimize function in Scypy. See the Scypy documentation page for more information about all the optimizing methods, with the function .minimize. The output produced by Code 18, is the following:
fun: 0.2558201286363281 hess_inv: array([[12.66527269, -1.22908954, -2.82539649],       [-1.22908954, 71.16261306,  7.07658929],       [-2.82539649,  7.07658929, 13.39777084]])      jac: array([7.36409258e-07, 3.24760454e-08, 9.55291040e-07])  message: 'Optimization terminated successfully.'     nfev: 20      nit: 19     njev: 20   status: 0  success: True        x: array([-0.70755981,  3.72528774,  0.93824469])
Decision Boundary
The BFGS algorithm has found θ = [-0.70755981, 3.72528774, 0.93824469], corresponding to a vector containing the Result.x argument, of the scypy.minimize function. In Step 3, we mentioned that, for Logistic Regression, the Hypothesis hθ(x) calculates the probability that the output can be 0 or 1. To map this probability to a discrete class (Bening/Malignant), we select a threshold that is 0.5 above which we will classify values as “1” and below which we classify values as “0”. So, what we want to do is tracking the so-called Decision Boundary. A Decision Boundary is a property of the Hypothesis and its parameters θ, and not a property of the dataset. Let’s plot the Radius and Texture features again, but with the adding of a red line, that represents the Decision Boundary for the found θ:
I think that it’s important to highlight that, surprisingly, in spite of Logistic Regression uses a non-linear (sigmoid) function into its Hypothesis model, the Decision Boundary is linear!
Calculate the accuracy
Now we want to calculate the accuracy of our algorithm. The function CalcAccuracy described in Step 8 will do this job:
The output of CalculateAccuracy is 89.1, that is a good accuracy score!
Make a prediction
Now that we have tested our algorithm and evaluated its accuracy, we want to make predictions. The following code represents a possible example of a Query: we want to know what is the outcome for a Radius = 18.00 and a Texture = 10.12:
Note that we have to normalize the Query with mu and sigma for the Scaling and Normalization. The outcome predicted is 0.79, which means that for a Radius = 18 and a Texture =10.12, the risk of malignancy is nearby to 1.
In this step, we will modify the previous code for the handling of multiple variables. There are few things to update: the first concerns the Feature Scaling and Normalization function:
Other minor revisions concern the code for uploading the X vector and the code for the Query. The following code reassembles what we have done until now, extending all the functions to the use of the Logistic Regression with multiple variables. Copy and paste the following code in a new Jupyter Notebook cell:
Code 24 will predict the risk of malignancy for: Radius = 5.00, Texture = 1.10, and W_Concave_points = 0.4. After all the calculations it will produce the following output:
Your Query is: 5.0, 1.1, 0.4BFGS Message: Optimization terminated successfully.Theta found: -1.40776926  3.13487429  1.52552603  4.02250701Train Accuracy: 95.43057996485061Prediction: 0.8154762110038786
The prediction of malignancy for these input values is nearby 1. Modifying the Query, you can experiment by yourself how the probability changes.
Logistic Regression is a powerful Machine Learning tool, and we can use it successfully for predicting categorical outputs of biomedical data. Data wrangling and data mining can benefit from excellent performances offered by Python and its libraries so well supported by the community. Linear Algebra programming has intrinsic advantages in avoiding, where possible, ‘while’ and ‘for’ loops. It is implementable by numpy, a package that vectorizes the matrixes. Numpy makes working on them more comfortable, and guarantees better control over the operations, especially for large arrays.
Moreover, the Machine Learning scenario with Python is enriched by the presence of many powerful packages (i.e., Tensorflow, Scikit-learn, and other that, for technical reasons, we haven’t mentioned in this post), which provide excellently optimized classifications and predictions on data.
While Machine Learning, equipped with Python and all its accessories, can represent the path toward the future of preventive and diagnostic Medicine, limitations in comprehending biological variables could make this path an awkward road.
The Wisconsin Breast Cancer (Diagnostic) Data Set, with its 569 patients and 30 features, offers an exhaustive assortment of parameters for classification and for this reason represents a perfect example for Machine Learning applications. Anyway, many of these features seem to be redundant, and a definite impact on classification and prediction by some of them remains still unknown.
We have introduced a Bivariate Analysis in Step 3, to reduce the number of redundant features. An in-depth discussion on the role of these features in the prediction should deserve a dedicated article.
I hope you find this post useful!
Mangasarian, Olvi & Street, Nick & Wolberg, William. (1970). Breast Cancer Diagnosis and Prognosis Via Linear Programming. Operations Research. 43. 10.1287/opre.43.4.570.https://www.cancer.org/cancer/breast-cancer/screening-tests-and-early-detection/breast-biopsy/fine-needle-aspiration-biopsy-of-the-breast.html#referencesAndrew NG, Machine Learning | Coursera.Chris Albon, Machine Learning with Python Cookbook, O’Really, ISBN-13: 978–1491989388.Nocedal, J, and S J Wright. 2006. Numerical Optimization. Springer New York.
Mangasarian, Olvi & Street, Nick & Wolberg, William. (1970). Breast Cancer Diagnosis and Prognosis Via Linear Programming. Operations Research. 43. 10.1287/opre.43.4.570.
https://www.cancer.org/cancer/breast-cancer/screening-tests-and-early-detection/breast-biopsy/fine-needle-aspiration-biopsy-of-the-breast.html#references
Andrew NG, Machine Learning | Coursera.
Chris Albon, Machine Learning with Python Cookbook, O’Really, ISBN-13: 978–1491989388.
Nocedal, J, and S J Wright. 2006. Numerical Optimization. Springer New York.