Healthcare Dataset with Spark
By: Alena Nazarava
Tags: Sql, Spark, Data Science

Spark is an open source project from Apache. It is also the most commonly used analytics engine for big data and machine learning.

This post will be focused on a quick start to develop a prediction algorithm with Spark.

I chose ‘Healthcare Dataset Stroke Data’ dataset to work with from kaggle.com, the world’s largest community of data scientists and machine learning.

Content:

According to the World Health Organization, ischaemic heart disease and stroke are the world’s biggest killers.

Information from official site: http://www.who.int/news-room/fact-sheets/detail/the-top-10-causes-of-death

What we need to do is to predict the stroke probability using the given information of patients. It is a classification problem, where we will try to predict the probability of an observation belonging to a category (in our case probability of having a stroke).

There are lot of algorithms to solve classification problems I will use the Decision Tree algorithm.

Setting up Spark and getting data

from pyspark.sql import SparkSession

import pyspark.sql as sparksql

spark = SparkSession.builder.appName('stroke').getOrCreate() train = spark.read.csv('train_2v.csv', inferSchema=True,header=True)

Exploration of data

The first operation to perform after importing data is to get some information of what it looks like. It’s possible to do with the following commands:

df.printSchema()

df.describe()

df.dtypes

train.groupBy('stroke').count().show() +------+-----+

|stroke|count|

+------+-----+

| 1| 783|

| 0|42617|

+------+-----+

As can be seen from this observation. This is an Imbalanced dataset, where the number of observations belonging to one class is significantly lower than those belonging to the other classes. In this case, the predictive model could be biased and inaccurate. There are different strategies to handling Imbalanced Datasets, hence it is out of scope for this post, instead I will focus on Spark. To find more information about imbalanced dataset:

https://www.analyticsvidhya.com/blog/2017/03/imbalanced-classification-problem/

Here we have clinical measurements (e.g. Hypertension, heart_disease, age, family history of disease) for a number of patients, as well as information about whether each patient has had a stroke. In practice, we want this method to accurately predict stroke risk for future patients based on their clinical measurements.

Analysis

Perform brief analysis using basic operations. It’s possible to do in several ways:

DataFrames provide a domain-specific language for structured data manipulation, access to a DataFrame’s columns can be by attribute or by indexing.

to run SQL queries programmatically and return the result as a DataFrame

For instance, to see what type of work has more cases of stroke we can do the following:

# create DataFrame as a temporary view

train.createOrReplaceTempView('table')

SQL query

spark.sql("SELECT work_type, count(work_type) as work_type_count FROM table WHERE stroke == 1 GROUP BY work_type ORDER BY work_type_count DESC").show() +-------------+---------------+

| work_type|work_type_count|

+-------------+---------------+

| Private| 441|

|Self-employed| 251|

| Govt_job| 89|

| children| 2|

+-------------+---------------+

Looks like Private occupation is the most dangerous work type in this dataset.

Let’s find out who participated in this clinic measurement.

spark.sql("SELECT gender, count(gender) as count_gender, count(gender)*100/sum(count(gender)) over() as percent FROM table GROUP BY gender").show() +------+------------+-------------------+

|gender|count_gender| percent|

+------+------------+-------------------+

|Female| 25665| 59.13594470046083|

| Other| 11|0.02534562211981567|

| Male| 17724| 40.83870967741935|

+------+------------+-------------------+

59% of all people are Female and only 40% are Male that participated in stroke research.

From this information there is possibility to retrieve information about how many Female/Male have a stroke:

spark.sql("SELECT gender, count(gender), (COUNT(gender) * 100.0) /(SELECT count(gender) FROM table WHERE gender == 'Male') as percentage FROM table WHERE stroke = '1' and gender = 'Male' GROUP BY gender").show() +------+-------------+--------------------+

|gender|count(gender)| percentage|

+------+-------------+--------------------+

| Male| 352|1.986007673211464...|

+------+-------------+--------------------+

spark.sql("SELECT gender, count(gender), (COUNT(gender) * 100.0) /(SELECT count(gender) FROM table WHERE gender == 'Female') as percentage FROM table WHERE stroke = '1' and gender = 'Female' GROUP BY gender").show() +------+-------------+--------------------+

|gender|count(gender)| percentage|

+------+-------------+--------------------+

|Female| 431|1.679329826612117...|

+------+-------------+--------------------+

1,68% Female and almost 2% Male have had a stroke.

We also can see if the age has an influence on stroke and what is the risk by age.

spark.sql("SELECT age, count(age) as age_count FROM table WHERE stroke == 1 GROUP BY age ORDER BY age_count DESC").show() +----+---------+

| age|age_count|

+----+---------+

|79.0| 70|

|78.0| 57|

|80.0| 49|

|81.0| 43|

|82.0| 36|

|70.0| 25|

|77.0| 24|

|74.0| 24|

|76.0| 24|

|67.0| 23|

|75.0| 23|

|72.0| 21|

|68.0| 20|

|59.0| 20|

|69.0| 20|

|71.0| 19|

|57.0| 19|

|63.0| 18|

|65.0| 18|

|66.0| 17|

+----+---------+

only showing top 20 rows

I can use filter operation to calculate the number of stroke cases for people after 50 years.

train.filter((train['stroke'] == 1) & (train['age'] > '50')).count() result: 708

As we can see Age is an important risk factor for developing a stroke.

Cleaning data

The next step of exploration is to deal with categorical and missing values. There are missing values for smoking_status and bmi parameters.

I will fill out smoking_status with a value of ‘No info’ and bmi parameter with mean value.

# fill in missing values

train_f = train.na.fill('No Info', subset=['smoking_status']) # fill in miss values with mean

from pyspark.sql.functions import mean

mean = train_f.select(mean(train_f['bmi'])).collect()

mean_bmi = mean[0][0]

train_f = train_f.na.fill(mean_bmi,['bmi'])

Most of ML algorithms cannot work directly with categorical data. The encoding allows algorithms which expect continuous features to use categorical features.

StringIndexer -> OneHotEncoder -> VectorAssembler

from pyspark.ml.feature import (VectorAssembler,OneHotEncoder,

StringIndexer)

It does not need to know how many categories in a feature beforehand the combination of StringIndexer and OneHotEncoder take care of it.

The next step is to create an assembler, that combines a given list of columns into a single vector column to train ML model. I will use the vector columns, that we got after one_hot_encoding.

assembler = VectorAssembler(inputCols=['genderVec',

'age',

'hypertension',

'heart_disease',

'ever_marriedVec',

'work_typeVec',

'Residence_typeVec',

'avg_glucose_level',

'bmi',

'smoking_statusVec'],outputCol='features')

Then we will create a DecisionTree object. To do this we need to import DecisionTreeClassifier.

from pyspark.ml.classification import DecisionTreeClassifier dtc = DecisionTreeClassifier(labelCol='stroke',featuresCol='features')

So far we have kind of a complex task that contains bunch of stages, that need to be performed to process data. To wrap all of that Spark ML represents such a workflow as a Pipeline, which consists of a sequence of PipelineStages to be run in a specific order.

from pyspark.ml import Pipeline pipeline = Pipeline(stages=[gender_indexer, ever_married_indexer, work_type_indexer, Residence_type_indexer,

smoking_status_indexer, gender_encoder, ever_married_encoder, work_type_encoder,

Residence_type_encoder, smoking_status_encoder, assembler, dtc])

The next step is to split dataset to train and test.

train_data,test_data = train_f.randomSplit([0.7,0.3])

What I’m going to do now is to fit the model. For this I will use the pipeline that was created and train_data

model = pipeline.fit(train_data)

After that transform the test_data.

dtc_predictions = model.transform(test_data)

Now it’s time to evaluate a model

from pyspark.ml.evaluation import MulticlassClassificationEvaluator # Select (prediction, true label) and compute test error

acc_evaluator = MulticlassClassificationEvaluator(labelCol="stroke", predictionCol="prediction", metricName="accuracy") dtc_acc = acc_evaluator.evaluate(dtc_predictions) print('A Decision Tree algorithm had an accuracy of: {0:2.2f}%'.format(dtc_acc*100))

A Decision Tree algorithm had an accuracy of: 98.08%

As was defined in the beginning the predictive model of an imbalanced dataset could be with misleading accuracy.

Conclusion

Apache Spark is an open-source framework, it is very concise and easy to use.