Healthcare Dataset with Spark. Spark is an open source project from... | by Alena Nazarava | Towards Data Science

Spark is an open source project from Apache. It is also the most commonly used analytics engine for big data and machine learning.
This post will be focused on a quick start to develop a prediction algorithm with Spark.
I chose ‘Healthcare Dataset Stroke Data’ dataset to work with from kaggle.com, the world’s largest community of data scientists and machine learning.
Content:
According to the World Health Organization, ischaemic heart disease and stroke are the world’s biggest killers.
Information from official site: http://www.who.int/news-room/fact-sheets/detail/the-top-10-causes-of-death
What we need to do is to predict the stroke probability using the given information of patients. It is a classification problem, where we will try to predict the probability of an observation belonging to a category (in our case probability of having a stroke).
There are lot of algorithms to solve classification problems I will use the Decision Tree algorithm.
Setting up Spark and getting data
from pyspark.sql import SparkSessionimport pyspark.sql as sparksqlspark = SparkSession.builder.appName('stroke').getOrCreate()train = spark.read.csv('train_2v.csv', inferSchema=True,header=True)
The first operation to perform after importing data is to get some information of what it looks like. It’s possible to do with the following commands:
df.printSchema()
df.describe()
df.dtypes
train.groupBy('stroke').count().show()+------+-----+|stroke|count|+------+-----+|     1|  783||     0|42617|+------+-----+
As can be seen from this observation. This is an Imbalanced dataset, where the number of observations belonging to one class is significantly lower than those belonging to the other classes. In this case, the predictive model could be biased and inaccurate. There are different strategies to handling Imbalanced Datasets, hence it is out of scope for this post, instead I will focus on Spark. To find more information about imbalanced dataset:
https://www.analyticsvidhya.com/blog/2017/03/imbalanced-classification-problem/
Here we have clinical measurements (e.g. Hypertension, heart_disease, age, family history of disease) for a number of patients, as well as information about whether each patient has had a stroke. In practice, we want this method to accurately predict stroke risk for future patients based on their clinical measurements.
Perform brief analysis using basic operations. It’s possible to do in several ways:
DataFrames provide a domain-specific language for structured data manipulation, access to a DataFrame’s columns can be by attribute or by indexing.
to run SQL queries programmatically and return the result as a DataFrame
For instance, to see what type of work has more cases of stroke we can do the following:
# create DataFrame as a temporary viewtrain.createOrReplaceTempView('table')
SQL query
spark.sql("SELECT work_type, count(work_type) as work_type_count FROM table WHERE stroke == 1 GROUP BY work_type ORDER BY work_type_count DESC").show()+-------------+---------------+|    work_type|work_type_count|+-------------+---------------+|      Private|            441||Self-employed|            251||     Govt_job|             89||     children|              2|+-------------+---------------+
Looks like Private occupation is the most dangerous work type in this dataset.
Let’s find out who participated in this clinic measurement.
spark.sql("SELECT gender, count(gender) as count_gender, count(gender)*100/sum(count(gender)) over() as percent FROM table GROUP BY gender").show()+------+------------+-------------------+|gender|count_gender|            percent|+------+------------+-------------------+|Female|       25665|  59.13594470046083|| Other|          11|0.02534562211981567||  Male|       17724|  40.83870967741935|+------+------------+-------------------+
59% of all people are Female and only 40% are Male that participated in stroke research.
From this information there is possibility to retrieve information about how many Female/Male have a stroke:
spark.sql("SELECT gender, count(gender), (COUNT(gender) * 100.0) /(SELECT count(gender) FROM table WHERE gender == 'Male') as percentage FROM table WHERE stroke = '1' and gender = 'Male' GROUP BY gender").show()+------+-------------+--------------------+|gender|count(gender)|          percentage|+------+-------------+--------------------+|  Male|          352|1.986007673211464...|+------+-------------+--------------------+spark.sql("SELECT gender, count(gender), (COUNT(gender) * 100.0) /(SELECT count(gender) FROM table WHERE gender == 'Female') as percentage FROM table WHERE stroke = '1' and gender = 'Female' GROUP BY gender").show()+------+-------------+--------------------+|gender|count(gender)|          percentage|+------+-------------+--------------------+|Female|          431|1.679329826612117...|+------+-------------+--------------------+
1,68% Female and almost 2% Male have had a stroke.
We also can see if the age has an influence on stroke and what is the risk by age.
spark.sql("SELECT age, count(age) as age_count FROM table WHERE stroke == 1 GROUP BY age ORDER BY age_count DESC").show()+----+---------+| age|age_count|+----+---------+|79.0|       70||78.0|       57||80.0|       49||81.0|       43||82.0|       36||70.0|       25||77.0|       24||74.0|       24||76.0|       24||67.0|       23||75.0|       23||72.0|       21||68.0|       20||59.0|       20||69.0|       20||71.0|       19||57.0|       19||63.0|       18||65.0|       18||66.0|       17|+----+---------+only showing top 20 rows
I can use filter operation to calculate the number of stroke cases for people after 50 years.
train.filter((train['stroke'] == 1) & (train['age'] > '50')).count()result: 708
As we can see Age is an important risk factor for developing a stroke.
The next step of exploration is to deal with categorical and missing values. There are missing values for smoking_status and bmi parameters.
I will fill out smoking_status with a value of ‘No info’ and bmi parameter with mean value.
# fill in missing valuestrain_f = train.na.fill('No Info', subset=['smoking_status'])# fill in miss values with meanfrom pyspark.sql.functions import meanmean = train_f.select(mean(train_f['bmi'])).collect()mean_bmi = mean[0][0]train_f = train_f.na.fill(mean_bmi,['bmi'])
Most of ML algorithms cannot work directly with categorical data. The encoding allows algorithms which expect continuous features to use categorical features.
StringIndexer -> OneHotEncoder -> VectorAssembler
from pyspark.ml.feature import (VectorAssembler,OneHotEncoder,                                StringIndexer)
It does not need to know how many categories in a feature beforehand the combination of StringIndexer and OneHotEncoder take care of it.
The next step is to create an assembler, that combines a given list of columns into a single vector column to train ML model. I will use the vector columns, that we got after one_hot_encoding.
assembler = VectorAssembler(inputCols=['genderVec', 'age', 'hypertension', 'heart_disease', 'ever_marriedVec', 'work_typeVec', 'Residence_typeVec', 'avg_glucose_level', 'bmi', 'smoking_statusVec'],outputCol='features')
Then we will create a DecisionTree object. To do this we need to import DecisionTreeClassifier.
from pyspark.ml.classification import DecisionTreeClassifierdtc = DecisionTreeClassifier(labelCol='stroke',featuresCol='features')
So far we have kind of a complex task that contains bunch of stages, that need to be performed to process data. To wrap all of that Spark ML represents such a workflow as a Pipeline, which consists of a sequence of PipelineStages to be run in a specific order.
from pyspark.ml import Pipelinepipeline = Pipeline(stages=[gender_indexer, ever_married_indexer, work_type_indexer, Residence_type_indexer,                           smoking_status_indexer, gender_encoder, ever_married_encoder, work_type_encoder,                           Residence_type_encoder, smoking_status_encoder, assembler, dtc])
The next step is to split dataset to train and test.
train_data,test_data = train_f.randomSplit([0.7,0.3])
What I’m going to do now is to fit the model. For this I will use the pipeline that was created and train_data
model = pipeline.fit(train_data)
After that transform the test_data.
dtc_predictions = model.transform(test_data)
Now it’s time to evaluate a model
from pyspark.ml.evaluation import MulticlassClassificationEvaluator# Select (prediction, true label) and compute test erroracc_evaluator = MulticlassClassificationEvaluator(labelCol="stroke", predictionCol="prediction", metricName="accuracy")dtc_acc = acc_evaluator.evaluate(dtc_predictions)print('A Decision Tree algorithm had an accuracy of: {0:2.2f}%'.format(dtc_acc*100))
A Decision Tree algorithm had an accuracy of: 98.08%
As was defined in the beginning the predictive model of an imbalanced dataset could be with misleading accuracy.
Apache Spark is an open-source framework, it is very concise and easy to use.