Machine Learning with Koalas and Spark | by MA Raza, Ph.D. | Towards Data Science

Pandas is considered a de-facto data analysis library written in python. Most of the data scientists or machine learning engineers start with Pandas and Numpy before moving to other libraries. No once can debate around the use of Pandas as a standard data processing library. There are a lot of benefits of using Pandas however one key bottleneck of Pandas API to adapt with distributed processing. Solutions like Modin and Dask solve this problem to some extent.
When it comes to using distributed processing frameworks, Spark is the de-facto choice for professionals and large data processing hubs. Recently, Databricks’s team open-sourced a library called Koalas to implemented the Pandas API with spark backend. This library is under active development and covering more than 60% of Pandas API. To read more about using Koalas, refer to my earlier article Spark-ifying Pandas: Databrick’s Koalas with Google Colab.
In this tutorial, I will walk you through to perform exploratory data analysis using Koalas and PySpark to build a regression model using the Spark distributed framework. There are a lot of benefits of using Koalas instead of Pandas API when dealing with large datasets. Some of the key points are
Big data processing made easyQuick transformation from Pandas to KoalasIntegration with PySpark is seamless
Big data processing made easy
Quick transformation from Pandas to Koalas
Integration with PySpark is seamless
The objective of this tutorial is to leverage the Spark backend for a complete machine learning development cycle using Koalas. The working google collaboratory will be embedded.
As a first step, I configure the google colab runtime with spark installation. For details, readers may read my article Getting Started Spark 3.0.0 in Google Colab om medium.
We will install the below programs
Java 8
spark-3.0.1
Hadoop3.2
Findspark
you can install the LATEST version of Spark using the below set of commands.
# Run below commands!apt-get install openjdk-8-jdk-headless -qq > /dev/null!wget -q http://apache.osuosl.org/spark/spark-3.0.1/spark-3.0.1-bin-hadoop3.2.tgz!tar xf spark-3.0.1-bin-hadoop3.2.tgz!pip install -q findspark
After installing the spark and Java, set the environment variables where Spark and Java are installed.
import osos.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"os.environ["SPARK_HOME"] = "/content/spark-3.0.1-bin-hadoop3.2"
Let us test the installation of spark in our google colab environment.
import findsparkfindspark.init()from pyspark.sql import SparkSessionspark = SparkSession.builder.master("local[*]").getOrCreate()# Test the spark df = spark.createDataFrame([{"hello": "world"} for x in range(1000)])df.show(3, False)/content/spark-3.0.1-bin-hadoop3.2/python/pyspark/sql/session.py:381: UserWarning: inferring schema from dict is deprecated,please use pyspark.sql.Row instead  warnings.warn("inferring schema from dict is deprecated,"+-----+|hello|+-----+|world||world||world|+-----+only showing top 3 rows
After installing spark and making sure, it is working, we can now install the databrick’s Koalas using pip.
! pip install koalasRequirement already satisfied: koalas in /usr/local/lib/python3.6/dist-packages (1.2.0)Requirement already satisfied: numpy<1.19.0,>=1.14 in /usr/local/lib/python3.6/dist-packages (from koalas) (1.18.5)Requirement already satisfied: matplotlib<3.3.0,>=3.0.0 in /usr/local/lib/python3.6/dist-packages (from koalas) (3.2.2)Requirement already satisfied: pyarrow>=0.10 in /usr/local/lib/python3.6/dist-packages (from koalas) (0.14.1)Requirement already satisfied: pandas<1.1.0,>=0.23.2 in /usr/local/lib/python3.6/dist-packages (from koalas) (1.0.5)Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib<3.3.0,>=3.0.0->koalas) (2.4.7)Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib<3.3.0,>=3.0.0->koalas) (0.10.0)Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib<3.3.0,>=3.0.0->koalas) (1.2.0)Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib<3.3.0,>=3.0.0->koalas) (2.8.1)Requirement already satisfied: six>=1.0.0 in /usr/local/lib/python3.6/dist-packages (from pyarrow>=0.10->koalas) (1.15.0)Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas<1.1.0,>=0.23.2->koalas) (2018.9)Requirement already satisfied: pyarrow in /usr/local/lib/python3.6/dist-packages (0.14.1)Requirement already satisfied: numpy>=1.14 in /usr/local/lib/python3.6/dist-packages (from pyarrow) (1.18.5)Requirement already satisfied: six>=1.0.0 in /usr/local/lib/python3.6/dist-packages (from pyarrow) (1.15.0)# Install compatible version of pyarrow! pip install pyarrowimport seaborn as sns/usr/local/lib/python3.6/dist-packages/statsmodels/tools/_testing.py:19: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.  import pandas.util.testing as tmimport databricks.koalas as ks
The standard practice for developing machine learning models is to perform the exploratory data analysis, perform feature engineering, and build a model. I will try to touch basis on the above point using Koalas. I believe most of the ML engineers/scientists use Pandas API to perform EDA and Feature engineering. I demonstrate the use of Koalas to do the job and use PySpark to build the model.
I will use the Combined Cycle Power Plant data set to predict the net hourly electrical output (EP). I have uploaded the data to my GitHub so that users can reproduce the results.
As a first step, I want to explore the given data, its distribution, and dependency using Koalas API. I will include a simple example to demonstrate the idea, users can extend it for the problem in-hand.
Download the data and saved it locally
# Downloading the clustering dataset!wget -q 'https://raw.githubusercontent.com/amjadraza/blogs-data/master/spark_ml/ccpp.csv'
Read the data using the Koalas read_csv method. To read more about the API follow the Koalas official documentation
# Read the iris datakdf_ccpp = ks.read_csv("ccpp.csv")kdf_ccpp.columnsIndex(['AT', 'V', 'AP', 'RH', 'PE'], dtype='object')kdf_ccpp.head()
Just like Pandas, Koalas has a feature to plot data to understand the variables. In the below example, I plotted the original data and smoother versions of it. This example demonstrates the use of plot and rolling window methods
kdf_ccpp['AT'].plot(figsize=(12,6))kdf_ccpp['AT'].rolling(window=20).mean().plot()kdf_ccpp['AT'].rolling(window=200).mean().plot()<matplotlib.axes._subplots.AxesSubplot at 0x7f9992f63320>
Just like the above plotting, users can plot all the columns. See below command
kdf_ccpp.plot(figsize=(12,6))<matplotlib.axes._subplots.AxesSubplot at 0x7f9993b7e668>
Plotting all columns using a moving average of 20 data points.
kdf_ccpp.rolling(window=20).mean().plot(figsize=(12,6))<matplotlib.axes._subplots.AxesSubplot at 0x7f99939bd358>
The below command is to demonstrate the use of describe a method similar to in Pandas API.
kdf_ccpp.describe()
The feature engineering step usually comes in junction with EDA. Users prepare the features with an objective to have the most predictive and nicely distributed feature set. In this article, I will demonstrate the use of Koalas to perform feature engineering.
To understand the relationship between different variables, paiprplot function seaborn is widely used. Use the below command to plot the pairplot of variables in the dataset. Since Seaborn does not support the Koala dataframe, users have to convert it into a pandas dataframe before calling pairplot.
sns.pairplot(kdf_ccpp.to_pandas())<seaborn.axisgrid.PairGrid at 0x7f99989459e8>
By looking at the above figure, we can see a lot of outliers and for some variable relationship with Target is not clear. To remove the outliers, the simplest solution is to calculate the moving average and I demonstrate that using Koalas.
sns.pairplot(kdf_ccpp.rolling(window=20).mean().to_pandas())<seaborn.axisgrid.PairGrid at 0x7f9993a5ada0>
Looks like the 20-day moving average has a better relationship with the Target variable hence using the 20-day average features makes more sense.
Once the EDA and Feature engineering done, it is time to build the predictive model. One of the benefits of using the Koalas dataframe is that users can create a Spark dataframe seamlessly. In the below section, I demonstrate the use of PySpark API to build and train Gradient Boosting Machines (GBM).
# Create the moving average featureskdf = kdf_ccpp.rolling(window=20, min_periods=1).mean()# Convert the Koalas DataFrame into Spark DataFramesdf = kdf.to_spark()sdf.show(5,False)+------------------+------+------------------+-----------------+-----------------+|AT                |V     |AP                |RH               |PE               |+------------------+------+------------------+-----------------+-----------------+|14.96             |41.76 |1024.07           |73.17            |463.26           ||20.07             |52.36 |1022.055          |66.125           |453.815          ||15.083333333333334|48.04 |1018.7566666666667|74.79666666666667|465.3966666666667||16.5275           |50.36 |1016.6275         |75.2575          |460.6675         ||15.386000000000001|47.788|1015.1479999999999|79.53            |463.314          |+------------------+------+------------------+-----------------+-----------------+only showing top 5 rows
Now, build the model using PySpark API. For more details on building models using PySpark refer to my article Machine Learning With Spark.
from pyspark.ml.regression import LinearRegressionfrom pyspark.ml.feature import VectorAssemblerfrom pyspark.ml.evaluation import RegressionEvaluatorfrom pyspark.ml.regression import GBTRegressor
Prepare the features compatible with PySpark models
# Create the feature column using VectorAssembler classvectorAssembler = VectorAssembler(inputCols =["AT", "V", "AP", "RH"], outputCol = "features")vpp_sdf = vectorAssembler.transform(sdf)vpp_sdf.show(2, False)+-----+-----+--------+------+-------+-----------------------------+|AT   |V    |AP      |RH    |PE     |features                     |+-----+-----+--------+------+-------+-----------------------------+|14.96|41.76|1024.07 |73.17 |463.26 |[14.96,41.76,1024.07,73.17]  ||20.07|52.36|1022.055|66.125|453.815|[20.07,52.36,1022.055,66.125]|+-----+-----+--------+------+-------+-----------------------------+only showing top 2 rows
Create the tarin and test splits
# Define train and test data splitsplits = vpp_sdf.randomSplit([0.7,0.3])train_df = splits[0]test_df = splits[1]
Build and train the model
# Define the GBT Modelgbt = GBTRegressor(featuresCol="features", labelCol="PE")gbt_model = gbt.fit(train_df)gbt_predictions = gbt_model.transform(test_df)
Evaluate the model accuracy
# Evaluate the GBT Modelgbt_evaluator = RegressionEvaluator(labelCol="PE", predictionCol="prediction", metricName="rmse")gbt_rmse = gbt_evaluator.evaluate(gbt_predictions)print("The RMSE of GBT Tree regression Model is {}".format(gbt_rmse))The RMSE of GBT Tree regression Model is 1.077464978110743
Converting prediction back to Koalas DataFrame
kdf_predictions = ks.DataFrame(gbt_predictions)kdf_predictions.head()
Let us plot the actual and predictions from the model.
kdf_predictions[['PE', 'prediction']].plot(figsize=(12,8))<matplotlib.axes._subplots.AxesSubplot at 0x7f9992ba4828> 
Below is the working google colab notebook to recreate the tutorial. Give it a try and develop machine learning algorithms on top of the presented use of Koalas as Pandas replacement where possible.
In this tutorial, I have demonstrated the use of Koalas to perform exploratory data analysis and feature engineering. For Pandas users, switching to Koalas is straight forward with the benefit of using Spark backend for distributed computations. Below the key points discussed
Koalas use to perform EDA
Feature Engineering using Koalas
PySpark integration with Koalas