Building A Linear Regression with PySpark and MLlib | by Susan Li | Towards Data Science

Apache Spark has become one of the most commonly used and supported open-source tools for machine learning and data science.
In this post, I’ll help you get started using Apache Spark’s spark.ml Linear Regression for predicting Boston housing prices. Our data is from the Kaggle competition: Housing Values in Suburbs of Boston. For each house observation, we have the following information:
CRIM — per capita crime rate by town.
ZN — proportion of residential land zoned for lots over 25,000 sq.ft.
INDUS — proportion of non-retail business acres per town.
CHAS — Charles River dummy variable (= 1 if tract bounds river; 0 otherwise).
NOX — nitrogen oxides concentration (parts per 10 million).
RM — average number of rooms per dwelling.
AGE — proportion of owner-occupied units built prior to 1940.
DIS — weighted mean of distances to five Boston employment centres.
RAD — index of accessibility to radial highways.
TAX — full-value property-tax rate per $10,000.
PTRATIO — pupil-teacher ratio by town.
BLACK — 1000(Bk — 0.63)2 where Bk is the proportion of blacks by town.
LSTAT — lower status of the population (percent).
MV — median value of owner-occupied homes in $1000s. This is the target variable.
The input data set contains data about details of various houses. Based on the information provided, the goal is to come up with a model to predict median value of a given house in the area.
from pyspark import SparkConf, SparkContextfrom pyspark.sql import SQLContextsc= SparkContext()sqlContext = SQLContext(sc)house_df = sqlContext.read.format('com.databricks.spark.csv').options(header='true', inferschema='true').load('boston.csv')house_df.take(1)
[Row(CRIM=0.00632, ZN=18.0, INDUS=2.309999943, CHAS=0, NOX=0.537999988, RM=6.574999809, AGE=65.19999695, DIS=4.090000153, RAD=1, TAX=296, PT=15.30000019, B=396.8999939, LSTAT=4.980000019, MV=24.0)]
Print Schema in a tree format.
house_df.cache()house_df.printSchema()root |-- CRIM: double (nullable = true) |-- ZN: double (nullable = true) |-- INDUS: double (nullable = true) |-- CHAS: integer (nullable = true) |-- NOX: double (nullable = true) |-- RM: double (nullable = true) |-- AGE: double (nullable = true) |-- DIS: double (nullable = true) |-- RAD: integer (nullable = true) |-- TAX: integer (nullable = true) |-- PT: double (nullable = true) |-- B: double (nullable = true) |-- LSTAT: double (nullable = true) |-- MV: double (nullable = true)
Perform descriptive analytics
house_df.describe().toPandas().transpose()
Scatter matrix is a great way to roughly determine if we have a linear correlation between multiple independent variables.
import pandas as pdnumeric_features = [t[0] for t in house_df.dtypes if t[1] == 'int' or t[1] == 'double']sampled_data = house_df.select(numeric_features).sample(False, 0.8).toPandas()axs = pd.scatter_matrix(sampled_data, figsize=(10, 10))n = len(sampled_data.columns)for i in range(n):    v = axs[i, 0]    v.yaxis.label.set_rotation(0)    v.yaxis.label.set_ha('right')    v.set_yticks(())    h = axs[n-1, i]    h.xaxis.label.set_rotation(90)    h.set_xticks(())
It’s hard to see. Let’s find correlation between independent variables and target variable.
import sixfor i in house_df.columns:    if not( isinstance(house_df.select(i).take(1)[0][0], six.string_types)):        print( "Correlation to MV for ", i, house_df.stat.corr('MV',i))Correlation to MV for  CRIM -0.3883046116575088Correlation to MV for  ZN 0.36044534463752903Correlation to MV for  INDUS -0.48372517128143383Correlation to MV for  CHAS 0.17526017775291847Correlation to MV for  NOX -0.4273207763683772Correlation to MV for  RM 0.695359937127267Correlation to MV for  AGE -0.37695456714288667Correlation to MV for  DIS 0.24992873873512172Correlation to MV for  RAD -0.3816262315669168Correlation to MV for  TAX -0.46853593528654536Correlation to MV for  PT -0.5077867038116085Correlation to MV for  B 0.3334608226834164Correlation to MV for  LSTAT -0.7376627294671615Correlation to MV for  MV 1.0
The correlation coefficient ranges from –1 to 1. When it is close to 1, it means that there is a strong positive correlation; for example, the median value tends to go up when the number of rooms goes up. When the coefficient is close to –1, it means that there is a strong negative correlation; the median value tends to go down when the percentage of the lower status of the population goes up. Finally, coefficients close to zero mean that there is no linear correlation.
We are going to keep all the variables, for now.
Prepare data for Machine Learning. And we need two columns only — features and label(“MV”):
from pyspark.ml.feature import VectorAssemblervectorAssembler = VectorAssembler(inputCols = ['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD', 'TAX', 'PT', 'B', 'LSTAT'], outputCol = 'features')vhouse_df = vectorAssembler.transform(house_df)vhouse_df = vhouse_df.select(['features', 'MV'])vhouse_df.show(3)
splits = vhouse_df.randomSplit([0.7, 0.3])train_df = splits[0]test_df = splits[1]
from pyspark.ml.regression import LinearRegressionlr = LinearRegression(featuresCol = 'features', labelCol='MV', maxIter=10, regParam=0.3, elasticNetParam=0.8)lr_model = lr.fit(train_df)print("Coefficients: " + str(lr_model.coefficients))print("Intercept: " + str(lr_model.intercept))
Coefficients: [0.0,0.007302310571175137,-0.03286303124593804,1.4134773328268,-7.91932366863737,5.341921692409693,0.0,-0.5791187396097941,0.0,-0.0010503197747184644,-0.7748333592630333,0.01126108224671488,-0.3932170620689197]Intercept: 11.327590788070061
Summarize the model over the training set and print out some metrics:
trainingSummary = lr_model.summaryprint("RMSE: %f" % trainingSummary.rootMeanSquaredError)print("r2: %f" % trainingSummary.r2)
RMSE: 4.675914r2: 0.743627
RMSE measures the differences between predicted values by the model and the actual values. However, RMSE alone is meaningless until we compare with the actual “MV” value, such as mean, min and max. After such comparison, our RMSE looks pretty good.
train_df.describe().show()
R squared at 0.74 indicates that in our model, approximate 74% of the variability in “MV” can be explained using the model. This is in align with the result from Scikit-Learn. It is not bad. However, we must be cautious that the performance on the training set may not a good approximation of the performance on the test set.
lr_predictions = lr_model.transform(test_df)lr_predictions.select("prediction","MV","features").show(5)from pyspark.ml.evaluation import RegressionEvaluatorlr_evaluator = RegressionEvaluator(predictionCol="prediction", \                 labelCol="MV",metricName="r2")print("R Squared (R2) on test data = %g" % lr_evaluator.evaluate(lr_predictions))
test_result = lr_model.evaluate(test_df)print("Root Mean Squared Error (RMSE) on test data = %g" % test_result.rootMeanSquaredError)
Root Mean Squared Error (RMSE) on test data = 5.52048
Sure enough, we achieved worse RMSE and R squared on the test set.
print("numIterations: %d" % trainingSummary.totalIterations)print("objectiveHistory: %s" % str(trainingSummary.objectiveHistory))trainingSummary.residuals.show()
numIterations: 11objectiveHistory: [0.49999999999999956, 0.4281126976069304, 0.22539203628598917, 0.20185326295592582, 0.1686847843494657, 0.16588096079648484, 0.16543041085178495, 0.16508485781434112, 0.16484472289473545, 0.16454785266359198, 0.16447743850144508]
Using our Linear Regression model to make some predictions:
predictions = lr_model.transform(test_df)predictions.select("prediction","MV","features").show()
from pyspark.ml.regression import DecisionTreeRegressordt = DecisionTreeRegressor(featuresCol ='features', labelCol = 'MV')dt_model = dt.fit(train_df)dt_predictions = dt_model.transform(test_df)dt_evaluator = RegressionEvaluator(    labelCol="MV", predictionCol="prediction", metricName="rmse")rmse = dt_evaluator.evaluate(dt_predictions)print("Root Mean Squared Error (RMSE) on test data = %g" % rmse)
Root Mean Squared Error (RMSE) on test data = 4.39053
Feature Importance
dt_model.featureImportances
SparseVector(13, {0: 0.0496, 1: 0.0, 4: 0.0118, 5: 0.624, 6: 0.0005, 7: 0.1167, 8: 0.0044, 10: 0.013, 12: 0.1799})
house_df.take(1)
[Row(CRIM=0.00632, ZN=18.0, INDUS=2.309999943, CHAS=0, NOX=0.537999988, RM=6.574999809, AGE=65.19999695, DIS=4.090000153, RAD=1, TAX=296, PT=15.30000019, B=396.8999939, LSTAT=4.980000019, MV=24.0)]
Apparently, the number of rooms is the most important feature to predict the house median price in our data.
from pyspark.ml.regression import GBTRegressorgbt = GBTRegressor(featuresCol = 'features', labelCol = 'MV', maxIter=10)gbt_model = gbt.fit(train_df)gbt_predictions = gbt_model.transform(test_df)gbt_predictions.select('prediction', 'MV', 'features').show(5)
gbt_evaluator = RegressionEvaluator(    labelCol="MV", predictionCol="prediction", metricName="rmse")rmse = gbt_evaluator.evaluate(gbt_predictions)print("Root Mean Squared Error (RMSE) on test data = %g" % rmse)
Root Mean Squared Error (RMSE) on test data = 4.19795
Gradient-boosted tree regression performed the best on our data.
Source code can be found on Github. I am happy to hear any feedback or questions.