Univariate Linear Regression in Pyspark
By: Jayaprakash Nallathambi
Tags: Regression, Linear Regression, Data Science, Pyspark

Building machine learning models for big data has gained lot of pace in the recent years, I like to demonstrate univariate linear regression in Pyspark along with my other posts in R and Python.

Introduction

Linear regression is a technique to identify the linear relationship between independent variables and dependent variables. In LR the dependent variable is always a continuous variable.

Linear regression with one independent variable is called Univariate linear regression

The equation for linear regression is

Linear Equation

Loading Libraries

from pyspark.sql import SparkSession

from pyspark.ml.regression import LinearRegression

from pyspark.ml.feature import VectorAssembler

import matplotlib.pyplot as plt

Create Spark Session

spark = SparkSession.builder.appName('Univariate LR').getOrCreate()

spark

Loading Data

file_path = 'input/ex1data1.txt'

data = spark.read.csv(file_path, inferSchema=True, header=False).toDF('population','profit')

data.printSchema() root

|-- population: double (nullable = true)

|-- profit: double (nullable = true) data.show(5) +----------+------+

|population|profit|

+----------+------+

| 6.1101|17.592|

| 5.5277|9.1302|

| 8.5186|13.662|

| 7.0032|11.854|

| 5.8598|6.8233|

+----------+------+

only showing top 5 rows data.describe().show() +-------+-----------------+-----------------+

|summary| population| profit|

+-------+-----------------+-----------------+

| count| 97| 97|

| mean|8.159800000000002| 5.83913505154639|

| stddev|3.869883527882331|5.510262255231546|

| min| 5.0269| -2.6807|

| max| 22.203| 24.147|

+-------+-----------------+-----------------+



Feature Transformation

In spark MLib, input variables or features are represented as vectors and by default carries the name as ‘features’. The below code uses VectorAssembler to assemble input variables as vector (in this case only one predictor ‘population’)

assembler = VectorAssembler(inputCols=['population'], outputCol='features')

df = assembler.transform(data)

df.printSchema() root

|-- population: double (nullable = true)

|-- profit: double (nullable = true)

|-- features: vector (nullable = true) df = df.select(['features','profit'])



Model

lr = LinearRegression(maxIter=10, labelCol='profit')

lrModel = lr.fit(df)

Intercept and Coefficients

print(f'Intercept: {lrModel.intercept}

Coefficient: {lrModel.coefficients.values}') Intercept: -3.895780878311882

Coefficient: [1.19303364]

Model Evaluation Metrics

Spark MLib gives all required evaluation metrics for the respective models which can be accessed as attributes of model summary as shown below.

modelsummary = lrModel.summary print(f'Explained Variance: {modelsummary.explainedVariance}

R Squared: {modelsummary.r2}')

print(f'Std. Error: {modelsummary.coefficientStandardErrors}

Root Mean Squared Err: {modelsummary.rootMeanSquaredError}')

print(f'Mean Absolute Err: {modelsummary.meanAbsoluteError}

Mean Squared Err: {modelsummary.meanSquaredError}')

print(f'P-value: {modelsummary.pValues}')

modelsummary.residuals.show(5)

print(f'Num Iterations: {modelsummary.totalIterations}

Objective History: {modelsummary.objectiveHistory}')

Explained Variance: 21.0960268527634

R Squared: 0.70203155378414

Std. Error: [0.07974394383258744, 0.7194828241406706]

Root Mean Squared Err: 2.9923139460876023

Mean Absolute Err: 2.1942453988270034

Mean Squared Err: 8.953942751950358

P-value: [0.0, 4.607886650020987e-07]

+------------------+

| residuals|

+------------------+

|14.198226008949028|

| 6.43124880332505|

| 7.394804476918388|

|7.3947276613233015|

|3.7281423300896854|

+------------------+

only showing top 5 rows



Num Iterations: 1

Objective History: [0.0]

Objective history gives the history of p-values for each iterations from starting point.

Predictions

The model summary has a collection called predictions which contains the features (inputs to model), label (target) and predictions as dataframe.

modelsummary.predictions.show(5) +--------+------+------------------+

|features|profit| prediction|

+--------+------+------------------+

|[6.1101]|17.592|3.3937739910509706|

|[5.5277]|9.1302| 2.69895119667495|

|[8.5186]|13.662| 6.267195523081613|

|[7.0032]|11.854| 4.459272338676698|

|[5.8598]|6.8233|3.0951576699103143|

+--------+------+------------------+

only showing top 5 rows

Conclusion

With other posts in R and Python, the intercept and coefficient we got using pyspark is also same. It is better to know how to achieve similar things in multiple languages as data scientist and with recent increase in demand for spark (big data ML), I advise to try your hands on your variant of Spark.