PySpark + MySQL Tutorial
By: Gregory Janesch
Tags: MySQL, Pyspark, Python, Linux, Data Science

PySpark + MySQL Tutorial

This post is meant as a short tutorial on how to set up PySpark to access a MySQL database and run a quick machine learning algorithm with it. Both PySpark and MySQL are locally installed onto a computer running Kubuntu 20.04 in this example, so this can be done without any external resources.

Requirements

Installing MySQL onto a Linux machine is fairly quick thanks to the apt package manager with sudo apt install mysql-server . Once it’s installed, you can run sudo mysql in a terminal to access MySQL from the command line:

Screenshot of the MySQL prompt in a console window.

For PySpark, just running pip install pyspark will install Spark as well as the Python interface. For this example, I’m also using mysql-connector-python and pandas to transfer the data from CSV files into the MySQL database. Spark can load CSV files directly, but that won’t be used for the sake of this example.

Finally, we need the Java drivers that will let Spark connect to MySQL. I didn’t see a good way to install them through apt , so I downloaded the driver from the MySQL website and installed it manually. You may need the location of the driver file later on, depending on where it was installed, so finding it may be necessary. On Kubuntu, the driver file was installed to “/usr/share/java/”.

The data that I’m using is the wine quality dataset from the UCI machine learning repository. It’s not especially large, with only around 6500 examples total, but it’s clean and easy to use for this example. There are a few possible ways to use it for machine learning or other predictive purposes, but I’m going to focus on predicting whether the wine in question is red or white.

Writing Data to MySQL

The first step is to read in the data. The only things worth noting here are that the files are semicolon-delimited, and we need to create the column for whether a wine is white or red ourselves:

First and last five rows of the data, including the new “is_red” column, as they appear in a dataframe.

MySQL is similarly straightforward as you just set up the new database and an appropriate table. You will need to ensure that your user has privileges to edit the table, though; if you need to change privileges, that will have to be done from the MySQL prompt.

And then load the data. MySQL can load multiple rows into a table at once if the contents of each row are contained within parentheses and comma-delimited, which a simple str.join() can do. Then upload into the database, using FLUSH TABLES to actually update the table with the rows (otherwise the changes are merely staged and would eventually be discarded once the connection is closed).

Accessing MySQL with PySpark

Starting a Spark session from Python is fairly straightforward. Again, I had to specify the location of the MySQL Java driver, which is the only subtlety that I found. Loading the table afterward is similarly simple, despite the number of options that need to be specified, though you’ll need the port number that MySQL is on when loading the data from the database (the default port is 3306).

If you’re loading data into Spark from a file, you’ll probably want to specify a schema to avoid making Spark infer it. For a MySQL database, however, that’s not necessary since it has its own schema and Spark can translate it.

Training the Model

Finally, training the model. As always, split the train and test data first, then define the target and predictor variables. Specifying the model can be done a few ways, including the ability to use an R-like formula to specify the model. Here I’ll use the VectorAssembler , which basically just concatenates all the features into a single list:

Then a logistic regression model can be trained on the data, and we can get the predictions that the model makes for the test data.

PySpark also has a Pipeline class, which can intelligently connect up all of the separate steps into a single operation, if you prefer:

Regardless, evaluating the model is necessary. Spark seems a little limited in the native options for evaluating models — the BinaryClassificationEvaluator below only seems to support area under the curve for ROC (default) or the precision-recall curve.

Note that the logistic regression model will actually return three columns of prediction data:

rawPrediction depends on the model used, but here refers to the value of the linear part of the logistic regression model before being transformed

depends on the model used, but here refers to the value of the linear part of the logistic regression model before being transformed probability is an array of the actual probabilities for each class

is an array of the actual probabilities for each class prediction is the actual class prediction

You can use the toPandas() method to return the prediction data as a pandas dataframe, so other metrics are possible to calculate with either pandas or numpy .

Pandas version of PySpark’s prediction columns.

Finally, if you want to look at an overview of Spark’s activity during the session, you can open a browser tab to localhost:4040 and see an overview of it:

SparkUI after running the above code.

So that is a quick, not-too-into-the-weeds overview connecting MySQL to PySpark and fitting a logistic regression model. Obviously, there is much more power in both tools, but this will hopefully provide a window into your own work.