PySpark debugging — 6 common issues | by Maria Karanasou | Towards Data Science

Debugging a spark application can range from a fun to a very (and I mean very) frustrating experience.
I’ve started gathering the issues I’ve come across from time to time to compile a list of the most common problems and their solutions.
This is the first part of this list. I hope you find it useful and it saves you some time. Most of them are very simple to resolve but their stacktrace can be cryptic and not very helpful.
When you add a column to a dataframe using a udf but the result is Null: the udf return datatype is different than what was defined
For example, if you define a udf function that takes as input two numbers a and b and returns a / b , this udf function will return a float (in Python 3). If the udf is defined as:
udf_ratio_calculation = F.udf(calculate_a_b_ratio, T.BooleanType())# or udf_ratio_calculation = F.udf(calculate_a_b_ratio, T.DoubleType())
instead of:
udf_ratio_calculation = F.udf(calculate_a_b_ratio, T.FloatType())
then the outcome of using the udf will be something like this:
df = df.withColumn('a_b_ratio', udf_ratio_calculation('a', 'b'))df.show() +---+---+---------+|  a|  b|a_b_ratio|+---+---+---------+|  1|  0|     null|| 10|  3|     null|+---+---+---------+
Full example:
This exception usually happens when you are trying to connect your application to an external system, e.g. a database. The stacktrace below is from an attempt to save a dataframe in Postgres.
This means that spark cannot find the necessary jar driver to connect to the database. We need to provide our application with the correct jars either in the spark configuration when instantiating the session
from pyspark import SparkConffrom pyspark.sql import SparkSessionconf = SparkConf()conf.set('spark.jars', '/full/path/to/postgres.jar,/full/path/to/other/jar')spark_session = SparkSession.builder \    .config(conf=conf) \    .appName('test') \    .getOrCreate()
or as a command line argument — depending on how we run our application.
spark-submit --jars /full/path/to/postgres.jar,/full/path/to/other/jar ...
Note 1: It is very important that the jars are accessible to all nodes and not local to the driver.
Note 2: This error might also mean a spark version mismatch between the cluster components. There other more common telltales, like AttributeError. More on this here.
Note 3: Make sure there is no space between the commas in the list of jars.
Spark driver memory and spark executor memory are set by default to 1g. It is in general very useful to take a look at the many configuration parameters and their defaults, because there are many things there that can influence your spark application.
spark.apache.org
When spark is running locally, you should adjust the spark.driver.memory to something that’s reasonable for your system, e.g. 8g and when running on a cluster, you might also want to tweak the spark.executor.memory also, even though that depends on your kind of cluster and its configuration.
Again as in #2, all the necessary files/ jars should be located somewhere accessible to all of the components of your cluster, e.g. an FTP server or a common mounted drive.
spark-submit --master yarn --deploy-mode cluster http://somewhere/accessible/to/master/and/workers/test.py
Or if the error happens while trying to save to a database, you’ll get a java.lang.NullPointerException :
This usually means that we forgot to set the driver , e.g. org.postgresql.Driver for Postgres:
df = spark.read.format('jdbc').options(    url= 'db_url',    driver='org.postgresql.Driver',  # <-- here    dbtable='table_name',    user='user',    password='password').load()
Please, also make sure you check #2 so that the driver jars are properly set.
You might get the following horrible stacktrace for various reasons.
Two of the most common are:
You are using pyspark functions without having an active spark session
from pyspark.sql import SparkSession, functions as Fclass A(object):    def __init__(self):        self.calculations = F.col('a') / F.col('b')...a = A() # instantiating A without an active spark session will give you this error 
Or you are using pyspark functions within a udf:
from pyspark import SparkConffrom pyspark.sql import SparkSession, functions as F, types as Tconf = SparkConf()spark_session = SparkSession.builder \    .config(conf=conf) \    .appName('test') \    .getOrCreate()# create a dataframedata = [{'a': 1, 'b': 0}, {'a': 10, 'b': 3}]df = spark_session.createDataFrame(data)df.show()# +---+---+# |  a|  b|# +---+---+# |  1|  0|# | 10|  3|# +---+---+# define a simple function that returns a / b# we *cannot* use pyspark functions inside a udf# udfs operate on a row per row basis while pyspark functions on a column basisdef calculate_a_b_max(a, b):    return F.max([a, b])# and a udf for this function - notice the return datatypeudf_max_calculation = F.udf(calculate_a_b_ratio, T.FloatType())df = df.withColumn('a_b_max', udf_max_calculation('a', 'b'))df.show()
Both will give you this error.
In the last example F.max needs a column as an input and not a list, so the correct usage would be:
df = df.withColumn('a_max', F.max('a'))
Which would give us the maximum of column a — not what the udf is trying to do.
The correct way to set up a udf that calculates the maximum between two columns for each row would be:
def calculate_a_b_max(a, b):    return max([a, b]) 
Assuming a and b are numbers. (There are other ways to do this of course without a udf.)
I hope this was helpful. I plan to continue with the list and in time go to more complex issues, like debugging a memory leak in a pyspark application.Any thoughts, questions, corrections and suggestions are very welcome :)
If you want to know a bit about how Spark works, take a look at: