PySpark debugging — 6 common issues
By: Maria Karanasou
Tags: Pyspark, Apache Spark, Programming, Debugging, Towards Data Science

1. Null column returned from a udf

When you add a column to a dataframe using a udf but the result is Null: the udf return datatype is different than what was defined

For example, if you define a udf function that takes as input two numbers a and b and returns a / b , this udf function will return a float (in Python 3). If the udf is defined as:

udf_ratio_calculation = F.udf(calculate_a_b_ratio, T.BooleanType())

# or

udf_ratio_calculation = F.udf(calculate_a_b_ratio, T.DoubleType())

instead of:

udf_ratio_calculation = F.udf(calculate_a_b_ratio, T.FloatType())

then the outcome of using the udf will be something like this:

df = df.withColumn('a_b_ratio', udf_ratio_calculation('a', 'b'))

df.show()

+---+---+---------+

| a| b|a_b_ratio|

+---+---+---------+

| 1| 0| null|

| 10| 3| null|

+---+---+---------+

Full example:

Example of wrongly defined udf return datatype

2. ClassNotFoundException

This exception usually happens when you are trying to connect your application to an external system, e.g. a database. The stacktrace below is from an attempt to save a dataframe in Postgres.

This means that spark cannot find the necessary jar driver to connect to the database. We need to provide our application with the correct jars either in the spark configuration when instantiating the session

from pyspark import SparkConf

from pyspark.sql import SparkSession





conf = SparkConf()

conf.set('spark.jars', '/full/path/to/postgres.jar,/full/path/to/other/jar')

spark_session = SparkSession.builder \

.config(conf=conf) \

.appName('test') \

.getOrCreate()

or as a command line argument — depending on how we run our application.

spark-submit --jars /full/path/to/postgres.jar,/full/path/to/other/jar ...

Note 1: It is very important that the jars are accessible to all nodes and not local to the driver.

Note 2: This error might also mean a spark version mismatch between the cluster components. There other more common telltales, like AttributeError . More on this here.

Note 3: Make sure there is no space between the commas in the list of jars.

3. Low memory consumption — why doesn’t spark use up all of my resources?

Spark driver memory and spark executor memory are set by default to 1g . It is in general very useful to take a look at the many configuration parameters and their defaults, because there are many things there that can influence your spark application.

When spark is running locally, you should adjust the spark.driver.memory to something that’s reasonable for your system, e.g. 8g and when running on a cluster, you might also want to tweak the spark.executor.memory also, even though that depends on your kind of cluster and its configuration.

4. File does not exist: Spark runs ok in local mode but can’t find file when running in YARN

Again as in #2, all the necessary files/ jars should be located somewhere accessible to all of the components of your cluster, e.g. an FTP server or a common mounted drive.

spark-submit --master yarn --deploy-mode cluster http://somewhere/accessible/to/master/and/workers/test.py

5. Trying to connect to a database: java.sql.SQLException: No suitable driver

Or if the error happens while trying to save to a database, you’ll get a java.lang.NullPointerException :

This usually means that we forgot to set the driver , e.g. org.postgresql.Driver for Postgres:

df = spark.read.format('jdbc').options(

url= 'db_url',

driver='org.postgresql.Driver', # <-- here

dbtable='table_name',

user='user',

password='password'

).load()

Please, also make sure you check #2 so that the driver jars are properly set.

6. ‘NoneType’ object has no attribute ‘ _jvm'

You might get the following horrible stacktrace for various reasons.

Two of the most common are:

You are using pyspark functions without having an active spark session

from pyspark.sql import SparkSession, functions as F class A(object):

def __init__(self):

self.calculations = F.col('a') / F.col('b') ... a = A() # instantiating A without an active spark session will give you this error

Or you are using pyspark functions within a udf:

from pyspark import SparkConf

from pyspark.sql import SparkSession, functions as F, types as T

conf = SparkConf()

spark_session = SparkSession.builder \

.config(conf=conf) \

.appName('test') \

.getOrCreate()



# create a dataframe

data = [{'a': 1, 'b': 0}, {'a': 10, 'b': 3}]

df = spark_session.createDataFrame(data)

df.show()



# +---+---+

# | a| b|

# +---+---+

# | 1| 0|

# | 10| 3|

# +---+---+





# define a simple function that returns a / b

# we *cannot* use pyspark functions inside a udf

# udfs operate on a row per row basis while pyspark functions on a column basis

def calculate_a_b_max(a, b):

return F.max([a, b])



# and a udf for this function - notice the return datatype

udf_max_calculation = F.udf(calculate_a_b_ratio, T.FloatType())

df = df.withColumn('a_b_max', udf_max_calculation('a', 'b'))

df.show()

Both will give you this error.

In the last example F.max needs a column as an input and not a list, so the correct usage would be:

df = df.withColumn('a_max', F.max('a'))

Which would give us the maximum of column a — not what the udf is trying to do.

The correct way to set up a udf that calculates the maximum between two columns for each row would be:

def calculate_a_b_max(a, b):

return max([a, b])