How To Change The Column Type in PySpark DataFrames | Towards Data Science

A fairly common operation in PySpark is type casting that is usually required when we need to change the data type of specific columns in DataFrames. For instance, it’s quite common (and a bad practise!) to have datetimes stored as strings, or even integers and doubles stored as StringType.
In today’s short guide we will explore how to change the column types of certain DataFrame columns in PySpark. Specifically, we will discuss how to do so using
the cast() function
the selectExpr() function
Spark SQL
First, let’s create an example DataFrame that we’ll reference throughout this article in order to demonstrate a few concepts.
from pyspark.sql import SparkSession# Create an instance of spark sessionspark_session = SparkSession.builder \    .master('local[1]') \    .appName('Example') \    .getOrCreate()df = spark_session.createDataFrame(    [        (1, '10-01-2020', '1.0', '100'),        (2, '14-02-2021', '2.0', '200'),        (3, '15-06-2019', '3.0', '300'),        (4, '12-12-2020', '4.0', '400'),        (5, '01-09-2019', '5.0', '500'),    ],    ['colA', 'colB', 'colC', 'colD'])df.show()+----+----------+----+----+|colA|      colB|colC|colD|+----+----------+----+----+|   1|10-01-2020| 1.0| 100||   2|14-02-2021| 2.0| 200||   3|15-06-2019| 3.0| 300||   4|12-12-2020| 4.0| 400||   5|01-09-2019| 5.0| 500|+----+----------+----+----+df.printSchema()root |-- colA: long (nullable = true) |-- colB: string (nullable = true) |-- colC: string (nullable = true) |-- colD: string (nullable = true)
In the following sections, we will showcase how to change the column type of columns colB, colC and colD to DateType, DoubleType and IntegerType respectively.
The first option you have when it comes to converting data types is pyspark.sql.Column.cast() function that converts the input column to the specified data type.
from datetime import datetimefrom pyspark.sql.functions import col, udffrom pyspark.sql.types import DoubleType, IntegerType, DateType# UDF to process the date columnfunc = udf(lambda x: datetime.strptime(x, '%d-%m-%Y'), DateType())df = df \    .withColumn('colB', func(col('colB'))) \    .withColumn('colC', col('colC').cast(DoubleType())) \    .withColumn('colD', col('colD').cast(IntegerType()))df.show()+----+----------+----+----+|colA|      colB|colC|colD|+----+----------+----+----+|   1|2020-01-10| 1.0| 100||   2|2021-02-14| 2.0| 200||   3|2019-06-15| 3.0| 300||   4|2020-12-12| 4.0| 400||   5|2019-09-01| 5.0| 500|+----+----------+----+----+df.printSchema()root |-- colA: long (nullable = true) |-- colB: date (nullable = true) |-- colC: double (nullable = true) |-- colD: integer (nullable = true)
Note that in order to cast the string into DateType we need to specify a UDF in order to process the exact format of the string date.
Alternatively, you can use pyspark.sql.DataFrame.selectExpr function by specifying the corresponding SQL expressions that can cast the data type of desired columns, as shown below.
df = df.selectExpr(    'colA',    'to_date(colB, \'dd-MM-yyyy\') colB',    'cast(colC as double) colC',    'cast(colD as int) colD',)df.show()+----+----------+----+----+|colA|      colB|colC|colD|+----+----------+----+----+|   1|2020-01-10| 1.0| 100||   2|2021-02-14| 2.0| 200||   3|2019-06-15| 3.0| 300||   4|2020-12-12| 4.0| 400||   5|2019-09-01| 5.0| 500|+----+----------+----+----+df.printSchema()root |-- colA: long (nullable = true) |-- colB: date (nullable = true) |-- colC: double (nullable = true) |-- colD: integer (nullable = true)
Lastly, you can even use Spark SQL in order to cast the desired columns in a similar way we have used selectExpr function to do so.
# First we need to register the DF as a global temporary viewdf.createGlobalTempView("df")df = spark_session.sql(    """    SELECT         colA,        to_date(colB, 'dd-MM-yyyy') colB,        cast(colC as double) colC,        cast(colD as int) colD    FROM global_temp.df    """)df.show()+----+----------+----+----+|colA|      colB|colC|colD|+----+----------+----+----+|   1|2020-01-10| 1.0| 100||   2|2021-02-14| 2.0| 200||   3|2019-06-15| 3.0| 300||   4|2020-12-12| 4.0| 400||   5|2019-09-01| 5.0| 500|+----+----------+----+----+df.printSchema()root |-- colA: long (nullable = true) |-- colB: date (nullable = true) |-- colC: double (nullable = true) |-- colD: integer (nullable = true)
In today’s short guide we discussed a few different ways for changing column types of DataFrame columns in PySpark. Specifically, we explored how you can use withColumn() function in combination with cast() as well as using more SQL-like approaches such as selectExpr() or Spark SQL.
Become a member and read every story on Medium. Your membership fee directly supports me and other writers you read. You’ll also get full access to every story on Medium.
You may also like