How To Change The Column Names Of PySpark DataFrames | Towards Data Science

In today’s short guide we will discuss 4 ways for changing the name of columns in a Spark DataFrame. Specifically, we are going to explore how to do so using:
selectExpr() method
withColumnRenamed() method
toDF() method
alias
Spark Session and Spark SQL
and rename one or more columns at a time.
First, let’s create an example PySpark DataFrame that we’ll reference throughout this guide to demonstrate a few concepts.
>>> from pyspark.sql import SparkSession# Create an instance of spark session>>> spark_session = SparkSession.builder \    .master('local[1]') \    .appName('Example') \    .getOrCreate()>>> df = spark_session.createDataFrame(    [       (1, 'a', True, 1.0),       (2, 'b', False, 2.0),       (3, 'c', False, 3.0),       (4, 'd', True, 4.0),    ],    ['colA', 'colB', 'colC', 'colD'])>>> df.show()+----+----+-----+----+                                                          |colA|colB| colC|colD|+----+----+-----+----+|   1|   a| true| 1.0||   2|   b|false| 2.0||   3|   c|false| 3.0||   4|   d| true| 4.0|+----+----+-----+----+>>> df.printSchema()root |-- colA: long (nullable = true) |-- colB: string (nullable = true) |-- colC: boolean (nullable = true) |-- colD: double (nullable = true)
The first option you have is pyspark.sql.DataFrame.selectExpr() method which is a variant of select() method that accepts SQL expressions.
>>> df =  df.selectExpr(    'colA AS A',    'colB AS B',    'colC AS C',    'colD AS D',)>>> df.show()+---+---+-----+---+|  A|  B|    C|  D|+---+---+-----+---+|  1|  a| true|1.0||  2|  b|false|2.0||  3|  c|false|3.0||  4|  d| true|4.0|+---+---+-----+---+
Note however that this approach is mostly suitable when you need to rename most of the columns and when you also have to deal with a relatively small number of columns.
At this point, you may also want to recall the difference between select() and selectExpr() in Spark:
towardsdatascience.com
The second option you have when it comes to rename columns of PySpark DataFrames is the pyspark.sql.DataFrame.withColumnRenamed(). This method returns a new DataFrame by renaming an existing column.
>>> df = df.withColumnRenamed('colA', 'A')>>> df.show()+---+----+-----+----+|  A|colB| colC|colD|+---+----+-----+----+|  1|   a| true| 1.0||  2|   b|false| 2.0||  3|   c|false| 3.0||  4|   d| true| 4.0|+---+----+-----+----+
This method is mostly suitable when you need to rename one column at a time. If you need to rename multiple columns in one go then other methods discussed in this article will be more helpful.
pyspark.sql.DataFrame.toDF() method returns a new DataFrame with the new specified column names.
>>> df = df.toDF('A', 'colB', 'C', 'colD')>>> df.show()+---+----+-----+----+|  A|colB|    C|colD|+---+----+-----+----+|  1|   a| true| 1.0||  2|   b|false| 2.0||  3|   c|false| 3.0||  4|   d| true| 4.0|+---+----+-----+----+
This method is useful when you need to rename more than one columns at the same time.
Note that if you have a list containing the new column names then the following should do the trick as well:
>>> new_col_names = ['A', 'colB', 'C', 'colD']>>> df = df.toDF(*new_col_names)
Another option we have is the use of alias method that returns a new DataFrame with an alias set.
>>> from pyspark.sql.functions import col>>> df = df.select(    col('colA').alias('A'),     col('colB'),    col('colC').alias('C'),    col('colD'))>>> df.show()+---+----+-----+----+|  A|colB|    C|colD|+---+----+-----+----+|  1|   a| true| 1.0||  2|   b|false| 2.0||  3|   c|false| 3.0||  4|   d| true| 4.0|+---+----+-----+----+
Again this method should be used when multiple columns need to be renamed and when you don’t have to deal with numerous columns, otherwise this could get really verbose.
Finally, you can rename columns using Spark SQL using the traditional SQL syntax for DataFrames that were stored as tables. For example,
>>> df.createOrReplaceTempView('test_table')>>> df = spark_session.sql(    'SELECT colA AS A, colB, colC AS C, colD FROM test_table')>>> df.show()+---+----+-----+----+|  A|colB|    C|colD|+---+----+-----+----+|  1|   a| true| 1.0||  2|   b|false| 2.0||  3|   c|false| 3.0||  4|   d| true| 4.0|+---+----+-----+----+
In today’s short guide we discussed how to rename columns of PySpark DataFrames in many different ways. Depending on whether you need to rename one or multiple columns, you have to choose the method which is most suitable for your specific use case.
Become a member and read every story on Medium. Your membership fee directly supports me and other writers you read.