How to change dataframe column names in PySpark ?

15 Feb, 2022
In this article, we are going to see how to change the column names in the pyspark data frame. 
Let’s create a Dataframe for demonstration:
Python3
# Importing necessary librariesfrom pyspark.sql import SparkSession # Create a spark sessionspark = SparkSession.builder.appName('pyspark - example join').getOrCreate() # Create data in dataframedata = [(('Ram'), '1991-04-01', 'M', 3000),        (('Mike'), '2000-05-19', 'M', 4000),        (('Rohini'), '1978-09-05', 'M', 4000),        (('Maria'), '1967-12-01', 'F', 4000),        (('Jenis'), '1980-02-17', 'F', 1200)] # Column names in dataframecolumns = ["Name", "DOB", "Gender", "salary"] # Create the spark dataframedf = spark.createDataFrame(data=data,                           schema=columns) # Print the dataframedf.show()
Output :
We will use of withColumnRenamed() method to change the column names of pyspark data frame.
Syntax: DataFrame.withColumnRenamed(existing, new)
Parameters
existingstr: Existing column name of data frame to rename.
newstr: New column name.
Returns type: Returns a data frame by renaming an existing column.
Example 1: Renaming the single column in the data frame
Here we’re Renaming the column name ‘DOB’ to ‘DateOfBirth’.
Python3
# Rename the column name from DOB to DateOfBirth# Print the dataframedf.withColumnRenamed("DOB","DateOfBirth").show()
Output :
Example 2: Renaming multiple column names
Python3
# Rename the column name 'Gender' to 'Sex'# Then for the returning dataframe# again rename the 'salary' to 'Amount'df.withColumnRenamed("Gender","Sex").withColumnRenamed("salary","Amount").show()
Output :
Renaming the column names using selectExpr() method
Syntax : DataFrame.selectExpr(expr)
Parameters :
expr : It’s an SQL expression.
Here we are renaming Name as a name.
Python3
# Select the 'Name' as 'name'# Select remaining with their original namedata = df.selectExpr("Name as name","DOB","Gender","salary") # Print the dataframedata.show()
Output :
Syntax: DataFrame.select(cols)
Parameters :
cols: List of column names as strings.
Return type: Selects the cols in the dataframe and returns a new DataFrame.
Here we Rename the column name ‘salary’ to ‘Amount’
Python3
# Import col method from pyspark.sql.functionsfrom pyspark.sql.functions import col # Select the 'salary' as 'Amount' using aliasing# Select remaining with their original namedata = df.select(col("Name"),col("DOB"),                 col("Gender"),                 col("salary").alias('Amount')) # Print the dataframedata.show()
Output :
This function returns a new DataFrame that with new specified column names.
Syntax: toDF(*col)
Where, col is a new column name
In this example, we will create an order list of new column names and pass it into toDF function
Python3
Data_list = ["Emp Name","Date of Birth",             " Gender-m/f","Paid salary"] new_df = df.toDF(*Data_list)new_df.show()
Output:
clintra
kapoorsagar226
germanshephered48
Picked
Python-Pyspark
Python
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.
How to Install PIP on Windows ?
Python Classes and Objects
Python OOPs Concepts
Introduction To PYTHON
Python | os.path.join() method
How to drop one or multiple columns in Pandas Dataframe
How To Convert Python Dictionary To JSON?
Check if element exists in list in Python
Python | Get unique values from a list
Create a directory in Python