Add a column with the literal value in PySpark DataFrame - GeeksforGeeks

24 Sep, 2021
In this article, we are going to see how to add a column with the literal value in PySpark Dataframe.
Creating dataframe for demonstration:
Python3
# import SparkSession from the pysparkfrom pyspark.sql import SparkSession # build and create the# SparkSession with name "lit_value"spark = SparkSession.builder.appName("lit_value").getOrCreate() # create the spark dataframe with columns A,Bdata = spark.createDataFrame([('x',5),('Y',3),                            ('Z',5) ],['A','B']) # showing the schema and  tabledata.printSchema()data.show()
 Output: 
Here we can add the constant column ‘literal_values_1’ with value 1 by Using the select method. The lit() function will insert constant values to all the rows.
Select table by using select() method and pass the arguments first one is the column name, or  “*”  for selecting the whole table and second argument pass the lit() function with constant values. 
Python3
# Import the lit() function# from the pyspark.sql.functionsfrom pyspark.sql.functions import lit # select all the columns from data# table and insert new columns# 'literal_values_1' with values 1df2 = data.select('*' ,lit("1").alias("literal_values_1"))  # showing the schema and updated tabledf2.printSchema()df2.show()
Output:
In this method first, we have to create the temp view of the table with the help of createTempView we can create the temporary view. The life of this temp is up to the life of the sparkSession. CreateOrReplace will create the temp table if it is not available or if it is available then replace it.
Then after creating the table select the table by SQL clause which will take all the values as a string 
Python3
# this will create a temp view of the table as lit_valdf2.createOrReplaceTempView("temp") # select all the columns and rows# from data table and insert new# columns 'literal_values_2' with values 2df2 = spark.sql("select *, 2 as literal_values_2 from temp") # showing the schema and updated tabledf2.printSchema()df2.show()
Output:
This function allows us to create the new function as per our requirements that’s why this is also called a user-defined function. Now we define the datatype of the UDF function and create the functions which will return the values in the form of a new column
Python3
# import the udf from pysparkfrom pyspark.sql.functions import udf # defining the data types of udf which is# integer type@udf("int") # defining the lit_col() function which# will return literal values to  data framedef lit_col():    return 3 # create new column as# 'literal_values_3' with values 3df2 = df2.withColumn('literal_values_3', lit_col()) # showing the schema and updated tabledf2.printSchema()df2.show()
Output:
arorakashish0911
Picked
Python-Pyspark
Python
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.
Comments
Old Comments
Python Dictionary
Read a file line by line in Python
Enumerate() in Python
How to Install PIP on Windows ?
Iterate over a list in Python
Different ways to create Pandas Dataframe
Python program to convert a list to string
Python String | replace()
Reading and Writing to text files in Python
sum() function in Python