Most Useful Date Manipulation Functions in Spark | by Sivakar Sivarajah | Towards Data Science

Pyspark and Spark SQL provide many built-in functions. The functions such as the date and time functions are useful when you are working with DataFrame which stores date and time type values. Sometimes you may be tempted to write a UDF(user-defined functions) only later to realize that it would have been better to check out the documentation because it already exists. In this article, we will check what are the most important Spark SQL date functions with some examples.
The Spark SQL built-in date functions are user and performance-friendly. Spark SQL supports almost all date functions that are supported in Apache Hive. Following in the table below are the Spark SQL date functions these can be used to manipulate the data frame columns that contain data type values. The list contains pretty much all date functions that are supported in Apache Spark.
For this tutorial I am using the airport dataset, the dataset is open-sourced and can be found on Kaggle.
Reading CSV File>>> df = spark.read.csv("Airports2.csv", header=True, inferSchema=True)Visualizing the first 5 rows of the dataset>>> df.show(5)
Checking the Schema>>> df.printSchema()
As you can check above the “Fly_Date” columns which contain dates is string type.
To_date(col) → Convert the string type containing date value to date format
To_date(col) → Convert the string type containing date value to date format
Example:Converting the string type "Fly_date" column containing dates value to date type/format.>>> df = df.withColumn(‘Fly_date’, to_date("Fly_date"))>>> df.printSchema()
2. Date_format(date, format) → Converts a date/timestamp/string to a value of the string in the format specified by the date format given by the second argument.
Example:Format "Fly_date" column with the "dd/MM/yyyy" format >>> df.select("Fly_date", date_format("Fly_date", "dd/MM/yyyy").alias("Formatted_date")).show(3)
3) Date_add (start, days) → Add days to the date
Example:Adding two days to the Fly_date columns>>> df.select("Fly_date", date_add("Fly_date",2).alias("date_added")).show(5)
4) Add_months(start, months)→Add Months to Date
Example:Adding two months to the date columns and saving into a new dataframe>>> test_df=df.select('Fly_date', add_months(df.Fly_date,2).alias("months_added"))>>> test_df.show(5)
5) Datediff(end, start) → Returns the difference between two dates in days.
Example:Returns the difference between two dates in days>>> test_df.select("Fly_date","months_added", datediff("months_added", "Fly_date").alias("date_diff")).show(3)
6) Extracting Single “date” Elements
Year(Col) → Extract the corresponding year of a given date as an integer.
Quarter(Col) → Extract the corresponding quarter of a given date as an integer.
Month(Col) → Extract the corresponding month of a given date as an integer.
Dayofmonth(Col) → Extract the corresponding day of the month of a given date as an integer.
Weekofyear(Col) → Extract the corresponding week number of a given date as an integer.
Hour(Col) → Extract the corresponding hours of a given date as an integer.
Minute(Col) → Extract the corresponding minutes of a given date as an integer.
Examples:# Year>>> test_df.select("date_days", year( "date_days").alias("year")).show(3)# Month>>> test_df.select("date_days", Month( "date_days").alias("Month")).show(3)# Dayofmonth>>> test_df.select("date_days", Dayofmonth( "date_days").alias("Day of the Month")).show(3)# Weekoftheyear>>> test_df.select("date_days", Weekofyear( "date_days").alias("Week of the year")).show(3)
7) Date_sub(start, days) → Subtract the days from the date field.
Example:Subtract three days to the current date >>> df.select('Fly_date', date_sub(df.Fly_date,3).alias("date_sub")).show(3)
8) Last_day(date) → Returns the last day of the month to which the given date belongs to.
Example:Get the last day of the month using "last_day" function>>> df.select("Fly_date", last_day( "Fly_date").alias("last_date")).show(4)
9) Months_between(date 1, date 2) → Returns the number of months between two dates.
Example:Identify the months between "Month Added"  and "Fly date" using months_between function>>> new_df=test_df.select("Fly_date","month_added", months_between("month_added", "Fly_date").alias("months")).show(3)
10) Next_day(date, dayOfWeek) → Returns the first date which is later than the value of the date column.
Example:Getting next Tuesday date using the "next_day" function>>> df.select("Fly_date", next_day("Fly_date", "tue").alias("Next_Tuesday")).show(3)
11) Sorting By ‘date’ → sorting the Dataframe by dates
Example:Sorting the selected columns by "Fly date" column in descending order>>> df.select("Passengers","Fly_date")./   sort(df.Fly_date.desc()).show(3)
12) Current Timestamp → Returns the current timestamp as a timestamp column
Example:Returns the current timestamp as a timestamp column>>> df.select(current_timestamp().alias( "current_timestamp")).show(1)
13) Current Date → Returns the current date as a date column.
Example: Returns the current date as a date column>>> df.select(current_date().alias(“current_date”)).show(1)
In this post, I have tried to show the different date functions with a relevant example. These functions provide great flexibility in the manipulation of the date type columns as well as in coming up with interesting features for machine learning and creating interesting insights out of the data. Use these functions whenever possible instead of Spark SQL user-defined functions. Thank you for reading :)