Spark 3.2: Session Windowing Feature for Streaming Data | by Pınar Ersoy | Towards Data Science

The latest version of Spark 3.2 was released on October 13, 2021 [1]. In addition to its improvements on different topics,
The existing windowing framework for streaming data processing provides only tumbling and sliding windows as highlighted in the Spark technical documentation[2]. In the terminology, there exists an additional windowing operation which is called a session window.
Unlike the other two windowing approaches, the session window doesn’t hold solid window start and end times. Session window creation depends on a defined session period which can be static or dynamic.
In the static version of session windowing, the events which are happening in a specific period of time are regarded as a session window. A session window stops when it does not collect events for the specified time period.
In the dynamic version of session windowing, the period could be varied from event to event.
For each windowing approach, you may find a corresponding implementation in PySpark language in the upcoming sections. For the development platform, and programming language, Google Colab, and PySpark will be used respectively.
Before installing Spark 3.2, initially, we are required to install the matching version of Java. With the script below, you can set up Java 8 version.
!apt-get install openjdk-8-jdk-headless -qq > /dev/null
After an appropriate version of Java installation is completed, Spark 3.2 installation can be started. The following script will help you to set up Spark 3.2. version [3].
!wget -q https://archive.apache.org/dist/spark/spark-3.2.0/spark-3.2.0-bin-hadoop2.7.tgz
When the installation concluded successfully, you are required to extract the spark file with the help of the following script.
!tar xf spark-3.2.0-bin-hadoop2.7.tgz
The installation of Java and Spark 3.2. is finished. The next step is to assign the environment variables for both of them. If you work on Google Colab, you may set the path environments like the following.
import osos.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"os.environ["SPARK_HOME"] = "/content/spark-3.2.0-bin-hadoop2.7"
For testing purposes, you may run the findspark library like the following script.
import findsparkfindspark.init()
With the newest updates, Spark 3.2 contains three unique types of windowing functions as Tumbling, Sliding, and Session. To be able to test them individually, a spark session shall be created with the latest Spark version.
The following script creates a session by using all the local cores on the Google Colab environment.
from pyspark.sql import SparkSessionspark = SparkSession.builder.master("local[*]").getOrCreate()
To test the version of the Spark, the .version function can be executed for the spark session.
spark.version
To be able to apply windowing functions, a spark session and a sample dataframe are required. A sample spark session can be initialized as the following code snippet.
from pyspark.sql import SparkSessionspark = SparkSession.builder.master("local[*]").getOrCreate()spark.version
Sample data can be generated with the following script.
# Sample Data is generated for windowing exampleswindowingData = (("12", "2019-01-02 15:30:00"),("12",  "2019-01-02 15:30:30"),("12",  "2019-01-02 15:31:00"),("12",  "2019-01-02 15:31:50"),("12",  "2019-01-02 15:31:55"),("16",  "2019-01-02 15:33:00"),("16",  "2019-01-02 15:35:20"),("16",  "2019-01-02 15:37:00"),("20",  "2019-01-02 15:30:30"),("20",  "2019-01-02 15:31:00"),("20",  "2019-01-02 15:31:50"),("20",  "2019-01-02 15:31:55"),("20",  "2019-01-02 15:33:00"),("20",  "2019-01-02 15:35:20"),("20",  "2019-01-02 15:37:00"),("20",  "2019-01-02 15:40:00"),("20",  "2019-01-02 15:45:00"),("20",  "2019-01-02 15:46:00"),("20",  "2019-01-02 15:47:30"),("20",  "2019-01-02 15:48:00"),("20",  "2019-01-02 15:48:10"),("20",  "2019-01-02 15:48:20"),("20",  "2019-01-02 15:48:30"),("20",  "2019-01-02 15:50:00"),("20",  "2019-01-02 15:53:00"),("20",  "2019-01-02 15:54:30"),("20",  "2019-01-02 15:55:00"),("22",  "2019-01-02 15:50:30"),("22",  "2019-01-02 15:52:00"),("22",  "2019-01-02 15:50:30"),("22",  "2019-01-02 15:52:00"),("22",  "2019-01-02 15:50:30"),("22",  "2019-01-02 15:52:00"))columns = ["eventId", "timeReceived"]windowing_df = spark.createDataFrame(data = windowingData, schema = columns)windowing_df.printSchema()windowing_df.show(truncate=False)
The tumbling windows can be represented as a group of equally partitioned, adjacent time periods without having any intersecting intervals. The inputted data may be subject to an individual window.
from pyspark.sql.functions import *tumblingWindows = windowing_df.withWatermark("timeReceived", "10 minutes").groupBy("eventId", window("timeReceived", "10 minutes")).count()tumblingWindows.show(truncate = False)
When observed on a timeline, the windows can be viewed as a sequence followed by each other in a static manner as in the following image.
The sliding windows may have intersecting time periods when the time span contains a shorter interval than the range of the window. In such cases, timestamped items can be found in more than one window.
from pyspark.sql.functions import *slidingWindows = windowing_df.withWatermark("timeReceived", "10 minutes").groupBy("eventId", window("timeReceived", "10 minutes", "5 minutes")).count()slidingWindows.show(truncate = False)
When observed on a timeline, the windows can be overlapping or not overlapping according to the length of the time range when compared to the length of the main sliding window with a static manner as the tumbling window.
Session windows may contain the nature of a dynamic length by means of the window size relative to the inputting item type.
A session window begins with a single data point and broadens itself in case of the upcoming element has been collected inside of the gap period.
When the last item is accepted, the session window ends when no items are acknowledged inside of the gap period. This feature permits the user to cluster the events when there are no advanced events injected for a selected period of time as also described in the official article as an inactive period [4].
In the real world, the logic of the session windowing operates as timeout occurrences. In some specific time gap, you shall be performing an activity, otherwise, the existing session closes when an adequate amount of time is passed. The session period can be enlarged in case having additional activity is recorded.
For the specific usage of the session window, it can be depicted as creating a session window with upcoming events, and the continually gathering events inside the timeout period will be added in the current session.
from pyspark.sql.functions import *sessionWindows = windowing_df.withWatermark("timeReceived", "10 minutes").groupBy("eventId", session_window("timeReceived", "5 minutes")).count()sessionWindows.show(truncate = False)
Every event that is acquired will be capable of increasing the range of the session timeout. With this feature, the time period of the session window can be accepted as not static compared to the other two windowing approaches.
For the specific example that is added above, the event with an id of 20, session windowing flow can be observed as one closes after a specified time period and begins again when a new element is acknowledged.
The session window functionality has an additional feature which is called dynamic gap duration as mentioned in the Databricks blog post [4]. The period of the session may have various values when requested. The sessions that have an overlapping behavior can be grouped into a single session. The total duration of this session will be equal to the sum of the time duration of the intersecting events.
You may group the data by the eventId, session window, and calculate its occurrences with the following script by inserting two values for the session_window() function as described in the latest programming guide of Spark 3.2 [5].
windowedCountsDF = windowing_df.withWatermark("timeReceived", "10 minutes").groupBy(windowing_df.eventId, session_window(windowing_df.timeReceived, \when(windowing_df.eventId == "20", "10 seconds").when(windowing_df.eventId == "12","30 seconds").otherwise("10 minutes"))).count()windowedCountsDF.show(100, truncate = False)
In the literature, there exist three main windowing functionalities. The ones that have a static manner are tumbling and sliding windows which were already available in the previous version of Spark 3.2. The dynamic natured session windows capability is now available with the latest version of Spark 3.2 [6].
You can find the full PySpark scripted code on the Google Colab environment.
Questions and comments are highly appreciated! Thank you so much for reading!
Spark Release 3.2.0Structured Streaming Programming GuideAdvanced Spark Streaming
Spark Release 3.2.0
Structured Streaming Programming Guide
Advanced Spark Streaming